Speech generation method and apparatus, electronic device, and storage medium

Abstract

A speech generation method and apparatus, an electronic device, and a storage medium. The speech generation method comprises: acquiring content constraint information for content constraint during speech generation, and extracting an information embedding of the content constraint information (401); acquiring style prompt speech for style constraint during the speech generation, and extracting from the style prompt speech a plurality of raw speech tokens for representing speech style features (402); inputting the information embedding and the plurality of raw speech tokens into a first large language model to obtain a plurality of target speech tokens generated by the first large language model by means of prediction (403); acquiring timbre prompt speech for timbre constraint during the speech generation, and extracting from the timbre prompt speech a first speaker embedding for representing a timbre feature of a speaker (404); and generating speech on the basis of the first speaker embedding and the plurality of target speech tokens (405).

Description

Speech generation methods, devices, electronic devices and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411747099.1, filed on December 2, 2024, entitled “Speech Generation Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence technology, and in particular to a speech generation method, apparatus, electronic device, and storage medium. Background Technology

[0003] With the development of artificial intelligence technology, zero-shot speech generation is becoming increasingly widespread, finding applications in scenarios such as voice assistants, audiobooks, and map navigation. However, the speech generated by current zero-shot speech generation systems is relatively fixed in timbre and style, and its realism and flexibility need to be improved.

[0004] Technical content

[0005] The following is an overview of the subject matter described in detail in this disclosure. This overview is not intended to limit the scope of the claims.

[0006] This disclosure provides a speech generation method, apparatus, electronic device, and storage medium that can improve the realism and flexibility of the generated speech.

[0007] On one hand, embodiments of this disclosure provide a speech generation method, including:

[0008] Obtain content constraint information for content constraint during speech generation, and extract the information embedding of the content constraint information;

[0009] Obtain style-cue speech for style constraint during speech generation, and extract multiple original speech tags from the style-cue speech to characterize speech style features;

[0010] The embedded information and multiple original speech tags are input into a first large language model to obtain multiple target speech tags predicted and generated by the first large language model.

[0011] Acquire timbre-cue speech for timbre constraint during speech generation, and extract a first speaking object embedding from the timbre-cue speech to characterize the timbre features of the speaking object;

[0012] Speech is generated based on the first speaking object embedding and multiple target speech tags.

[0013] On the other hand, embodiments of this disclosure also provide a speech generation apparatus, including:

[0014] The content processing module is used to acquire content constraint information for content constraint during speech generation and extract information embedding from the content constraint information.

[0015] The style processing module is used to acquire style-cue speech for style constraints during speech generation, and extract multiple original speech tags from the style-cue speech to characterize speech style features.

[0016] The prediction module is used to input the embedded information and multiple original speech tags into the first large language model to obtain multiple target speech tags predicted by the first large language model;

[0017] The timbre processing module is used to acquire timbre-cue speech for timbre constraint during speech generation, and extract a first speaking object embedding from the timbre-cue speech to characterize the timbre features of the speaking object;

[0018] The generation module is used to generate speech based on the first speaking object embedding and multiple target speech tags.

[0019] On the other hand, this disclosure also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described speech generation method.

[0020] On the other hand, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the above-described speech generation method.

[0021] On the other hand, this disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the speech generation method described above.

[0022] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing this disclosure.

[0023] Brief description of the attached figures

[0024] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0025] Figure 1 is a schematic diagram of the implementation environment provided in the embodiments of this disclosure;

[0026] Figure 2 is a schematic diagram of the speech generation method provided in this embodiment of the present disclosure applied in a virtual human voice assistant scenario;

[0027] Figure 3 is a schematic diagram of the speech generation method provided in this embodiment of the present disclosure applied to a live chatbot scenario.

[0028] Figure 4 is a schematic flowchart of the speech generation method provided in the embodiments of this disclosure;

[0029] Figure 5 is a schematic diagram of the raw data processing process provided in the embodiments of this disclosure;

[0030] Figure 6 is a schematic diagram of the first prompt text and the second prompt text provided in the embodiments of this disclosure;

[0031] Figure 7 is a schematic diagram of the overall technical architecture of the speech generation method provided in the embodiments of this disclosure;

[0032] Figure 8 is a schematic diagram of the internal processing of the overall technical architecture of the speech generation method provided in the embodiments of this disclosure;

[0033] Figure 9 is a schematic diagram of the speech generation module framework based on a large language model provided in an embodiment of this disclosure;

[0034] Figure 10 is a schematic diagram of the configuration process of the live chatbot provided in the embodiments of this disclosure;

[0035] Figure 11 is a schematic diagram of the speech generation device provided in an embodiment of this disclosure;

[0036] Figure 12 is a partial structural block diagram of the terminal provided in an embodiment of this disclosure;

[0037] Figure 13 is a partial structural block diagram of the server provided in an embodiment of this disclosure. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0039] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. The target object can be a user. In addition, when embodiments of this disclosure require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this disclosure be obtained.

[0040] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0041] To facilitate understanding of the technical solutions provided in the embodiments of this disclosure, some key terms used in the embodiments of this disclosure will be explained below:

[0042] Speech generation is an artificial intelligence technology centered on text-to-speech (TTS). It transforms text into natural speech waveforms by simulating the human vocal mechanism. Compared to text synthesis, speech generation focuses more on extended functions, such as timbre conversion, prosody adjustment, and voice cloning.

[0043] Content constraint information refers to the information that is expected to be converted into speech during the speech generation process. It can be text information, voice information, or image information, etc., and is used to impose content constraints on the generated speech during speech generation.

[0044] Embedding: Embedding is the process of transforming information into a numerical representation or feature vector. For example, for content constraint information, by extracting the information embedding of the content constraint information, the features in the content constraint information can be mapped to a high-dimensional vector space.

[0045] Style-cued speech: Represents the style that the target object wants to imitate in the generated speech. It is used to constrain the style during speech generation. This style can include multiple aspects such as the speaker's intonation, speech rate, pronunciation characteristics, and emotional expression, providing a specific style reference for speech generation.

[0046] Phonological cue speech: This is a timbre sample that the target object wants to imitate in speech generation. It can be a speech produced by a specific speaker, which can represent the unique timbre characteristics of the speaker to be imitated, and provide a specific timbre reference for speech generation.

[0047] Large Language Models (LLMs) are deep learning models trained on large amounts of text data that can generate natural language text or understand the meaning of language text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0048] Prompt text represents a command or instruction used to instruct a large language model to perform an action or generate output; that is, to instruct the model what action to take or what output to generate when performing a specific task.

[0049] With the development of artificial intelligence technology, zero-shot speech generation is becoming increasingly widespread, finding applications in scenarios such as voice assistants, audiobooks, and map navigation. However, the speech generated by current zero-shot speech generation systems is relatively fixed in timbre and style, and its realism and flexibility need to be improved.

[0050] Based on this, the present disclosure provides a speech generation method, apparatus, electronic device, and storage medium, which can improve the realism and flexibility of the generated target speech.

[0051] Referring to Figure 1, which is a schematic diagram of an implementation environment provided in an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0052] For example, server 102 can obtain content constraint information for content constraint during speech generation, and extract information embeddings from the content constraint information; obtain style cue speech for style constraint during speech generation, and extract multiple original speech tags from the style cue speech; input the information embeddings and multiple original speech tags into a first language model for prediction to obtain multiple target speech tags; obtain timbre cue speech for timbre constraint during speech generation, and extract the first speaker object embedding from the timbre cue speech; and generate speech based on the first speaker object embedding and multiple target speech tags to obtain target speech. The content constraint information, style cue speech, and timbre cue speech can be sent from terminal 101 to server 102, and server 102 can send the generated target speech to terminal 101.

[0053] In addition, the process of generating the target speech described above can also be performed only on the terminal 101 side.

[0054] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 102 can also be a node server in a blockchain network.

[0055] Terminal 101 may be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 101 and server 102 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the disclosure does not impose any limitations.

[0056] For example, the speech generation method in this disclosure is applicable to various specific application scenarios, such as virtual human voice assistants, live chatbots, audio and video conferencing, system AI intelligent assistants, and in-vehicle voice interaction systems. Virtual human voice assistants and live chatbots are used as examples for illustration:

[0057] (1) Virtual human voice assistant.

[0058] Referring to Figure 2, which is a schematic diagram of the speech generation method provided in this embodiment applied to a virtual human voice assistant scenario, the figure shows a virtual human voice assistant named "Little Assistant," designed to provide various services such as information query, schedule management, and entertainment interaction. By employing the above speech generation method, the target user can ask "Little Assistant" "How's the weather today?" After receiving the request, "Little Assistant" extracts content constraint information from the request and embeds it. Then, based on the target user's preferences or the atmosphere of the current scene, "Little Assistant" selects a suitable style prompt voice. To maintain the personalized characteristics of "Little Assistant," it selects a specific timbre as the timbre prompt voice. The final target voice, similar in timbre to the timbre prompt voice and similar in style to the style prompt voice, makes "Little Assistant's" response more natural and realistic, enhancing the target user's immersion and trust. This makes "Little Assistant" a more personalized voice assistant, enabling more effective interaction with the target user and improving the user experience.

[0059] (2) Live chatbot.

[0060] Referring to Figure 3, which is a schematic diagram of the speech generation method provided in this embodiment applied to a live-stream chatbot scenario, in a live-stream, the chatbot not only needs to respond quickly to audience questions but also needs to communicate in a natural and engaging manner. Using the aforementioned speech generation method, the live-stream chatbot can reply with a voice similar to the host's style or the target audience's style, enhancing the immersion and interactivity of the live stream. Furthermore, by adjusting style and tone prompts, the live stream can possess a unique "sound identifier," enhancing brand image and memorability. The speech generation method can also quickly process large amounts of text input and convert it into natural and fluent speech output, ensuring the chatbot can respond to audience questions instantly. Therefore, whether it's a formal product launch live stream or a lighthearted entertainment live stream, the chatbot can adjust its voice style according to the scenario, maintaining consistency with the live stream atmosphere. In addition, during peak audience question periods, the chatbot can share some of the reply workload, allowing the host to focus more on presenting the live stream content while maintaining interaction with the audience. Therefore, applying the above-mentioned voice generation method to live chatbots not only enhances the interactivity and personalization of the live stream, but also improves the robot's response efficiency, bringing viewers a more natural and smooth live streaming experience.

[0061] The methods provided in this disclosure can also be applied to different scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0062] Referring to Figure 4, which is a flowchart of the speech generation method provided in the embodiments of this disclosure, the speech generation method can be executed by the server or the terminal alone, or by the terminal and the server in cooperation. The speech generation method includes, but is not limited to, the following steps 401 to 405.

[0063] Step 401: Obtain content constraint information for content constraint during speech generation, and extract the information embedding of the content constraint information;

[0064] In step 401 above, the information used for content constraints during speech generation can be called content constraint information or speech generation input content (referred to as input content). This is text, speech, or image information that is intended to be converted into sound during speech generation, used for content constraints (or semantic constraints) to ensure the semantic accuracy of the generated speech. The input content can be obtained through user input or extracted from existing information. Specifically, when the input content is text, it can be text information that the user wants to be read aloud or converted into speech; this text can be sentences, paragraphs, articles, or any form of text content. When the input content is speech, it can be an existing speech segment or an input speech segment. When the input content is an image, it can be an input picture, a video frame selected from the target object's input video, or a screenshot from the current live broadcast scene, etc.

[0065] Information embedding transforms content constraint information into a numerical representation or feature vector. By extracting the information embedding from the content constraint information, the features within the content constraint information can be mapped into a high-dimensional vector space. It should be noted that information embedding can be obtained by processing the content constraint information using a deep learning model, such as by performing word embedding processing on the content constraint information using a word embedding model (Word2Vec).

[0066] Step 402: Obtain style cue speech for style constraint during speech generation, and extract multiple original speech tags from the style cue speech to characterize speech style features;

[0067] In step 402 above, the style-cued speech can be used to represent the style that the target object wants to imitate in the generated speech, and to constrain the style during speech generation. This style can include multiple aspects such as the speaker's intonation, speech rate, pronunciation characteristics, and emotional expression, providing a specific style reference for speech generation.

[0068] Original speech tokens, also known as style acoustic features, are a type of speech token. There are multiple original speech tokens, which can be a series of acoustic features or speech units extracted from style-cued speech. These units provide stylistic references and material for subsequent speech generation. It's important to note that original speech tokens are a discrete representation of style-cued speech; they can be any identifiable unit within the style-cued speech that reflects its stylistic features. These units can be phonemes, syllables, words, or other acoustic features. Original speech tokens can be obtained by processing style-cued speech using deep learning models, such as speech recognition models, acoustic feature extraction algorithms, or speech tokenizers.

[0069] Step 403: Input the embedded information and multiple original speech tags into the first large language model to obtain multiple target speech tags predicted and generated by the first large language model;

[0070] In step 403 above, the first large language model is a pre-trained large language model. In this embodiment of the disclosure, information embedding and original speech tags can be used as inputs to the first large language model. These input information together constitute the context information for model prediction. Based on the input context information, the first large language model predicts multiple target speech tags that are similar to the style of the style prompt speech style.

[0071] Target speech tokens are also a type of speech token. There are multiple target speech tokens, which are the output of the primary language model. They represent speech features similar to the style-cued speech style. These tokens are learned during model training and can capture subtle differences in speech, such as intonation, speech rate, and pronunciation habits. The introduction of target speech tokens makes the speech generation process more flexible and controllable. In this embodiment, by adjusting the input of the primary language model, such as information embedding and style-cued speech, speech with different content and styles can be generated, thereby meeting the needs of various application scenarios.

[0072] Step 404: Obtain timbre-cue speech for timbre constraint during speech generation, and extract a first speaking object embedding from the timbre-cue speech to characterize the timbre features of the speaking object;

[0073] In step 404 above, the timbre-cue speech is a timbre sample that the target object wants to imitate in speech generation. It can be a speech produced by a specific speaker, which can represent the unique timbre characteristics of the speaker to be imitated, and provide a specific timbre reference for speech generation.

[0074] The first speaker embedding is a numerical representation or feature vector extracted from the timbre-based cues, capturing the feature information of the speaker in the timbre-based cues. It should be noted that the first speaker embedding can be obtained by processing the timbre-based cues using a deep learning model, such as a speaker recognition model, or by encoding the timbre-based cues using a speaker encoder.

[0075] Step 405: Generate speech based on the embedding of the first speaking object and multiple target speech tags.

[0076] In step 405 above, the generated speech can be referred to as target speech, which is the final speech signal generated by the speech generation method in this embodiment of the disclosure. It is generated based on multiple factors such as content constraint information provided by the target object, style prompts, and timbre prompts, aiming to achieve flexible control over content, style, and timbre. The target speech has similar style features to the style prompts, similar timbre features to the timbre prompts, and includes information from the content constraint information. Therefore, the target speech is more natural, realistic, and flexible, and can meet the speech generation needs of the target object in different scenarios.

[0077] It should be noted that, in the embodiments of this disclosure, the first speaking object embedding and multiple target speech tags can be used as input features and input into a pre-trained generative model or framework. The model will comprehensively consider these input features, including content, style, timbre and acoustic environment, to generate high-quality speech signals and obtain the desired target speech. The speech signals generated in this way are more natural and realistic, and can flexibly adapt to different content and style requirements.

[0078] For example, embodiments of this disclosure may embed a first speaking object and multiple target speech tags into a pre-trained flow matching module. However, it should be noted that the flow matching module is just an example. In addition, other conditionally generated artificial intelligence generated content (AIGC) models, such as diffusion-based or other technology-based generation models, can be used. These models can generate high-quality speech signals based on input features.

[0079] In summary, by executing steps 401 to 405, this embodiment of the present disclosure can obtain content constraint information for content constraint during speech generation, extract information embedding of the content constraint information, and constrain the content in the target speech through information embedding during speech generation. Furthermore, by obtaining style cue speech for style constraint during speech generation, extracting multiple original speech tags from the style cue speech, and inputting the information embedding and multiple original speech tags into a first large language model for prediction, multiple target speech tags with styles similar to the style cue speech can be predicted based on the first large language model. On this basis, by obtaining content constraint information for content constraint during speech generation... The method involves constraining the timbre of the timbre-cued speech, extracting the first speaker object embedding from the timbre-cued speech, and then generating speech based on the first speaker object embedding and multiple target speech tags. This results in target speech with a timbre similar to the timbre-cued speech and a style similar to the style-cued speech, thus achieving control over style and timbre and making the target speech more natural, thereby improving the realism of the target speech. Furthermore, since content constraint information, style-cued speech, and timbre-cued speech are introduced at different stages of the generation process, the content, style, and timbre of the target speech are decoupled, supporting flexible combinations of arbitrary content, style, and timbre, thereby improving the flexibility of the target speech.

[0080] The above steps 401 to 405 have described the overall process of the speech generation method in this embodiment of the present disclosure. The following steps will describe in detail the further contents included in the steps.

[0081] The style-cue speech can include the style-cue speech corresponding to each content segment in the content constraint information (also called segment style-cue speech). The style-cue speech of each content segment can be retrieved from a preset knowledge base. The above-mentioned acquisition of style-cue speech for style constraint during speech generation can specifically involve inputting the content constraint information into a second language model to obtain the first style description text predicted by the second language model and the first sentiment description text of each content segment in the content constraint information. The first style description text is used to describe the speaking style of the content constraint information, and the first sentiment description text of each content segment is used to describe its corresponding sentiment attribute. Based on the embedding of the first style description text, a first global embedding of each content segment is constructed, and based on the embedding of the first sentiment description text of each content segment, a first detail embedding of each content segment is constructed. Wherein, the first global embedding is the embedding representation of the first style description text, and the first detail embedding is the embedding representation of the first sentiment description text. For each content segment, based on the first global embedding and the first detail embedding of the content segment, the style-cue speech corresponding to the content segment is retrieved from the preset knowledge base.

[0082] The knowledge base is a database or data set that stores various styles of prompting voices and their related information. It can be used to retrieve style prompting voices that match the input descriptive text. It should be noted that the various style prompting voices pre-stored in the knowledge base can be configured as needed. For example, it can include multiple types of style prompting voices, such as different types of intonation, speaking speed, pronunciation characteristics, and emotional expression. Each style type can have multiple different instances. These voice instances are labeled or indexed so that one or more of the most matching voices can be quickly retrieved based on the input descriptive text. This embodiment of the disclosure does not impose specific limitations on this.

[0083] The second large language model is a pre-trained large language model. In this embodiment, content constraint information can be used as the input of the second large language model, and the content constraint information constitutes the context for model prediction. Based on the input context information, the second large language model predicts a first style description text for describing the speaking style of the content constraint information, and a first sentiment description text for describing the sentiment of each content fragment in the content constraint information.

[0084] The first style description text describes the speaking style of the speaker within the content constraints. It can contain vocabulary related to various style types, such as intonation, speaking speed, pronunciation characteristics, and emotional expression. These words reflect the overall speaking style or intonation adopted by the speaker. The first emotion description text describes the emotion of each content segment within the content constraints. Different content segments may have different first emotion description texts. Through the first emotion description text, the emotional state expressed in the content segment, such as happiness, sadness, surprise, and anger, can be specifically reflected, providing more detailed guidance for the speech generation process.

[0085] The first global embedding is an embedding representation based on the first style description text. It captures the overall speaking style features of the speaker from the content constraint information. For example, the first style description text contains words or phrases that describe the speaking style, such as "formal," "casual," and "cheerful." This text is input into an embedding layer, such as the Word2Vec module, to generate corresponding embedding vectors. Then, these embedding vectors are combined into a global embedding vector, which is the first global embedding.

[0086] The first detail embedding is an embedding representation based on the first sentiment description text. It captures the specific sentiment features of each content fragment within the content constraint information. A content fragment is a segment within the content constraint information; for example, when the content constraint information is text, a content fragment could be a sentence within the content constraint information. Similar to the first global embedding, the first sentiment description text is also input into the embedding layer to generate corresponding embedding vectors. However, unlike global embedding, detail embedding generates individual embeddings for each content fragment or sentiment description, or combines them into multiple detail embedding vectors using methods such as sequence models and attention mechanisms to reflect the sentiment changes in different parts of the content. Ultimately, the first detail embeddings for each content fragment can be obtained.

[0087] It should be noted that, when retrieving style-cue speech, this embodiment uses a first global embedding and a first detailed embedding as query vectors to search the knowledge base for speech instances that best match these query vectors. This can be achieved by matching based on the similarity between the embedding vectors and each speech instance, or by scoring each speech instance based on the embedding vectors. Ultimately, by comprehensively considering global style and detailed sentiment information, this embodiment can retrieve style-cue speech that best matches the content constraint information in both overall style and specific sentiment, thus enabling style constraints to be applied during speech generation.

[0088] Furthermore, the knowledge base stores reference speech segments (or candidate speech segments) and second global embeddings and second detail embeddings associated with the reference speech segments. The second global embedding indicates the speaker style features of the reference speech segment, and the second detail embedding indicates the emotional attributes of the reference speech segment. For example, the knowledge base stores multiple reference speech segments and second global embeddings and second detail embeddings associated with each reference speech segment. The second global embedding associated with each reference speech segment indicates the speaker style features of that reference speech segment, and the second detail embedding associated with each reference speech segment indicates the emotional attributes of that reference speech segment.

[0089] The reference speech segments are speech sample instances stored in a preset knowledge base. These speech samples may have different speaking style characteristics and emotional attributes. In this embodiment, the reference speech segments are used as candidates for style-cue speech to be selected based on the style requirements of the content constraint information.

[0090] The second global embedding is an embedding vector associated with a reference speech segment, used to indicate the speaker's style features. It's important to note that in the knowledge base, each reference speech segment is associated with a second global embedding. This embedding vector captures the overall style features of the speech segment, and multiple reference speech segments can be associated with the same second global embedding. The second detail embedding is another embedding vector associated with a reference speech segment, used to indicate its emotional attributes. Similar to the second global embedding, each reference speech segment is associated with a second detail embedding. This embedding vector captures the emotional features within the speech segment. In this case, one reference speech segment can correspond to one content segment.

[0091] For each content segment, based on the first global embedding and the first detail embedding of that content segment, the style-related audio prompts are retrieved from a pre-defined knowledge base. Specifically, this can be done as follows:

[0092] For each content segment: based on the similarity between the first global embedding of the content segment and the second global embedding associated with each reference speech segment, multiple first reference speech segments are retrieved from the knowledge base; based on the similarity between the first detail embedding of the content segment and the second detail embedding associated with each first reference speech segment, the style cue speech corresponding to the content segment is retrieved from the multiple first reference speech segments.

[0093] It should be noted that, in this embodiment of the disclosure, by calculating the similarity between the first global embedding and the second global embedding of each reference speech segment in the knowledge base, such as cosine similarity or Euclidean distance, multiple first reference speech segments with similar styles to the content constraint information can be retrieved. These first reference speech segments match the requirements of the content constraint information in terms of the speaking style. Subsequently, among the multiple first reference speech segments retrieved in the first step, by calculating the similarity between the first detail embedding and the second detail embedding of each first reference speech segment, reference speech segments that match the emotional needs of the content constraint information can be further filtered out. This ensures that the final selected style prompt speech not only matches the content constraint information in terms of the speaking style but also in terms of emotional details, thereby retrieving style prompt speech more accurately and precisely.

[0094] For example, in this embodiment of the present disclosure, after calculating the similarity between the first global embedding and the second global embedding of each reference speech segment in the knowledge base, a first similarity value between the current first global embedding and each second global embedding can be obtained. Further, this embodiment of the present disclosure can sort the first similarity values ​​and select the top M first similarity values ​​in descending order, and use the reference speech segments to which the second global features corresponding to the top M first similarity values ​​belong as multiple first reference speech segments with a style similar to the content constraint information, where the size of M can be set according to actual needs; or, this embodiment of the present disclosure can set a first similarity threshold and select first similarity values ​​greater than or equal to the first similarity threshold, using the reference speech segments to which the second global features corresponding to these first similarity values ​​belong as multiple first reference speech segments with a style similar to the content constraint information, where the size of the first similarity threshold can be set according to actual needs.

[0095] Similarly, in this embodiment of the present disclosure, after calculating the similarity between the first detail embedding and the second detail embeddings of the plurality of first reference speech segments retrieved in the first step, a second similarity value between the current first detail embedding and each of the second detail embeddings can be obtained. Further, this embodiment of the present disclosure can sort the various second similarity values ​​and select the top K second similarity values ​​in descending order, and use the reference speech segments to which the second detail embeddings corresponding to the top K second similarity values ​​belong as style prompt speech corresponding to the content segment, where the size of K can be set according to actual needs; or, this embodiment of the present disclosure can set a second similarity threshold and select second similarity values ​​greater than or equal to the second similarity threshold, using the reference speech segments to which the second detail features corresponding to these second similarity values ​​belong as style prompt speech corresponding to the content segment, where the size of the second similarity threshold can be set according to actual needs.

[0096] Before retrieving the style prompt speech corresponding to each content segment from the preset knowledge base based on the first global embedding and the first detail embedding of each content segment, the embodiments of this disclosure may further obtain the original speech, segment the original speech according to the speaker to obtain multiple speech blocks; perform speech recognition on each speech block to obtain the speech block text corresponding to each speech block; for each speech block text, segment the speech block text into multiple reference text segments, and segment the corresponding speech block into reference speech segments corresponding to each of its reference text segments; based on the multiple speech block texts, construct the second global embedding of each reference text segment of each speech block text, construct its second detail embedding based on each reference text segment of each speech block text, and associate and store each reference speech segment, its second global embedding, and its second detail embedding in the knowledge base.

[0097] The original speech is a segment of speech data to be processed, which may contain dialogue, monologue, or speech by multiple speakers. The original speech can be obtained from audio files, audio streams, or based on input from the target audience. In this embodiment, after acquiring the original speech, it can be segmented into multiple independent speech blocks, each containing the speech of only one speaker. Furthermore, this embodiment can perform other audio processing on the original speech, including noise reduction, scoring, and filtering, to improve the quality of the segmented speech blocks.

[0098] After obtaining each speech block, this embodiment of the present disclosure can perform speech recognition on each speech block separately. During the speech recognition process, the audio features of each speech block can be analyzed to extract the contained text content, and each speech block can be converted into text corresponding to the contained speech audio, ultimately obtaining the speech block text in each speech block. Furthermore, this embodiment of the present disclosure can implement speech recognition of each speech block based on a preset speech recognition model, and this embodiment of the present disclosure does not impose specific limitations on this.

[0099] After obtaining the text of each speech block, embodiments of this disclosure can segment each speech block text separately, breaking down speech block text containing multiple sentences or paragraphs into smaller text units, thereby obtaining multiple reference text segments. Each reference text segment can represent a sentence or a paragraph, or each reference text segment can represent at least two of multiple sentences or paragraphs. Similarly, embodiments of this disclosure can segment each speech block separately to obtain reference speech segments corresponding to each reference text segment.

[0100] Finally, for each speech block text, embodiments of this disclosure can construct a corresponding second global embedding using word embedding or sentence embedding to capture the overall semantic information of the speech block text. Furthermore, a more granular analysis is performed on the speech block text; for each reference text segment, a similar method is used to construct a detailed embedding to capture the local semantic information of the text segment. Subsequently, embodiments of this disclosure associate each reference speech segment with its corresponding second global embedding and second detailed embedding, and store them in a knowledge base, so that each reference speech segment in the knowledge base has a corresponding second global embedding and second detailed embedding for retrieval and use in subsequent speech generation processes.

[0101] The above embodiments will now be illustrated with reference to the figures.

[0102] Referring to Figure 5, which is a schematic diagram of the raw data processing process provided in this embodiment, after acquiring the raw speech, noise reduction processing can be performed on the raw speech. After noise reduction, language identification is performed on the speech to determine its language, facilitating subsequent text recognition. After language identification, speaker diarization processing is performed on the speech. Speaker diarization processing can segment the raw speech into speech segments, determining the start and end times of each speech segment in the raw speech, and obtaining speech blocks corresponding to each speaker ID after segmentation. Subsequently, this embodiment can perform segmentation on each speech block based on voice activity detection (VAD). Based on VAD, a score can be obtained for each speech block, and speech blocks with scores higher than a preset value are selected for subsequent processing. Next, this embodiment obtains the text data corresponding to each speech block through Automatic Speech Recognition (ASR) of different languages, such as obtaining the speech block text under Chinese or English recognition, and after filtering by word error rate (WER), the speech block text is saved into JSON structured data.

[0103] Next, this embodiment performs chunking processing on the speech block text and its corresponding speech corpus into smaller blocks, obtaining multiple reference text segments and corresponding reference speech segments for each reference text segment. Finally, a second global embedding is constructed for each reference text segment based on the multiple speech block texts, and a second detail embedding is constructed for each of the multiple reference text segments of each speech block text. Each reference speech segment, its corresponding second global embedding, and its second detail embedding are then associated and stored in a knowledge base. Chunking is particularly important for improving the accuracy of sentiment style retrieval, refining the granularity of retrieval, and avoiding length issues. Furthermore, this embodiment employs three different levels of text chunking methods based on text. For example, a tag-level chunking method can be used to tag each speech block text, and chunking can be performed based on the tagging results, which is simple and straightforward. Alternatively, a semantic-level chunking method can be used, employing a pre-trained large language model to determine the breakpoints of the speech block text, thereby enabling more contextual consideration in chunking. Or, a sentence-level chunking method can be used to divide the text into chunks corresponding to each sentence, achieving a balance between preserving text semantics and conciseness and efficiency.

[0104] The above-mentioned construction of a second global embedding of each reference text segment of each speech block text based on multiple speech block texts, and construction of a second detailed embedding of each reference text segment of each speech block text based on each reference text segment of each speech block text, can specifically be as follows: A first prompt text is constructed based on multiple speech block texts; the first prompt text is input into a second large language model, and a second style description text is output to describe the style features of each speaker; the first prompt text is used to prompt the second large language model to output the second style description text of each speaker based on multiple speech block texts; a second global embedding of each reference text segment of the speech block text of each speaker is constructed based on the embedding of the second style description text of each speaker; for each reference text segment in the speech block text of each speaker, a second prompt text is constructed based on the second style description text of the speaker, the reference text segment, and multiple speech block texts; the second prompt text is input into the second large language model, and a second sentiment description text is output to describe the sentiment attributes of the reference text segment; the second prompt text is used to prompt the second large language model to output the second sentiment description text based on the second style description text, the reference text segment, and multiple speech block texts; and a second detailed embedding is constructed based on the embedding of the second sentiment description text of each reference text segment.

[0105] The first prompt text is a prompt text constructed based on multiple speech block texts. It is used to prompt the second language model to generate a text describing the stylistic features of each speaker based on multiple speech block texts. Therefore, the first prompt text may contain some guiding words or sentence structures to guide the second language model in generating the text.

[0106] After obtaining the first prompt text, this embodiment of the present disclosure can input the first prompt text and multiple speech block texts together into a trained second language model. The second language model can generate a text describing the speaking style of each speech block text, i.e., a second style description text. Finally, this embodiment of the present disclosure converts the second style description text into an embedding form, which is a second global embedding representing the overall style of the speech block of the speaking object.

[0107] The second prompt text is similar to the first prompt text. It is a prompt text constructed based on the second style description text of the speaker, the reference text fragment, and multiple speech block texts for each reference text fragment in the speech block text of each speaker. It is used to prompt the second language model to generate a text describing the emotion of the reference text fragment based on the second style description text, the reference text fragment, and multiple speech block texts. Therefore, the second prompt text may also contain some guiding words or sentence structures.

[0108] After obtaining the second prompt text, this embodiment of the present disclosure can input the second prompt text, the second style description text, the reference text fragment, and multiple speech block texts together into a second large language model. Based on the input second prompt text and other relevant information, the second large language model generates a text describing the emotion of the reference text fragment, namely, the second emotion description text. Finally, this embodiment of the present disclosure converts the second emotion description text into an embedded form, which is the second detail embedding representing the specific emotion of the reference text fragment.

[0109] Through the above steps, each reference speech segment in the knowledge base is assigned a second global embedding and a second detail embedding. This embedding information will be used in the subsequent speech generation process to retrieve and match style-cued speech, thereby ensuring that the generated speech has the required style and emotional features.

[0110] The first and second prompt texts in the above embodiments are illustrated below:

[0111] Referring to Figure 6, which is a schematic diagram of the first prompt text and the second prompt text provided in an embodiment of this disclosure. In this embodiment, the first prompt text may include not only multiple speech blocks but also some guiding words or sentence structures. For example, the first prompt text may be "Given the following dialogue between speakers {dialogue text content} Based on the above dialogue, what kind of style and personality do you think the speaker {speaker ID} has? (Note: limited to 150 words.)", where the dialogue text content consists of multiple speech blocks, and the speaker ID is the corresponding speaker ID. Similarly, in addition to the second style description text, reference text fragments, and multiple speech block texts, the second prompt text can also include some guiding words or sentence structures. For example, the second prompt text could be "system: You are an expert in analyzing the emotional state of a speaker in a dialogue. Given the personality style description of the speaker {speaker ID}: {speaker persona}; given the following dialogue as context {conversation}", "user: Based on the above dialogue and the personality style description of the speaker, what is the emotional label of the sentence {utterance}?" and "assistant: The emotional label of the sentence {utterance} is {emotion}", where speaker persona is the second style description text, conversation is multiple speech block texts, utterance is a reference text fragment, and emotion is the second emotional description text to be output.

[0112] In addition to the above, there are other ways to construct a second global embedding based on multiple speech blocks and a second detail embedding based on a reference text segment. Furthermore, embodiments of this disclosure can also directly extract the corresponding second global embedding and second detail embedding based on the original speech. For example, embodiments of this disclosure can use audio feature extraction technology to capture the acoustic characteristics of the original speech to reflect the speaker's tone quality, speech rate, intonation, and other stylistic features. The extracted features are then input into a pre-trained neural network model, which can be a convolutional neural network (CNN), a recurrent neural network (RNN), or a Transformer. The model's task is to map these features into a low-dimensional embedding space, where the vectors represent the speaker's style. Through training, the model can learn the differences between different speaker styles and generate the corresponding second global embedding. Similarly, in addition to the acoustic features mentioned above, embodiments of this disclosure can also extract emotion-related features, such as prosodic changes, timbre changes, pauses and stresses in speech, and generate corresponding second detail embeddings from the extracted features. The generation process of the second detail embedding is similar to that of the second global embedding, and this disclosure does not impose specific limitations on it.

[0113] The above-mentioned method of retrieving the matching speech segment from multiple first reference speech segments based on the similarity between the first detail embedding of the content segment and the second detail embedding associated with each first reference speech segment can specifically involve retrieving multiple second reference speech segments from multiple first reference speech segments based on the similarity between the first detail embedding and the second detail embedding associated with each first reference speech segment; determining the contextual relevance between each second reference speech segment and the content segment, and identifying the second reference speech segment with the highest contextual relevance as the style cue speech corresponding to the content segment; or, further determining the similarity between the second global embedding and the first global embedding of each second reference speech segment, and identifying the second reference speech segment with the highest similarity as the style cue speech corresponding to the content segment; or, performing a speech quality assessment on each second reference speech segment, and identifying the second reference speech segment with the highest speech quality as the style cue speech corresponding to the content segment.

[0114] Furthermore, after retrieving multiple first reference speech segments from the knowledge base based on the similarity between the first global embedding and the second global embedding, this embodiment of the present disclosure can also retrieve multiple second reference speech segments from the multiple first reference speech segments based on the similarity between the first detail embedding and the second detail embedding. Specifically, for each reference speech segment in the knowledge base, this embodiment of the present disclosure can calculate the similarity between its second detail embedding and the first detail embedding of the content constraint information, then set a similarity threshold, and regard those reference speech segments with similarity exceeding the threshold as second reference speech segments, or sort all first reference speech segments from high to low similarity and select the top N as second reference speech segments.

[0115] Next, there are several ways to select the style-cue speech corresponding to the content segment from multiple second reference speech segments, for example:

[0116] This disclosure allows for the determination of the contextual relevance between each second reference speech segment and the content segment. This enables a better understanding of the contextual relationships between candidate speech segments and content segments, and a better grasp of subtle contextual differences, thereby providing higher ranking accuracy. For example, text analysis is performed on the content segment to extract key information, themes, or sentiments. Similarly, a similar text analysis is performed on each second reference speech segment to obtain its semantic content and sentiment expression. Then, the semantic similarity or contextual coherence between the content segment and the second reference speech segment is evaluated. Subsequently, based on the calculated contextual relevance, the second reference speech segment with the highest relevance is selected as the style cue speech corresponding to the content segment. This ensures that the style cue speech not only matches the style of the content constraint information but also maintains consistency in semantics and context.

[0117] Alternatively, in this embodiment, the similarity between the second global embedding and the first global embedding of each second reference speech segment can be determined again, global consistency rearrangement can be performed, and the second reference speech segment with the highest similarity can be selected as the style prompt speech based on the similarity calculation result, thereby ensuring that the speaking style of the style prompt speech is highly consistent with the speaking style of the content constraint information.

[0118] Alternatively, embodiments of this disclosure may also perform speech quality assessment on each second reference speech segment, using speech quality assessment tools or models to evaluate the speech quality of each second reference speech segment. This involves analyzing the acoustic features of the speech segments, such as spectrum, fundamental frequency, and noise level. For example, WER evaluation can be performed on each candidate speech segment to obtain the speech quality assessment results for each second reference speech segment. Subsequently, the second reference speech segment with the highest speech quality is determined as the style cue speech corresponding to the content segment, thereby ensuring that the style cue speech achieves optimal speech quality, which helps to improve the realism and naturalness of the target speech.

[0119] The above-mentioned construction of the first global embedding of each content fragment based on the embedding of the first style description text can specifically be to obtain the first relevant description text; and to fuse the embedding of the first style description text and the embedding of the first relevant description text to obtain the first global embedding of each content fragment.

[0120] The first relevant descriptive text is used to provide additional information or supplementary descriptions when retrieving style-cue speech, which can further enhance the understanding of the speaker's style and help to more accurately locate style-cue speech that matches the content constraint information. For example, the first relevant descriptive text may include attributes describing the speaker, such as the speaker's region and education level, which have a significant impact on determining the speaker's speaking style, tone, and word choice. In addition, the first relevant descriptive text may also include information related to the speaker's application scenario, gender, style and emotion, age, and region, etc., which are not specifically limited in this embodiment.

[0121] After obtaining the first relevant descriptive text, embodiments of this disclosure can fuse the embedding of the first style descriptive text and the embedding of the first relevant descriptive text to obtain the first global embedding of the content fragment. Specifically, the first style descriptive text and the first relevant descriptive text can be converted into embedded representations, and then the two embedded representations can be merged into a global embedding to obtain the desired first global embedding.

[0122] For example, there are several ways to fuse the embeddings of the first style description text and the first related description text. For instance, the embeddings of the first style description text and the first related description text can be fused by averaging, adding or averaging the two embedding vectors to obtain the first global embedding; alternatively, the embeddings of the first style description text and the first related description text can be fused by weighting, including assigning different weights according to the importance of each embedding and then summing them to obtain the desired first global embedding; alternatively, the embeddings of the first style description text and the first related description text can be fused by concatenation, combining the two embedding vectors into a longer vector to obtain the desired first global embedding; or alternatively, a pre-built attention mechanism can be used to determine the weights of the embeddings of the first style description text and the first related description text, and then summing them to obtain the desired first global embedding.

[0123] The first detailed embedding of each content fragment is constructed based on the embedding of the first sentiment description text of each content fragment. Specifically, for each content fragment, the first global embedding and the embedding of the first sentiment description text of that content fragment are fused to obtain the first detailed embedding of that content fragment.

[0124] It should be noted that the embodiments of this disclosure can integrate the first global embedding and the first emotional description text embedding, combining the two embedding representations to form a new embedding representation, obtaining the first detail embedding of the content fragment. This constructs an embedding representation that includes both the overall style of the speaker and the emotional details of the content fragment, helping to more accurately simulate the style and emotional state of the speaker during speech generation, thereby generating more natural and realistic speech. The fusion process may include, but is not limited to, vector concatenation, weighted averaging, attention mechanisms, etc., aiming to effectively combine the information from the two embeddings. The embodiments of this disclosure do not impose specific limitations on this.

[0125] In addition, in this embodiment, the embedding of the first emotional description text can be directly used as the first detail embedding of the content segment. It should be noted that when the embedding of the first emotional description text is chosen as the first detail embedding, it means that when retrieving style-cue speech, the primary basis is the emotional description of the content segment, without further considering the overall style characteristics of the speaker. This makes this embodiment applicable to scenarios where emotional expression dominates speech style, or when the overall style characteristics of the speaker have little impact on speech generation.

[0126] The content constraint information can be speech, and the content fragment is a speech fragment. The first detail embedding of the content fragment is constructed based on the embedding of the first emotion description text. Specifically, it can also be the speech emotion embedding of the speech fragment. The first global embedding, the embedding of the first emotion description text, and the speech emotion embedding are fused to obtain the first detail embedding of the content fragment.

[0127] Furthermore, when the content constraint information is speech and the content segment is a speech segment, embodiments of this disclosure can extract the speech emotion embedding of the speech segment. Here, speech emotion embedding refers to the embedded representation extracted from the speech segment that reflects its emotional characteristics. This embedding can be generated by analyzing features such as intonation, rhythm, and volume in the speech segment to capture emotional information in the speech.

[0128] After obtaining the speech emotion embedding of a speech segment, embodiments of this disclosure can fuse the first global embedding, the first emotion description text embedding, and the speech emotion embedding to obtain a more comprehensive and accurate first detail embedding of the content segment. This first detail embedding not only includes the overall style features of the speaker but also the emotional details and emotional information in the speech segment, thereby providing richer and more accurate references when retrieving style-cued speech from the knowledge base. The fusion process may include, but is not limited to, vector concatenation, weighted averaging, and attention mechanisms, aiming to effectively combine the information from the first global embedding, the first emotion description text embedding, and the speech emotion embedding. Embodiments of this disclosure do not impose specific limitations on this.

[0129] In addition, if the content constraint information is an image, the technical means of constructing the first detail embedding of the content fragment in the above embodiments is also applicable. Specifically, when the content constraint information is an image and the content fragment is an image block, the embodiments of this disclosure can extract the image sentiment embedding of the image block. Image sentiment embedding refers to the embedding representation extracted from the image block that can reflect its emotional characteristics. This embedding can be generated by analyzing features such as color, texture, character characteristics, or environmental characteristics in the image block to capture the emotional information in the image. Then, the first global embedding, the embedding of the first emotional description text, and the image sentiment embedding are fused to obtain the first detail embedding of the content fragment. The embodiments of this disclosure will not elaborate further on this.

[0130] As can be seen, in this embodiment, the first detail embedding can exist in multiple types depending on the modality of the content constraint information. When the content constraint information is text, the first global embedding and the first emotional description text embedding are fused to obtain the first detail embedding of the content segment. When the content constraint information is speech, the first global embedding, the first emotional description text embedding (the text obtained after speech recognition of the corresponding speech), and the speech emotion embedding are fused to obtain the first detail embedding of the content segment. When the content constraint information is an image, the first global embedding, the first emotional description text embedding (the text obtained after image recognition of the corresponding image), and the image emotion embedding are fused to obtain the first detail embedding of the content segment. Accordingly, different types of second detail embeddings can be associated with reference speech segments in the knowledge base and stored in different partitions. The second detail embedding is retrieved from the corresponding partition for retrieval based on the specific modality of the content constraint information.

[0131] The above-mentioned style prompt speech corresponding to the content segment is retrieved from the preset knowledge base based on the first global embedding and the first detail embedding. Specifically, the first global embedding and the first detail embedding can be input into a classifier for classification to obtain the style sentiment classification result of the content segment. The style sentiment classification result is used to indicate whether the style sentiment of the content segment is neutral or rich. When the style sentiment classification result indicates that the style sentiment of the content segment is rich, the style prompt speech corresponding to the content segment is retrieved from the preset knowledge base based on the first global embedding and the first detail embedding.

[0132] In this context, a classifier is a machine learning or deep learning model whose main task is to assign input data to predefined categories. In this embodiment, the classifier is used to analyze the first global embedding and the first detail embedding of a content fragment and output a style sentiment classification result. This result indicates whether the style sentiment of the content fragment is insufficient or sufficient. Insufficient style sentiment means that the content fragment has no obvious stylistic features or emotional expression, while sufficient style sentiment indicates that the content fragment has rich stylistic features or emotional expression.

[0133] It should be noted that the classifier can be one of many types of machine learning models, including but not limited to support vector machines, decision trees, random forests, neural networks, and their variants, and this disclosure does not impose specific limitations on these models. During the training phase, the classifier is trained using a large amount of labeled data, which includes embedding representations of content fragments and their corresponding style and sentiment categories. Through training, the classifier learns how to map new content fragment embeddings to the correct style and sentiment categories.

[0134] When the style and sentiment classification result indicates that the style and sentiment of the content fragment is rich, subsequent retrieval is required in order to express the style and sentiment. Based on this, the embodiments of this disclosure use the first global embedding and the first detailed embedding as retrieval conditions to search in a preset knowledge base. In the knowledge base, the style prompt speech that best matches these embedding representations is found and used in the subsequent speech generation steps to ensure that the generated speech matches the content fragment in style and sentiment.

[0135] The above-mentioned style prompt speech corresponding to the content segment is retrieved from the preset knowledge base based on the first global embedding and the first detail embedding. Specifically, when the style sentiment classification result indicates that the style sentiment of the content segment is neutral, the style prompt speech corresponding to the content segment is retrieved from the knowledge base based on the first global embedding.

[0136] Furthermore, when the style sentiment classification result indicates that the style sentiment of the content fragment is neutral, it means that the content does not have a particularly significant or diverse expression in style or sentiment. In this case, because the first global embedding summarizes the overall style of the speaker, the process of retrieving style-cue speech will mainly rely on the first global embedding. Based on this, the embodiments of this disclosure use the first global embedding as a retrieval condition to search in a preset knowledge base, and finally retrieve the style-cue speech corresponding to the content fragment, thereby ensuring that the generated speech is consistent with the overall style of the content constraint information in style, while avoiding the introduction of unnecessary emotional diversity.

[0137] However, not all content constraint information requires retrieval of style-cue voice prompts, especially when the style and sentiment of the content constraint information are classified as neutral. In this case, since the content itself does not have a strong style or sentiment tendency, this embodiment of the disclosure assumes that using any style-cue voice prompt from the preset knowledge base will not have a significant impact on the generated speech. Therefore, the additional step of retrieving a specific style-cue voice prompt is unnecessary, or because the system has default style settings, speech can be generated without a specific style prompt. Therefore, this embodiment of the disclosure can also further refine the process by retrieving style-cue voice prompts when the style and sentiment classification result indicates that the style and sentiment of the content segment is neutral, thereby improving the efficiency of speech generation.

[0138] The above-mentioned information embedding and multiple original speech tags are input into the first large language model for prediction to obtain multiple target speech tags. Specifically, it can be to extract the second speaking object embedding of style-cue speech, and input the information embedding, the second speaking object embedding and multiple original speech tags into the first large language model for prediction to obtain multiple target speech tags.

[0139] The second speaker embedding is a numerical representation or feature vector extracted from the style-cued speech, capturing the feature information of the speaker in the style-cued speech. It should be noted that the second speaker embedding can be obtained by processing the style-cued speech using a deep learning model, such as a speaker recognition model or a speaker encoder.

[0140] It should be noted that, in this embodiment of the present disclosure, by extracting the second speaking object embedding of the style prompt speech and multiple original speech tags in the style prompt speech, the information embedding, the second speaking object embedding and multiple original speech tags are input into the first large language model for prediction. Based on the first large language model, multiple target speech tags similar in style to the style prompt speech can be predicted. Furthermore, due to the addition of the second speaking object embedding, multiple target speech tags that are even more similar in style to the style prompt speech can be predicted.

[0141] The above-mentioned information embedding, second speaker embedding, and multiple original speech tags are input into the first large language model for prediction to obtain multiple target speech tags. Specifically, the sequence start marker, second speaker embedding, information embedding, speech conversion marker, and multiple original speech tags are sequentially concatenated and input into the first large language model for prediction to obtain multiple target speech tags.

[0142] The sequence start marker is a special marker used to inform the first language model that the input sequence has begun, helping the model identify the starting point of the input sequence and thus better process the input data. The speech conversion marker is also a special marker used to indicate to the first language model the conversion from information embedding to speech tag generation, helping the model understand that the current task is the conversion of text content to speech tags.

[0143] In this embodiment, the sequence start marker, the second speaking object embedding, the information embedding, the speech conversion marker, and multiple original speech markers can be sequentially concatenated to form a complete input sequence, which is then input into the first language model for prediction. The first language model will predict a series of target speech markers based on the input sequence, combined with its pre-trained knowledge and the current task, so that they can be used in subsequent speech generation steps to finally generate the target speech that meets the requirements.

[0144] The above-mentioned speech generation based on the embedding of the first speaker and multiple target speech tags yields the target speech. Specifically, it can be to extract the acoustic features of style-cue speech or timbre-cue speech, and then generate speech based on the acoustic features, the embedding of the first speaker, and multiple target speech tags to obtain the target speech.

[0145] The acoustic features are used to simulate the acoustic environment of style-cue or timbre-cue speech. For example, the acoustic features may include masked Mel features or Mel frequency features. Masked Mel features are Mel spectral features processed by masking operations, which can convert style-cue or timbre-cue speech into a frequency domain representation to simulate different acoustic environments or emphasize specific sound features. Mel frequency features can map the spectrum of style-cue or timbre-cue speech to a Mel frequency scale and then extract the coefficients of the spectral envelope. Therefore, Mel frequency features can capture the spectral information of style-cue or timbre-cue speech. Further, the embodiments of this disclosure will use masked Mel features as examples of acoustic features in the following examples; however, this disclosure does not impose specific limitations on these embodiments.

[0146] The acoustic features in this embodiment are extracted from style-cue speech or timbre-cue speech. Specifically, by extracting acoustic features from style-cue speech, the acoustic environment of the style-cue speech can be simulated, and these features help generate speech with a style similar to the style-cue speech. Similarly, by extracting acoustic features from timbre-cue speech, the acoustic environment of the timbre-cue speech can be simulated, and these features help generate speech with a timbre similar to the timbre-cue speech. It should be noted that in practical applications, the target object can choose to generate speech with both a specific style and a specific timbre. Therefore, this embodiment allows the target object to control these two aspects separately by providing style-cue speech and timbre-cue speech. Furthermore, the acoustic features can also be extracted from style-cue speech and timbre-cue speech, such as extracting sub-acoustic features from style-cue speech or timbre-cue speech respectively, and then fusing these two sub-acoustic features to obtain the desired acoustic features. This embodiment does not impose specific limitations on this approach.

[0147] It should be noted that, in this embodiment of the present disclosure, by extracting the acoustic features of the style prompt speech or the timbre prompt speech, the acoustic environment of the style prompt speech or the timbre prompt speech can be simulated through the acoustic features. Then, based on the acoustic features, the embedding of the first speaking object, and multiple target speech tags, speech generation is performed to obtain target speech that is similar in timbre to the timbre prompt speech and similar in style to the style prompt speech, thereby realizing the control of style and timbre.

[0148] It can also perform quality checks on style prompts or timbre prompts based on the acoustic environment. When the quality check result indicates poor acoustic environment quality (e.g., loud reverberation and noise), the recording function can be invoked to obtain a segment of near-end speech to extract acoustic features. Alternatively, pre-stored template speech can be read to extract acoustic features, thereby improving the stability and reliability of acoustic feature extraction.

[0149] The speech generation method in this disclosure will be illustrated below with specific examples.

[0150] Referring to Figure 7, which is a schematic diagram of the overall technical architecture of the speech generation method provided in this embodiment, the speech generation model architecture in this embodiment can include three parts: a knowledge base retrieval module, a style retrieval module, and a speech generation module based on a large language model. The style retrieval module can also be called a Retrieval Augmented Generation (RAG) style retrieval module. The knowledge base retrieval module is responsible for constructing a multimodal knowledge base based on text and speech corpora. The knowledge base contains style and sentiment information that can be retrieved, as well as corresponding speech and text data. The RAG style retrieval module is responsible for analyzing style and sentiment information based on input constraint information, retrieving the most relevant speech and text data from the multimodal knowledge base, and organizing and outputting it as input prompts for the next module. The speech generation module based on a large language model deploys a first large language model, responsible for generating target speech that conforms to the style and timbre based on style prompts, content constraint information, and timbre prompts from the RAG style retrieval module.

[0151] The following is a further detailed description of the specific technical implementation and processing details of the three major modules. Referring to Figure 8, which is a schematic diagram of the internal processing of the overall technical architecture of the speech generation method provided in this embodiment,...

[0152] In the knowledge base retrieval module, constructing a multimodal knowledge base requires deciding how to preprocess and segment the original speech into blocks, selecting which semantic embedding representations to use, and choosing a suitable vector database for efficient storage of embedding vectors. Specifically, after acquiring the original speech, noise reduction processing can be performed, followed by language identification to determine the language, facilitating subsequent text recognition. After language identification, speaker diarization processing is performed, which segments the original speech into segments, determining the start and end times of each speech segment, resulting in speech blocks corresponding to each speaker ID. Subsequently, this embodiment can use Voice Activity Detection (VAD) to prune each speech block. Based on VAD, a score can be obtained for each speech block, and speech blocks with scores higher than a preset value are selected for further processing. Next, this embodiment obtains the text data corresponding to each speech block through Automatic Speech Recognition (ASR) in different languages, such as obtaining the speech block text in Chinese or English, and saves it into JSON structured data after filtering by word error rate (WER).

[0153] Next, this embodiment performs chunking processing on the speech block text and the corresponding speech corpus into smaller blocks, resulting in multiple reference text segments and corresponding reference speech segments for each reference text segment. Chunking is particularly important for improving the accuracy of sentiment style retrieval, refining retrieval granularity, and avoiding length issues. Furthermore, this embodiment employs three different levels of text chunking methods based on text. For example, a tag-level chunking method can be used to tag each speech block text, thereby chunking based on the tagging results—simple and clear. Alternatively, a semantic-level chunking method can be used, employing a pre-trained large language model to determine the breakpoints of the speech block text, thus better combining context for chunking. Or, a sentence-level chunking method can be used to divide the text into chunks corresponding to each sentence, achieving a balance between preserving text semantics and conciseness and efficiency.

[0154] In the process of constructing a second global embedding based on multiple speech blocks and a second detail embedding based on a reference text fragment, to extract embedding semantics that can represent style and sentiment information, this embodiment uses a model called PersoERC, fine-tuned based on LLM, as the second major language model. It utilizes the context of the dialogue text blocks to perform style profiling on each speaker, and then combines the style profiling to perform sentiment recognition on each sentence spoken by the speaker. Specifically, this embodiment constructs a first cue text based on multiple speech blocks, inputs the first cue text into the second major language model for prediction, and obtains a second style description text describing the style of the speaker in the multiple speech blocks. The second global embedding is constructed based on the embedding of the second style description text. Furthermore, a second cue text is constructed based on the second style description text, the reference text fragment, and the multiple speech blocks. The second cue text is input into the second major language model for prediction, and obtains a second sentiment description text describing the sentiment of the reference text fragment. The second detail embedding is constructed based on the embedding of the second sentiment description text, including using emotion2vec to extract the embedding of the second sentiment description text and using it as the second detail embedding. Finally, the reference speech fragment, the second global embedding, and the second detail embedding are associated and stored in a knowledge base.

[0155] In addition to the aforementioned embedding of the speaker's style, this embodiment can also obtain a first relevant descriptive text for supplementary descriptions when retrieving style-cue speech. Then, the embeddings of the first style descriptive text and the first relevant descriptive text are fused to obtain the first global embedding of the content segment. The first relevant descriptive text may include attributes describing the speaker, such as the speaker's region and education level. This information has a significant impact on determining the speaker's speaking style, tone, and word choice. Furthermore, the first relevant descriptive text may also include information related to the speaker's application scenario, gender, style and emotion, age, and region; this embodiment does not impose specific limitations on these aspects. After obtaining the first relevant descriptive text, this embodiment can use an LLM Embedder model to compress the aforementioned text information into a fixed-length embedding vector.

[0156] It should be noted that in the style profile of the speaking object and the embeddings extracted from the first relevant descriptive text, the style profile text information itself is used to calculate an embedding that represents global semantics, namely the second global embedding, which is used for subsequent RAG retrieval. In addition, as global information, the second global embedding can also be used to fine-tune the first or second language model to improve the accuracy of sentiment recognition by fine-grained text sentiment embedding at the sentence level of the dialogue text.

[0157] Based on the above information, a multimodal knowledge base for retrieval can be constructed in different ways. For example, the second global embedding and the second detailed embedding can be concatenated; or, the second global embedding and the second detailed embedding can be added together and then L2 normalized to 1, with the resulting embedding serving as the vector representation of the corresponding speech corpus in the retrieval knowledge base.

[0158] Multimodal knowledge bases can be built using various vector database tools, including but not limited to Milvus, Weaviate, Faiss, Chroma, and Qdrant. In this embodiment, Milvus is used to index the knowledge base, and retrieval is performed using a maximum inner product search (MID) method.

[0159] In the RAG-style retrieval module, it is necessary to determine whether retrieval enhancement is needed based on the content constraint information input by the target object, in order to improve system efficiency. It should be noted that the first language model itself has certain general generation capabilities, continuation capabilities, and expressiveness. Although the RAG-style retrieval module can make the generated speech expression more accurate, realistic, natural, and personalized, frequent retrieval increases response time and computational cost. Therefore, as shown in Figure 8, this embodiment first needs to determine whether retrieval is necessary based on the content constraint information. For example, taking text as the content constraint information, if the target object provides not only text but also a clear reference audio sample, retrieval enhancement is not needed. In this case, the reference audio sample provided by the target object can be directly used as the prompt speech input to the first speech model for generation. Conversely, if the target object provides a query description or a first relevant description text, then further retrieval enhancement processing is performed.

[0160] The role of retrieval enhancement is to select the top k most relevant corpora from a pre-built knowledge base based on the similarity of the embeddings. First, the query needs to be rewritten based on the information input by the target object to improve the query and better match relevant corpora. Specifically, a second language model can be used to perform style and sentiment profiling on each speaker in the text, obtaining a first style description text. Based on the embedding of the first style description text, a first global embedding of the content fragment is constructed. Then, the style profile is combined to perform sentiment recognition on each sentence spoken by the speaker, and a first detail embedding of the content fragment is constructed based on the embedding of the first sentiment description text.

[0161] If the target object provides a first relevant descriptive text related to the query, the HyDE method can be used to generate a pseudo-document based on the first relevant descriptive text, and the query description embedding can be calculated to obtain the embedding of the first style descriptive text, which can be used to represent other text information related to the retrieval.

[0162] After the above rewrite query steps, it is necessary to make another judgment on "whether subsequent retrieval enhancement is needed". Specifically, a classifier is trained, and the first global embedding and the first detail embedding are input into the classifier for classification to obtain the style sentiment classification result of the content fragment. The style sentiment classification result indicates whether the style sentiment of the content fragment is insufficient or sufficient. In this embodiment, neutral style sentiment does not need to be retrieved.

[0163] If the style and sentiment classification result indicates that the style and sentiment of the content segment is rich, then further retrieval is required to express the style and sentiment. In this embodiment, multiple reference speech segments are first retrieved from the knowledge base based on the similarity between the first and second global embeddings, obtaining the top M similarity reference speech segments as a candidate global set. Next, referring to the second detail embeddings corresponding to these M reference speech segments in the knowledge base, detail retrieval is performed based on the similarity between the first detail embedding and these second detail embeddings, obtaining the top K similarity reference speech segments.

[0164] This embodiment can use a reranker for fine-grained sorting. The purpose of this reranking step is to enhance the relevance of the retrieved files and ensure that the most relevant information appears at the top of the list. Specifically, this embodiment can determine the contextual relevance between each candidate speech segment and the content segment to better understand the contextual relationship between the candidate speech segments and the content segment, and to better understand subtle differences in context, thereby providing higher sorting accuracy. Subsequently, based on the calculation result of the contextual relevance, the candidate speech segment with the highest relevance is selected as the style cue speech corresponding to the content segment. Alternatively, this embodiment can further determine the similarity between the second global embedding and the first global embedding of each candidate speech segment, perform global consistency reranking (i.e., reranking based on the similarity between the second global embedding and the first global embedding), and select the candidate speech segment with the highest similarity as the style cue speech based on the similarity calculation result. Alternatively, this embodiment can also perform speech quality assessment on each candidate speech segment, using speech quality assessment tools or models to evaluate the speech quality of each candidate speech segment. For example, WER assessment can be performed on each candidate speech segment to obtain the speech quality assessment result of each candidate speech segment. Subsequently, the candidate speech segment with the highest speech quality is determined as the style cue speech corresponding to the content segment.

[0165] In the process following the coarse or fine sorting described above, this embodiment can also provide an interactive interface that allows the target object to manually select and sort candidate speech segments.

[0166] Finally, after determining the style-cue speech, the retrieved style-cue speech corpus files can be repackaged into an aligned, structured form. In the repackaging step, since style retrieval is not performed when the style sentiment of the content segment is "neutral," consecutive sentences of this type can be merged. Then, "neutral" corpus files that meet global consistency (i.e., the similarity between the second global embedding and the first global embedding) are retrieved and supplemented. Thus, all retrieved corpus files can be aligned one-to-one with the sentences of each speaker in the target text.

[0167] In the speech generation module based on a large language model, referring to Figure 9, which is a schematic diagram of the framework of the speech generation module based on a large language model provided in this embodiment, a first large language model is deployed in the speech generation module based on the large language model. This model is responsible for generating target speech that conforms to the style and timbre based on style prompts, content constraint information, and timbre prompts from the RAG style retrieval module. The target can also input other prompts as needed to generate speech. The dashed lines in the figure represent the data flow required only during the training phase.

[0168] In this module, the speech generation task is modeled in two main parts. One part is modeled as the task of generating autoregressive speech tag sequences based on the first language model, which is also the task of generating target speech tags. The other part is modeled as the target speech tag sequence generated above, and the speech Mel spectrum of the target speech is generated by the Flow Matching module.

[0169] During the LLM training phase, the input sequence is constructed as [(S),v,{t}]. i∈[1:I] ,(T),{x} k∈[1:K] [,(E)], where (S), (T) and (E) represent the sequence start marker, the speech transition marker between information embedding and speech marker, and the sequence end marker, respectively. v represents the speech object embedding vector extracted by the speaker encoder from the style cue speech, i.e., the second speaker object embedding; {t} i∈[1:I] This represents the information embedding obtained by a text encoder after text T is processed by the encoder, where i is the index of the embedded element and I is the total number of embedded elements; {x} k∈[1:K]The diagram shows the sequence of labeled speech tokens extracted from labeled speech X by a speech tokenizer. Labeled speech is the speech used as labels during loss calculation. k is the index of the labeled speech token in the sequence, and K is the total number of labeled speech tokens in the sequence. Z′ in the diagram represents the acoustic features extracted from the timbre-cued speech, designed to make the acoustic environment of the generated speech sound similar to that of the timbre-cued speech. The objective function for training the first language model uses cross-entropy loss.

[0170] Where q(x) k ) represents the x predicted by the first language model and the softmax layer. k The posterior probability.

[0171] As shown in Figure 9, the input to the Flow Matching module includes the second speaker embedding, acoustic features, and the target speech tag output by the first large language model. Given these input conditions, the parameters of the Flow Matching module are trained to match the vector field of the speech data distribution.

[0172] During the inference process, the input sequence of the first large language model is [(S), v, {t}]. i∈[1：I] ,(T),{y} j∈[1：J] ], where {y} j∈[1：J] The `j` represents the original speech token extracted by the Speech Tokenizer from the style-cued speech, where `j` is the index of the original speech token and `J` is the total number of original speech tokens. `v` represents the speaker object embedding vector extracted from the style-cued speech, i.e., the second speaker object embedding. `v′` represents the timbre-cued speech object embedding vector extracted by the Speaker Encoder, i.e., the first speaker object embedding. The input to the Flow Matching module includes the first speaker object embedding, acoustic features, and the target speech tokens output by the first large language model. It is important to emphasize that when the acoustic features are masked Mel features, in addition to using masked Mel features extracted from the timbre-cued speech, masked Mel features extracted from the style-cued speech can also be used. In this case, the acoustic environment of the generated speech sounds similar to the acoustic environment in the style-cued speech. In actual system deployment, the choice of which type of cues to extract Mel representations from can be made based on business needs.

[0173] This embodiment can be applied to various projects and product applications such as virtual human voice assistants, live chatbots, audio and video conferencing, system AI intelligent assistants, and in-vehicle voice interaction systems. Taking the application in a live chatbot as an example, referring to Figure 10, which is a schematic diagram of the configuration process of a live chatbot provided in this embodiment, as shown in the figure, this embodiment can provide a configuration box on one side of the interface, such as a live chat script configuration box. In the live chat script configuration box, the conversion function or other column functions can be selected. Under the conversion function, the content that the chatbot in the live room needs to pronounce can be entered. After entering text such as "Dear friends, welcome to my live room!", clicking the generate button below will use the entered text as content constraint text and generate voice through the voice generation method in the above embodiment. Finally, after confirming that the generated target voice meets the requirements, the deployment button at the bottom of the interface can be clicked, and the target voice can be applied in the live broadcast and emitted by the chatbot to realize the interaction of the live chatbot.

[0174] In this embodiment, Mel representations extracted from timbre-cued speech are used as acoustic features. Furthermore, in this embodiment, the Text Encoder can be obtained by augmenting and fine-tuning the SentencePiece Tokenizer and Text Encoder from the open-source pre-trained large model based on Non-Language Left Behind (NLLB); the Speech Tokenizer can be obtained by augmenting and fine-tuning the open-source w2v-bert 2.0 large model; and the first large language model is obtained by augmenting and fine-tuning a pre-trained large language model. Alternatively, the Speech Tokenizer in this embodiment can also be obtained by augmenting and fine-tuning the supervised Speech Tokenizer from cosyvoice, and correspondingly, the Text Encoder is obtained by training a Byte Pair Encoded (BPE) tokenizer. The Text Encoder is used to align the semantic spaces of the text token and the speech token.

[0175] It should be noted that, in addition to the Text Encoder based on the NLLB pre-trained large model or the Text Encoder based on CosyVoice that were trained and fine-tuned as mentioned in the above technical solutions, other Text Encoders can also be used as alternatives. Similarly, in addition to the Speech Tokenizer based on the w2v-bert 2.0 open-source pre-trained large model or the Speech Tokenizer based on CosyVoice that were trained and fine-tuned as mentioned in the above technical solutions, other Speech Tokenizers can also be used as alternatives. These alternatives need to extract, encode, and tokenize only the semantic content of the speech, without including other acoustic information (such as the speaker's timbre, ambient sound, etc.).

[0176] The first large language model in this embodiment can also be other large language models, and the model size can be selected, distilled, and optimized according to the resource situation of the actual application scenario.

[0177] The speech generation module based on Flow Matching used in this embodiment can be replaced with other condition-based AIGC models, including state-of-the-art (SOTA) or future extension models based on Diffusion or Flow Matching.

[0178] The multimodal knowledge base construction and RAG-style retrieval module described in this embodiment includes text modality and speech modality. Other alternatives include, but are not limited to, speech and text modality, and may also include multimodal knowledge bases and retrieval including visual and image modalities.

[0179] In summary, this embodiment successfully decouples the content, style, and timbre of the target speech, supports flexible combinations of any style and any speaker's timbre, generates speech timbre that is highly similar to the target timbre, and produces speech with a natural, realistic, and appropriate style and emotion. Its expressiveness can be personalized and controlled, and it can achieve a high degree of consistency with the target style and emotion. The speech generation quality, accuracy, and stability all meet the requirements for practical applications.

[0180] Furthermore, this embodiment features simple training and deployment, high user-friendliness, and supports zero-shot speech generation without requiring additional fine-tuning data or models, which are relatively cumbersome training and deployment processes. Moreover, it only requires a very short (e.g., 3 seconds) reference style and timbre as cues to generate highly expressive speech with a high degree of consistency in style and timbre, making it extremely user-friendly.

[0181] This embodiment supports the automatic, flexible, highly precise, and personalized matching of appropriate style and emotion to text content, achieving realistic, natural, fluent, fitting, personalized, varied, and flexible speech expression. Furthermore, the cost of achieving this beneficial effect is extremely low (almost zero cost), unlike traditional technical solutions that require enormous human and financial resources, lengthy timeframes, hiring voice actors for customized recordings, and fine-tuning training models. Instead, through the innovative RAG+LLM framework design of this embodiment, it fully, automatically, and intelligently utilizes the massive, abundant, and free data and knowledge resources of the network and media era to achieve better results.

[0182] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0183] Referring to Figure 11, which is a schematic diagram of the structure of a speech generation apparatus provided in an embodiment of this disclosure, the speech generation apparatus 1100 can execute the speech generation method in any of the above embodiments. The apparatus includes:

[0184] Content processing module 1101 is used to obtain content constraint information for content constraint during speech generation and extract information embedding of content constraint information;

[0185] The style processing module 1102 is used to acquire style-cue speech for style constraints during speech generation and extract multiple original speech tags from the style-cue speech.

[0186] Prediction module 1103 is used to input information embedding and multiple original speech tags into the first language model for prediction to obtain multiple target speech tags;

[0187] The timbre processing module 1104 is used to acquire the timbre-cue speech used for timbre constraint during speech generation, and to extract the first speaking object embedding of the timbre-cue speech;

[0188] The generation module 1105 is used to generate speech based on the embedding of the first speaking object and multiple target speech tags to obtain the target speech.

[0189] The functions and implementations of the various modules of the speech generation device 1100 in this embodiment can be found in the preceding method embodiments, and will not be repeated here. By acquiring content constraint information for content constraints during speech generation and extracting information embeddings of the content constraint information, the content of the target speech can be constrained through information embedding during speech generation. Furthermore, by acquiring style cue speech for style constraints during speech generation and extracting multiple original speech tags from the style cue speech, the information embeddings and multiple original speech tags are input into a first-level language model for prediction. Based on the first-level language model, multiple target speech tags with styles similar to the style cue speech can be predicted. On this basis, by acquiring timbre cue speech for timbre constraints during speech generation and extracting the first speaker object embedding of the timbre cue speech, speech generation is performed based on the first speaker object embedding and multiple target speech tags, resulting in target speech with timbre similar to both the timbre cue speech and the style cue speech. This achieves control over style and timbre, making the target speech more natural and improving its realism. Moreover, because content constraint information, style cue speech, and timbre cue speech are introduced at different stages of the generation process, the content, style, and timbre of the target speech are decoupled, supporting flexible combinations of arbitrary content, style, and timbre, thereby improving the flexibility of the target speech.

[0190] The electronic device provided in this disclosure for executing the above-described speech generation method can be a terminal. Referring to FIG12, FIG12 is a partial structural block diagram of a terminal provided in this disclosure. The terminal includes: a camera assembly 1210, a first memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a wireless fidelity (WiFi) module 1270, a first processor 1280, and a first power supply 1290, etc. Those skilled in the art will understand that the terminal structure shown in FIG12 does not constitute a limitation on the terminal, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0191] The camera assembly 1210 can be used to capture images or videos.

[0192] The first memory 1220 can be used to store software programs and modules.

[0193] The first processor 1280 executes various terminal functions and data processing by running software programs and modules stored in the first memory 1220.

[0194] The input unit 1230 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 1230 may include a touch panel 1231 and other input devices 1232.

[0195] The display unit 1240 can be used to display input or provided information, as well as various menus of the terminal. The display unit 1240 may include a display panel 1241.

[0196] Audio circuitry 1260, speaker 1261, and microphone 1262 provide an audio interface.

[0197] The first power source 1290 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery.

[0198] The number of sensors 1250 can be one or more, including but not limited to: accelerometers, gyroscopes, pressure sensors, optical sensors, etc.

[0199] In this embodiment, the first processor 1280 included in the terminal can execute the speech generation method of the previous embodiment.

[0200] The electronic device for executing the above-described speech generation method provided in this disclosure can also be a server. Referring to Figure 13, which is a partial structural block diagram of a server provided in this disclosure, the server can vary considerably due to different configurations or performance. It may include one or more second processors 1310 and second memories 1330, and one or more storage media 1340 (e.g., one or more mass storage devices) for storing application programs 1343 or data 1342. The second memories 1330 and storage media 1340 can be temporary or persistent storage. The program stored in the storage media 1340 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the second processor 1310 may be configured to communicate with the storage media 1340 and execute a series of instruction operations in the storage media 1340 on the server.

[0201] The server may also include one or more secondary power supplies 1320, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1360, and / or one or more operating systems 1341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0202] The second processor 1310 in the server can be used to execute the speech generation method.

[0203] This disclosure also provides a computer-readable storage medium for storing a computer program for executing the speech generation methods of the foregoing embodiments.

[0204] This disclosure also provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the speech generation method described above.

[0205] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate to describe embodiments of this disclosure, for example, those that can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0206] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0207] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0208] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0209] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0210] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0211] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0212] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0213] The above is a detailed description of the preferred embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A speech generation method, executed by an electronic device, comprising: Obtain content constraint information for content constraint during speech generation, and extract the information embedding of the content constraint information; Obtain style-cue speech for style constraint during speech generation, and extract multiple original speech tags from the style-cue speech to characterize speech style features; The embedded information and multiple original speech tags are input into a first large language model to obtain multiple target speech tags predicted and generated by the first large language model. Acquire timbre-cue speech for timbre constraint during speech generation, and extract a first speaking object embedding from the timbre-cue speech to characterize the timbre features of the speaking object; Speech is generated based on the first speaking object embedding and multiple target speech tags.

2. The method of claim 1, wherein, The style prompt voice includes the style prompt voice corresponding to each content segment in the information; The step of obtaining style-cue speech for style constraint during speech generation includes: The content constraint information is input into the second language model to obtain the first style description text predicted by the second language model and the first sentiment description text of each content fragment in the information. The first style description text is used to describe the speaking style features of the content constraint information, and the first sentiment description text of each content fragment is used to describe its sentiment attributes. Based on the embedding of the first style description text, a first global embedding of each content fragment is constructed, and based on the embedding of the first sentiment description text of each content fragment, a first detail embedding of each content fragment is constructed. For each content segment, based on the first global embedding and the first detail embedding of the content segment, a matching speech segment is retrieved from a preset knowledge base and used as the style prompt speech corresponding to the content segment.

3. The method of claim 2, wherein, The knowledge base stores multiple reference speech segments and second global embeddings and second detail embeddings associated with each reference speech segment. The second global embedding associated with each reference speech segment indicates the speaking style characteristics of the reference speech segment, and the second detail embedding associated with each reference speech segment indicates the emotional attribute of the reference speech segment. For each content segment, based on the first global embedding and the first detail embedding of the content segment, the style prompt speech corresponding to the content segment is retrieved from the preset knowledge base, including: For each content segment: Based on the similarity between the first global embedding of the content fragment and the second global embedding associated with each of the reference speech fragments, multiple first reference speech fragments are retrieved from the knowledge base; Based on the similarity between the first detail embedding of the content fragment and the second detail embedding associated with each of the first reference speech fragments, the style prompt speech corresponding to the content fragment is retrieved from the plurality of first reference speech fragments.

4. The method of claim 3, further comprising: The original speech is acquired and segmented according to the speaker to obtain multiple speech blocks; Speech recognition is performed on the multiple speech blocks respectively to obtain the speech block text corresponding to each speech block; For each speech block text: the speech block text is divided into multiple reference text segments, and its corresponding speech block is divided into reference speech segments corresponding to each of its reference text segments respectively; Based on the multiple speech block texts, construct the second global embedding of each reference text segment of each speech block text, construct its second detailed embedding based on each reference text segment of each speech block text, and associate and store each reference speech segment, its second global embedding and the second detailed embedding in the knowledge base.

5. The method of claim 4, wherein, The construction of the second global embedding of each reference text segment of each speech block text based on multiple speech block texts, and the construction of its second detailed embedding based on each reference text segment of each speech block text, include: A first prompt text is constructed based on multiple speech block texts. The first prompt text is input into the second large language model, and a second style description text is output to describe the style features of each speaker. Based on the embedding of the second style description text for each speaking object, construct the second global embedding of each reference text segment in the speech block text of that speaking object; For each reference text segment in the speech block text of each speaking object, a second prompt text is constructed based on the second style description text of the speaking object, the reference text segment, and the multiple speech block texts. The second prompt text is input into the second large language model, and a second sentiment description text is output to describe the sentiment attributes of the reference text segment. A second detail embedding is constructed based on the embedding of the second sentiment description text of each of the reference text fragments.

6. The method of claim 3, wherein, The step of retrieving the style-cue speech corresponding to the content segment from the plurality of first reference speech segments based on the similarity between the first detail embedding of the content segment and the second detail embedding associated with each of the first reference speech segments includes: Based on the similarity between the first detail embedding and the second detail embedding associated with each of the first reference speech segments, a plurality of second reference speech segments are retrieved from the plurality of first reference speech segments; Determine the contextual relevance between each second reference speech segment and the content segment, and determine the second reference speech segment with the highest contextual relevance as the style cue speech corresponding to the content segment; or, determine the similarity between the second global embedding and the first global embedding of each second reference speech segment, and determine the second reference speech segment with the highest similarity as the style cue speech corresponding to the content segment; or, perform a speech quality assessment on each second reference speech segment, and determine the second candidate reference speech segment with the highest speech quality as the style cue speech corresponding to the content segment.

7. The method of claim 2, wherein, The construction of the first global embedding for each content fragment based on the embedding of the first style description text includes: Obtain a first relevant descriptive text, which is used to provide supplementary descriptions when retrieving the style prompt speech; By combining the embedding of the first style description text and the embedding of the first related description text, the first global embedding of each fragment is obtained.

8. The method of claim 2, wherein, The first sentiment description text based on each content fragment constructs the first detail embedding of each content fragment, including: For each content fragment, the first global embedding and the embedding of the first sentiment description text of the content fragment are merged to obtain the first detail embedding of the content fragment.

9. The method of claim 8, wherein, The content constraint information is speech, the content segment is a speech segment, and the construction of the first detail embedding of each content segment based on the embedding of the first emotion description text of each content segment includes: For each speech segment, extract the speech emotion embedding of the speech segment; By fusing the first global embedding, the embedding of the first emotional description text of the speech segment, and the speech emotion embedding, the first detail embedding of the speech segment is obtained.

10. The method of claim 2, wherein, The step of retrieving the style-guided speech corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding includes: The first global embedding and the first detail embedding are input into a classifier for classification to obtain the style sentiment classification result of the content fragment, wherein the style sentiment classification result is used to indicate that the style sentiment of the content fragment is neutral or rich; When the style sentiment classification result indicates that the style sentiment of the content segment is rich, the style prompt voice corresponding to the content segment is retrieved from the preset knowledge base based on the first global embedding and the first detail embedding.

11. The method of claim 10, wherein, The step of retrieving the style prompt speech corresponding to the content segment from a preset knowledge base based on the first global embedding and the first detail embedding further includes: When the style sentiment classification result indicates that the style sentiment of the content segment is neutral, the style prompt speech corresponding to the content segment is retrieved from the knowledge base based on the first global embedding.

12. The method of claim 1, wherein, The process of embedding the information and inputting the original speech tags into a first language model to obtain multiple target speech tags predicted by the first language model includes: Extract the second speaking object embedding of the style-cue speech, input the information embedding, the second speaking object embedding, and multiple original speech tags into the first large language model, and output the multiple target speech tags.

13. The method of claim 12, wherein, The process of inputting the embedded information, the embedded second speaking object, and the multiple original speech tags into the first large language model, and outputting the multiple target speech tags, includes: The sequence start marker, the second speaking object embedding, the information embedding, the speech conversion marker, and multiple original speech markers are sequentially concatenated and input into the first large language model, and the multiple target speech markers are output.

14. The method of claim 1, wherein, The step of generating speech based on the first speaking object embedding and multiple target speech tags includes: The acoustic features of the style-cue speech or the timbre-cue speech are extracted, and the speech is generated based on the acoustic features, the first speaker embedding, and multiple target speech tags.

15. A speech generation apparatus, comprising: The content processing module is used to acquire content constraint information for content constraint during speech generation and extract information embedding from the content constraint information. The style processing module is used to acquire style-cue speech for style constraints during speech generation, and extract multiple original speech tags from the style-cue speech to characterize speech style features. The prediction module is used to input the embedded information and multiple original speech tags into a first large language model, and the first large language model predicts and generates multiple target speech tags; The timbre processing module is used to acquire timbre-cue speech for timbre constraint during speech generation, and extract a first speaking object embedding from the timbre-cue speech to characterize the timbre features of the speaking object; The generation module is used to generate speech based on the first speaking object embedding and multiple target speech tags.

16. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the speech generation method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech generation method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program that, when executed by a processor, implements the speech generation method according to any one of claims 1 to 14.

Images