Speech generation method, apparatus, device, and storage medium

Abstract

Embodiments of this disclosure relate to speech generation methods, apparatus, devices, and storage media. The proposed method includes: in response to receiving at least one first input content, extracting a reference speech feature representation corresponding to at least one first input content, wherein the reference speech feature representation at least indicates timbre-related feature information in at least one first input content; providing reference speech generated based on reference text and the reference speech feature representation; and in response to detecting confirmation of the reference speech, generating target speech based on target text and the reference speech, wherein the timbre of the target speech matches the timbre of the reference speech.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to a speech generation method, apparatus, device, and storage medium. Background Technology

[0002] Text-to-speech (TTS) is a technology that converts text into speech. In TTS, timbre is a characteristic that determines the personalization and realism of speech; it determines the "sound quality" of the speech.

[0003] In current practical applications, users can select the desired timbre from a preset timbre library and then use TTS technology to generate target speech with the selected timbre. However, the timbre library has a limited number of selectable timbre types, making it difficult to meet the increasingly diverse speech generation needs. Summary of the Invention

[0004] In a first aspect of this disclosure, a speech generation method is provided. The method includes: in response to receiving at least one first input content, extracting a reference speech feature representation corresponding to at least one first input content, wherein the reference speech feature representation at least indicates timbre-related feature information in at least one first input content; providing reference speech generated based on reference text and the reference speech feature representation; and in response to detecting confirmation of the reference speech, generating target speech based on target text and the reference speech, wherein the timbre of the target speech matches the timbre of the reference speech.

[0005] In a second aspect of this disclosure, an apparatus for speech generation is provided. The apparatus includes: a feature representation extraction module configured to, in response to receiving at least one first input content, extract a reference speech feature representation corresponding to at least one first input content, wherein the reference speech feature representation at least indicates timbre-related feature information in at least one first input content; a reference speech providing module configured to provide reference speech generated based on reference text and the reference speech feature representation; and a target speech generation module configured to, in response to detecting confirmation of the reference speech, generate target speech based on target text and the reference speech, wherein the timbre of the target speech matches the timbre of the reference speech.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment according to an embodiment of the present disclosure is shown;

[0012] Figure 2 A flowchart illustrating an example process of a speech generation method according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram illustrating an example scenario of a speech generation method according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating an example scenario of generating a reference speech feature representation according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram illustrating an example scenario of generating reference speech according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic diagram illustrating an example scenario of generating target speech according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A schematic structural block diagram of an apparatus for speech generation according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0021] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0023] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0024] As briefly described above, users can select the desired timbre from a preset timbre library and then use TTS technology to generate target speech with the selected timbre. Currently, there are generally three ways to select a timbre. The first way is to directly select a specific type of timbre from the timbre library. The second way is to filter the timbres in the timbre library using tags. The third way is to search for the desired timbre in the timbre library using keywords, etc.

[0025] However, while these three methods offer users diverse choices, the limited range of timbre types covered by the timbre library itself makes it difficult for users to accurately obtain a timbre that perfectly matches their individual needs, regardless of the method chosen. This limitation in timbre selection restricts the development of TTS technology, making it difficult for TTS to meet the increasingly diverse needs of speech generation.

[0026] In view of this, embodiments of the present disclosure provide a speech generation scheme. According to this scheme, firstly, in response to receiving at least one first input content, a reference speech feature representation corresponding to at least one first input content is extracted. The reference speech feature representation at least indicates timbre-related feature information in at least one first input content. Then, reference speech generated based on reference text and the reference speech feature representation is provided. Subsequently, in response to detecting confirmation of the reference speech, target speech is generated based on target text and the reference speech. The timbre of the target speech matches the timbre of the reference speech.

[0027] This disclosed solution achieves in-depth mining and precise capture of timbre features by extracting the reference speech feature representation corresponding to the input content. Regardless of the user's input content, this disclosed solution can parse and extract feature information closely related to timbre. Furthermore, this disclosed solution generates reference speech based on reference text and reference speech feature representation. The reference speech provides the user with an intuitive and perceptible speech sample. The user can select and adjust the reference speech according to their personal needs until they find a timbre that meets their requirements. In addition, after detecting the user's confirmation of the reference speech, this disclosed solution can generate target speech that matches the timbre of the reference speech based on the target text and reference speech. This step achieves a seamless connection from timbre selection to speech generation, ensuring that the final generated speech is consistent with the user's expectations in terms of timbre.

[0028] In this way, the disclosed solution breaks through the limitations of voice selection in traditional TTS technology, giving users greater freedom in voice customization. Furthermore, the disclosed solution also achieves a low user entry barrier, allowing users to easily participate in the voice customization process without needing complex speech processing knowledge or skills, thereby expanding the application scenarios of TTS.

[0029] The following will further describe in detail various example implementations of this scheme with reference to the accompanying drawings.

[0030] Figure 1 A schematic diagram of an example environment 100 according to an embodiment of the present disclosure is shown. (Refer to...) Figure 1 Example environment 100 may include electronic device 110, user 120, text 130, and voice 140.

[0031] In example environment 100, user 120 can provide text 130 to electronic device 110. Electronic device 110 can generate speech 140 based on text 130 using TTS technology.

[0032] As an example, user 120 can input text 130 into electronic device 110 in various ways. For instance, user 120 can input text content directly on electronic device 110 or through an external device associated with electronic device 110. Alternatively, user 120 can use voice input to have electronic device 110 convert their speech into text to provide text 130.

[0033] After receiving text 130 from user 120, electronic device 110 can generate speech 140 based on TTS technology and output the generated speech 140. For example, electronic device 110 can play the speech 140 through a speaker, or it can save the speech 140 as an audio file for later use. In this way, user 120 can use electronic device 110 to convert text 130 into speech, thereby realizing diverse information interaction and application needs.

[0034] In some embodiments, electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0035] In some embodiments, electronic device 120 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Electronic device 110 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. In this scenario, user 120 can interact with electronic device 110 through any type of terminal device, thereby enabling electronic device 120 to generate voice 140 for text 130.

[0036] In some embodiments, a communication connection can be established between the electronic device 110 and the terminal device. The communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth connections, mobile network connections, Universal Serial Bus connections, Wi-Fi connections, etc., and the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the electronic device 110 and the terminal device can achieve signaling interaction through the communication connection between them.

[0037] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0038] Figure 2 A flowchart of an example process 200 of a speech generation method according to some embodiments of the present disclosure is shown.

[0039] Figure 3 A schematic diagram of an example scenario 300 of a speech generation method according to some embodiments of the present disclosure is shown below. Figure 1 and Figure 3 The process 200 will be described in detail below. The process 200 can be implemented at the electronic device 110.

[0040] Reference Figure 2 In box 210, in response to receiving at least one first input content, electronic device 110 extracts a reference speech feature representation corresponding to at least one first input content. The reference speech feature representation at least indicates timbre-related feature information in at least one first input content.

[0041] In box 220, electronic device 110 provides reference speech generated based on reference text and reference speech feature representation.

[0042] As an example, the first input content may be the original information provided by user 120 for generating reference speech. The reference speech, also known as audition speech, is used by user 120 to adjust the voice tone. In the embodiments of this disclosure, the specific form of the first input content is not limited. For example, the first input content may be an input image, input speech, or input text, etc. As an example, the first input content may be provided by user 120 through the interactive interface of electronic device 110. For example, the interactive interface of electronic device 110 may display text boxes and / or file upload controls, etc. User 120 can provide input text through text boxes, and submit input images and / or input speech through file upload controls, etc.

[0043] As an example, user 120 can provide one first input content or multiple first input contents simultaneously to electronic device 110 based on its own needs. When there are multiple first input contents, these first input contents can belong to different types to support multimodal input. For example, example scenario 300 shows multiple first input contents 301-1, 301-2, ..., 301-N, where N is a positive integer. First input content 301-1 can be an input image, first input content 301-2 can be input speech, ..., first input content 301-N can be input text, etc. For ease of discussion, the multiple first input contents 301-1, 301-2, ..., 301-N will be referred to individually or collectively as first input content 301.

[0044] It should be noted that the above description of the first input content 301 is merely illustrative and does not constitute a limitation on the embodiments of this disclosure. For example, depending on actual needs, the first input content 301 may also include other content besides input images, input voice, and input text. As another example, in addition to providing input text through a text box, user 120 may also provide input text through speech-to-text conversion, and so on.

[0045] Timbre is one of the fundamental attributes of sound, and it can be used to distinguish the voices of different speakers. In embodiments of this disclosure, timbre-related feature information can be used to describe and distinguish different types of timbre. For example, timbre-related feature information can indicate "deep voice," "high-pitched voice," or similar content. As an example, electronic device 110 can identify timbre-related feature information in the first input content 301 through any suitable method (e.g., a machine learning model). As an example, electronic device 110 can perform operations such as embedding on the identified feature information to obtain a reference speech feature representation 322. As an example, the reference speech feature representation 322 can be a vector, matrix, or other data structure, which can be determined according to actual needs, and embodiments of this disclosure do not limit this. In embodiments of this disclosure, the reference speech feature representation 322 can indicate the speech characteristics of the first input content 301 in the form of annotations; such a reference speech feature representation 322 can also be called a captioning speech feature representation.

[0046] It should be noted that the above description of timbre-related features is merely illustrative and does not constitute a limitation on the embodiments of this disclosure. For example, in addition to "deep voice" and "high-pitched voice," timbre-related features may also indicate more, such as "hoarse voice," which will not be listed here in the embodiments of this disclosure.

[0047] In some embodiments, in addition to indicating timbre-related feature information, the reference speech feature representation 322 may also indicate pitch-related feature information in each of the first input contents 301. Alternatively or additionally, the reference speech feature representation 322 may also indicate loudness-related feature information in each of the first input contents 301, and so on.

[0048] In some embodiments, when there are multiple first input contents 301, the reference speech feature representation 322 can indicate the timbre-related feature information of each of the multiple first input contents 301. Such a reference speech feature representation 322 can also be called a fused speech feature representation of multiple first input contents 301.

[0049] Continue to refer to Figure 3In some embodiments, the electronic device 110 may utilize a second machine learning model 320 to extract a reference speech feature representation 322 from at least one first input content 301. In some embodiments, the at least one first input content 301 includes multiple first input content 301s of different types. The second machine learning model 320 includes a speech feature representation extraction module 321, which extracts speech feature representations corresponding to each of the multiple first input content 301s. Subsequently, the speech feature representation extraction module 321 performs weighted fusion of the speech feature representations corresponding to each of the multiple first input content 301s based on their respective weights to obtain the reference speech feature representation 322.

[0050] As described above, the types of the multiple first input contents 301 are different. In some embodiments, the speech feature representation extraction module 321 can extract the speech feature representations corresponding to each of the multiple first input contents 301 based on the feature extraction methods associated with each of the multiple first input contents 301.

[0051] Figure 4 A schematic diagram of an example scenario 400 for generating a reference speech feature representation according to some embodiments of the present disclosure is shown.

[0052] Reference Figure 4 The speech feature representation extraction module 321 includes multiple extraction sub-modules 401-1, 401-2, ..., 401-N corresponding to multiple first input contents 301. For ease of discussion, the multiple extraction sub-modules 401-1, 401-2, ..., 401-N will be referred to individually or collectively as extraction sub-modules 401 below. The speech feature representation extraction module 321 uses these extraction sub-modules 401-1, 401-2, ..., 401-N to determine the speech feature representation corresponding to each first input content 301. In some embodiments, the extraction sub-modules 401-1, 401-2, ..., 401-N may be constructed based on a Mixture-of-Experts (MoE) architecture.

[0053] Assuming the first input content 301-1 is an input image, the speech feature representation extraction module 321 can use the extraction submodule 401-1 to extract timbre-related feature information from the input image. Furthermore, the extraction submodule 401-1 can perform an encoding operation on this feature information to obtain the speech feature representation 402-1 corresponding to the first input content 301-1. As an example, the extraction submodule 401-1 can be constructed based on a Vision Language Model (VLM) or other models suitable for processing image content.

[0054] Assuming the first input content 301-2 is input speech, the speech feature representation extraction module 321 can use the extraction submodule 401-2 to extract timbre-related feature information from the input speech. Furthermore, the extraction submodule 401-2 can perform an encoding operation on this feature information to obtain the speech feature representation 402-2 corresponding to the first input content 301-2. As an example, the extraction submodule 401-2 can be constructed based on a Speech Language Model (SLM) or other models suitable for processing speech content.

[0055] Assuming the first input content 301-N is input text, the speech feature representation extraction module 321 can utilize the extraction submodule 401-N to extract timbre-related feature information from the input text. Furthermore, the extraction submodule 401-N can perform an encoding operation on this feature information to obtain the speech feature representation 402-N corresponding to the first input content 301-N. As an example, the extraction submodule 401-N can be constructed based on a Large Language Model (LLM) or other models suitable for processing text content.

[0056] It should be noted that, unlike input text and input speech, the timbre-related feature information in the input image may be implicit in the image features. In some embodiments, the extraction submodule 401-1 can first extract image features from the input image that may be associated with the timbre of the target object, such as image features related to the target object (e.g., a person). Then, the electronic device 110 can determine the possible timbre of the target object based on the extracted image features, thereby obtaining the timbre-related feature information of the first input content 301-1. For example, assuming that the features related to the target object indicate that the target object is of a first type, then the electronic device 110 can determine "deep voice" as the timbre-related feature information in the first input content 301-1. As another example, assuming that the features related to the target object indicate that the target object is of a second type, then the electronic device 110 can determine "high-pitched voice" as the timbre-related feature information in the first input content 301-1.

[0057] In some embodiments, the speech feature representations 402-1, 402-2, ..., 402-N have the same feature dimension, or in other words, the speech feature representations 402-1, 402-2, ..., 402-N are speech feature representations 402 mapped to the same feature space, thereby facilitating subsequent feature fusion.

[0058] Continue to refer to Figure 4The speech feature representation extraction module 321 also includes a feature fusion module 403. After obtaining speech feature representations 402-1, 402-2, ..., 402-N, the feature fusion module 403 can perform weighted fusion of speech feature representations 402-1, 402-2, ..., 402-N based on the respective weights of the multiple first input contents 301 to obtain a reference speech feature representation 322.

[0059] In this manner, embodiments of the present disclosure are able to extract reference speech feature representations 322 from a first input content 301 of different modalities (such as images, speech, text, etc.). This multimodal input allows for more accurate extraction of the reference speech feature representations 322.

[0060] In some embodiments, the plurality of first input contents 301 include input images, input speech, and / or input text. During the weighted fusion process, the input speech has a greater weight than the input images and input text.

[0061] Input speech is a direct recording of speech signals. Compared to input images and input text, input speech can more directly and accurately reflect the physical characteristics of speech, such as pitch, timbre, and intonation. Therefore, embodiments of this disclosure assign greater weight to input speech during the weighted fusion process, thereby ensuring that the generated reference speech feature representation 322 is closer to the real speech features.

[0062] Return to reference Figure 3 The second machine learning model 320 also includes a reference speech generation module 324. After determining the reference speech feature representation 322, the reference speech generation module 324 can convert the reference text 323 into speech form based on the reference speech feature representation 322, thereby obtaining reference speech 302. As an example, the timbre of the reference speech 302 matches the reference speech feature representation 322. For example, suppose the content of the reference text 323 is AABBCC, and the timbre-related feature information indicated by the reference speech feature representation 322 is "deep voice". In this case, the reference speech generation module 324 will generate a reference speech 302 that reads "AABBCC" in a "deep voice" based on this information.

[0063] Figure 5 A schematic diagram of an example scenario 500 for generating reference speech according to some embodiments of the present disclosure is shown.

[0064] Reference Figure 5The reference speech generation module 324 may include a first processing module 501 and a first vocoder 502. The first processing module 501 may output an intermediate feature representation or feature sequence based on the reference speech feature representation 322 and the reference text 323. The first vocoder 502 may decode the reference speech 302 based on the intermediate feature representation or feature sequence output by the first processing module 501. As an example, the first processing module 501 may be constructed based on a content-generative model (e.g., LLM), which facilitates providing more accurate and effective input to the first vocoder 502. In this way, embodiments of this disclosure can fully utilize the text processing capabilities of LLM and the speech generation capabilities of the vocoder to convert the reference text 323 into reference speech 302 with high quality.

[0065] In some embodiments, the electronic device 110 determines the reference text 323 based on the target text. Alternatively or additionally, the electronic device 110 determines the reference text 323 based on the reference speech feature representation 322.

[0066] Return to reference Figure 3 As an example, target text 303 may be text provided by user 120 for generating target speech 305. In some embodiments, electronic device 110 may combine target text 303 and reference speech feature representation 322 to determine reference text 323. In other embodiments, electronic device 110 may select either target text 303 or reference speech feature representation 322 to determine reference text 323.

[0067] As an example, in the process of generating reference speech 302, in response to the detection of target text 303, the second machine learning model 320 can directly use target text 303 as reference text 323; in response to the absence of detection of target text 303, the second machine learning model 320 can generate reference text 323 based on reference speech feature representation 322.

[0068] In some embodiments, when the second machine learning model 320 generates reference text 323 based on reference speech feature representation 322, the second machine learning model 320 can generate reference text 323 that is similar in speech style to the reference speech feature representation 322. Thus, after converting the reference text 323 into reference speech 302, the reference speech 302 will have better expressiveness.

[0069] In some embodiments, the reference speech feature representation 322 is extracted using a second machine learning model 320. The second machine learning model 320 may be trained in the following manner: First, the electronic device 110 provides at least one sample input content to the second machine learning model 320. The second machine learning model 320 extracts a sample speech feature representation corresponding to the at least one sample input content. The sample speech feature representation at least indicates timbre-related feature information in the at least one sample input content. Then, the second machine learning model 320 generates a sample text representation describing the sample speech feature representation. Subsequently, the electronic device 110 trains the second machine learning model 320 based on the similarity between the sample text representation and the labeled text representation associated with the at least one sample input content.

[0070] To distinguish it from the application process of the second machine learning model 320, the input content and speech feature representation involved in the training process are represented by "sample input content" and "sample speech feature representation". For relevant content about "sample input content" and "sample speech feature representation", please refer to the part related to the first input content and reference speech feature representation in the previous text, so it will not be repeated here.

[0071] As an example, the labeled text representation can refer to the text labels marked by the sample provider for the sample input content. Considering that the sample speech feature representation is an intermediate representation of the second machine learning model 320, in order to achieve alignment between the sample speech feature representation and the labeled text representation, during the training process of the second machine learning model 320, embodiments of this disclosure enable the second machine learning model 320 to further generate sample text representations for describing the sample speech feature representations. Then, based on the similarity between the sample text representation and the labeled text representation, the second machine learning model 320 is trained.

[0072] In some embodiments, in the second machine learning model 320, the speech feature representation extraction module 321 and the reference speech generation module 324 may be trained in an end-to-end manner, respectively.

[0073] After determining the reference voice 302, the electronic device 110 can play the reference voice 302 directly, or play the reference voice 302 at the instruction of the user 120. After providing the reference voice 302 to the user 120, the electronic device 110 will detect the user 120's instruction regarding the reference voice 302.

[0074] In some embodiments, in response to detecting an adjustment instruction for reference speech 302, electronic device 110 regenerates reference speech 302 based on reference text 323 and reference speech feature representation 322.

[0075] As an example, the adjustment instruction could be triggered by user 120 via voice command. Alternatively, the adjustment instruction could also be triggered by user 120 via text command. Alternatively, the adjustment instruction could also be triggered by user 120 by clicking a specific control.

[0076] The adjustment instructions can include user 120's adjustment requests regarding aspects such as speech rate, intonation, and timbre. For example, the adjustment instructions could be text such as "The reference speech intonation is too harsh; I hope it will be softer." As an example, the reference speech generation module 324 can use the reference text 323 and the reference speech feature representation 322 as a basis, and based on the adjustment instructions, regenerate the reference speech 302 by appropriately changing the timbre, intonation, speech rate, etc.

[0077] In this way, the embodiments of this disclosure can flexibly adjust the reference voice 302 according to the adjustment instructions of the user 120, thereby ensuring that the final generated reference voice 302 meets the personalized needs of the user 120.

[0078] In some embodiments, in response to receiving at least one second input content associated with an adjustment instruction, the electronic device 110 updates the reference speech feature representation 322 based on at least one second input content. Subsequently, the electronic device 110 regenerates the reference speech 302 based on the updated reference speech feature representation 322 and the reference text 323.

[0079] As an example, at least one second input content can be newly added by user 120, or it can be input content obtained by user 120 after modifying the original first input content. The relevant content regarding the electronic device 110 updating the reference speech feature representation 322 based on the second input content can be found above, specifically the part related to the speech feature representation extraction module 321 extracting the reference speech feature representation 322 from the first input content 301; therefore, it will not be repeated here.

[0080] After the reference speech feature representation 322 is updated, the reference speech generation module 324 will reconvert the reference text 323 into reference speech 302 based on the updated reference speech feature representation 322.

[0081] In this way, embodiments of the present disclosure can allow user 120 to provide more input content or change the original input content to regenerate reference speech 302, thereby further enhancing the personalization capability of reference speech 302.

[0082] Return to reference Figure 2In box 230, in response to the detection of confirmation of reference speech 302, electronic device 110 generates target speech 305 based on target text 303 and reference speech 302. The timbre of target speech 305 matches the timbre of reference speech 302.

[0083] As an example, confirmation of reference voice 302 can be triggered by user 120 via voice command. Alternatively, confirmation of reference voice 302 can also be triggered by user 120 via text command. Alternatively, confirmation of reference voice 302 can also be triggered by user 120 by clicking a specific control.

[0084] After confirming the reference speech, the electronic device 110 can convert the target text 303 into speech form by referring to the speech expression of the reference speech 302, thereby obtaining the target speech 305. In this process, the electronic device 110 does not directly copy the reference speech 302, but uses the reference speech 302 as a benchmark, combined with the actual content of the target text 303, to dynamically adjust the specific expression of the target speech 305. For example, the electronic device 110 can make fine adjustments to the tone, speech rate, etc., while retaining the timbre of the reference speech 302, so that the target speech 305 is consistent with the context described by the target text 303.

[0085] In some embodiments, the electronic device 110 constructs prompt word information 304 for a first machine learning model 310 from the reference speech and the target text. Subsequently, the electronic device 110 generates target speech 305 based on the prompt word information 304 and the first machine learning model 310.

[0086] In the embodiments of this disclosure, the prompt word information 304 includes information from both speech and text modalities; such prompt word information 304 can also be referred to as multimodal prompt word information. Through multimodal prompt word information, the first machine learning model 310 can better learn the inherent correlation and mapping relationship between speech and text. For example, in some text contexts, speech may exhibit characteristics such as a high pitch and a fast speaking speed; while in other text contexts, speech may become low and slow. On the other hand, the first machine learning model 310 can also learn how the structure and semantics of the text affect the expression of speech. For example, when telling a story, the tone and rhythm of the speech can be adjusted according to the development of the plot to enhance the story's appeal. This in-depth learning of the speech-text relationship enables the first machine learning model 310 to more accurately grasp the expression of speech when generating target speech 305, thereby generating higher quality target speech that conforms to auditory habits and semantic understanding.

[0087] Figure 6A schematic diagram of an example scenario 600 for generating target speech according to some embodiments of the present disclosure is shown.

[0088] Reference Figure 6 The reference speech generation module 324 may include a second processing module 601 and a second vocoder 602. The second processing module 601 may output an intermediate feature representation or feature sequence based on the prompt word information 304. The second vocoder 602 may decode the target speech 305 based on the intermediate feature representation or feature sequence output by the second processing module 601. As an example, the second processing module 601 may be constructed based on a content-generative model (e.g., LLM), thereby facilitating the provision of more accurate and effective input to the second vocoder 602. In this manner, embodiments of the present disclosure can fully utilize the text processing capabilities of LLM and the speech generation capabilities of the vocoder to convert the target text 303 into the target speech 305 with high quality.

[0089] In some embodiments, in the prompt word information 304, the reference speech 302 is constructed as the system prompt word of the first machine learning model 310.

[0090] In some embodiments, the electronic device 110 can convert reference speech into system prompts for the first machine learning model 310 by performing a system prompt registration operation. Unlike general prompts, system prompts exert greater control over the first machine learning model 310, thereby ensuring that the timbre characteristics of the reference speech 302 are preserved to the greatest extent possible during the generation of the target speech 305. Furthermore, after completing the registration operation for the reference speech 302, when the user 120 needs to generate a target speech 305 with the same or similar timbre in the future, they only need to select a pre-saved system prompt. This not only greatly simplifies the user 120's operation process but also improves the consistency and reliability of the generated results.

[0091] As can be clearly understood from the various embodiments described above, the embodiments of this disclosure propose an end-to-end speech generation method. The embodiments of this disclosure not only support multimodal input content (such as input images, input speech, and input text), but also significantly reduce the difficulty for users to describe speech features through automated speech feature extraction, thereby making the entire process more intuitive and user-friendly. This design avoids the problem of separating timbre selection and speech synthesis in traditional TTS, achieving a more coherent user experience.

[0092] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 7A schematic structural block diagram of a speech generation apparatus 700 according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0093] Reference Figure 7 The apparatus 700 includes a feature representation extraction module 710, a reference speech providing module 720, and a target speech generation module 730. The feature representation extraction module 710 is configured to extract a reference speech feature representation corresponding to at least one first input content in response to receiving at least one first input content. The reference speech feature representation indicates at least timbre-related feature information in at least one first input content. The reference speech providing module 720 is configured to provide reference speech generated based on reference text and the reference speech feature representation. The target speech generation module 730 is configured to generate target speech based on target text and the reference speech in response to detecting confirmation of the reference speech. The timbre of the target speech matches the timbre of the reference speech.

[0094] In some embodiments, at least one first input content includes multiple first input contents of different types. The feature representation extraction module 710 is further configured to: extract speech feature representations corresponding to each of the multiple first input contents; and perform weighted fusion of the speech feature representations corresponding to each of the multiple first input contents based on their respective weights to obtain a reference speech feature representation.

[0095] In some embodiments, the plurality of first input contents include an input image, input speech and / or input text, and wherein, during the weighted fusion process, the input speech has a greater weight than the input image and the input text.

[0096] In some embodiments, the reference text is determined by: determining the reference text based on the target text; and / or determining the reference text based on the reference speech feature representation.

[0097] In some embodiments, the target speech generation module 730 is further configured to: construct cue word information for a first machine learning model from the reference speech and the target text; and generate target speech based on the cue word information and using the first machine learning model.

[0098] In some embodiments, in the prompt word information, the reference speech is constructed as the system prompt word for the first machine learning model.

[0099] In some embodiments, the apparatus 700 further includes an adjustment module. The adjustment module is configured to regenerate the reference speech based on the reference text and the feature representation of the reference speech in response to detecting an adjustment instruction for the reference speech.

[0100] In some embodiments, the adjustment module is further configured to: update the reference speech feature representation based at least on at least one second input content associated with the adjustment instruction in response to receiving the second input content; and regenerate the reference speech based on the updated reference speech feature representation and the reference text.

[0101] In some embodiments, the reference speech feature representation is extracted using a second machine learning model. Training the second machine learning model includes: extracting sample speech feature representations corresponding to at least one sample input content based on at least one sample input content, wherein the sample speech feature representations at least indicate timbre-related feature information in at least one sample input content; generating sample text representations describing the sample speech feature representations; and training the second machine learning model based on the similarity between the sample text representations and the labeled text representations associated with at least one sample input content.

[0102] Figure 8 A block diagram is shown of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. The electronic device 800 may, for example, be used to implement... Figure 1 The electronic device 110 shown or such Figure 7 The device 700 shown. It should be understood that, Figure 8 The electronic device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0103] Reference Figure 8 The electronic device 800 is in the form of a general-purpose electronic device. Components of the electronic device 800 may include, but are not limited to, one or more processors 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processor 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0104] Electronic device 800 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 800.

[0105] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0106] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0107] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0108] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0109] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0113] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A speech generation method, comprising: In response to receiving at least one first input content, a reference speech feature representation corresponding to the at least one first input content is extracted, wherein the reference speech feature representation at least indicates timbre-related feature information in the at least one first input content; Provide reference speech generated based on reference text and the reference speech feature representation; as well as In response to the detection of confirmation of the reference speech, a target speech is generated based on the target text and the reference speech, wherein the timbre of the target speech matches the timbre of the reference speech.

2. The method according to claim 1, wherein the at least one first input content includes a plurality of first input contents of different types, and wherein extracting the reference speech feature representation corresponding to the at least one first input content includes: Extract the speech feature representations corresponding to each of the multiple first input contents; as well as The speech feature representations corresponding to each of the plurality of first input contents are weighted and fused based on their respective weights to obtain the reference speech feature representation.

3. The method of claim 2, wherein the plurality of first input contents include input images, input speech and / or input text, and wherein during the weighted fusion process, the weight of the input speech is greater than the weight of the input images and the input text.

4. The method of claim 1, wherein the reference text is determined by: Based on the target text, determine the reference text; and / or The reference text is determined based on the reference speech feature representation.

5. The method of claim 1, wherein generating the target speech comprises: The reference speech and the target text are used to construct prompt word information for the first machine learning model; as well as Based on the prompt word information, the target speech is generated using the first machine learning model.

6. The method of claim 5, wherein in the prompt word information, the reference speech is constructed as a system prompt word for the first machine learning model.

7. The method according to claim 1, further comprising: In response to the detection of an adjustment instruction for the reference speech, the reference speech is regenerated based on the reference text and the reference speech feature representation.

8. The method according to claim 7, further comprising: In response to receiving at least one second input content associated with the adjustment instruction, the reference speech feature representation is updated based at least on the at least one second input content; as well as The reference speech is regenerated based on the updated reference speech feature representation and the reference text.

9. The method of claim 1, wherein the reference speech feature representation is extracted using a second machine learning model, and training the second machine learning model comprises: Based on at least one sample input, using the second machine learning model, Extract sample speech feature representations corresponding to the at least one sample input content, wherein the sample speech feature representations at least indicate timbre-related feature information in the at least one sample input content. Generate a sample text representation to describe the sample speech feature representation; and The second machine learning model is trained based on the similarity between the sample text representation and the labeled text representation associated with the at least one sample input content.

10. An apparatus for speech generation, comprising: The feature representation extraction module is configured to extract a reference speech feature representation corresponding to the at least one first input content in response to receiving at least one first input content, wherein the reference speech feature representation at least indicates timbre-related feature information in the at least one first input content; A reference speech providing module is configured to provide reference speech generated based on reference text and the reference speech feature representation; as well as A target speech generation module is configured to generate target speech based on target text and the reference speech in response to detecting confirmation of the reference speech, wherein the timbre of the target speech matches the timbre of the reference speech.

11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images