Speech synthesis method, device, equipment and computer medium

Abstract

The invention provides a speech synthesis method, device and equipment and a computer readable medium, and the method comprises the steps: obtaining input data, the input data comprises a reference audio and a target synthesis text, and then determining a target tone feature and a reference speech content feature sequence according to the reference audio, reference voice content features in the reference voice content feature sequence are used for representing at least one of audio content, intonation, rhythm and speaking habits of the reference voice, and inputting the reference voice content feature sequence and the target synthetic text into a pre-trained voice content feature generation model, according to the method, the target voice content feature sequence is generated, and then the target synthetic audio is generated according to the target timbre feature and the target voice content feature sequence, so that the target synthetic text can be converted into the target synthetic audio conforming to the timbre, the intonation, the rhythm, the speaking habit and the like of the reference audio, and the authenticity of voice synthesis is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech synthesis method, apparatus, device and computer-readable medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

[0003] With the development of artificial intelligence technology, Text-to-Speech (TTS) systems have become a part of people's daily lives. TTS is a technology that can convert text information into audible sounds. This technology can convert any written text into oral expressions and has been widely used in human-computer interaction, virtual assistants, intelligent customer service and other fields.

[0004] Traditional text-to-speech technology outputs a single machine-synthesized sound, which is often mechanical and monotonous, lacking the emotional color of natural language. It is difficult to imitate the timbre and speaking style of a specific individual, and it is impossible to generate personalized, human-like speech.

[0005] In view of the above problems, there is a need to provide a more realistic speech synthesis solution that can imitate the timbre and speaking style of a specific individual. Summary of the invention

[0006] Multiple aspects of the present application provide a speech synthesis method, apparatus, device and computer-readable storage medium for comprehensively imitating the timbre and speaking style of a specific individual to improve the authenticity of speech synthesis.

[0007] In one aspect of the present application, a speech synthesis method is provided, wherein the method comprises:

[0008] Acquire input data, wherein the input data includes reference audio and target synthesized text;

[0009] Determine a target timbre feature and a reference speech content feature sequence according to the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech;

[0010] Inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence;

[0011] A target synthesized audio is generated according to the target timbre feature and the target speech content feature sequence.

[0012] Another aspect of the present application provides a speech synthesis device, wherein the device comprises:

[0013] An input data acquisition module, used to acquire input data, wherein the input data includes reference audio and target synthesized text;

[0014] A reference audio processing module, used to determine a target timbre feature and a reference speech content feature sequence according to the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech;

[0015] A content feature sequence generation module, used for inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence;

[0016] The audio synthesis module is used to generate a target synthesized audio according to the target timbre feature and the target speech content feature sequence.

[0017] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech synthesis method as described above.

[0021] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the speech synthesis method as described above.

[0022] In the solution provided by the embodiment of the present application, input data is obtained, the input data includes reference audio and target synthesized text, and then according to the reference audio, a target timbre feature and a reference speech content feature sequence are determined, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech, and then the reference speech content feature sequence and the target synthesized text are input into a pre-trained speech content feature generation model to generate a target speech content feature sequence, and then a target synthesized audio is generated according to the target timbre feature and the target speech content feature sequence, so that the target synthesized text can be generated. The target synthesized audio is converted into a target synthesized audio that is consistent with the reference audio, wherein, since the synthesis elements of the target synthesized audio include the target timbre features extracted from the reference audio, the target synthesized audio can inherit the timbre of the reference audio, and since the target speech content feature sequence is generated by fusion of the reference speech content feature sequence of the reference audio and the target synthesized text, the target synthesized speech can also inherit the speech content features of the reference audio, including intonation, rhythm and speaking habits, etc., to ensure that the generated target synthesized speech can effectively inherit the timbre, intonation, rhythm and speaking habits of the reference audio, thereby achieving comprehensive imitation of the timbre and speaking style of a specific individual, and improving the authenticity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0025] Figure 1 A flowchart of a speech synthesis method provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of a model architecture provided for an embodiment of the present application;

[0027] Figure 3 A flowchart of a model training process provided in one embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of the present application;

[0029] Figure 5A schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiments of the present application;

[0030] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0033] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0034] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0035] In view of the problem that the speech synthesis technology in the prior art is difficult to imitate the timbre and speaking style of a specific individual, and is therefore unable to generate personalized, human-like speech, the embodiments of the present application provide a speech synthesis method, apparatus, device, and computer-readable storage medium to improve the authenticity and human-like effect of speech synthesis, which obtains input data, the input data including reference audio and target synthesized text, and then determines a target timbre feature and a reference speech content feature sequence based on the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm, and speaking habits of the reference speech, and then inputs the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence. The target synthesized audio is then generated according to the target timbre features and the target speech content feature sequence, so that the target synthesized text can be converted into a target synthesized audio that matches the reference audio, wherein, since the synthesis elements of the target synthesized audio include the target timbre features extracted from the reference audio, the target synthesized audio can inherit the timbre of the reference audio, and since the target speech content feature sequence is generated by fusing the reference speech content feature sequence of the reference audio and the target synthesized text, the target synthesized speech can also inherit the speech content features of the reference audio, including intonation, rhythm and speaking habits, etc., to ensure that the generated target synthesized speech can effectively inherit the timbre, intonation, rhythm and speaking habits of the reference audio, thereby achieving a comprehensive imitation of the timbre and speaking style of a specific individual, and improving the authenticity of speech synthesis.

[0036] The following is an exemplary description of the speech synthesis method, apparatus, device and computer-readable storage medium provided in the present application.

[0037] In actual scenarios, the execution subject of the speech synthesis method provided in the embodiment of the present application can be a user device, or a device formed by integrating a user device and a network device through a network, or an application program (such as a speech synthesis application, a speech synthesis system, etc.) running on the above-mentioned device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0038] Figure 1 The processing flow of a speech synthesis method provided in an embodiment of the present application is shown, and the method at least includes the following processing steps:

[0039] Step S101, obtaining input data, wherein the input data includes reference audio and target synthesized text.

[0040] The reference audio refers to an audio sample provided by a user that contains a specific timbre and speaking style, and the target synthesized text refers to the text content that the user wishes to synthesize.

[0041] When implementing this step, the reference audio can be obtained through audio files provided by the user. These audio files can be voice samples of a few seconds, which contain the user's timbre and speaking habits. By reading these audio files, the timbre features and voice content features can be extracted. A specific implementation method includes using an audio processing tool (such as Python's torchaudio library) to read the audio file and convert it into a digital signal suitable for processing. This process can ensure the accurate acquisition and processing of audio data, providing a reliable basis for subsequent feature extraction and matching.

[0042] In addition, the acquisition of the target synthetic text can be achieved through the text content provided by the user. The text content can be any form of written text, such as sentences, paragraphs or articles. By reading these text contents, the specific text information that needs to be synthesized can be determined. A specific implementation method includes using a text processing tool (such as Python's open function) to read the text file and convert it into a string format suitable for processing. This process can ensure the accurate acquisition and processing of text data, providing a reliable basis for subsequent feature generation and synthesis.

[0043] By obtaining reference audio and target synthesized text, it is possible to ensure that the synthesized speech generated in the subsequent steps is not only consistent with the reference audio in terms of timbre, but also completely matches the target synthesized text in terms of content. The provision of reference audio enables the speech synthesis application or system to capture the timbre and speaking style of a specific individual, while the provision of target synthesized text enables the speech synthesis application or system to generate synthesized speech that is completely consistent with user needs. This step can ensure the accuracy and reliability of the entire speech synthesis process and provide users with high-quality speech synthesis services.

[0044] Step S102, determining a target timbre feature and a reference speech content feature sequence based on the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech.

[0045] The audio content may refer to the specific information and meaning contained in the reference speech, that is, the text content expressed by the speech. For example, an audio content describing the weather may include specific text information such as "It is sunny today and the temperature is moderate".

[0046] Intonation refers to the ups and downs in speech, which reflects the speaker's emotional state and tone. For example, an interrogative tone is usually expressed as a rising tone, while a declarative tone is usually a falling tone. By performing feature analysis and extraction on the intonation of the reference speech, it is possible to generate synthetic audio with specific emotions and tones, making the synthetic audio sound more natural and expressive.

[0047] Prosody can refer to the rhythm and stress pattern in speech, including the duration, pauses and stress of syllables. Prosody determines the fluency and naturalness of speech. For example, a formal speech usually has a steady rhythm with appropriate pauses, while a spoken communication may have a faster rhythm with fewer pauses. By performing feature analysis and extraction on the prosody of a reference speech, it is possible to generate synthetic audio with a rhythm and stress pattern consistent with the reference speech, making the synthetic audio sound more fluent and natural.

[0048] Speaking habits can refer to the personal characteristics of the speaker in terms of pronunciation, speaking speed, pauses, etc., which reflect the speaker's personality and style. For example, some people may have specific catchphrases or habitual pauses when speaking, while others may speak faster and pronounce clearly. By performing feature analysis and extraction on the speaking habits of the reference voice, it is possible to generate synthetic audio with specific personal characteristics, making the synthetic audio sound more personalized and realistic.

[0049] One of the purposes of this step is to ensure that the generated synthetic speech is not only consistent with the reference audio in timbre, but also captures many aspects of the reference audio, such as audio content, intonation, rhythm, and speaking habits.

[0050] When this step is implemented, the target timbre features may refer to features extracted from the reference audio that can characterize the speaker's timbre. These features usually include frequency response, formants, fundamental frequency (pitch), etc. A specific implementation method is to use a pre-trained timbre encoder to process the reference audio and extract the timbre features. For example, a deep learning model (such as a convolutional neural network CNN) can be used to extract features from the audio signal to generate a feature vector that can characterize the timbre. This process can ensure that the timbre features in the reference audio are accurately captured, providing a basis for subsequent synthetic speech.

[0051] In addition, the reference speech content feature sequence refers to a series of features extracted from the reference audio, which are used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference audio. One of the specific implementation methods is to use a pre-trained speech content representation model (such as the HuBERT model) to process the reference audio and extract the speech content features. For example, the HuBERT model learns the high-level representation of speech signals through mask prediction, which can capture features such as intonation, rhythm and speaking habits in the audio. The extracted reference speech content features are organized into a feature sequence, namely the reference speech content feature sequence, which is used for subsequent feature generation and synthesis.

[0052] Through this step, it can be ensured that the generated synthesized speech is consistent with the reference audio not only in timbre, but also in speech content, intonation, rhythm, speaking habits, etc. For example, assuming that the reference audio provided by the user is a sentence with a specific intonation and rhythm, these features can be extracted through step S102 and retained in the subsequent synthesis process, so that the generated synthesized speech sounds more natural and realistic.

[0053] This step can significantly improve the naturalness and realism of speech synthesis. Specifically, by accurately extracting the target timbre features and the reference speech content feature sequence, it can ensure that the generated synthesized speech is highly consistent with the reference audio in terms of timbre, intonation, rhythm and speaking habits, thereby achieving more personalized and human-like speech output. This step can not only improve the quality of synthesized speech, but also enhance the flexibility and adaptability of speech synthesis, ensuring that high-quality speech synthesis services can be provided in various application scenarios.

[0054] Step S103: input the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence.

[0055] One of the purposes of this step is to ensure that the generated synthesized speech completely matches the target synthesized text in terms of content while retaining the speech content characteristics in the reference audio.

[0056] The reference speech content feature sequence is a series of features extracted from the reference audio, which are used to characterize at least one of the audio content, intonation, prosody, and speaking habits of the reference audio. These features can be extracted by a pre-trained speech content representation model (such as the HuBERT model) and organized into a feature sequence. The target synthetic text is the text content that the user wants to synthesize, usually provided in the form of a character string. The target speech content feature sequence is a feature sequence generated based on the reference speech content feature sequence and the target synthetic text, which can characterize the audio content, intonation, prosody, and speaking habits of the target synthetic text.

[0057] This step may include the following sub-steps during implementation:

[0058] Input of feature sequence and text: The extracted reference speech content feature sequence and target synthesized text are input into a pre-trained speech content feature generation model. This model is usually a deep learning model, such as a Transformer-based model, which can process sequence data and generate new feature sequences.

[0059] Feature generation: The model maps the reference speech content feature sequence and the target synthesized text to a new feature space through the internal multi-layer neural network structure to generate the target speech content feature sequence. This process includes encoding the target synthesized text and combining the information in the reference speech content feature sequence to generate a feature sequence corresponding to the target synthesized text.

[0060] Output of feature sequence: The generated target speech content feature sequence can characterize the audio content, intonation, rhythm, speaking habits and other features of the target synthesized text, ensuring that the generated synthesized speech completely matches the target synthesized text in content while retaining the speech content features in the reference audio.

[0061] Through the above sub-steps, it is possible to ensure that the generated synthesized speech is not only consistent with the target synthesized text in terms of content, but also retains the speech content features in the reference audio. For example, assuming that the target synthesized text provided by the user is a sentence describing the weather, this step can generate a target speech content feature sequence corresponding to this text, while retaining features such as intonation and rhythm in the reference audio, so that the generated synthesized speech sounds more natural and realistic.

[0062] Through this step, the naturalness and realism of speech synthesis can be significantly improved. Specifically, by inputting the reference speech content feature sequence and the target synthesized text into the pre-trained speech content feature generation model, a target speech content feature sequence that completely matches the target synthesized text can be generated. This process not only ensures the accuracy of the content of the generated synthesized speech, but also retains the speech content features in the reference audio, making the synthesized speech more natural and realistic in terms of intonation, rhythm, and speaking habits, thereby improving the overall quality of the synthesized speech and user experience, making it more widely and effectively used in human-computer interaction, virtual assistants, intelligent customer service and other fields.

[0063] Step S104, generating a target synthesized audio according to the target timbre feature and the target speech content feature sequence.

[0064] One of the purposes of this step is to ensure that the generated synthetic audio is not only consistent with the reference audio in timbre, but also completely matches the target synthetic text in content, while retaining the speech content characteristics in the reference audio.

[0065] This step may include the following sub-steps during implementation:

[0066] Feature fusion: The target timbre feature and the target speech content feature sequence are fused. This process usually forms a comprehensive feature vector by splicing the two feature sequences together. For example, the target timbre feature vector and the target speech content feature sequence can be spliced ​​into a high-dimensional feature vector.

[0067] Feature decoding: The fused feature vector is input into the decoding part of the pre-trained audio codec model to generate the target synthetic audio. The audio codec model is usually a deep learning model, such as a Transformer-based model or a convolutional neural network (CNN), which can convert high-dimensional feature vectors into time domain signals. The decoding process involves gradually converting the feature vector into an audio waveform to generate the final synthetic audio.

[0068] Audio output: The generated target synthesized audio can represent the content of the target synthesized text and retain the characteristics of the reference audio such as timbre, intonation, rhythm, and speaking habits.

[0069] Through this step, it is possible to ensure that the generated synthesized audio is not only consistent with the reference audio in terms of timbre, but also completely matches the target synthesized text in terms of content, while retaining the speech content features in the reference audio. For example, assuming that the target synthesized text provided by the user is a sentence describing the weather, after this step, the synthesized audio corresponding to this text can be generated, while retaining the timbre and intonation features in the reference audio, making the generated synthesized audio sound more natural and realistic.

[0070] The above-mentioned speech synthesis method provided in the embodiment of the present application obtains input data, wherein the input data includes reference audio and target synthesized text, and then determines target timbre features and a reference speech content feature sequence based on the reference audio, wherein the reference speech content features in the reference speech content feature sequence are used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech, and then inputs the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence, and then generates a target synthesized audio based on the target timbre features and the target speech content feature sequence, so that the target The synthesized text is converted into a target synthesized audio that matches the reference audio, wherein, since the synthesis elements of the target synthesized audio include the target timbre features extracted from the reference audio, the target synthesized audio can inherit the timbre of the reference audio, and since the target speech content feature sequence is generated by fusion of the reference speech content feature sequence of the reference audio and the target synthesized text, the target synthesized speech can also inherit the speech content features of the reference audio, including intonation, rhythm and speaking habits, etc., to ensure that the generated target synthesized speech can effectively inherit the timbre, intonation, rhythm and speaking habits of the reference audio, thereby achieving comprehensive imitation of the timbre and speaking style of a specific individual, and improving the authenticity of speech synthesis.

[0071] In some modified implementations, the above step S102, determining the target timbre feature and the reference speech content feature sequence according to the reference audio, may include:

[0072] Performing timbre feature extraction and speech content feature extraction on the reference audio according to a pre-trained audio codec model to obtain a timbre feature extraction result and a content feature extraction result;

[0073] Determining a target timbre feature according to the timbre feature extraction result, and determining a reference speech content feature sequence according to the content feature extraction result;

[0074] Accordingly, the above step S104, generating the target synthesized audio according to the target timbre feature and the target speech content feature sequence, may include:

[0075] The target timbre feature and the target speech content feature sequence are decoded according to the pre-trained audio codec model to obtain a target synthesized audio.

[0076] Among them, the audio codec model is a pre-trained deep learning model that can encode and decode audio signals. The encoding part is responsible for extracting timbre features and speech content features from the audio signal, and the decoding part is responsible for converting the feature sequence back into an audio signal.

[0077] Tone feature extraction refers to extracting features that can characterize the speaker's tone from reference audio, such as frequency response, formants, fundamental frequency (pitch), etc.

[0078] Speech content feature extraction refers to extracting features from reference audio that can represent audio content, intonation, rhythm, and speaking habits.

[0079] The timbre feature extraction result refers to the feature vector or feature sequence obtained after the timbre feature extraction.

[0080] The content feature extraction result refers to the feature vector or feature sequence extracted from the reference audio that can represent the audio content, intonation, rhythm and speaking habits.

[0081] The target timbre feature refers to a feature determined from the timbre feature extraction result and used to characterize the target timbre.

[0082] The reference speech content feature sequence refers to a feature sequence determined from the content feature extraction result and used to characterize the audio content, intonation, rhythm and speaking habits of the reference audio.

[0083] Please refer to Figure 2 To understand, it schematically shows a schematic diagram of the model architecture provided by an embodiment of the present application. In this embodiment, the model architecture on which the speech synthesis method relies includes two parts: an audio codec model and a speech content feature generation model. Speech synthesis is performed through the collaborative work of these two parts. The audio codec model includes two stages in the entire synthesis process, namely the encoding stage and the decoding stage. In the encoding stage, it extracts timbre features and speech content features from the reference audio, and obtains a reference speech content feature sequence and a target timbre feature based on the extraction results. On this basis, the speech content feature generation model generates a target speech content feature sequence based on the reference speech content feature sequence and the target synthesized text. Then, in the decoding stage of the audio codec model, it decodes according to the aforementioned target timbre features and target speech content feature sequence to obtain the target synthesized audio. Both the audio codec model and the speech content feature generation model need to be trained in advance. Figure 2 The model parameter file in the contains the trained model parameters and can be directly loaded and used. It should be noted that since the audio codec model and the speech content feature generation model work together and have a strong coupling relationship, the two are jointly trained, that is, the two are trained synchronously using the same samples. Therefore, the model parameter file obtained after training can be stored as two different files or as one file, both of which can achieve the purpose of the embodiments of the present application and should be within the scope of protection of the present application.

[0084] In the specific implementation of this embodiment, correspondingly, step S102 may include the following sub-steps:

[0085] Timbre feature extraction: Use a timbre encoder to process the reference audio and extract the timbre feature extraction results. The timbre encoder converts the audio signal into a feature vector that can represent the timbre through a multi-layer neural network structure.

[0086] Speech content feature extraction: Use a speech content representation model (such as the HuBERT model) to process the reference audio and extract content feature extraction results. The speech content representation model can learn the high-level representation of the audio signal through mask prediction and extract feature vectors that can represent the audio content, intonation, rhythm and speaking habits as speech content features.

[0087] Determine the target timbre features: Determine the target timbre features based on the timbre feature extraction results. This step usually generates a feature vector that can characterize the target timbre by selecting key features from the timbre feature extraction results.

[0088] Determine the reference speech content feature sequence: Determine the reference speech content feature sequence based on the content feature extraction results. This step is usually performed by organizing the content feature extraction results into a feature sequence for subsequent feature generation and synthesis.

[0089] Accordingly, step S104 may include the following sub-steps:

[0090] Feature fusion: The target timbre feature and the target speech content feature sequence are fused to form a comprehensive feature vector. This step is usually performed by splicing the two feature sequences together to generate a high-dimensional feature vector.

[0091] Feature decoding: The fused feature vector is input into the decoding part of the pre-trained audio codec model to generate the target synthetic audio. The decoding part can gradually convert the high-dimensional feature vector into a time domain signal through a multi-layer neural network structure to generate the final synthetic audio.

[0092] Audio output: The generated target synthesized audio can represent the content of the target synthesized text and retain the characteristics of the reference audio such as timbre, intonation, rhythm, and speaking habits.

[0093] Through the above implementation, the naturalness and fidelity of speech synthesis can be significantly improved. The specific technical effects are as follows:

[0094] Timbre consistency: By using a pre-trained audio codec model to extract timbre features, it is possible to ensure that the generated synthetic audio is highly consistent with the reference audio in timbre, and that the target synthetic audio has the timbre characteristics of a specific individual, making the target synthetic audio more personalized and realistic.

[0095] Content matching: By using a pre-trained audio codec model to extract speech content features, it is possible to ensure that the generated synthetic audio fully matches the target synthetic text in terms of content. This step ensures the accuracy of the content of the synthetic audio, allowing the synthetic audio to accurately convey the information of the target text.

[0096] Naturalness improvement: By inputting the target timbre features and target speech content feature sequences into a pre-trained audio codec model for decoding, it is possible to generate synthetic audio with natural intonation, rhythm, and speaking habits, thereby improving the naturalness of the synthetic audio and making it sound more smooth and natural.

[0097] Based on the above implementation, in some modified implementations, the audio codec model includes a timbre encoder and a speech content representation model;

[0098] The step of extracting timbre features and speech content from the reference audio according to the pre-trained audio codec model to obtain timbre feature extraction results and content feature extraction results may include:

[0099] Performing timbre feature encoding on the reference audio according to the timbre encoder to obtain a timbre feature extraction result;

[0100] The speech content features of the reference audio are extracted and represented according to the speech content representation model to obtain a content feature extraction result.

[0101] Among them, the timbre encoder is a part of the audio codec model, which is specifically used to extract timbre features from audio signals.

[0102] The speech content representation model is also part of the audio codec model and is specifically used to extract speech content features from audio signals. These features are used to characterize audio content, intonation, prosody, and speaking habits.

[0103] In the specific implementation of this embodiment, a timbre encoder can be used to process the reference audio to extract the timbre feature extraction result. The timbre encoder converts the audio signal into a feature vector that can characterize the timbre through a multi-layer neural network structure. For example, a convolutional neural network (CNN) or a recurrent neural network (RNN) can be used to extract features of the audio signal to generate a feature vector that can characterize the timbre.

[0104] At the same time, the speech content representation model can be used to process the reference audio and extract the content feature extraction results. The speech content representation model can learn the high-level representation of the audio signal through mask prediction and extract the feature vector that can represent the audio content, intonation, rhythm and speaking habits. For example, the HuBERT model can be used to process the audio signal to generate a feature vector that can represent the speech content characteristics.

[0105] This embodiment can extract accurate timbre features through the timbre encoder to ensure that the generated synthesized audio is highly consistent with the reference audio in timbre, thereby achieving personalized and realistic timbre output, and can extract accurate speech content features through the speech content representation model to ensure that the generated synthesized audio fully matches the target synthesized text in content, thereby ensuring the accuracy of the synthesized audio. On this basis, by inputting the target timbre features and target speech content feature sequences into a pre-trained audio codec model for decoding, it is possible to generate synthesized audio with natural intonation, rhythm and speaking habits, significantly improving the naturalness of the synthesized audio.

[0106] Based on the above implementation, in some modified implementations, the speech content representation model includes a HuBERT model, and correspondingly, the content feature extraction result includes a HuBERT feature.

[0107] Among them, the HuBERT (Hidden Unit BERT) model is a speech content representation model based on self-supervised learning. It learns the high-level representation of audio signals through mask prediction and can extract rich speech content features.

[0108] In this embodiment, the HuBERT model can be used to process the reference audio to extract the content feature extraction result, i.e., the HuBERT feature. The HuBERT model learns the high-level representation of the audio signal by mask prediction and extracts the feature vector that can characterize the audio content, intonation, prosody, and speaking habits. For example, the HuBERT model can be used to process the audio signal to generate a feature vector that can characterize the speech content characteristics.

[0109] By processing the reference audio with the HuBERT model, precise speech content features can be extracted to ensure that the generated synthetic audio fully matches the target synthetic text in content.

[0110] In some modified implementations, determining the target timbre feature according to the timbre feature extraction result, and determining the reference speech content feature sequence according to the content feature extraction result, include:

[0111] Determine the timbre feature extraction result as the target timbre feature, and determine the feature sequence composed of the content feature extraction result as the reference speech content feature sequence; or,

[0112] A timbre feature that matches the timbre feature extraction result is queried from the underlying timbre feature library corresponding to the pre-trained audio codec model as a target timbre feature, and a speech content feature sequence that matches the content feature extraction result is queried from the underlying content feature library corresponding to the pre-trained audio codec model as a reference speech content feature sequence.

[0113] This embodiment exemplarily provides two methods for determining the target timbre features and the reference speech content feature sequence. The first method is to directly use the extraction results, and the second method is to query the matching features from the base model feature library.

[0114] In the first method, the features extracted from the reference audio by the timbre encoder and the speech content representation model (such as the HuBERT model) are directly used as the target timbre features and the reference speech content feature sequence. This method is particularly suitable for scenarios with long and large reference audio. Specifically, when the reference audio provided by the user is long and large in number, directly using the extraction results can make full use of these high-quality samples to generate high-quality synthetic audio. In this scenario, the longer reference audio can provide more timbre and content feature information, allowing the timbre encoder and the speech content representation model to extract features more accurately. In addition, for real-time applications that need to quickly generate synthetic audio, such as online virtual assistants and real-time translation, directly using the extraction results can reduce query time and improve system response speed. For one-time or small-scale speech synthesis tasks, directly using the extraction results can simplify the process and reduce the consumption of computing resources.

[0115] The advantages of the first method are: 1. Fast and efficient. The method of directly using the extraction results is simple and direct, which reduces the time of querying the base model feature library and can quickly generate high-quality synthetic audio. 2. High precision. When the reference audio is long and the number is large, the timbre encoder and speech content representation model can accurately extract the timbre and content features to ensure that the generated synthetic audio is highly consistent with the reference audio in timbre and content. 3. Resource saving. There is no need to maintain and query the base model feature library, which reduces the demand for storage and computing resources and is suitable for resource-constrained environments.

[0116] In the second method, features matching the extraction results are queried from the underlying timbre feature library and underlying content feature library corresponding to the pre-trained audio codec model as target timbre features and reference speech content feature sequences. This method is particularly suitable for scenarios with short and small reference audio. Specifically, when the reference audio provided by the user is short and small in number, the directly extracted features may not be accurate enough or sufficient to represent the target timbre and content features. In this case, querying matching features from the underlying feature library can provide more stable feature matching and improve the accuracy of synthesized audio. The underlying feature library stores a variety of timbre features and content features, which can provide a variety of choices to meet the needs of different scenarios. In addition, for scenarios that need to handle a large number of speech synthesis tasks, such as intelligent customer service, voice broadcast systems, etc., the underlying feature library can provide stable feature matching and improve the robustness and consistency of the system.

[0117] The advantages of the second method are: 1. High robustness. By querying matching features from the base model feature library, even if the reference audio is short and small in number, the closest timbre and content features can be found to ensure the consistency of the generated synthetic audio in timbre and content. 2. Diversified timbre. The base model feature library stores a variety of timbre features, which can meet the timbre requirements in different scenarios and improve the diversity of synthetic audio. 3. Stability. The base model feature library provides pre-trained and verified features to ensure that the generated synthetic audio maintains stable quality in different tasks and scenarios. Strong adaptability. With the support of the base model feature library, it can flexibly respond to different types of reference audio and target synthetic text, improving the adaptability and flexibility of the system.

[0118] In some modified implementations, the input data further includes a reference text corresponding to the reference audio;

[0119] The method further comprises:

[0120] Fine-tuning the pre-trained audio codec model and speech content feature generation model according to the reference audio and the reference text;

[0121] Accordingly, the step of inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence includes:

[0122] Inputting the reference speech content feature sequence and the target synthesized text into the fine-tuned speech content feature generation model to generate a target speech content feature sequence;

[0123] The step of decoding the target timbre feature and the target speech content feature sequence according to the pre-trained audio codec model to obtain the target synthesized audio comprises:

[0124] The target timbre feature and the target speech content feature sequence are decoded according to the fine-tuned audio codec model to obtain the target synthesized audio.

[0125] The reference text is the text content corresponding to the reference audio, which is used to assist in fine-tuning the model. Fine-tuning means further training the model using specific reference audio and reference text based on the pre-trained model to optimize the model's performance on a specific task.

[0126] The step of fine-tuning the pre-trained audio codec model and the speech content feature generation model according to the reference audio and the reference text may be deployed before step S102, and the fine-tuning process may include:

[0127] Data preparation: Pair reference audio and reference text to form a training dataset.

[0128] Fine-tuning process: Use the training data set to jointly fine-tune the pre-trained audio codec model and the speech content feature generation model. During the fine-tuning process, the parameters of the two models are updated synchronously to optimize the performance of the model on a specific task. The specific method of fine-tuning may include adjusting model parameters, increasing the number of training rounds, etc., which will not be described in detail in the embodiments of this application.

[0129] This implementation method can achieve the following effects by fine-tuning the pre-trained audio codec model and speech content feature generation model: First, in terms of personalized timbre, by fine-tuning the audio codec model, the timbre and speaking style in the reference audio can be better captured, which means that the generated synthetic audio is highly consistent with the reference audio in timbre, thereby achieving personalized and realistic timbre output. Secondly, in terms of content matching, by fine-tuning using reference text, the content of the reference audio can be better understood, and the generated synthetic audio is completely matched with the target synthetic text in content, thereby ensuring the accuracy of the synthetic audio. In addition, the fine-tuned audio codec model and speech content feature generation model can generate synthetic audio with natural intonation, rhythm and speaking habits, significantly improving the naturalness of the synthetic audio. Finally, this implementation method can not only process different types of reference audio and target synthetic text, but also provide high-quality speech synthesis services in a variety of application scenarios. Through fine-tuning, it can better adapt to the needs of specific users and improve flexibility and adaptability.

[0130] In some implementation variations, the method further comprises:

[0131] Acquire a training data set, wherein the training data set includes a plurality of training samples, each of the training samples includes an audio sample and its corresponding text sample;

[0132] The audio codec model and the speech content feature generation model are trained according to the training data set to obtain a pre-trained audio codec model and a speech content feature generation model.

[0133] In one example, training can be performed as follows:

[0134] First, a training data set is obtained, where the training data set includes a plurality of training samples, each of which includes an audio sample and its corresponding text sample. An exemplary implementation method may include:

[0135] Data collection: Collect a large number of audio samples and their corresponding text samples from multiple sources to form a training dataset. These samples can come from public voice datasets, user-provided audio data, etc.

[0136] Data preprocessing: Preprocess the collected audio samples and text samples, including audio noise reduction, text cleaning, etc., to ensure the quality and consistency of the data.

[0137] Data pairing: Pair each audio sample with its corresponding text sample to form a training sample.

[0138] Next, the audio codec model and the speech content feature generation model are trained according to the training data set to obtain a pre-trained audio codec model and a speech content feature generation model. An exemplary implementation method may include:

[0139] Model initialization: Initialize the audio codec model and speech content feature generation model, and set initial parameters.

[0140] Training process: Use the training data set to train the audio codec model and the speech content feature generation model. During the training process, the model continuously adjusts parameters through the back propagation algorithm to optimize the performance of the model on specific tasks. The specific training methods may include batch gradient descent, stochastic gradient descent, etc., which will not be described here.

[0141] Model evaluation: Regularly evaluate the performance of the model during the training process, and use the validation set data to check the accuracy and generalization ability of the model. Based on the evaluation results, you can adjust the training parameters to further optimize the model performance.

[0142] Please continue to refer to Figure 3For understanding, it schematically shows a flow chart of the model training process provided by an embodiment of the present application. As shown in the figure, it is necessary to first prepare a training data set, which can be a long audio training set. Then, the audio samples in the long audio training set are preprocessed by removing background sound, normalizing the format, etc., and then cut into sections to obtain a short audio training set. Each audio sample in the short audio training set is automatically labeled with a text sample to obtain a short audio training set with annotated text. BERT feature extraction and HuBERT feature extraction are performed on the samples in the short audio training set, and the extracted features are formatted to meet the data format requirements of the audio codec model and the speech content feature generation model. Then, the audio codec model and the speech content feature generation model can be input for model training, and the model parameter file is output after the training is completed.

[0143] This implementation method, by using a large amount of training data sets to train the audio codec model and the speech content feature generation model, can optimize the model's parameters, improve the model's performance on specific tasks, and make the generated synthetic audio more natural and realistic in terms of timbre, intonation, rhythm, and speaking habits; the audio samples and text samples contained in the training data set can help the model better understand and generate audio content, ensure that the synthetic audio is completely matched with the target synthetic text in content, and improve the accuracy of the synthetic audio; at the same time, by using a variety of training data sets, the generalization ability of the model can be improved, so that it still performs well when facing new and unseen audio samples and text samples, and enhances robustness and adaptability; in addition, the pre-trained audio codec model and speech content feature generation model are highly flexible in different application scenarios, and can better meet the needs of different users and improve user experience.

[0144] To facilitate understanding, further explanation is provided below with reference to specific examples.

[0145] In a specific example, the training process of the above audio codec model and speech content feature generation model may include the following:

[0146] 1.1. Prepare training data set: The training data set of this embodiment consists of a series of paired text samples and audio samples. Each sample contains corresponding text content and its corresponding audio file.

[0147] 1.2 Preprocessing: Standardize the audio files, including but not limited to removing background noise, normalizing the volume, etc., to ensure that the audio quality meets the training requirements. Perform preprocessing operations such as cleaning, word segmentation, and encoding on the text content to facilitate subsequent processing.

[0148] 1.3. Feature extraction, including text feature extraction and audio feature extraction:

[0149] 1.3.1. For text features, the BERT model is used to process the text content corresponding to each audio file to extract semantic features. For example, suppose a text: "Today the weather is fine, suitable for going out for a walk." BERT will analyze the meaning of this text, including the positive emotion conveyed by "the weather is fine" and the suggested action of "going out for a walk". This information will be used to guide the tone and intonation of speech synthesis.

[0150] Among them, the BERT (Bidirectional Encoder Representations from Transformers) model is a deep learning model based on the Transformer architecture, specially designed for natural language processing tasks. BERT can understand the contextual meaning of text, which is crucial for accurately generating contextual speech. For example, BERT can extract the emotional color, grammatical structure and lexical meaning of sentences from text, which will be used to guide the intonation and stress in the speech synthesis process.

[0151] 1.3.2. For audio features, the Hubert model is used to process the audio files of each sample and extract audio features. Audio features include frequency components, intensity changes, etc. These features can help identify the timbre characteristics of the speaker. For example, Hubert can detect the softness of a speaker's voice or the hoarseness of another person's voice. These features are crucial for later speech synthesis, as they ensure that the synthesized speech has natural sound quality and clear recognizability.

[0152] 1.4. Audio codec model training: Use the audio codec model to encode the audio file and generate a discrete feature token sequence (i.e., a reference speech content feature sequence). Then, use the extracted discrete feature token sequence and the corresponding text features to train the audio codec model. For example, if a text feature describing a happy mood and the corresponding speaker timbre feature are extracted, the audio codec model will learn how to combine these features to produce speech with the corresponding timbre and emotional color. The goal of this stage is to ensure that the input audio and output audio are as consistent as possible, so that the model can learn the essential characteristics of the audio.

[0153] 1.5. Speech content feature generation model training: The extracted discrete feature token sequence and text features are input into the speech content feature generation model to train the speech content feature generation model to learn the speaker's speaking habits such as intonation, rhythm, pauses, etc. The speech content feature generation model can capture the subtle differences when the speaker expresses different emotions, such as how to adjust the speaking speed and pitch according to the text content. For example, when encountering text describing a tense scene, the speech content feature generation model will learn how to use a faster speaking speed and a higher pitch to express this tension. This ability makes the synthesized speech more natural and fluent, and can better convey the emotion and intent of the text.

[0154] 1.6. Model fine-tuning: If a small number of samples are used for synthesis, fine-tuning is performed based on the pre-trained model. A small number of samples refers to using minute-level speech samples for model training to obtain highly similar timbre simulation.

[0155] 1.7. Output model parameters, including:

[0156] Audio codec model parameters: The training parameters of the audio codec model, including the learning results of the timbre encoder.

[0157] Speech content feature generation model parameters: The training parameters of the speech content feature generation model record the results of speaking habit learning.

[0158] Based on the above example, the model reasoning process based on the above audio codec model and speech content feature generation model may include the following:

[0159] 2.1. Provide input data, including:

[0160] Reference audio: used to extract timbre characteristics for use in the synthesis process.

[0161] Reference text: used to help the speech content feature generation model understand the semantics of the reference audio to better guide the speech synthesis process.

[0162] Target synthesized text: the text that needs to be synthesized, that is, the speech content that the end user wants to synthesize.

[0163] The trained audio codec model and speech content feature generation model parameters are used to drive the inference process to ensure that the synthesized speech meets the user's intention and has the correct timbre.

[0164] 2.2. Feature extraction, including:

[0165] Timbre feature extraction: Extract timbre features from reference audio to ensure that the synthesized speech has the specified timbre.

[0166] Discrete feature token (i.e., speech content feature) extraction: Feature tokens are extracted from the reference audio for use in the subsequent completion stage.

[0167] 2.3. Audio codec model encoding: Use the audio codec model to encode the reference audio and generate a discrete feature token sequence (i.e., a reference speech content feature sequence).

[0168] 2.4. Speech content feature generation model completion: The discrete feature tokens are input as history tokens into the speech content feature generation model. At the same time, the target synthesized text is input into the speech content feature generation model, and the speech content feature generation model is used to complete the feature token sequence (i.e., the target speech content feature sequence) to reflect the semantics of the target synthesized text.

[0169] 2.5. Audio codec model decoding: Combining the timbre features and the completed feature token sequence, use the audio codec model for decoding to generate the final synthesized audio.

[0170] 2.6. Output synthesized audio: audio with specified timbre and speaking habits, that is, the final speech synthesis result.

[0171] In addition, based on the audio codec model and speech content feature generation model provided in the above example, the process of performing speech synthesis using a small number of samples may include the following:

[0172] Small-sample synthesis refers to the use of short-term (usually minute-level) speech samples for model training to obtain highly similar timbre simulation. This technology is based on a pre-trained model (such as a 2,000-hour open source training set) and quickly adapts to the timbre of a specific speaker through fine-tuning. In this way, even with only a small number of speech samples, high-quality timbre simulation effects can be achieved.

[0173] 3.1. Prepare input data, including:

[0174] A small amount of sample audio: typically 5 to 10 minutes of speech samples, used to fine-tune the model to the timbre of a specific speaker.

[0175] Trained audio codec model and speech content feature generation model parameters: basic model for fine-tuning.

[0176] 3.2 Model fine-tuning, including:

[0177] Fine-tune the audio codec model: Fine-tune the audio codec model based on a small amount of sample audio to adapt to the timbre characteristics of a specific speaker.

[0178] Fine-tune the speech content feature generation model: Fine-tune the speech content feature generation model based on a small amount of sample audio to adapt to the speaking habits of a specific speaker.

[0179] After fine-tuning is completed, the fine-tuned audio codec model and speech content feature generation model parameters are output.

[0180] 3.3. Audio codec model encoding: Use the fine-tuned audio codec model to encode a small amount of sample audio to generate a discrete feature token sequence (i.e., a reference speech content feature sequence).

[0181] 3.4. Speech content feature generation model completion: The discrete feature tokens are input as history tokens into the fine-tuned speech content feature generation model. At the same time, the target synthetic text is input into the speech content feature generation model, and the speech content feature generation model is used to complete the feature token sequence (i.e., the target speech content feature sequence) to reflect the semantics of the target text.

[0182] 3.5. Audio codec model decoding: Combining the timbre features and the completed feature token sequence, use the fine-tuned audio codec model for decoding to generate the final synthesized audio.

[0183] 3.6. Output synthesized audio: audio with specified timbre and speaking habits, that is, the final speech synthesis result.

[0184] In addition, based on the audio codec model and speech content feature generation model provided in the above example, the process of performing speech synthesis using fewer samples may include the following:

[0185] The process of using fewer samples for speech synthesis can be called zero-sample synthesis, which allows the generation of speech with similar timbre using only 3 to 10 seconds of reference audio without additional training. This function mainly relies on the existing timbre library in the model base model, and finds the closest timbre through query and matching. It should be noted that the timbre library in the base model, namely the base model timbre feature library, contains timbre features of various different timbres. These timbre features are derived from real speech samples and are stored in the library after processing and extraction. The timbre features in these timbre libraries not only cover different genders, ages, regions and emotional states, but also reflect the personalized characteristics of different speakers. In particular, in order to ensure the legality and compliance of the timbre library, all real human voices used to build the timbre library should come from legal channels, such as public voice data sets, authorized voice material libraries or audio samples voluntarily provided by users. When using these real human voices, relevant laws and regulations should be strictly observed to ensure that the data collection, processing and use process meet the requirements of privacy protection and data security.

[0186] 4.1. Prepare input data, including:

[0187] Reference audio: 3 to 10 seconds of speech samples for extracting timbre features.

[0188] Target synthesis text: the text to be synthesized.

[0189] 4.2. Feature extraction: Extract timbre features and speech content feature sequences from the reference audio to obtain timbre feature extraction results and content feature extraction results.

[0190] 4.3. Feature matching: query the timbre feature that matches the timbre feature extraction result from the underlying timbre feature library corresponding to the pre-trained audio codec model as the target timbre feature, and query the speech content feature sequence that matches the content feature extraction result from the underlying content feature library corresponding to the pre-trained audio codec model as the reference speech content feature sequence.

[0191] 4.4. Feature completion: Input the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence.

[0192] 4.5. Synthesis: Generate target synthesized audio according to the target timbre feature and the target speech content feature sequence.

[0193] 4.6. Output the synthesized target audio, that is, the synthesized audio having a similar timbre to the reference audio.

[0194] Based on the same inventive concept, a speech synthesis device is also provided in an embodiment of the present application. The method corresponding to the device may be the speech synthesis method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The speech synthesis device provided in an embodiment of the present application may implement the aforementioned speech synthesis method, and the speech synthesis device may be implemented in software, hardware, or a combination of software and hardware. For example, the speech synthesis device may include integrated or separate functional modules or units to perform the corresponding steps in the aforementioned methods. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment. The device embodiment described below is merely illustrative. Please refer to Figure 4 , the speech synthesis device may include:

[0195] An input data acquisition module 101 is used to acquire input data, wherein the input data includes reference audio and target synthesized text;

[0196] A reference audio processing module 102, configured to determine a target timbre feature and a reference speech content feature sequence according to the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech;

[0197] A content feature sequence generation module 103 is used to input the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence;

[0198] The audio synthesis module 104 is used to generate a target synthesized audio according to the target timbre feature and the target speech content feature sequence.

[0199] In some modified implementations, the reference audio processing module 102 includes:

[0200] A feature extraction unit, used to extract timbre features and speech content features from the reference audio according to a pre-trained audio codec model, to obtain timbre feature extraction results and content feature extraction results;

[0201] A feature determination unit, configured to determine a target timbre feature according to the timbre feature extraction result, and to determine a reference speech content feature sequence according to the content feature extraction result;

[0202] Accordingly, the audio synthesis module 104 includes:

[0203] The audio synthesis unit is used to decode the target timbre feature and the target speech content feature sequence according to the pre-trained audio codec model to obtain the target synthesized audio.

[0204] In some variant implementations, the audio codec model includes a timbre encoder and a speech content representation model;

[0205] The feature extraction unit comprises:

[0206] A timbre feature extraction subunit, used for performing timbre feature encoding on the reference audio according to the timbre encoder to obtain a timbre feature extraction result;

[0207] The content feature extraction subunit is used to extract and represent the speech content features of the reference audio according to the speech content representation model to obtain a content feature extraction result.

[0208] In some modified implementations, the speech content representation model includes a HuBERT model, and correspondingly, the content feature extraction result includes a HuBERT feature.

[0209] In some modified implementations, the feature determination unit includes:

[0210] A first feature determination subunit is used to determine the timbre feature extraction result as a target timbre feature, and to determine a feature sequence composed of the content feature extraction result as a reference speech content feature sequence; or

[0211] The second feature determination subunit is used to query the timbre feature that matches the timbre feature extraction result from the underlying model timbre feature library corresponding to the pre-trained audio codec model as the target timbre feature, and to query the speech content feature sequence that matches the content feature extraction result from the underlying model content feature library corresponding to the pre-trained audio codec model as the reference speech content feature sequence.

[0212] In some modified implementations, the input data further includes a reference text corresponding to the reference audio;

[0213] The device also includes:

[0214] A model fine-tuning module, used to fine-tune the pre-trained audio codec model and speech content feature generation model according to the reference audio and the reference text;

[0215] Accordingly, the content feature sequence generating module 103 includes:

[0216] A content feature sequence generating unit, configured to input the reference speech content feature sequence and the target synthesized text into the fine-tuned speech content feature generating model to generate a target speech content feature sequence;

[0217] The audio synthesis module 104 includes:

[0218] The fine-tuning audio synthesis unit is used to decode the target timbre feature and the target speech content feature sequence according to the fine-tuned audio coding and decoding model to obtain the target synthesized audio.

[0219] In some variations, the device further comprises:

[0220] A training data acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of training samples, each of which includes an audio sample and its corresponding text sample;

[0221] The model training module is used to train the audio codec model and the speech content feature generation model according to the training data set to obtain a pre-trained audio codec model and a speech content feature generation model.

[0222] The speech synthesis device provided in the embodiment of the present application is based on the same inventive concept as the speech synthesis method provided in the aforementioned embodiment of the present application and has the same beneficial effects.

[0223] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the speech synthesis method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech synthesis methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0224] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0225] Figure 5 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0226] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0227] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0228] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0229] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination therewith.

[0230] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0231] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0232] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0233] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0234] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0235] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0236] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0238] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0240] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A speech synthesis method, wherein: The method comprises: Acquire input data, wherein the input data includes reference audio and target synthesized text; Determine a target timbre feature and a reference speech content feature sequence according to the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech; Inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence; A target synthesized audio is generated according to the target timbre feature and the target speech content feature sequence.

2. The speech synthesis method according to claim 1, wherein: The step of determining the target timbre feature and the reference speech content feature sequence according to the reference audio comprises: Performing timbre feature extraction and speech content feature extraction on the reference audio according to a pre-trained audio codec model to obtain a timbre feature extraction result and a content feature extraction result; Determining a target timbre feature according to the timbre feature extraction result, and determining a reference speech content feature sequence according to the content feature extraction result; Accordingly, generating a target synthesized audio according to the target timbre feature and the target speech content feature sequence includes: The target timbre feature and the target speech content feature sequence are decoded according to the pre-trained audio codec model to obtain a target synthesized audio.

3. The speech synthesis method according to claim 2, wherein: The audio codec model includes a timbre encoder and a speech content representation model; The step of performing timbre feature extraction and speech content extraction on the reference audio according to the pre-trained audio codec model to obtain a timbre feature extraction result and a content feature extraction result includes: Performing timbre feature encoding on the reference audio according to the timbre encoder to obtain a timbre feature extraction result; The speech content features of the reference audio are extracted and represented according to the speech content representation model to obtain a content feature extraction result.

4. The speech synthesis method according to claim 3, wherein: The speech content representation model includes a HuBERT model, and correspondingly, the content feature extraction result includes a HuBERT feature.

5. The speech synthesis method according to claim 2, wherein: The step of determining a target timbre feature according to the timbre feature extraction result, and determining a reference speech content feature sequence according to the content feature extraction result, comprises: Determine the timbre feature extraction result as the target timbre feature, and determine the feature sequence composed of the content feature extraction result as the reference speech content feature sequence; or, A timbre feature that matches the timbre feature extraction result is queried from the underlying timbre feature library corresponding to the pre-trained audio codec model as a target timbre feature, and a speech content feature sequence that matches the content feature extraction result is queried from the underlying content feature library corresponding to the pre-trained audio codec model as a reference speech content feature sequence.

6. The speech synthesis method according to claim 2, wherein: The input data also includes a reference text corresponding to the reference audio; The method further comprises: Fine-tuning the pre-trained audio codec model and speech content feature generation model according to the reference audio and the reference text; Accordingly, the step of inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence includes: Inputting the reference speech content feature sequence and the target synthesized text into the fine-tuned speech content feature generation model to generate a target speech content feature sequence; The step of decoding the target timbre feature and the target speech content feature sequence according to the pre-trained audio codec model to obtain the target synthesized audio comprises: The target timbre feature and the target speech content feature sequence are decoded according to the fine-tuned audio codec model to obtain the target synthesized audio.

7. The speech synthesis method according to claim 2, wherein: The method further comprises: Acquire a training data set, wherein the training data set includes a plurality of training samples, each of the training samples includes an audio sample and its corresponding text sample; The audio codec model and the speech content feature generation model are trained according to the training data set to obtain a pre-trained audio codec model and a speech content feature generation model.

8. A speech synthesis device, wherein: The device comprises: An input data acquisition module, used to acquire input data, wherein the input data includes reference audio and target synthesized text; A reference audio processing module, used to determine a target timbre feature and a reference speech content feature sequence according to the reference audio, wherein the reference speech content feature in the reference speech content feature sequence is used to characterize at least one of the audio content, intonation, rhythm and speaking habits of the reference speech; A content feature sequence generation module, used for inputting the reference speech content feature sequence and the target synthesized text into a pre-trained speech content feature generation model to generate a target speech content feature sequence; The audio synthesis module is used to generate a target synthesized audio according to the target timbre feature and the target speech content feature sequence.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images