A method, apparatus, device and medium for training a speech generation model

Abstract

The application belongs to the field of artificial intelligence, and relates to a training method of a speech generation model, comprising the following steps: obtaining reference timbre spectrum, phoneme information and speech spectrum of a target object; training a preset initial speech generation model based on the reference timbre spectrum, the phoneme information and the speech spectrum to obtain model parameters; and adjusting parameters of a multi-timbre feature extraction network, a phoneme feature extraction network, a prosody feature discretization network, a time sequence alignment module, an attention fusion module and a speech reconstruction decoding network of the initial speech generation model based on the model parameters to construct the speech generation model. The application also provides an apparatus, a device and a medium. In addition, the application also relates to blockchain technology, and speech training data and model parameters can be stored in a blockchain. The application can realize decoupling of timbre and prosody information, and flexibly adjust the timbre and prosody information to generate synthesized speech with diversity and flexibility.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and medium for a speech generation model. Background Technology

[0002] In recent years, adaptive speech generation systems have been able to accurately reproduce the characteristics of a target voice, such as pitch, speech rate, and timbre, using only a few minutes of speech data, providing a natural and realistic auditory experience. However, their performance is limited by the quality and quantity of data; insufficient or poor data can lead to harsh and blurry generated audio.

[0003] To break free from these dependencies, the industry has explored new solutions, among which generative models have attracted significant attention in zero-shot speech generation. This system can quickly clone speech with just one voice prompt, eliminating the need for tedious fine-tuning and improving efficiency. However, zero-shot speech generation systems also have drawbacks: the prosodic information and timbre in the prompt are highly coupled and difficult to transmit independently, limiting the diversity and flexibility of the generated speech and making the system inadequate for different scenarios and needs. Summary of the Invention

[0004] The purpose of this application is to propose a training method, apparatus, device, and medium for a speech generation model to solve the problem that the prosodic information and timbre in existing prompt speech are highly coupled, making it difficult to transmit them separately, thus limiting the diversity and flexibility of generated speech.

[0005] To address the aforementioned technical problems, this application provides a method for training a speech generation model, employing the following technical solution:

[0006] Acquire speech training data for the target object, including reference timbre spectrum, phoneme information, and speech spectrum. Based on the reference timbre spectrum, phoneme information, and speech spectrum, train a pre-defined initial speech generation model to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module, and a speech reconstruction decoding network. Based on the model parameters, adjust the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network to construct the speech generation model.

[0007] Furthermore, the step of training a pre-defined initial speech generation model based on the reference timbre spectrum, phoneme information, and speech spectrum to obtain model parameters specifically includes:

[0008] A multi-timbre feature extraction network is employed to encode the reference timbre spectrum, obtaining the first timbre feature encoding information, which serves as a latent representation of the timbre information of the reference timbre spectrum. A phoneme feature extraction network is then used to extract features from the phoneme information, obtaining the first phoneme feature encoding information, which serves as a latent representation of the phoneme sequence corresponding to the phoneme information. A prosodic feature discretization network is used to encode and filter the speech spectrum, obtaining the first prosodic bottleneck representation information, which serves as a latent representation of the prosodic information of the speech spectrum. Finally, a temporal alignment module is used to perform temporal alignment on the speech spectrum, obtaining the various... The first duration information of the phoneme is obtained; an attention fusion module is used to process the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information; a speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data; based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data, the total loss function is calculated; based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, and the speech reconstruction decoding network are trained to obtain the model parameters.

[0009] Furthermore, the step of using a prosodic feature discretization network to encode and filter the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum specifically includes:

[0010] A prosodic feature discretization network is used to discretize and encode the speech spectrum to obtain the prosodic feature encoding information of the speech spectrum. A preset information bottleneck mechanism is used to filter the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0011] Furthermore, the step of employing an attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information specifically includes:

[0012] An attention fusion module is used to convert the first phoneme feature encoding information into a query vector and the first timbre feature encoding information into a key vector and a value vector. The key vector and the query vector are multiplied by a dot product to obtain the attention weight, which is used to characterize the correlation between the phoneme feature and the timbre feature. The value vector and the attention weight are weighted and summed to obtain the fused first speech multidimensional feature information.

[0013] Furthermore, a speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data. Specifically, this includes:

[0014] Upsampling is performed on the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; a speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data.

[0015] Furthermore, the steps for calculating the total loss function based on the baseline speech spectrum and reconstructed speech spectrum of the speech training data specifically include:

[0016] Based on the difference between the baseline speech spectrum and the reconstructed speech spectrum from the speech training data, the reconstruction loss is calculated. The prosodic codebook vector indicated by the prosodic feature encoding information is obtained, as well as the target codebook vector of the reference codebook in the prosodic feature discretization network. The target codebook vector is the codebook vector in the reference codebook whose similarity to the prosodic codebook vector reaches a threshold. The mean square error between the prosodic codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss. A pre-defined discriminator is used to classify the baseline and reconstructed speech spectra to obtain truth probability values. Based on the truth probability values, the adversarial loss is calculated. The reconstruction loss, discretization codebook loss, and adversarial loss are fused to obtain the total loss function.

[0017] Furthermore, after the steps of adjusting the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters to construct the speech generation model, the following steps are also included:

[0018] The process involves: acquiring target timbre information, content information, and target prosodic information for the speech to be synthesized; using a multi-timbre feature extraction network of the speech generation model to encode the target timbre information, obtaining the second timbre feature encoding information; using a phoneme feature extraction network of the speech generation model to extract features from the content information, obtaining the second phoneme feature encoding information for the content information; using a prosodic feature discretization network of the speech generation model to encode and filter the target prosodic information, obtaining the second prosodic bottleneck representation information for the target prosodic information; using a temporal alignment module to temporally align the target prosodic information, obtaining the second duration information for each phoneme of the target prosodic information; using an attention fusion module of the speech generation model to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information, obtaining the fused second speech multidimensional feature information; and using a speech reconstruction decoding network of the speech generation model to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information, and the second duration information to generate the speech to be synthesized.

[0019] To address the aforementioned technical problems, this application also provides a training device for a speech generation model, employing the following technical solution:

[0020] The acquisition module is used to acquire the speech training data of the target object. The speech training data includes the reference timbre spectrum, phoneme information, and speech spectrum.

[0021] The training module is used to train a preset initial speech generation model based on the reference timbre spectrum, phoneme information, and speech spectrum to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module, and a speech reconstruction decoding network.

[0022] The adjustment module is used to adjust the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters, in order to construct a speech generation model.

[0023] To address the aforementioned technical problems, this application also provides a computer device, including a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the training method for the speech generation model described above.

[0024] To address the aforementioned technical problems, embodiments of this application also provide a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the training method for the speech generation model described above.

[0025] Compared with existing technologies, the embodiments of this application have the following main advantages: By acquiring the speech training data of the target object, including the reference timbre spectrum, phoneme information, and speech spectrum, a rich and accurate information foundation is provided for the subsequent training of the preset initial speech generation model. During the initial speech generation model training process, multiple modules such as multi-timbre feature extraction networks, phoneme feature extraction networks, and prosodic feature discretization networks can be used to comprehensively and deeply extract features from the speech training data. For example, by encoding and filtering the speech spectrum through the prosodic feature discretization network, a potential representation of prosodic information is obtained, effectively achieving the initial separation of prosodic information and timbre. Furthermore, the introduction of the temporal alignment module and the attention fusion module further enhances the initial speech generation model's ability to capture and process speech features. The temporal alignment module can accurately calculate the first duration information of each phoneme, providing strong support for accurate speech reconstruction. The attention fusion module, through attention processing of timbre features and phoneme features, achieves deep fusion of the two at the feature level, thereby improving the naturalness and clarity of the generated speech. Finally, the initial speech generation model was trained based on the total loss function, constructing a speech generation model that includes multiple modules such as a multi-timbre feature extraction network, a phoneme feature extraction network, and a prosodic feature discretization network. This speech generation model can not only achieve efficient speech cloning, but also flexibly adjust timbre and prosodic information to generate diverse and flexible speech outputs when facing different scenarios and needs. Attached Figure Description

[0026] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0028] Figure 2 This is a flowchart illustrating a training method for a speech generation model provided in this application;

[0029] Figure 3 This is a schematic diagram of the structure of a training device for a speech generation model provided in this application;

[0030] Figure 4 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0035] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0036] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers.

[0037] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0038] It should be noted that the training method of the speech generation model provided in this application embodiment is generally executed by the server, and correspondingly, the training device of the speech generation model is generally set in the server.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a training method for a speech generation model according to this application. The training method for the speech generation model includes the following steps:

[0041] Step S201: Obtain the speech training data of the target object. The speech training data includes the reference timbre spectrum, phoneme information, and speech spectrum.

[0042] In this embodiment, the training method for the speech generation model runs on an electronic device (e.g., Figure 1 The server shown can acquire the target's voice training data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods.

[0043] The target object refers to the individual whose speech features will be used to train the initial speech generation model to generate speech with specific attributes of the target object. For example, the target object can be a person or an animal.

[0044] The speech training data refers to a dataset containing the speech features of the target object, including reference timbre spectra, phoneme information, and speech spectra. It is used to train the initial speech generation model so that it can mimic the speech of the target object.

[0045] The reference timbre spectrum refers to the spectral data used to extract timbre features, such as other mel spectra. It reflects the unique properties of timbre in speech and serves as a timbre template for training the initial speech generation model.

[0046] Phoneme information refers to the information describing each phoneme (the basic unit of speech) in speech, and it is an important input for speech synthesis. It is used to guide the initial speech generation model to generate accurate pronunciation.

[0047] The speech spectrum refers to the spectral representation of a speech signal, such as the target mel spectrum. It contains the frequency domain features of speech and is the foundation for model training and speech reconstruction.

[0048] Step S202: Based on the reference timbre spectrum, phoneme information, and speech spectrum, the preset initial speech generation model is trained to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module, and a speech reconstruction decoding network.

[0049] The initial speech generation model refers to the untrained speech generation network framework, which includes multiple components such as a multi-timbre feature extraction network and a phoneme feature extraction network, providing the basic structure for subsequent training.

[0050] The model parameters refer to a set of values ​​that are continuously adjusted and optimized during training. They determine the output characteristics of the speech generation model, including the weights and biases of each network layer.

[0051] Among them, the multi-timbre feature extraction network refers to a network specifically designed to extract timbre features from the reference timbre spectrum. It can learn the latent representation of timbre for subsequent speech generation.

[0052] Among them, the phoneme feature extraction network is used to process phoneme information and extract phoneme features. It can convert phoneme information into a potential representation that the model can understand.

[0053] Among them, the prosodic feature discretization network refers to a network that encodes and filters the speech spectrum to extract prosodic features. Its output is a latent representation of prosodic information, which helps the initial speech generation model generate natural speech rhythm.

[0054] The timing alignment module is used to align each phoneme in the speech spectrum with its position on the time axis, and outputs the duration information of each phoneme to ensure the timing accuracy of speech generation.

[0055] The attention fusion module combines timbre features and phoneme features, and uses an attention mechanism to fuse information to generate multi-dimensional feature information containing rich speech features.

[0056] Among them, the speech reconstruction decoding network is used to decode the fused multidimensional feature information, phoneme features, prosodic features and time sequence information into the reconstructed speech spectrum.

[0057] Step S203: Based on the model parameters, adjust the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network respectively to construct a speech generation model.

[0058] The parameters of the multi-tone feature extraction network are the numerical values ​​that determine the output characteristics of the network. They can be optimized through training.

[0059] The parameters of the phoneme feature extraction network refer to a set of values ​​that affect its performance. These parameters can be adjusted through training to extract phoneme features more accurately.

[0060] The parameters of the prosodic feature discretization network refer to the parameters that control the prosodic feature extraction and discretization process. These parameters influence the initial speech generation model's understanding and representation of prosodic information.

[0061] The parameters of the timing alignment module are a set of values ​​that determine the accuracy and efficiency of timing alignment. These parameters can be optimized through training to improve the timing consistency of speech generation.

[0062] The parameters of the attention fusion module refer to those that affect the effect of the attention mechanism in fusing timbre and phoneme features. These parameters can be adjusted through training to optimize the fusion results.

[0063] The parameters of the speech reconstruction and decoding network refer to a set of values ​​that control the speech reconstruction and decoding process. These parameters can be optimized through training to improve the quality of speech generation.

[0064] Among them, a speech generation model refers to a model that, after being trained, can generate speech output with the speech characteristics of the target object based on the input feature information.

[0065] This application embodiment can acquire speech training data of the target object, including reference timbre spectrum, phoneme information, and speech spectrum, providing a rich and accurate information foundation for the subsequent training of a pre-set initial speech generation model. During the initial speech generation model training process, multiple modules such as a multi-timbre feature extraction network, a phoneme feature extraction network, and a prosodic feature discretization network can be used to comprehensively and deeply extract features from the speech training data. For example, by encoding and filtering the speech spectrum through a prosodic feature discretization network, a potential representation of prosodic information is obtained, effectively achieving the initial separation of prosodic information from timbre. Furthermore, the introduction of a temporal alignment module and an attention fusion module further enhances the initial speech generation model's ability to capture and process speech features. The temporal alignment module can accurately calculate the first duration information of each phoneme, providing strong support for accurate speech reconstruction. The attention fusion module, through attention processing of timbre features and phoneme features, achieves deep fusion of the two at the feature level, thereby improving the naturalness and clarity of the generated speech. Finally, the initial speech generation model was trained based on the total loss function, constructing a speech generation model that includes multiple modules such as a multi-timbre feature extraction network, a phoneme feature extraction network, and a prosodic feature discretization network. This speech generation model can not only achieve efficient speech cloning, but also flexibly adjust timbre and prosodic information to generate diverse and flexible speech outputs when facing different scenarios and needs.

[0066] In some optional implementations of this embodiment, step 202, training a preset initial speech generation model based on the reference timbre spectrum, phoneme information, and speech spectrum to obtain model parameters, specifically includes the following steps:

[0067] A multi-timbre feature extraction network is employed to encode the reference timbre spectrum, obtaining the first timbre feature encoding information, which serves as a latent representation of the timbre information of the reference timbre spectrum. A phoneme feature extraction network is then used to extract features from the phoneme information, obtaining the first phoneme feature encoding information, which serves as a latent representation of the phoneme sequence corresponding to the phoneme information. A prosodic feature discretization network is used to encode and filter the speech spectrum, obtaining the first prosodic bottleneck representation information, which serves as a latent representation of the prosodic information of the speech spectrum. Finally, a temporal alignment module is used to perform temporal alignment on the speech spectrum, obtaining the various... The first duration information of the phoneme is obtained; an attention fusion module is used to process the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information; a speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data; based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data, the total loss function is calculated; based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, and the speech reconstruction decoding network are trained to obtain the model parameters.

[0068] The encoding process refers to the process of converting the reference timbre spectrum into a latent representation that the model can process. For example, feature extraction and dimensionality reduction are performed on the reference timbre spectrum to obtain the first timbre feature encoding information.

[0069] Timbre information refers to the characteristics that describe the timbre in speech, including the brightness and fullness of the timbre.

[0070] The first timbre feature encoding information refers to the potential representation of the timbre features output by the multi-timbre feature extraction network, which is used in the subsequent speech generation process.

[0071] Feature extraction refers to the process of extracting phoneme features from phoneme information.

[0072] Among them, the first phoneme feature encoding information refers to the potential representation of the phoneme features output by the phoneme feature extraction network, which is used to guide the speech generation model to pronounce accurately.

[0073] Among them, the phoneme sequence refers to the arrangement order of a series of phonemes in speech, which is the basis of speech.

[0074] Among them, the encoding and filtering process refers to the process by which the prosodic feature discretization network encodes the input speech spectrum, filters redundant information, and extracts key features.

[0075] Among them, the first prosodic bottleneck representation information refers to the potential representation of prosodic features output by the prosodic feature discretization network, which is the key to speech rhythm and naturalness.

[0076] Among them, prosodic information refers to information describing the rhythm, intonation and other features of speech, which is an important way for speech to express emotions and content.

[0077] Temporal alignment refers to the process of mapping each phoneme in speech to its position on the timeline, ensuring the temporal accuracy of speech generation.

[0078] Among them, the first duration information refers to the duration of each phoneme output by the timing alignment module, which is an important basis for the speech generation model to control the pronunciation speed.

[0079] Attention processing refers to the process of weighted fusion of multiple feature information through an attention mechanism to improve the overall performance of the speech generation model.

[0080] Among them, the first speech multidimensional feature information refers to the fusion result of the attention fusion module, which includes multidimensional feature information such as timbre, phonemes, and prosody.

[0081] Decoding refers to the process of converting the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information into an understandable output (such as reconstructing the speech spectrum).

[0082] Among them, the reconstructed speech spectrum refers to the speech spectrum output by the speech reconstruction decoding network. It is the final output of the speech generation model and reflects the speech characteristics of the target object.

[0083] The total loss function measures the difference between the initial output of the speech generation model and the expected output. It guides the optimization of model parameters to achieve more accurate speech generation.

[0084] In one example, in a financial customer service system, to improve customer service quality, it is often necessary to convert text information into speech with specific timbre and customized prosodic features. The technical solution proposed in this embodiment aims to train a model capable of accurately simulating and generating the speech of a target audience. First, speech training data of the target audience is obtained from a financial customer service recording library. This data includes a reference timbre spectrum, phoneme information (such as pinyin or phonetic symbols), and a speech spectrum. This data is preprocessed to ensure it matches the context of financial customer service, such as removing noise and standardizing volume. Next, a multi-timbre feature extraction network is used to deeply encode the reference timbre spectrum, generating first timbre feature encoding information, which serves as a potential expression of the target timbre features. Simultaneously, a phoneme feature extraction network extracts the first phoneme feature encoding of the phoneme information, reflecting the potential structure of the phoneme sequence. Further, a prosodic feature discretization network encodes and filters the speech spectrum, extracting first prosodic bottleneck representation information. This information captures prosodic features such as rhythm and intonation in the speech, which is crucial for simulating the professional and fluent tone of financial customer service. The temporal alignment module is responsible for accurately aligning the speech spectrum with the phoneme sequence, obtaining the first duration information of each phoneme, which is crucial for maintaining the natural fluency of the speech. The attention fusion module combines the first timbre feature encoding information and the first phoneme feature encoding information, enhancing the expression of key information through an attention mechanism to form the fused first multi-dimensional speech feature information. Finally, the speech reconstruction decoding network receives the above multi-dimensional feature information, the first phoneme feature encoding, the first prosodic bottleneck representation, and the first duration information, and decodes to generate the reconstructed speech spectrum. By comparing the baseline speech spectrum with the reconstructed speech spectrum, the total loss function is calculated, and each network module is trained accordingly until the model parameters are optimized to the best state. In financial customer service applications, this model can generate highly realistic voice responses based on the input text and specified timbre and prosodic requirements, significantly improving customer experience while reducing the pressure on human customer service and improving service efficiency.

[0085] This application embodiment utilizes deep neural networks to extract and fuse multi-dimensional features from the target object's speech training data, significantly improving the accuracy and naturalness of speech synthesis. A multi-timbre feature extraction network accurately captures the latent features of the reference timbre spectrum, endowing the synthesized speech with a highly personalized timbre. A phoneme feature extraction network and a prosodic feature discretization network respectively analyze the phoneme sequence and speech prosody, ensuring accurate pronunciation and fluent rhythm in the synthesized speech. A temporal alignment module accurately calculates the duration of each phoneme, maintaining the temporal structure of the speech. An attention fusion module cleverly integrates timbre and phoneme features, enhancing the correlation between speech features, enabling the synthesized speech to accurately convey semantic information while maintaining a personalized timbre. A speech reconstruction decoding network integrates multi-dimensional feature information, efficiently decodes and generates a reconstructed speech spectrum, compares it with the baseline speech spectrum to calculate the loss, and provides feedback to optimize model parameters, forming a closed-loop training system.

[0086] In some optional implementations of this embodiment, the step "using a prosodic feature discretization network to encode and filter the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum" specifically includes the following steps:

[0087] A prosodic feature discretization network is used to discretize and encode the speech spectrum to obtain the prosodic feature encoding information of the speech spectrum. A preset information bottleneck mechanism is used to filter the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0088] In one example, this embodiment can be described in detail by combining it with an intelligent customer service system application in the financial field. First, the speech spectrum is extracted from the recorded data of financial customer service and used as input for subsequent processing. The speech spectrum is the frequency domain representation of the speech signal and contains rich information about the speech. Next, the prosodic feature discretization network of this invention is used to discretize and encode the speech spectrum. This network consists of multiple convolutional and pooling layers, which can extract prosodic features from the speech spectrum layer by layer and convert them into discrete encoded information. This encoded information is represented in the form of high-dimensional vectors, each vector corresponding to a segment of prosodic features in the speech spectrum. However, directly using these encoded information for speech synthesis may lead to overfitting problems because the encoded information may contain redundant information that is not important for speech synthesis. To solve this problem, a preset information bottleneck mechanism is introduced. This mechanism filters the prosodic feature encoded information by setting a bottleneck layer, removing timbre information in the audio and retaining only the information most critical to speech synthesis. Specifically, the bottleneck layer uses a low-dimensional fully connected layer to compress the high-dimensional prosodic feature encoded information into low-dimensional first prosodic bottleneck representation information. In this way, key prosodic features in the speech spectrum are preserved while effectively avoiding overfitting.

[0089] This application embodiment utilizes a prosodic feature discretization network to perform fine-grained discretization encoding of the speech spectrum. This process effectively transforms the continuous speech spectrum into highly representative prosodic feature encoding information, thereby accurately capturing and preserving key prosodic features such as rhythm and intonation in speech. Furthermore, a pre-defined information bottleneck mechanism plays a crucial role in processing the prosodic feature encoding information. Like a precise filter, it rigorously filters out redundant and noisy information, retaining only the core representations most critical to speech prosody—the first prosodic bottleneck representation information. This processing not only significantly reduces the data dimensionality but also significantly enhances the model's generalization ability, effectively avoiding overfitting.

[0090] In some optional implementations of this embodiment, the step "using an attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information" specifically includes the following steps:

[0091] An attention fusion module is used to convert the first phoneme feature encoding information into a query vector and the first timbre feature encoding information into a key vector and a value vector. The key vector and the query vector are multiplied by a dot product to obtain the attention weight, which is used to characterize the correlation between the phoneme feature and the timbre feature. The value vector and the attention weight are weighted and summed to obtain the fused first speech multidimensional feature information.

[0092] In one example, this embodiment can be explained in detail by combining it with an intelligent customer service system application in the financial field. First, the first timbre feature encoding information corresponding to the timbre spectrum and the first phoneme feature encoding information corresponding to the phoneme information are referenced from the financial customer service recording library. The first timbre feature encoding information represents the unique timbre of the customer service personnel, and the first phoneme feature encoding information reflects the phoneme sequence and its attributes in the speech. Next, an attention fusion module is used to fuse these two types of feature encoding information. Specifically, the first phoneme feature encoding information is converted into a query vector, and the first timbre feature encoding information is converted into a key vector and a value vector. This conversion process can be implemented through a fully connected layer or a linear transformation to ensure that the feature encoding information matches the requirements of the attention mechanism. Then, a dot product operation is performed on the key vector and the query vector to obtain the attention weight. The attention weight is a scalar value used to characterize the correlation between phoneme features and timbre features. Through the attention mechanism, the initial speech generation model can automatically learn and focus on the phoneme and timbre feature combinations that are most critical to synthesizing personalized speech. Finally, the value vector and the attention weight are weighted and summed to obtain the fused first speech multidimensional feature information. This information not only contains accurate phoneme sequence information, but also incorporates the unique timbre characteristics of customer service personnel, thus realizing the synthesis of personalized voice.

[0093] This application's embodiments achieve deep interaction and fusion of phoneme feature encoding information and timbre feature encoding information by introducing an attention fusion module. Specifically, this module cleverly transforms the first phoneme feature encoding information into a query vector, while the first timbre feature encoding information is decomposed into a key vector and a value vector. Through the dot product operation of the key vector and the query vector, the attention weight is accurately calculated. This weight precisely reflects the intrinsic connection and correlation between phoneme features and timbre features. Furthermore, by using these attention weights to perform a weighted summation of the value vector, the multidimensional features of phonemes and timbres are effectively fused, generating richer and more accurate first speech multidimensional feature information.

[0094] In some optional implementations of this embodiment, the step "using a speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data" specifically includes the following steps:

[0095] Upsampling is performed on the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; a speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data.

[0096] In one example, this embodiment can be elaborated in detail by combining it with the specific application of an intelligent voice customer service system in the financial field. First, upsampling technology is used to expand the first speech multidimensional feature information and the first phoneme feature encoding information in both time and space dimensions, thereby obtaining an enhanced feature set containing richer details. This step effectively improves the input quality of the subsequent speech reconstruction decoding network. Next, the speech reconstruction decoding network can be used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information. Among them, the first prosodic bottleneck representation information reflects the prosodic features of speech such as intonation and rhythm, while the first duration information provides the duration of each phoneme. These pieces of information together constitute a comprehensive representation of the speech signal. The speech reconstruction decoding network uses deep learning models, such as convolutional neural networks, recurrent neural networks, or their variants (such as long short-term memory networks, gated recurrent units), to achieve complex nonlinear transformation and fusion of input features. Through the layer-by-layer processing and feature fusion of the speech reconstruction decoding network, the reconstructed speech spectrum is finally generated.

[0097] This application embodiment significantly enhances the richness and detail of the feature set by upsampling the first speech multidimensional feature information and the first phoneme feature encoding information, providing a solid foundation for subsequent speech reconstruction. Based on this, an advanced speech reconstruction decoding network is employed to cleverly fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information. This process not only achieves comprehensive integration of speech features but also accurately captures the prosodic features and duration information of the speech signal, thereby successfully reconstructing a high-quality speech spectrum.

[0098] In some optional implementations of this embodiment, the step "calculating the total loss function based on the baseline speech spectrum and reconstructed speech spectrum of the speech training data" specifically includes the following steps:

[0099] Based on the difference between the baseline speech spectrum and the reconstructed speech spectrum from the speech training data, the reconstruction loss is calculated. The prosodic codebook vector indicated by the prosodic feature encoding information is obtained, as well as the target codebook vector of the reference codebook in the prosodic feature discretization network. The target codebook vector is the codebook vector in the reference codebook whose similarity to the prosodic codebook vector reaches a threshold. The mean square error between the prosodic codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss. A pre-defined discriminator is used to classify the baseline and reconstructed speech spectra to obtain truth probability values. Based on the truth probability values, the adversarial loss is calculated. The reconstruction loss, discretization codebook loss, and adversarial loss are fused to obtain the total loss function.

[0100] In one example, the technical solution proposed in this embodiment aims to calculate the mean square error between the prosodic codebook vector indicated by the prosodic feature encoding information and the target codebook vector by calculating the difference between the reference speech spectrum and the reconstructed speech spectrum, and to calculate the authenticity probability value obtained by classifying the reference speech spectrum and the reconstructed speech spectrum using a discriminator, thereby calculating the reconstruction loss, discretization codebook loss, and adversarial loss respectively, and then fusing these three to obtain the total loss function. Specifically, the reference speech spectrum of the speech training data can be extracted, and the reconstructed speech spectrum can be generated based on the proposed speech reconstruction decoding network. Then, the difference between the reference speech spectrum and the reconstructed speech spectrum is calculated to obtain the reconstruction loss. This step measures the error between the reconstructed speech spectrum and the real speech spectrum. Next, the prosodic codebook vector indicated by the prosodic feature encoding information can be obtained. At the same time, from the reference codebook of the prosodic feature discretization network, the codebook vector most similar to the prosodic codebook vector is found as the target codebook vector. Then, the mean square error between the prosodic codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss. This step aims to optimize the discretization representation of prosodic features and improve the naturalness and accuracy of speech synthesis. Then, a pre-defined discriminator is used to classify the baseline speech spectrum and the reconstructed speech spectrum to obtain authenticity probability values. Adversarial loss is calculated based on these authenticity probability values. This step, by introducing an adversarial training mechanism, further improves the authenticity of the reconstructed speech spectrum and its consistency with the baseline speech spectrum. Finally, the reconstruction loss, discretization codebook loss, and adversarial loss are fused to obtain the total loss function.

[0101] This application's embodiments effectively measure the accuracy of speech reconstruction by precisely calculating the difference between the baseline speech spectrum and the reconstructed speech spectrum, thereby guiding model optimization through reconstruction loss. Simultaneously, by matching prosodic feature encoding information with the reference codebook of the discretized network, the mean square error between the prosodic codebook vector and the target codebook vector is calculated, and the resulting discretized codebook loss significantly improves the accuracy of prosodic features. Furthermore, a pre-defined discriminator is introduced to classify the baseline and reconstructed speech spectra, and an adversarial loss calculated based on the realism probability value further enhances the realism and naturalness of the reconstructed speech. By fusing the reconstruction loss, discretized codebook loss, and adversarial loss, the resulting total loss function comprehensively guides the optimization direction of the speech training data, not only improving the accuracy of speech reconstruction and the expressiveness of prosodic features, but also significantly enhancing the realism and natural fluency of the reconstructed speech, providing strong technical support for applications such as speech synthesis and speech recognition.

[0102] In some optional implementations of this embodiment, after adjusting the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters in step S203 to construct the speech generation model, the following steps are further included:

[0103] The process involves: acquiring target timbre information, content information, and target prosodic information for the speech to be synthesized; using a multi-timbre feature extraction network of the speech generation model to encode the target timbre information, obtaining the second timbre feature encoding information; using a phoneme feature extraction network of the speech generation model to extract features from the content information, obtaining the second phoneme feature encoding information for the content information; using a prosodic feature discretization network of the speech generation model to encode and filter the target prosodic information, obtaining the second prosodic bottleneck representation information for the target prosodic information; using a temporal alignment module to temporally align the target prosodic information, obtaining the second duration information for each phoneme of the target prosodic information; using an attention fusion module of the speech generation model to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information, obtaining the fused second speech multidimensional feature information; and using a speech reconstruction decoding network of the speech generation model to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information, and the second duration information to generate the speech to be synthesized.

[0104] In one example, this embodiment can be illustrated in detail using a real-world case from the financial sector. First, target voice information (e.g., voice samples), content information (text content to be synthesized), and target prosodic information (preset speech rate and intonation information) of the target customer service representative can be obtained from a database. Then, the multi-voice feature extraction network of the speech generation model proposed in this embodiment is used to encode the target voice information, obtaining second voice feature encoding information. Simultaneously, the phoneme feature extraction network of the speech generation model is used to extract features from the text content, obtaining second phoneme feature encoding information for the text content. Furthermore, the prosodic feature discretization network of the speech generation model is used to encode and filter the preset speech rate and intonation information, obtaining second prosodic bottleneck representation information. Next, the speech rate and intonation information are temporally aligned using the temporal alignment module of the speech generation model, obtaining second duration information for each phoneme. Then, the attention fusion module of the speech generation model performs attention processing on the voice feature encoding and phoneme feature encoding, obtaining fused second multi-dimensional speech feature information. Finally, the speech reconstruction decoding network of the speech generation model is used to decode the multidimensional feature information, phoneme feature encoding, prosodic bottleneck representation information and duration information of the fused speech, so as to generate the speech to be synthesized with target timbre, accurate content and natural prosody.

[0105] This application's embodiments achieve high-quality speech synthesis by accurately capturing the target timbre, content information, and target prosodic information of the speech to be synthesized, and cleverly utilizing multiple network modules in the speech generation model. The multi-timbre feature extraction network deeply encodes the target timbre, ensuring high-fidelity timbre reproduction; the phoneme feature extraction network accurately extracts phoneme features from the content information, providing an accurate content foundation for speech synthesis. The prosodic feature discretization network effectively extracts the bottleneck representation of the target prosodic through encoding filtering, giving the speech a natural sense of rhythm. The temporal alignment module ensures accurate synchronization between prosodic and content information, while the attention fusion module further integrates timbre and phoneme features, enhancing the naturalness and expressiveness of the speech. Finally, the speech reconstruction decoding network integrates multi-dimensional feature information, phoneme feature encoding, prosodic bottleneck representation, and duration information, successfully generating a synthesized speech that conforms to the target timbre, accurately conveys the content, and has a natural prosodic rhythm, significantly improving the overall quality of speech synthesis and user experience.

[0106] It should be emphasized that, to further ensure the privacy and security of the aforementioned voice training data and model parameters, the aforementioned voice training data and model parameters can also be stored in a blockchain node.

[0107] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0108] The embodiments of this application can construct and optimize related models and networks based on artificial intelligence technology, such as multi-timbre feature extraction networks, phoneme feature extraction networks, prosodic feature discretization networks, temporal alignment modules, attention fusion modules, and speech reconstruction decoding networks. Artificial intelligence (AI) models are the culmination of theory and practice in simulating human intelligent decision-making processes through algorithms and data analysis to solve complex problems, predict future trends, or automate tasks. These models utilize large amounts of historical data and real-time information, trained and optimized through specific algorithmic frameworks to achieve efficient, accurate, and reliable performance.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0110] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0111] Further reference Figure 3 As a response to the above Figure 2The implementation of the method shown in this application provides an embodiment of a training device for a speech generation model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0112] like Figure 3 As shown, the training device 400 for the speech generation model in this embodiment includes: an acquisition module 401, a training module 402, and an adjustment module 403. Wherein:

[0113] The acquisition module 401 is used to acquire the speech training data of the target object. The speech training data includes the reference timbre spectrum, phoneme information and speech spectrum.

[0114] Training module 402 is used to train a preset initial speech generation model based on reference timbre spectrum, phoneme information and speech spectrum to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module and a speech reconstruction decoding network.

[0115] The adjustment module 403 is used to adjust the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters, in order to construct a speech generation model.

[0116] This application embodiment can obtain the speech training data of the target object, including reference timbre spectrum, phoneme information, and speech spectrum, to provide a precise information foundation for a preset initial speech generation model. During training, network modules such as multi-timbre feature extraction, phoneme feature extraction, and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network achieves initial separation of prosody and timbre. The temporal alignment module accurately calculates the duration of phonemes, supporting accurate speech reconstruction. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of the generated speech. Based on training with the total loss function, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning and flexibly adjust timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0117] In one embodiment, the training module 402 includes:

[0118] The encoding submodule is used to encode the reference timbre spectrum using a multi-timbre feature extraction network to obtain the first timbre feature encoding information of the reference timbre spectrum. The first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum.

[0119] The extraction submodule is used to extract features from phoneme information using a phoneme feature extraction network to obtain the first phoneme feature encoding information of the phoneme information. The first phoneme feature encoding information is the potential representation of the phoneme sequence corresponding to the phoneme information.

[0120] The filtering submodule is used to encode and filter the speech spectrum using a prosodic feature discretization network to obtain the first prosodic bottleneck representation information of the speech spectrum. The first prosodic bottleneck representation information is the potential representation of the prosodic information of the speech spectrum.

[0121] The alignment submodule is used to perform temporal alignment on the speech spectrum using the timing alignment module to obtain the first duration information of each phoneme in the speech spectrum;

[0122] The attention submodule is used to perform attention processing on the first timbre feature coding information and the first phoneme feature coding information using the attention fusion module to obtain the fused first speech multidimensional feature information.

[0123] The decoding submodule is used to use a speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information and the first duration information to obtain the reconstructed speech spectrum of the speech training data.

[0124] The training submodule is used to calculate the total loss function based on the baseline speech spectrum and reconstructed speech spectrum of the speech training data. Based on the loss function, the multi-timbre feature extraction network, phoneme feature extraction network, prosodic feature discretization network, temporal alignment module, and speech reconstruction decoding network are trained to obtain model parameters.

[0125] This application embodiment utilizes deep neural networks to extract and fuse multi-dimensional features from the target object's speech training data, significantly improving the accuracy and naturalness of speech synthesis. A multi-timbre feature extraction network accurately captures the latent features of the reference timbre spectrum, endowing the synthesized speech with a highly personalized timbre. A phoneme feature extraction network and a prosodic feature discretization network respectively analyze the phoneme sequence and speech prosody, ensuring accurate pronunciation and fluent rhythm in the synthesized speech. A temporal alignment module accurately calculates the duration of each phoneme, maintaining the temporal structure of the speech. An attention fusion module cleverly integrates timbre and phoneme features, enhancing the correlation between speech features, enabling the synthesized speech to accurately convey semantic information while maintaining a personalized timbre. A speech reconstruction decoding network integrates multi-dimensional feature information, efficiently decodes and generates a reconstructed speech spectrum, compares it with the baseline speech spectrum to calculate the loss, and provides feedback to optimize model parameters, forming a closed-loop training system.

[0126] In one embodiment, the filtering submodule is further configured to use a prosodic feature discretization network to discretize and encode the speech spectrum to obtain prosodic feature encoding information of the speech spectrum; and to use a preset information bottleneck mechanism to perform bottleneck filtering on the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0127] This application embodiment utilizes a prosodic feature discretization network to perform fine-grained discretization encoding of the speech spectrum. This process effectively transforms the continuous speech spectrum into highly representative prosodic feature encoding information, thereby accurately capturing and preserving key prosodic features such as rhythm and intonation in speech. Furthermore, a pre-defined information bottleneck mechanism plays a crucial role in processing the prosodic feature encoding information. Like a precise filter, it rigorously filters out redundant and noisy information, retaining only the core representations most critical to speech prosody—the first prosodic bottleneck representation information. This processing not only significantly reduces the data dimensionality but also significantly enhances the model's generalization ability, effectively avoiding overfitting.

[0128] In one embodiment, the attention submodule is further configured to employ an attention fusion module to convert the first phoneme feature encoding information into a query vector and the first timbre feature encoding information into a key vector and a value vector; perform a dot product operation on the key vector and the query vector to obtain attention weights, which are used to characterize the correlation between phoneme features and timbre features; and perform a weighted summation of the value vector and the attention weights to obtain the fused first speech multidimensional feature information.

[0129] This application's embodiments achieve deep interaction and fusion of phoneme feature encoding information and timbre feature encoding information by introducing an attention fusion module. Specifically, this module cleverly transforms the first phoneme feature encoding information into a query vector, while the first timbre feature encoding information is decomposed into a key vector and a value vector. Through the dot product operation of the key vector and the query vector, the attention weight is accurately calculated. This weight precisely reflects the intrinsic connection and correlation between phoneme features and timbre features. Furthermore, by using these attention weights to perform a weighted summation of the value vector, the multidimensional features of phonemes and timbres are effectively fused, generating richer and more accurate first speech multidimensional feature information.

[0130] In one embodiment, the decoding submodule is further configured to perform upsampling processing on the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; and to use a speech reconstruction decoding network to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data.

[0131] This application embodiment significantly enhances the richness and detail of the feature set by upsampling the first speech multidimensional feature information and the first phoneme feature encoding information, providing a solid foundation for subsequent speech reconstruction. Based on this, an advanced speech reconstruction decoding network is employed to cleverly fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information. This process not only achieves comprehensive integration of speech features but also accurately captures the prosodic features and duration information of the speech signal, thereby successfully reconstructing a high-quality speech spectrum.

[0132] In one embodiment, the training submodule is further configured to: calculate the reconstruction loss based on the difference between the baseline speech spectrum and the reconstructed speech spectrum of the speech training data; obtain the prosodic codebook vector indicated by the prosodic feature encoding information, and obtain the target codebook vector of the reference codebook of the prosodic feature discretization network, wherein the target codebook vector is the codebook vector in the reference codebook whose similarity to the prosodic codebook vector reaches a threshold; calculate the mean square error between the prosodic codebook vector and the target codebook vector to obtain the discretization codebook loss; classify the baseline speech spectrum and the reconstructed speech spectrum using a preset discriminator to obtain a truth probability value; calculate the adversarial loss based on the truth probability value; and fuse the reconstruction loss, the discretization codebook loss, and the adversarial loss to obtain the total loss function.

[0133] This application's embodiments effectively measure the accuracy of speech reconstruction by precisely calculating the difference between the baseline speech spectrum and the reconstructed speech spectrum, thereby guiding model optimization through reconstruction loss. Simultaneously, by matching prosodic feature encoding information with the reference codebook of the discretized network, the mean square error between the prosodic codebook vector and the target codebook vector is calculated, and the resulting discretized codebook loss significantly improves the accuracy of prosodic features. Furthermore, a pre-defined discriminator is introduced to classify the baseline and reconstructed speech spectra, and an adversarial loss calculated based on the realism probability value further enhances the realism and naturalness of the reconstructed speech. By fusing the reconstruction loss, discretized codebook loss, and adversarial loss, the resulting total loss function comprehensively guides the optimization direction of the speech training data, not only improving the accuracy of speech reconstruction and the expressiveness of prosodic features, but also significantly enhancing the realism and natural fluency of the reconstructed speech, providing strong technical support for applications such as speech synthesis and speech recognition.

[0134] In one embodiment, the training device 400 for the speech generation model further includes:

[0135] The information acquisition module is used to acquire the target timbre information, content information, and target prosody information of the speech to be synthesized.

[0136] The encoding module is used to encode the target timbre information using a multi-timbre feature extraction network of the speech generation model, so as to obtain the second timbre feature encoding information of the target timbre information;

[0137] The extraction module is used to extract features from the content information using a phoneme feature extraction network of the speech generation model, and obtain the second phoneme feature encoding information of the content information.

[0138] The filtering module is used to encode and filter the target prosodic information using a prosodic feature discretization network of the speech generation model to obtain the second prosodic bottleneck representation information of the target prosodic information.

[0139] The alignment module is used to perform temporal alignment on the target prosodic information using the temporal alignment module, so as to obtain the second duration information of each phoneme of the target prosodic information.

[0140] The attention module is used to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information in the speech generation model to obtain the fused second speech multidimensional feature information.

[0141] The decoding module is used in a speech reconstruction decoding network employing a speech generation model to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information, and the second duration information to generate the speech to be synthesized.

[0142] This application's embodiments achieve high-quality speech synthesis by accurately capturing the target timbre, content information, and target prosodic information of the speech to be synthesized, and cleverly utilizing multiple network modules in the speech generation model. The multi-timbre feature extraction network deeply encodes the target timbre, ensuring high-fidelity timbre reproduction; the phoneme feature extraction network accurately extracts phoneme features from the content information, providing an accurate content foundation for speech synthesis. The prosodic feature discretization network effectively extracts the bottleneck representation of the target prosodic through encoding filtering, giving the speech a natural sense of rhythm. The temporal alignment module ensures accurate synchronization between prosodic and content information, while the attention fusion module further integrates timbre and phoneme features, enhancing the naturalness and expressiveness of the speech. Finally, the speech reconstruction decoding network integrates multi-dimensional feature information, phoneme feature encoding, prosodic bottleneck representation, and duration information, successfully generating a synthesized speech that conforms to the target timbre, accurately conveys the content, and has a natural prosodic rhythm, significantly improving the overall quality of speech synthesis and user experience.

[0143] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0144] Computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0145] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0146] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for training methods of speech generation models. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or will be output.

[0147] In some embodiments, processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or to process data, such as computer-readable instructions for running a training method for a speech generation model.

[0148] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] This application embodiment can obtain the speech training data of the target object, including reference timbre spectrum, phoneme information, and speech spectrum, to provide a precise information foundation for a preset initial speech generation model. During training, network modules such as multi-timbre feature extraction, phoneme feature extraction, and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network achieves initial separation of prosody and timbre. The temporal alignment module accurately calculates the duration of phonemes, supporting accurate speech reconstruction. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of the generated speech. Based on training with the total loss function, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning and flexibly adjust timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0150] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the training method for the speech generation model as described above.

[0151] This application embodiment can obtain the speech training data of the target object, including reference timbre spectrum, phoneme information, and speech spectrum, to provide a precise information foundation for a preset initial speech generation model. During training, network modules such as multi-timbre feature extraction, phoneme feature extraction, and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network achieves initial separation of prosody and timbre. The temporal alignment module accurately calculates the duration of phonemes, supporting accurate speech reconstruction. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of the generated speech. Based on training with the total loss function, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning and flexibly adjust timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0153] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0154] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A training method for a speech generation model, characterized in that, Includes the following steps: Acquire speech training data of the target object, wherein the speech training data includes reference timbre spectrum, phoneme information and speech spectrum; Based on the reference timbre spectrum, the phoneme information, and the speech spectrum, a preset initial speech generation model is trained to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module, and a speech reconstruction decoding network. Based on the model parameters, the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network are adjusted respectively to construct a speech generation model. The step of training a preset initial speech generation model based on the reference timbre spectrum, the phoneme information, and the speech spectrum to obtain model parameters includes: The reference timbre spectrum is encoded using the multi-timbre feature extraction network to obtain the first timbre feature encoding information of the reference timbre spectrum, and the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum; The phoneme feature extraction network is used to extract features from the phoneme information to obtain the first phoneme feature encoding information of the phoneme information. The first phoneme feature encoding information is a potential representation of the phoneme sequence corresponding to the phoneme information. The prosodic feature discretization network is used to encode and filter the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum. The first prosodic bottleneck representation information is a potential representation of the prosodic information of the speech spectrum. The timing alignment module is used to perform timing alignment on the speech spectrum to obtain the first duration information of each phoneme in the speech spectrum; The attention fusion module is used to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information. The speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data. Based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data, a total loss function is calculated. Based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, and the speech reconstruction decoding network are trained to obtain model parameters.

2. The method according to claim 1, characterized in that, The step of using the prosodic feature discretization network to encode and filter the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum specifically includes: The prosodic feature discretization network is used to discretize and encode the speech spectrum to obtain the prosodic feature encoding information of the speech spectrum; A preset information bottleneck mechanism is used to perform bottleneck filtering on the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

3. The method according to claim 1, characterized in that, The step of using the attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first multidimensional speech feature information specifically includes: Using the attention fusion module, the first phoneme feature encoding information is converted into a query vector, and the first timbre feature encoding information is converted into a key vector and a value vector; The dot product operation is performed on the key vector and the query vector to obtain the attention weight, which is used to characterize the correlation between phoneme features and timbre features. The value vector and the attention weight are weighted and summed to obtain the fused first speech multidimensional feature information.

4. The method according to claim 1, characterized in that, The step of using the speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data specifically includes: Upsampling is performed on the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; The speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data.

5. The method according to claim 2, characterized in that, The step of calculating the total loss function based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data specifically includes: The reconstruction loss is calculated based on the difference between the baseline speech spectrum and the reconstructed speech spectrum of the speech training data. Obtain the prosodic codebook vector indicated by the prosodic feature encoding information, and obtain the target codebook vector of the reference codebook of the prosodic feature discretization network, wherein the target codebook vector is the codebook vector in the reference codebook whose similarity to the prosodic codebook vector reaches a threshold. Calculate the mean square error between the prosodic codebook vector and the target codebook vector to obtain the discretized codebook loss; A preset discriminator is used to classify the reference speech spectrum and the reconstructed speech spectrum to obtain a authenticity probability value. Based on the authenticity probability value, the adversarial loss is calculated. The reconstruction loss, the discretization codebook loss, and the adversarial loss are fused to obtain the total loss function.

6. The method according to claim 1, characterized in that, After the step of adjusting the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters to construct a speech generation model, the method further includes: Acquire the target timbre information, content information, and target prosody information of the speech to be synthesized; The multi-timbre feature extraction network of the speech generation model is used to encode the target timbre information to obtain the second timbre feature encoding information of the target timbre information; The phoneme feature extraction network of the aforementioned speech generation model is used to extract features from the content information to obtain the second phoneme feature encoding information of the content information; The prosodic feature discretization network of the speech generation model is used to encode and filter the target prosodic information to obtain the second prosodic bottleneck representation information of the target prosodic information. Using the aforementioned timing alignment module, the target prosodic information is timing aligned to obtain the second duration information of each phoneme in the target prosodic information; The attention fusion module of the speech generation model is used to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information to obtain the fused second speech multidimensional feature information. The speech reconstruction decoding network of the aforementioned speech generation model decodes the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information, and the second duration information to generate the speech to be synthesized.

7. A training device for a speech generation model, characterized in that, include: The acquisition module is used to acquire the speech training data of the target object, wherein the speech training data includes reference timbre spectrum, phoneme information and speech spectrum; The training module is used to train a preset initial speech generation model based on the reference timbre spectrum, the phoneme information, and the speech spectrum to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a temporal alignment module, an attention fusion module, and a speech reconstruction decoding network. The training module is further configured to use the multi-timbre feature extraction network to encode the reference timbre spectrum, obtaining first timbre feature encoding information of the reference timbre spectrum, wherein the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum; use the phoneme feature extraction network to extract features from the phoneme information, obtaining first phoneme feature encoding information of the phoneme information, wherein the first phoneme feature encoding information is a potential representation of the phoneme sequence corresponding to the phoneme information; and use the prosodic feature discretization network to encode and filter the speech spectrum, obtaining first prosodic bottleneck representation information of the speech spectrum, wherein the first prosodic bottleneck representation information is a potential representation of the prosodic information of the speech spectrum. The timing alignment module is used to perform timing alignment on the speech spectrum to obtain the first duration information of each phoneme in the speech spectrum; The attention fusion module is used to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multidimensional feature information. The speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data. Based on the baseline speech spectrum of the speech training data and the reconstructed speech spectrum, a total loss function is calculated. Based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, and the speech reconstruction decoding network are trained to obtain model parameters. The adjustment module is used to adjust the parameters of the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the temporal alignment module, the attention fusion module, and the speech reconstruction decoding network based on the model parameters, so as to construct a speech generation model.

8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the training method for the speech generation model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the training method for the speech generation model as described in any one of claims 1 to 6.

Images