Speech generation model training method and device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a voice generation model training method comprising the following steps: obtaining a reference tone frequency spectrum, phoneme information and a voice frequency spectrum of a target object; based on the reference tone frequency spectrum, the phoneme information and the voice frequency spectrum, training a preset initial voice generation model to obtain model parameters; and based on the model parameters, respectively adjusting parameters of a multi-timbre feature extraction network, a phoneme feature extraction network, a rhythm feature discretization network, a time sequence alignment module, an attention fusion module and a voice reconstruction decoding network of the initial voice generation model so as to construct the voice generation model. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and voice training data and model parameters can be stored in a block chain. According to the method and the device, the tone and the rhythm information can be decoupled, and the tone and the rhythm information can be flexibly adjusted so as to generate synthetic speech with diversity and flexibility.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment and medium for a speech generation model. Background Art

[0002] In recent years, adaptive speech generation systems have been able to accurately restore target sound characteristics, such as pitch, speaking speed, and timbre, with just a few minutes of speech data, providing a natural and realistic auditory experience. However, their performance is limited by the quality and quantity of data. Insufficient or poor data can result in stiff and fuzzy generated audio.

[0003] In order to get rid of the above dependence, the industry has explored new solutions, among which generative models have attracted much attention in zero-sample speech generation. The system can quickly clone speech with only one voice prompt, without tedious fine-tuning, improving work efficiency. However, the zero-sample speech generation system also has defects, that is, the rhythmic information in the prompt speech is highly coupled with the timbre, which is difficult to transmit separately, limiting the diversity and flexibility of the generated speech, making the system unable to cope with different scenarios and needs. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a training method, device, equipment and medium for a speech generation model to solve the problem that the prosodic information and timbre in the existing prompt speech are highly coupled, making it difficult to transmit them separately, thereby limiting the diversity and flexibility of the generated speech.

[0005] In order to solve the above technical problems, the present application provides a method for training a speech generation model, which adopts the following technical solutions:

[0006] Acquire speech training data of the target object, the speech training data includes reference timbre spectrum, phoneme information and speech spectrum; based on the reference timbre spectrum, phoneme information and speech spectrum, train the preset initial speech generation model to obtain model parameters, the initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a rhythmic feature discretization network, a timing alignment module, an attention fusion module, and a speech reconstruction decoding network; based on the model parameters, adjust the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythmic feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network to construct a speech generation model.

[0007] Furthermore, based on the reference timbre spectrum, phoneme information and speech spectrum, the preset initial speech generation model is trained to obtain model parameters, specifically including:

[0008] A multi-timbre feature extraction network is used to encode the reference timbre spectrum to obtain the first timbre feature encoding information of the reference timbre spectrum, and the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum; a phoneme feature extraction network is used to extract the phoneme information to obtain the first phoneme feature encoding information of the phoneme information, and the first phoneme feature encoding information is a potential representation of the phoneme sequence corresponding to the phoneme information; a rhythm feature discretization network is used to encode and filter the speech spectrum to obtain the first rhythm bottleneck representation information of the speech spectrum, and the first rhythm bottleneck representation information is a potential representation of the rhythm information of the speech spectrum; a timing alignment module is used to perform timing alignment on the speech spectrum to obtain each part of the speech spectrum. The first duration information of the phoneme; using the attention fusion module, the first timbre feature encoding information and the first phoneme feature encoding information are processed with attention to obtain the fused first speech multi-dimensional feature information; using the speech reconstruction decoding network, the first speech multi-dimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information and the first duration information are decoded to obtain the reconstructed speech spectrum of the speech training data; based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data, the total loss function is calculated, and based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the timing alignment module and the speech reconstruction decoding network are trained to obtain the model parameters.

[0009] Furthermore, the step of using a prosodic feature discretization network to encode and filter the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum specifically includes:

[0010] A prosodic feature discretization network is used to discretize and encode the speech spectrum to obtain the prosodic feature encoding information of the speech spectrum; a preset information bottleneck mechanism is used to bottleneck filter the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0011] Further, the step of using an attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multi-dimensional feature information specifically includes:

[0012] An attention fusion module is used to convert the first phoneme feature encoding information into a query vector, and the first timbre feature encoding information into a key vector and a value vector; a dot product operation is performed on the key vector and the query vector to obtain an attention weight, which is used to characterize the correlation between the phoneme feature and the timbre feature; a weighted sum is performed on the value vector and the attention weight to obtain the fused first speech multidimensional feature information.

[0013] Further, a speech reconstruction decoding network is used to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data, specifically comprising:

[0014] The first speech multidimensional feature information and the first phoneme feature coding information are upsampled to obtain an enhanced feature set; a speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature coding information, the first prosodic bottleneck representation information and the first duration information to obtain a reconstructed speech spectrum of the speech training data.

[0015] Furthermore, the step of calculating the total loss function based on the reference speech spectrum and the reconstructed speech spectrum of the speech training data specifically includes:

[0016] The reconstruction loss is calculated based on the difference between the baseline speech spectrum and the reconstructed speech spectrum of the speech training data; the prosody codebook vector indicated by the prosody feature encoding information is obtained, and the target codebook vector of the reference codebook of the prosody feature discretization network is obtained, and the target codebook vector is the codebook vector in the reference codebook whose similarity with the prosody codebook vector reaches a threshold; the mean square error between the prosody codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss; the baseline speech spectrum and the reconstructed speech spectrum are classified using a preset discriminator to obtain a truth probability value, and the adversarial loss is calculated based on the truth probability value; the reconstruction loss, the discretization codebook loss and the adversarial loss are integrated to obtain the total loss function.

[0017] Further, after the step of respectively adjusting the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the prosody feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network based on the model parameters to construct the speech generation model, the method further includes:

[0018] The target timbre information, content information and target prosody information of the speech to be synthesized are obtained; the multi-timbre feature extraction network of the speech generation model is used to encode the target timbre information to obtain the second timbre feature encoding information of the target timbre information; the phoneme feature extraction network of the speech generation model is used to extract the content information to obtain the second phoneme feature encoding information of the content information; the prosody feature discretization network of the speech generation model is used to encode and filter the target prosody information to obtain the second prosody bottleneck representation information of the target prosody information; the timing alignment module is used to perform timing alignment on the target prosody information to obtain the second duration information of each phoneme of the target prosody information; the attention fusion module of the speech generation model is used to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information to obtain the fused second speech multidimensional feature information; the speech reconstruction decoding network of the speech generation model is used to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosody bottleneck representation information and the second duration information to generate the speech to be synthesized.

[0019] In order to solve the above technical problems, the embodiment of the present application also provides a training device for a speech generation model, which adopts the following technical solutions:

[0020] An acquisition module is used to acquire speech training data of a target object, wherein the speech training data includes a reference timbre spectrum, phoneme information, and speech spectrum;

[0021] A training module is used to train a preset initial speech generation model based on a reference timbre spectrum, phoneme information, and speech spectrum to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a timing alignment module, an attention fusion module, and a speech reconstruction decoding network;

[0022] The adjustment module is used to adjust the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythm feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network based on the model parameters to build a speech generation model.

[0023] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the training method of the speech generation model as mentioned above are implemented.

[0024] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the training method of the speech generation model as mentioned above.

[0025] Compared with the prior art, the embodiment of the present application has the following beneficial effects: by acquiring the speech training data of the target object, including the reference timbre spectrum, phoneme information and speech spectrum, a rich and accurate information basis is provided for the subsequent training of the preset initial speech generation model. In the training process of the initial speech generation model, multiple modules such as the multi-timbre feature extraction network, the phoneme feature extraction network, and the prosody feature discretization network can be used to perform comprehensive and in-depth feature extraction on the speech training data. For example, the speech spectrum is encoded and filtered by the prosody feature discretization network to obtain the potential representation of the prosody information, which effectively realizes the preliminary separation of the prosody information and the timbre. In addition, the introduction of the timing alignment module and the attention fusion module further enhances the initial speech generation model's ability to capture and process speech features. The timing alignment module can accurately calculate the first duration information of each phoneme, providing strong support for the accurate reconstruction of speech. The attention fusion module realizes the deep fusion of the timbre features and the phoneme features at the feature level by paying attention to them, thereby improving the naturalness and clarity of the generated speech. Finally, the initial speech generation model was trained based on the total loss function, and a speech generation model was constructed that includes multiple modules such as multi-timbre feature extraction network, phoneme feature extraction network, and prosodic feature discretization network. This speech generation model can not only achieve efficient speech cloning, but also flexibly adjust the timbre and prosodic information in the face of different scenarios and needs, and generate diverse and flexible speech output. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0028] Figure 2 It is a flowchart of a method for training a speech generation model provided in this application;

[0029] Figure 3 It is a structural schematic diagram of a training device for a speech generation model provided by the present application;

[0030] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0035] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0037] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0038] It should be noted that the training method of the speech generation model provided in the embodiment of the present application is generally executed by a server, and accordingly, the training device of the speech generation model is generally set in the server.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for training a speech generation model according to the present application. The method for training a speech generation model comprises the following steps:

[0041] Step S201, obtaining speech training data of a target object, wherein the speech training data includes a reference timbre spectrum, phoneme information, and speech spectrum.

[0042] In this embodiment, the training method of the speech generation model is executed on the electronic device (eg Figure 1 The server shown in the figure) can obtain the speech training data of the target object through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0043] The target object refers to an individual whose speech features will be used to train the initial speech generation model to generate speech with specific attributes of the target object. For example, the target object can be a person or an animal.

[0044] The speech training data refers to a data set containing the speech features of the target object, including reference timbre spectrum, phoneme information, and speech spectrum, etc. It is used to train the initial speech generation model so that it can imitate the speech of the target object.

[0045] The reference timbre spectrum refers to the spectrum data used to extract timbre features, such as other Mel spectra. It reflects the unique nature of timbre in speech and serves as a reference timbre template when training the initial speech generation model.

[0046] Among them, phoneme information refers to the information describing each phoneme (the basic unit of speech) in speech, which is an important input for speech synthesis and is used to guide the initial speech generation model to generate accurate pronunciation.

[0047] The speech spectrum refers to the spectral representation of the speech signal, such as the target Mel spectrum. It contains the frequency domain characteristics of speech and is the basis for model training and speech reconstruction.

[0048] Step S202, based on the reference timbre spectrum, phoneme information and speech spectrum, the preset initial speech generation model is trained to obtain model parameters. The initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a rhythm feature discretization network, a timing alignment module, an attention fusion module, and a speech reconstruction decoding network.

[0049] Among them, the initial speech generation model refers to the untrained speech generation network framework, including multiple components such as multi-timbre feature extraction network and phoneme feature extraction network, which provides the basic structure for subsequent training.

[0050] Model parameters refer to a set of values ​​that are constantly adjusted and optimized during the training process. They determine the output characteristics of the speech generation model, including the weights and biases of each network layer.

[0051] Among them, the multi-timbre feature extraction network refers to a network specifically used to extract timbre features from a reference timbre spectrum, which can learn the potential representation of timbre for subsequent speech generation.

[0052] Among them, the phoneme feature extraction network is used to process phoneme information and extract phoneme features. It can convert phoneme information into potential representations that can be understood by the model.

[0053] Among them, the prosodic feature discretization network refers to a network that encodes and filters the speech spectrum and extracts prosodic features. Its output is a potential representation of prosodic information, which helps the initial speech generation model generate natural speech rhythm.

[0054] Among them, the timing alignment module is used to align the various phonemes in the speech spectrum with their positions on the time axis, output the duration information of each phoneme, and ensure the timing accuracy of speech generation.

[0055] Among them, the attention fusion module is used to combine timbre features and phoneme features, perform information fusion through the attention mechanism, and generate multi-dimensional feature information containing rich speech features.

[0056] Among them, the speech reconstruction decoding network is used to decode the fused multi-dimensional feature information, phoneme features, prosodic features and time series information into a reconstructed speech spectrum.

[0057] Step S203, based on the model parameters, respectively adjust the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythm feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network to construct a speech generation model.

[0058] The parameters of the multi-timbre feature extraction network refer to the values ​​that determine the output characteristics of the multi-timbre feature extraction network and can be optimized through training.

[0059] The parameters of the phoneme feature extraction network refer to a set of values ​​that affect the performance of the phoneme feature extraction network and can be adjusted through training to more accurately extract phoneme features.

[0060] The parameters of the prosodic feature discretization network refer to the parameters that control the prosodic feature extraction and discretization process, which affect the understanding and expression of the prosodic information by the initial speech generation model.

[0061] The parameters of the timing alignment module refer to a set of values ​​that determine the accuracy and efficiency of timing alignment, and can be optimized through training to improve the timing consistency of speech generation.

[0062] The parameters of the attention fusion module refer to the parameters that affect the effect of the attention mechanism in fusing timbre and phoneme features. They can be adjusted through training to optimize the fusion results.

[0063] The parameters of the speech reconstruction decoding network refer to a set of values ​​that control the speech reconstruction decoding process, and can be optimized through training to improve the quality of speech generation.

[0064] Among them, the speech generation model refers to a model that, after training, can generate speech output with the speech characteristics of the target object based on input feature information.

[0065] The embodiment of the present application can provide a rich and accurate information basis for the training of the subsequent preset initial speech generation model by obtaining the speech training data of the target object, including the reference timbre spectrum, phoneme information and speech spectrum. In the training process of the initial speech generation model, multiple modules such as the multi-timbre feature extraction network, the phoneme feature extraction network and the rhythm feature discretization network can be used to perform comprehensive and in-depth feature extraction on the speech training data. For example, the speech spectrum is encoded and filtered by the rhythm feature discretization network to obtain the potential representation of the rhythm information, which effectively realizes the preliminary separation of the rhythm information and the timbre. In addition, the introduction of the timing alignment module and the attention fusion module further enhances the initial speech generation model's ability to capture and process speech features. The timing alignment module can accurately calculate the first duration information of each phoneme, providing strong support for the accurate reconstruction of speech. The attention fusion module realizes the deep fusion of the timbre features and the phoneme features at the feature level by paying attention to them, thereby improving the naturalness and clarity of the generated speech. Finally, the initial speech generation model was trained based on the total loss function, and a speech generation model was constructed that includes multiple modules such as multi-timbre feature extraction network, phoneme feature extraction network, and prosodic feature discretization network. This speech generation model can not only achieve efficient speech cloning, but also flexibly adjust the timbre and prosodic information in the face of different scenarios and needs, and generate diverse and flexible speech output.

[0066] In some optional implementations of this embodiment, step 202, based on the reference timbre spectrum, phoneme information and speech spectrum, trains a preset initial speech generation model to obtain model parameters, specifically including the following steps:

[0067] A multi-timbre feature extraction network is used to encode the reference timbre spectrum to obtain the first timbre feature encoding information of the reference timbre spectrum, and the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum; a phoneme feature extraction network is used to extract the phoneme information to obtain the first phoneme feature encoding information of the phoneme information, and the first phoneme feature encoding information is a potential representation of the phoneme sequence corresponding to the phoneme information; a rhythm feature discretization network is used to encode and filter the speech spectrum to obtain the first rhythm bottleneck representation information of the speech spectrum, and the first rhythm bottleneck representation information is a potential representation of the rhythm information of the speech spectrum; a timing alignment module is used to perform timing alignment on the speech spectrum to obtain each part of the speech spectrum. The first duration information of the phoneme; using the attention fusion module, the first timbre feature encoding information and the first phoneme feature encoding information are processed with attention to obtain the fused first speech multi-dimensional feature information; using the speech reconstruction decoding network, the first speech multi-dimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information and the first duration information are decoded to obtain the reconstructed speech spectrum of the speech training data; based on the baseline speech spectrum and the reconstructed speech spectrum of the speech training data, the total loss function is calculated, and based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the timing alignment module and the speech reconstruction decoding network are trained to obtain the model parameters.

[0068] The encoding process refers to the process of converting the reference timbre spectrum into a potential representation that can be processed by the model. For example, feature extraction and dimension reduction are performed on the reference timbre spectrum to obtain the first timbre feature encoding information.

[0069] Among them, timbre information refers to the characteristics that describe the timbre of speech, including the brightness and fullness of the timbre.

[0070] Among them, the first timbre feature encoding information refers to the potential representation of the timbre feature output by the multi-timbre feature extraction network, which is used in the subsequent speech generation process.

[0071] Among them, feature extraction refers to the process of extracting phoneme features from phoneme information.

[0072] Among them, the first phoneme feature encoding information refers to the potential representation of the phoneme feature output by the phoneme feature extraction network, which is used to guide the speech generation model to pronounce accurately.

[0073] Among them, the phoneme sequence refers to the arrangement order of a series of phonemes in speech, which is the basis of speech.

[0074] Among them, the coding and filtering process refers to the process in which the prosodic feature discretization network encodes the input speech spectrum and filters redundant information to extract key features.

[0075] Among them, the first prosodic bottleneck representation information refers to the potential representation of the prosodic features output by the prosodic feature discretization network, which is the key to speech rhythm and naturalness.

[0076] Among them, prosodic information refers to the information that describes the rhythm, intonation and other characteristics of speech, and is an important way for speech to express emotions and content.

[0077] Among them, timing alignment refers to the process of matching each phoneme in the speech with its position on the timeline to ensure the timing accuracy of speech generation.

[0078] Among them, the first duration information refers to the duration of each phoneme output by the timing alignment module, which is an important basis for the speech generation model to control the pronunciation speed.

[0079] Among them, attention processing refers to the process of weighted fusion of multiple feature information through the attention mechanism to improve the overall performance of the speech generation model.

[0080] Among them, the first speech multi-dimensional feature information refers to the fusion result of multi-dimensional feature information including timbre, phonemes, rhythm, etc. output by the attention fusion module.

[0081] The decoding process refers to the process of converting the first speech multi-dimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information and the first duration information into an understandable output (such as a reconstructed speech spectrum).

[0082] Among them, the reconstructed speech spectrum refers to the speech spectrum output by the speech reconstruction decoding network, which is the final output of the speech generation model and reflects the speech characteristics of the target object.

[0083] The total loss function is a function that measures the difference between the output of the initial speech generation model and the expected output. It is used to guide the optimization of model parameters to achieve more accurate speech generation.

[0084] In one example, in a financial customer service system, in order to improve the quality of customer service, it is often necessary to convert text information into speech with specific timbre and customized prosodic features. The technical solution proposed in this embodiment is to train a model that can accurately simulate and generate the target object's speech. First, the speech training data of the target object is obtained from the financial customer service recording library, and the data contains a reference timbre spectrum, phoneme information (such as pinyin or phonetic symbols), and a speech spectrum. These data are preprocessed to ensure that they are consistent with the financial customer service context, such as removing noise, standardizing volume, etc. Then, using a multi-timbre feature extraction network, the reference timbre spectrum is deeply encoded to generate the first timbre feature encoding information, which is used as a potential expression of the target timbre feature. At the same time, through the phoneme feature extraction network, the first phoneme feature encoding of the phoneme information is extracted to reflect the potential structure of the phoneme sequence. Further, the prosodic feature discretization network encodes and filters the speech spectrum to extract the first prosodic bottleneck representation information, which captures the rhythm, intonation and other prosodic features in the speech, which is essential for simulating the professional and fluent intonation of financial customer service. The timing alignment module is responsible for accurately aligning the speech spectrum with the phoneme sequence and obtaining the first duration information of each phoneme, which is particularly important for maintaining the natural fluency of the speech. The attention fusion module combines the first timbre feature encoding information and the first phoneme feature encoding information, enhances the expression of key information through the attention mechanism, and forms the fused first speech multi-dimensional feature information. Finally, the speech reconstruction decoding network receives the above multi-dimensional feature information, the first phoneme feature encoding, the first prosodic bottleneck representation and the first duration information, and decodes and generates the reconstructed speech spectrum. By comparing the baseline speech spectrum with the reconstructed speech spectrum, the total loss function is calculated, and each network module is trained accordingly until the model parameters are optimized to the optimal state. In financial customer service applications, the model can generate highly realistic voice responses based on the input text and the specified timbre and prosody requirements, significantly improving the customer experience, while reducing the pressure on manual customer service and improving service efficiency.

[0085] The embodiment of the present application can perform multi-dimensional feature extraction and fusion on the speech training data of the target object through a deep neural network, significantly improving the accuracy and naturalness of speech synthesis. The multi-timbre feature extraction network accurately captures the potential features of the reference timbre spectrum, giving the synthesized speech a highly personalized timbre. The phoneme feature extraction network and the rhythm feature discretization network respectively analyze the phoneme sequence and speech rhythm to ensure the accurate pronunciation and smooth rhythm of the synthesized speech. The timing alignment module accurately calculates the duration of each phoneme and maintains the temporal structure of the speech. The attention fusion module cleverly integrates timbre and phoneme features, enhances the correlation between speech features, and enables the synthesized speech to accurately convey semantic information while maintaining individual timbre. The speech reconstruction decoding network integrates multi-dimensional feature information, efficiently decodes and generates a reconstructed speech spectrum, compares it with the benchmark speech spectrum to calculate the loss, and feeds back to optimize the model parameters to form a closed-loop training system.

[0086] In some optional implementations of this embodiment, the step of "using a prosodic feature discretization network to perform encoding and filtering processing on the speech spectrum to obtain first prosodic bottleneck representation information of the speech spectrum" specifically includes the following steps:

[0087] A prosodic feature discretization network is used to discretize and encode the speech spectrum to obtain the prosodic feature encoding information of the speech spectrum; a preset information bottleneck mechanism is used to bottleneck filter the prosodic feature encoding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0088] In one example, the present embodiment can be described in detail in combination with the application of intelligent customer service system in the financial field. First, a speech spectrum is extracted from the recording data of financial customer service as input for subsequent processing. The speech spectrum is the representation of the speech signal in the frequency domain, which contains rich information of the speech. Next, the prosodic feature discretization network in the present invention is used to discretize and encode the speech spectrum. The network consists of multiple convolutional layers and pooling layers, which can extract the prosodic features in the speech spectrum layer by layer and convert them into discrete coding information. These coding information are represented in the form of high-dimensional vectors, and each vector corresponds to a prosodic feature in the speech spectrum. However, directly using these coding information for speech synthesis may lead to overfitting problems, because the coding information may contain redundant information that is not important for speech synthesis. In order to solve this problem, a preset information bottleneck mechanism is introduced. The mechanism filters the prosodic feature coding information by setting a bottleneck layer, removes the timbre information in the audio, and only retains the most critical information for speech synthesis. Specifically, the bottleneck layer uses a lower-dimensional fully connected layer to compress the high-dimensional prosodic feature coding information into a low-dimensional first prosodic bottleneck representation information. In this way, the key prosodic features in the speech spectrum are retained and the overfitting problem is effectively avoided.

[0089] The embodiment of the present application can implement fine discretization encoding of the speech spectrum through a prosodic feature discretization network. This process effectively converts the continuous speech spectrum into highly representative prosodic feature coding information, thereby accurately capturing and retaining key prosodic features such as rhythm and intonation in the speech. Furthermore, the preset information bottleneck mechanism plays a key role in the processing of prosodic feature coding information. It is like a sophisticated filter. Through strict bottleneck filtering, it removes redundant and noise information and only retains the core representation that is most critical to speech prosody, that is, the first prosodic bottleneck representation information. This processing not only greatly reduces the data dimension, but also significantly enhances the generalization ability of the model, effectively avoiding overfitting.

[0090] In some optional implementations of this embodiment, the step of “using an attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain fused first speech multi-dimensional feature information” specifically includes the following steps:

[0091] An attention fusion module is used to convert the first phoneme feature encoding information into a query vector, and the first timbre feature encoding information into a key vector and a value vector; a dot product operation is performed on the key vector and the query vector to obtain an attention weight, which is used to characterize the correlation between the phoneme feature and the timbre feature; a weighted sum is performed on the value vector and the attention weight to obtain the fused first speech multidimensional feature information.

[0092] In one example, the present embodiment can be described in detail in combination with the application of intelligent customer service system in the financial field. First, the first timbre feature coding information corresponding to the timbre spectrum and the first phoneme feature coding corresponding to the phoneme information are referenced from the financial customer service recording library. Among them, the first timbre feature coding information represents the unique timbre of the customer service staff, and the first phoneme feature coding information reflects the phoneme sequence and its attributes in the voice. Next, the attention fusion module is used to fuse these two feature coding information. Specifically, the first phoneme feature coding information is converted into a query vector, and the first timbre feature coding information is converted into a key vector and a value vector. This conversion process can be achieved through a fully connected layer or a linear transformation to ensure that the feature coding information matches the requirements of the attention mechanism. Then, the key vector and the query vector are dot-producted to obtain the attention weight. The attention weight is a scalar value used to characterize the correlation between the phoneme feature and the timbre feature. Through the attention mechanism, the initial speech generation model can automatically learn and focus on the most critical phoneme and timbre feature combinations for synthesizing personalized voice. Finally, the value vector and the attention weight are weighted and summed to obtain the fused first voice multidimensional feature information. This information not only contains the accurate information of the phoneme sequence, but also incorporates the unique timbre characteristics of the customer service staff, thereby achieving personalized voice synthesis.

[0093] The embodiment of the present application can achieve deep interaction and fusion of phoneme feature coding information and timbre feature coding information by introducing an attention fusion module. Specifically, the module cleverly converts the first phoneme feature coding information into a query vector, and the first timbre feature coding information is decomposed into a key vector and a value vector. Through the dot product operation of the key vector and the query vector, the attention weight is accurately calculated, and this weight accurately reflects the intrinsic connection and correlation between the phoneme feature and the timbre feature. Furthermore, these attention weights are used to perform weighted summation on the value vector, which effectively integrates the multidimensional features of phonemes and timbre, and generates richer and more accurate first speech multidimensional feature information.

[0094] In some optional implementations of this embodiment, the step of "using a speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data" specifically includes the following steps:

[0095] The first speech multidimensional feature information and the first phoneme feature coding information are upsampled to obtain an enhanced feature set; a speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature coding information, the first prosodic bottleneck representation information and the first duration information to obtain a reconstructed speech spectrum of the speech training data.

[0096] In one example, the present embodiment can be described in detail in combination with the specific application of the intelligent voice customer service system in the financial field. First, the upsampling technology is used to expand the first voice multidimensional feature information and the first phoneme feature encoding information in the time and space dimensions, so as to obtain an enhanced feature set containing richer details. This step effectively improves the input quality of the subsequent voice reconstruction decoding network. Next, the voice reconstruction decoding network can be used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first continuous duration information. Among them, the first prosodic bottleneck representation information reflects the prosodic features such as intonation and rhythm of the voice, and the first continuous duration information provides the duration of each phoneme. These information together constitute a comprehensive representation of the voice signal. The voice reconstruction decoding network adopts a deep learning model, such as a convolutional neural network, a recurrent neural network or its variants (such as a long short-term memory network, a gated recurrent unit), etc., to achieve complex nonlinear transformation and fusion of input features. Through the layer-by-layer processing and feature fusion of the voice reconstruction decoding network, a reconstructed voice spectrum is finally generated.

[0097] The embodiment of the present application can process the first speech multi-dimensional feature information and the first phoneme feature coding information by upsampling, significantly enhancing the richness and detail expression of the feature set, and providing a solid foundation for subsequent speech reconstruction. On this basis, an advanced speech reconstruction decoding network is used to skillfully perform deep fusion and decoding of the enhanced feature set, the first phoneme feature coding information, the first prosodic bottleneck representation information, and the first duration information. This process not only achieves the comprehensive integration of speech features, but also accurately captures the prosodic features and duration information of the speech signal, thereby successfully reconstructing a high-quality speech spectrum.

[0098] In some optional implementations of this embodiment, the step of “calculating the total loss function based on the reference speech spectrum and the reconstructed speech spectrum of the speech training data” specifically includes the following steps:

[0099] The reconstruction loss is calculated based on the difference between the baseline speech spectrum and the reconstructed speech spectrum of the speech training data; the prosody codebook vector indicated by the prosody feature encoding information is obtained, and the target codebook vector of the reference codebook of the prosody feature discretization network is obtained, and the target codebook vector is the codebook vector in the reference codebook whose similarity with the prosody codebook vector reaches a threshold; the mean square error between the prosody codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss; the baseline speech spectrum and the reconstructed speech spectrum are classified using a preset discriminator to obtain a truth probability value, and the adversarial loss is calculated based on the truth probability value; the reconstruction loss, the discretization codebook loss and the adversarial loss are integrated to obtain the total loss function.

[0100] In one example, the technical solution proposed in this embodiment aims to calculate the difference between the reference speech spectrum and the reconstructed speech spectrum, obtain the mean square error between the prosody codebook vector indicated by the prosody feature encoding information and the target codebook vector, and use the discriminator to classify the reference speech spectrum and the reconstructed speech spectrum to obtain the authenticity probability value, respectively calculate the reconstruction loss, discretization codebook loss and adversarial loss, and merge the three to obtain the total loss function. Specifically, the reference speech spectrum of the speech training data can be extracted, and the reconstructed speech spectrum can be generated based on the proposed speech reconstruction decoding network. Then, the difference between the reference speech spectrum and the reconstructed speech spectrum is calculated to obtain the reconstruction loss. This step measures the error between the reconstructed speech spectrum and the real speech spectrum. Next, the prosody codebook vector indicated by the prosody feature encoding information can be obtained. At the same time, from the reference codebook of the prosody feature discretization network, the codebook vector most similar to the prosody codebook vector is found as the target codebook vector. Then, the mean square error between the prosody codebook vector and the target codebook vector is calculated to obtain the discretization codebook loss. This step aims to optimize the discretization representation of the prosody feature and improve the naturalness and accuracy of speech synthesis. Then, the preset discriminator is used to classify the reference speech spectrum and the reconstructed speech spectrum to obtain the authenticity probability value. The adversarial loss is calculated based on the authenticity probability value. This step further improves the authenticity of the reconstructed speech spectrum and its consistency with the reference speech spectrum by introducing the adversarial training mechanism. Finally, the reconstruction loss, discretization codebook loss and adversarial loss are fused to obtain the total loss function.

[0101] The embodiment of the present application can effectively measure the accuracy of speech reconstruction by accurately calculating the difference between the reference speech spectrum and the reconstructed speech spectrum, and then guide the model optimization through the reconstruction loss. At the same time, the prosodic feature coding information is matched with the reference codebook of the discretized network, and the mean square error between the prosodic codebook vector and the target codebook vector is calculated. The obtained discretized codebook loss significantly improves the accuracy of the prosodic feature. In addition, a preset discriminator is introduced to classify the reference and reconstructed speech spectra, and the adversarial loss calculated based on the authenticity probability value further enhances the realism and naturalness of the reconstructed speech. By integrating the reconstruction loss, the discretized codebook loss and the adversarial loss, the obtained total loss function comprehensively guides the optimization direction of the speech training data, which not only improves the accuracy of speech reconstruction and the expressiveness of prosodic features, but also significantly enhances the authenticity and natural fluency of the reconstructed speech, providing strong technical support for applications such as speech synthesis and speech recognition.

[0102] In some optional implementations of this embodiment, in step S203, based on the model parameters, the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the prosody feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network are adjusted respectively to construct the speech generation model, and the following steps are specifically included:

[0103] The target timbre information, content information and target prosody information of the speech to be synthesized are obtained; the multi-timbre feature extraction network of the speech generation model is used to encode the target timbre information to obtain the second timbre feature encoding information of the target timbre information; the phoneme feature extraction network of the speech generation model is used to extract the content information to obtain the second phoneme feature encoding information of the content information; the prosody feature discretization network of the speech generation model is used to encode and filter the target prosody information to obtain the second prosody bottleneck representation information of the target prosody information; the timing alignment module is used to perform timing alignment on the target prosody information to obtain the second duration information of each phoneme of the target prosody information; the attention fusion module of the speech generation model is used to perform attention processing on the second timbre feature encoding information and the second phoneme feature encoding information to obtain the fused second speech multidimensional feature information; the speech reconstruction decoding network of the speech generation model is used to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosody bottleneck representation information and the second duration information to generate the speech to be synthesized.

[0104] In one example, the present embodiment can be described in detail in combination with actual cases in the financial field. First, the target timbre information (e.g., timbre samples), content information (text content to be synthesized), and target prosody information (preset speech speed and intonation information) of the target customer service representative can be obtained from the database. Then, the multi-timbre feature extraction network of the speech generation model proposed in this embodiment is used to encode the target timbre information to obtain the second timbre feature encoding information. At the same time, the text content is feature extracted by the phoneme feature extraction network of the speech generation model to obtain the second phoneme feature encoding information of the text content. In addition, the prosody feature discretization network of the speech generation model is also used to encode and filter the preset speech speed and intonation information to obtain the second prosody bottleneck representation information. Next, the speech speed and intonation information are time-aligned by the timing alignment module of the speech generation model to obtain the second duration information of each phoneme. Then, the speech generation model uses the attention fusion module to perform attention processing on the timbre feature encoding and the phoneme feature encoding to obtain the fused second speech multidimensional feature information. Finally, the speech reconstruction decoding network of the speech generation model is used to decode the fused speech multi-dimensional feature information, phoneme feature encoding, prosodic bottleneck representation information, and duration information to generate the synthesized speech with target timbre, accurate content, and natural rhythm.

[0105] The embodiment of the present application can achieve high-quality speech synthesis by accurately capturing the target timbre, content information and target rhythm information of the speech to be synthesized, and cleverly using the multi-network modules in the speech generation model. The multi-timbre feature extraction network deeply encodes the target timbre to ensure a high degree of restoration of the timbre; the phoneme feature extraction network accurately extracts the phoneme features of the content information, providing an accurate content basis for speech synthesis. The rhythm feature discretization network effectively extracts the bottleneck representation of the target rhythm through encoding and filtering processing, giving the speech a natural sense of rhythm. The timing alignment module ensures the precise synchronization of rhythm information and content information, while the attention fusion module further integrates timbre and phoneme features to enhance the naturalness and expressiveness of the speech. Finally, the speech reconstruction decoding network integrates multi-dimensional feature information, phoneme feature encoding, rhythm bottleneck representation and duration information, and successfully generates a speech to be synthesized that not only meets the target timbre, but also accurately conveys the content and has a natural rhythm, significantly improving the overall quality and user experience of speech synthesis.

[0106] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned voice training data and model parameters, the above-mentioned voice training data and model parameters can also be stored in a node of a blockchain.

[0107] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0108] The embodiments of the present application can construct and optimize related models and networks based on artificial intelligence technology, such as multi-timbre feature extraction network, phoneme feature extraction network, rhythm feature discretization network, timing alignment module, attention fusion module, speech reconstruction decoding network, etc. Among them, the artificial intelligence (AI) model is the crystallization of theory and practice that simulates the decision-making process of human intelligence through algorithms and data analysis to solve complex problems, predict future trends or realize automation tasks. These models use a large amount of historical data and real-time information, and are trained and optimized through a specific algorithm framework to achieve efficient, accurate and reliable performance.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0111] Further references Figure 3 , as a response to the above Figure 2The present application provides an embodiment of a training device for a speech generation model, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0112] like Figure 3 As shown, the speech generation model training device 400 of this embodiment includes: an acquisition module 401, a training module 402 and an adjustment module 403. Among them:

[0113] An acquisition module 401 is used to acquire speech training data of a target object, wherein the speech training data includes a reference timbre spectrum, phoneme information, and speech spectrum;

[0114] A training module 402 is used to train a preset initial speech generation model based on a reference timbre spectrum, phoneme information and speech spectrum to obtain model parameters, wherein the initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a timing alignment module, an attention fusion module and a speech reconstruction decoding network;

[0115] The adjustment module 403 is used to adjust the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythm feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network based on the model parameters to construct a speech generation model.

[0116] The embodiment of the present application can provide an accurate information basis for the preset initial speech generation model by obtaining the speech training data of the target object, including reference timbre spectrum, phoneme information and speech spectrum. During the training process, network modules such as multi-timbre feature extraction, phoneme feature extraction and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network realizes the preliminary separation of rhythm and timbre. The timing alignment module accurately calculates the duration of phonemes and supports accurate reconstruction of speech. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of generated speech. Based on the total loss function training, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning, and flexibly adjust the timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0117] In one embodiment, the training module 402 includes:

[0118] The encoding submodule is used to use a multi-timbre feature extraction network to encode the reference timbre spectrum to obtain first timbre feature encoding information of the reference timbre spectrum, where the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum;

[0119] An extraction submodule, used to extract features of the phoneme information using a phoneme feature extraction network to obtain first phoneme feature encoding information of the phoneme information, where the first phoneme feature encoding information is a potential representation of a phoneme sequence corresponding to the phoneme information;

[0120] A filtering submodule, for performing encoding and filtering processing on the speech spectrum by using a prosodic feature discretization network to obtain first prosodic bottleneck representation information of the speech spectrum, where the first prosodic bottleneck representation information is a potential representation of the prosodic information of the speech spectrum;

[0121] An alignment submodule, used to perform time alignment on the speech spectrum using a time alignment module to obtain first duration information of each phoneme of the speech spectrum;

[0122] An attention submodule, configured to use an attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain fused first speech multi-dimensional feature information;

[0123] A decoding submodule, configured to use a speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data;

[0124] The training submodule is used to calculate the total loss function based on the baseline speech spectrum and reconstructed speech spectrum of the speech training data. Based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the rhythm feature discretization network, the timing alignment module and the speech reconstruction decoding network are trained to obtain the model parameters.

[0125] The embodiment of the present application can perform multi-dimensional feature extraction and fusion on the speech training data of the target object through a deep neural network, significantly improving the accuracy and naturalness of speech synthesis. The multi-timbre feature extraction network accurately captures the potential features of the reference timbre spectrum, giving the synthesized speech a highly personalized timbre. The phoneme feature extraction network and the rhythm feature discretization network respectively analyze the phoneme sequence and speech rhythm to ensure the accurate pronunciation and smooth rhythm of the synthesized speech. The timing alignment module accurately calculates the duration of each phoneme and maintains the temporal structure of the speech. The attention fusion module cleverly integrates timbre and phoneme features, enhances the correlation between speech features, and enables the synthesized speech to accurately convey semantic information while maintaining individual timbre. The speech reconstruction decoding network integrates multi-dimensional feature information, efficiently decodes and generates a reconstructed speech spectrum, compares it with the benchmark speech spectrum to calculate the loss, and feeds back to optimize the model parameters to form a closed-loop training system.

[0126] In one embodiment, the filtering submodule is also used to use a prosodic feature discretization network to discretize the speech spectrum to obtain prosodic feature coding information of the speech spectrum; and use a preset information bottleneck mechanism to bottleneck filter the prosodic feature coding information to obtain the first prosodic bottleneck representation information of the speech spectrum.

[0127] The embodiment of the present application can implement fine discretization encoding of the speech spectrum through a prosodic feature discretization network. This process effectively converts the continuous speech spectrum into highly representative prosodic feature coding information, thereby accurately capturing and retaining key prosodic features such as rhythm and intonation in the speech. Furthermore, the preset information bottleneck mechanism plays a key role in the processing of prosodic feature coding information. It is like a sophisticated filter. Through strict bottleneck filtering, it removes redundant and noise information and only retains the core representation that is most critical to speech prosody, that is, the first prosodic bottleneck representation information. This processing not only greatly reduces the data dimension, but also significantly enhances the generalization ability of the model, effectively avoiding overfitting.

[0128] In one embodiment, the attention submodule is also used to use an attention fusion module to convert the first phoneme feature encoding information into a query vector, and convert the first timbre feature encoding information into a key vector and a value vector; perform a dot product operation on the key vector and the query vector to obtain an attention weight, and the attention weight is used to characterize the correlation between the phoneme feature and the timbre feature; perform a weighted summation on the value vector and the attention weight to obtain the fused first speech multidimensional feature information.

[0129] The embodiment of the present application can achieve deep interaction and fusion of phoneme feature coding information and timbre feature coding information by introducing an attention fusion module. Specifically, the module cleverly converts the first phoneme feature coding information into a query vector, and the first timbre feature coding information is decomposed into a key vector and a value vector. Through the dot product operation of the key vector and the query vector, the attention weight is accurately calculated, and this weight accurately reflects the intrinsic connection and correlation between the phoneme feature and the timbre feature. Furthermore, these attention weights are used to perform weighted summation on the value vector, which effectively integrates the multidimensional features of phonemes and timbre, and generates richer and more accurate first speech multidimensional feature information.

[0130] In one embodiment, the decoding submodule is also used to upsample the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; and use a speech reconstruction decoding network to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data.

[0131] The embodiment of the present application can process the first speech multi-dimensional feature information and the first phoneme feature coding information by upsampling, significantly enhancing the richness and detail expression of the feature set, and providing a solid foundation for subsequent speech reconstruction. On this basis, an advanced speech reconstruction decoding network is used to skillfully perform deep fusion and decoding of the enhanced feature set, the first phoneme feature coding information, the first prosodic bottleneck representation information, and the first duration information. This process not only achieves the comprehensive integration of speech features, but also accurately captures the prosodic features and duration information of the speech signal, thereby successfully reconstructing a high-quality speech spectrum.

[0132] In one embodiment, the training submodule is further used to calculate the reconstruction loss based on the difference between the reference speech spectrum and the reconstructed speech spectrum of the speech training data; obtain the prosody codebook vector indicated by the prosody feature encoding information, and obtain the target codebook vector of the reference codebook of the prosody feature discretization network, the target codebook vector being the codebook vector in the reference codebook whose similarity with the prosody codebook vector reaches a threshold; calculate the mean square error between the prosody codebook vector and the target codebook vector to obtain the discretization codebook loss; use a preset discriminator to classify the reference speech spectrum and the reconstructed speech spectrum to obtain a truth probability value, and calculate the adversarial loss based on the truth probability value; and fuse the reconstruction loss, the discretization codebook loss, and the adversarial loss to obtain a total loss function.

[0133] The embodiment of the present application can effectively measure the accuracy of speech reconstruction by accurately calculating the difference between the reference speech spectrum and the reconstructed speech spectrum, and then guide the model optimization through the reconstruction loss. At the same time, the prosodic feature coding information is matched with the reference codebook of the discretized network, and the mean square error between the prosodic codebook vector and the target codebook vector is calculated. The obtained discretized codebook loss significantly improves the accuracy of the prosodic feature. In addition, a preset discriminator is introduced to classify the reference and reconstructed speech spectra, and the adversarial loss calculated based on the authenticity probability value further enhances the realism and naturalness of the reconstructed speech. By integrating the reconstruction loss, the discretized codebook loss and the adversarial loss, the obtained total loss function comprehensively guides the optimization direction of the speech training data, which not only improves the accuracy of speech reconstruction and the expressiveness of prosodic features, but also significantly enhances the authenticity and natural fluency of the reconstructed speech, providing strong technical support for applications such as speech synthesis and speech recognition.

[0134] In one embodiment, the training device 400 for the speech generation model further includes:

[0135] An information acquisition module, used to acquire target timbre information, content information and target rhythm information of the speech to be synthesized;

[0136] An encoding module, used for encoding the target timbre information using a multi-timbre feature extraction network of a speech generation model to obtain second timbre feature encoding information of the target timbre information;

[0137] An extraction module, used to extract features of the content information using a phoneme feature extraction network of a speech generation model, and obtain second phoneme feature encoding information of the content information;

[0138] A filtering module, for using a prosodic feature discretization network of a speech generation model to perform encoding and filtering processing on target prosodic information to obtain second prosodic bottleneck representation information of the target prosodic information;

[0139] An alignment module, configured to perform time alignment on the target prosodic information using the time alignment module, and obtain second duration information of each phoneme of the target prosodic information;

[0140] An attention module, used for performing attention processing on the second timbre feature encoding information and the second phoneme feature encoding information using an attention fusion module of a speech generation model, to obtain fused second speech multi-dimensional feature information;

[0141] The decoding module is used to use the speech reconstruction decoding network of the speech generation model to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information and the second duration information to generate the speech to be synthesized.

[0142] The embodiment of the present application can achieve high-quality speech synthesis by accurately capturing the target timbre, content information and target rhythm information of the speech to be synthesized, and cleverly using the multi-network modules in the speech generation model. The multi-timbre feature extraction network deeply encodes the target timbre to ensure a high degree of restoration of the timbre; the phoneme feature extraction network accurately extracts the phoneme features of the content information, providing an accurate content basis for speech synthesis. The rhythm feature discretization network effectively extracts the bottleneck representation of the target rhythm through encoding and filtering processing, giving the speech a natural sense of rhythm. The timing alignment module ensures the precise synchronization of rhythm information and content information, while the attention fusion module further integrates timbre and phoneme features to enhance the naturalness and expressiveness of the speech. Finally, the speech reconstruction decoding network integrates multi-dimensional feature information, phoneme feature encoding, rhythm bottleneck representation and duration information, and successfully generates a speech to be synthesized that not only meets the target timbre, but also accurately conveys the content and has a natural rhythm, significantly improving the overall quality and user experience of speech synthesis.

[0143] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0144] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0145] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0146] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the training method of the speech generation model, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0147] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions or process data stored in the memory 61, such as computer-readable instructions for running a training method for a speech generation model.

[0148] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] The embodiment of the present application can provide an accurate information basis for the preset initial speech generation model by obtaining the speech training data of the target object, including reference timbre spectrum, phoneme information and speech spectrum. During the training process, network modules such as multi-timbre feature extraction, phoneme feature extraction and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network realizes the preliminary separation of rhythm and timbre. The timing alignment module accurately calculates the duration of phonemes and supports accurate reconstruction of speech. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of generated speech. Based on the total loss function training, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning, and flexibly adjust the timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0150] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the training method of the speech generation model as described above.

[0151] The embodiment of the present application can provide an accurate information basis for the preset initial speech generation model by obtaining the speech training data of the target object, including reference timbre spectrum, phoneme information and speech spectrum. During the training process, network modules such as multi-timbre feature extraction, phoneme feature extraction and prosodic feature discretization are used to comprehensively and deeply extract speech features. The prosodic feature discretization network realizes the preliminary separation of rhythm and timbre. The timing alignment module accurately calculates the duration of phonemes and supports accurate reconstruction of speech. The attention fusion module fuses timbre and phoneme features to improve the naturalness and clarity of generated speech. Based on the total loss function training, a speech generation model containing multiple modules is constructed to achieve efficient speech cloning, and flexibly adjust the timbre and prosodic information to output diverse and flexible speech to meet the needs of different scenarios.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0153] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0154] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for training a speech generation model, characterized in that: The steps include: Acquire speech training data of a target object, wherein the speech training data includes a reference timbre spectrum, phoneme information, and a speech spectrum; Based on the reference timbre spectrum, the phoneme information and the speech spectrum, a preset initial speech generation model is trained to obtain model parameters, wherein the initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a timing alignment module, an attention fusion module and a speech reconstruction decoding network; Based on the model parameters, the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythm feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network are adjusted respectively to construct a speech generation model.

2. The method according to claim 1, characterized in that: The step of training a preset initial speech generation model based on the reference timbre spectrum, the phoneme information and the speech spectrum to obtain model parameters specifically includes: Using the multi-timbre feature extraction network, encoding the reference timbre spectrum to obtain first timbre feature encoding information of the reference timbre spectrum, where the first timbre feature encoding information is a potential representation of the timbre information of the reference timbre spectrum; Using the phoneme feature extraction network, extracting features from the phoneme information to obtain first phoneme feature encoding information of the phoneme information, where the first phoneme feature encoding information is a potential representation of a phoneme sequence corresponding to the phoneme information; Using the prosodic feature discretization network, the speech spectrum is coded and filtered to obtain first prosodic bottleneck representation information of the speech spectrum, where the first prosodic bottleneck representation information is a potential representation of the prosodic information of the speech spectrum; Using the timing alignment module, timing aligning the speech spectrum is performed to obtain first duration information of each phoneme of the speech spectrum; Using the attention fusion module, performing attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain fused first speech multi-dimensional feature information; Using the speech reconstruction decoding network, decoding the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data; Based on the benchmark speech spectrum of the speech training data and the reconstructed speech spectrum, a total loss function is calculated, and based on the loss function, the multi-timbre feature extraction network, the phoneme feature extraction network, the prosodic feature discretization network, the timing alignment module and the speech reconstruction decoding network are trained to obtain model parameters.

3. The method according to claim 2, characterized in that The step of using the prosodic feature discretization network to perform encoding and filtering processing on the speech spectrum to obtain the first prosodic bottleneck representation information of the speech spectrum specifically includes: Using the prosodic feature discretization network, discretize and encode the speech spectrum to obtain prosodic feature encoding information of the speech spectrum; A preset information bottleneck mechanism is used to perform bottleneck filtering on the prosodic feature coding information to obtain first prosodic bottleneck representation information of the speech spectrum.

4. The method according to claim 2, characterized in that: The step of using the attention fusion module to perform attention processing on the first timbre feature encoding information and the first phoneme feature encoding information to obtain the fused first speech multi-dimensional feature information specifically includes: Using the attention fusion module, converting the first phoneme feature encoding information into a query vector, and converting the first timbre feature encoding information into a key vector and a value vector; Performing a dot product operation on the key vector and the query vector to obtain an attention weight, where the attention weight is used to characterize the correlation between the phoneme feature and the timbre feature; The value vector and the attention weight are weightedly summed to obtain fused first speech multi-dimensional feature information.

5. The method according to claim 2, characterized in that: The step of using the speech reconstruction decoding network to decode the first speech multidimensional feature information, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain the reconstructed speech spectrum of the speech training data specifically includes: Performing upsampling processing on the first speech multidimensional feature information and the first phoneme feature encoding information to obtain an enhanced feature set; The speech reconstruction decoding network is used to fuse and decode the enhanced feature set, the first phoneme feature encoding information, the first prosodic bottleneck representation information, and the first duration information to obtain a reconstructed speech spectrum of the speech training data.

6. The method according to claim 3, characterized in that The step of calculating the total loss function based on the reference speech spectrum of the speech training data and the reconstructed speech spectrum specifically includes: Calculating a reconstruction loss based on a difference between a reference speech spectrum of the speech training data and the reconstructed speech spectrum; Acquire a prosody codebook vector indicated by the prosody feature encoding information, and acquire a target codebook vector of a reference codebook of the prosody feature discretization network, wherein the target codebook vector is a codebook vector in the reference codebook whose similarity with the prosody codebook vector reaches a threshold value; Calculating a mean square error between the prosody codebook vector and the target codebook vector to obtain a discretized codebook loss; Using a preset discriminator, classify the reference speech spectrum and the reconstructed speech spectrum to obtain a truth probability value, and calculate the adversarial loss based on the truth probability value; The reconstruction loss, the discretization codebook loss and the adversarial loss are integrated to obtain a total loss function.

7. The method according to claim 1, characterized in that After the step of adjusting the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the prosody feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network based on the model parameters to construct a speech generation model, the method further includes: Obtaining target timbre information, content information, and target prosody information of the speech to be synthesized; Using the multi-timbre feature extraction network of the speech generation model, encoding the target timbre information to obtain second timbre feature encoding information of the target timbre information; Using the phoneme feature extraction network of the speech generation model, extracting features from the content information to obtain second phoneme feature encoding information of the content information; Using the prosodic feature discretization network of the speech generation model, encoding and filtering the target prosodic information to obtain second prosodic bottleneck representation information of the target prosodic information; Using the timing alignment module, timing aligning the target prosody information is performed to obtain second duration information of each phoneme of the target prosody information; Using the attention fusion module of the speech generation model, the attention processing is performed on the second timbre feature encoding information and the second phoneme feature encoding information to obtain fused second speech multi-dimensional feature information; The speech reconstruction decoding network of the speech generation model is used to decode the second speech multidimensional feature information, the second phoneme feature encoding information, the second prosodic bottleneck representation information and the second duration information to generate the speech to be synthesized.

8. A training device for a speech generation model, characterized in that: include: An acquisition module, used to acquire speech training data of a target object, wherein the speech training data includes a reference timbre spectrum, phoneme information, and speech spectrum; A training module, for training a preset initial speech generation model based on the reference timbre spectrum, the phoneme information and the speech spectrum to obtain model parameters, wherein the initial speech generation model includes a multi-timbre feature extraction network, a phoneme feature extraction network, a prosodic feature discretization network, a timing alignment module, an attention fusion module and a speech reconstruction decoding network; An adjustment module is used to adjust the parameters of the multi-timbre feature extraction network, the parameters of the phoneme feature extraction network, the parameters of the rhythm feature discretization network, the parameters of the timing alignment module, the parameters of the attention fusion module, and the parameters of the speech reconstruction decoding network based on the model parameters to construct a speech generation model.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the training method of the speech generation model as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the method for training a speech generation model as described in any one of claims 1 to 7.

Images