Voice data reconstruction method and device, computer equipment and storage medium

Abstract

The invention discloses a voice data reconstruction method and device, computer equipment and a storage medium. The method comprises the following steps: encoding a phoneme text through a preset text encoder to obtain an output vector; carrying out encoding processing on the speaker prompt language through a preset speaker encoder to obtain global speaker representation; performing prediction processing on the output vector and the global speaker representation through a preset rhythm module to obtain a rhythm representation vector; and performing decoding processing on the rhythm representation vector through a preset decoder to obtain reconstructed voice data. According to the scheme, the preset rhythm module is used for processing the global speaker representation of the speaker encoder and the output vector of the text encoder, the global consistency and the local change of the rhythm are considered, the hierarchical change of the rhythm is captured, finally, the reconstructed voice with improved rhythm quality is obtained, and the performance in the aspects of naturalness and emotion expression is better.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis in artificial intelligence technology and the field of financial technology, and in particular to a method, device, computer equipment and storage medium for reconstructing speech data. Background Art

[0002] In recent years, zero-shot text-to-speech (TTS) technology has gradually become the mainstream in the industry with the support of large-scale corpora. This corpus is rich in diverse data, which significantly enhances the generalization ability of the voice model, enabling it to accurately capture and imitate the timbre, rhythm and style characteristics of different speakers. Even when facing speakers that have never been seen, it can achieve highly similar voice output. For example, in financial business scenarios, this technology can be used to output synthetic voices corresponding to salesmen, and use this synthetic voice to communicate with different customers online to increase the online transaction rate of financial products; Among them, the VALL-E technology quantizes speech into tokens by introducing a neural codec based on residual vector quantization (RVQ), and cleverly integrates text and audio tokens as input. At the same time, the technology also uses the language model to generate target speech tokens in an autoregressive manner, further improving the flexibility and accuracy of speech generation; the SPEAR-TTS technology improves the AudioLM and realizes a progressive transformation from text to semantic tokens and then to acoustic tokens. This transformation makes the speech generation process more delicate and layered. The NaturalSpeech2 technology adopts another strategy. It also uses the RVQ-based neural codec, but encodes the audio into a continuous vector as a data representation. On this basis, the technology introduces a latent diffusion model to predict these continuous vectors, thereby further improving the accuracy and naturalness of speech generation; However, it is worth noting that although these technologies have made remarkable achievements in speech generation, they still have deficiencies in explicit modeling and model application of prosody. This shortcoming may lead to the lack of naturalness and emotional expression in the generated speech, especially when dealing with reference speech with high expressiveness and rich prosodic changes. Therefore, those skilled in the art are in urgent need of finding a new technical solution to solve the above problems. Summary of the invention

[0003] Based on this, a method, apparatus, computer device and storage medium for reconstructing speech data are provided to solve the technical problem of lack of naturalness and emotional expression in the speech generation process in the prior art.

[0004] A method for reconstructing speech data, comprising: After encoding the phoneme text through a preset text encoder, an output vector is obtained; After encoding the speaker prompt language through a preset speaker encoder, a global speaker representation is obtained; After performing prediction processing on the output vector and the global speaker representation through a preset prosody module, a prosody representation vector is obtained; After the prosody representation vector is decoded by a preset decoder, reconstructed speech data is obtained.

[0005] A device for reconstructing speech data, comprising: A first encoding processing module, used for performing encoding processing on the phoneme text through a preset text encoder to obtain an output vector; A second encoding processing module, used for encoding the speaker prompt language through a preset speaker encoder to obtain a global speaker representation; A prediction processing module, configured to obtain a prosody representation vector by performing prediction processing on the output vector and the global speaker representation through a preset prosody module; The decoding processing module is used to obtain reconstructed speech data after performing decoding processing on the prosody representation vector through a preset decoder.

[0006] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned voice data reconstruction method when executing the computer program.

[0007] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for reconstructing speech data is implemented.

[0008] The above-mentioned speech data reconstruction method, device, computer equipment and storage medium obtain an output vector after encoding the phoneme text through a preset text encoder; obtain a global speaker representation after encoding the speaker prompt language through a preset speaker encoder; obtain a prosody representation vector after predicting the output vector and the global speaker representation through a preset prosody module; and obtain reconstructed speech data after decoding the prosody representation vector through a preset decoder. This solution uses a preset prosody module to process the global speaker representation of the speaker encoder and the output vector of the text encoder, taking into account the global consistency and local changes of the prosody, capturing the hierarchical changes of the prosody, and finally obtaining a reconstructed speech with improved prosody quality, which can perform better in terms of naturalness and emotional expression. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0009] Figure 1 is a schematic diagram of an application environment of a method for reconstructing speech data in one embodiment of the present invention; Figure 2 is a process diagram of a method for reconstructing speech data in one embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a device for reconstructing speech data in one embodiment of the present invention; Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention; Figure 5 4 is a system diagram of a method for reconstructing speech data in one embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0011] The method for reconstructing speech data provided by the present invention can be applied in Figure 1 In an application environment, a client communicates with a server via a network. The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0012] In one embodiment, if Figure 2 As shown, a method for reconstructing speech data is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps: S10, after encoding the phoneme text through a preset text encoder, an output vector is obtained.

[0013] Understandably, the preset text encoder is a type of model that is pre-trained on a large amount of text data, which can convert text into a numerical vector, wherein the numerical vector can capture the semantic and syntactic information in the text; phoneme text generally refers to a text representation in which text is converted into a phoneme sequence, wherein a phoneme is the smallest unit of speech, and is the smallest speech segment or smallest speech unit that constitutes a syllable; the encoding process can be to encode the phoneme sequence using a preset text encoder (such as the BERT encoder), and the encoder will process the input text word by word (or phoneme by phoneme) and convert it into a vector representation in a high-dimensional space. S20, after encoding the speaker prompt language through a preset speaker encoder, a global speaker representation is obtained.

[0014] Understandably, the preset speaker encoder is a neural network model, which is specifically used to extract the speaker's feature representation, wherein the feature representation usually includes personalized information such as the speaker's timbre, intonation, and speaking speed, which is the key to distinguishing different speakers; the speaker prompt language can be used as the speaker's timbre information; the global speaker representation can be used as a reference for synthesized speech, so that the generated speech can retain the characteristics of the original speaker, which can characterize the speaker's timbre and be used as a reference in the preset prosody module modeling process (because the expressiveness of speech is closely related to the speaker's characteristics); the encoding process can be to use the preset speaker encoder to encode the preprocessed speaker prompt language, and the encoder will extract the speaker's features and convert them into vector representations in a high-dimensional space, namely the global speaker representation; This embodiment takes into account that the speaker's timbre is a global attribute that does not change over time, and uses a preset speaker encoder to extract the global speaker representation. Since quantized audio will lose too many speech details and cannot effectively guide the recovery of the speaker's timbre, the speaker prompt language represented by the Mel-spectrogram can be used as the input of the preset speaker encoder. S30, performing prediction processing on the output vector and the global speaker representation through a preset prosody module to obtain a prosody representation vector.

[0015] Understandably, the preset prosody module includes a diffusion model-based pitch predictor and a hierarchical prosody adapter. The diffusion model-based pitch predictor uses a denoised diffusion probability model (DDPM), which is a generative model that gradually transforms data into pure noise through a predefined diffusion process, while the current diffusion model pitch predictor is the inverse process of learning the above-mentioned generative model denoising process, gradually recovering the required fundamental frequency data from the noise data; the hierarchical prosody adapter can process the fundamental frequency data generated by the diffusion model to obtain a sample-level sinusoidal excitation signal to express the pitch information in more detail; the prosody representation vector is a representation containing prosody and timbre information converted from the output vector under the guidance of the global speaker representation. S40, after decoding the prosody representation vector through a preset decoder, reconstructed speech data is obtained. Understandably, the preset decoders include a diffusion decoder and an acoustic decoder. Each encoder processes different information. The two encoders can obtain reconstructed speech data with enhanced timbre information. The diffusion decoder is an important component of the diffusion model, responsible for decoding the generated latent space samples back to the original data space. The acoustic decoder is mainly used in speech recognition and other audio processing tasks. Its function is to decode the input audio signal into meaningful text or audio features.

[0016] It should be noted that the embodiments of steps S10 to S40 can be applied to financial business scenarios. For example, in the process of consulting or purchasing online financial products, users raise questions or inquiries about financial products online. After a special recognition system understands the questions or inquiries raised by the user in advance, this solution can output voice data that truly corresponds to the salesperson (speaker) himself. The voice data performs better in terms of naturalness and emotional expression. In this way, not only does the salesperson not need to be online at all times, but the user can feel that he can communicate with a familiar salesperson or a real person, and the user's intention to purchase online financial products is improved.

[0017] In the embodiment of steps S10 to S40, a preset prosody module is used to process an output vector including a global speaker representation with global consistency and local changes, so as to capture the hierarchical changes of the prosody and finally obtain reconstructed speech data, thereby improving the prosodic quality of the reconstructed speech and achieving better performance in terms of naturalness and emotional expression. Furthermore, the encoding process is performed on the phoneme text by a preset text encoder to obtain an output vector, including: After performing normalization processing on the phoneme text, a normalized phoneme text is obtained; After encoding the standard phoneme text, output vectors corresponding to each phoneme are obtained. Understandably, the processing method of the standardized phoneme text may include phoneme standardization, text cleaning and phoneme mapping, wherein phoneme standardization includes converting the phoneme text into a standardized phoneme text in a standard form through phoneme standardization, such as unifying different phoneme representations (such as IPA symbols, ASCII characters, etc.) into the same representation method, text cleaning includes removing irrelevant characters or symbols in the phoneme text, such as spaces, punctuation marks, etc., and phoneme mapping includes mapping each phoneme in the phoneme text to a corresponding phoneme in a predefined set; encoding processing includes encoding the standardized phoneme text, wherein the text encoder processes each phoneme (or its embedded vector) one by one and outputs a hidden state or output vector corresponding to each phoneme; This embodiment can convert the phoneme text into a standard form and remove irrelevant characters or symbols. At the same time, an embedding vector can be assigned to each phoneme text to facilitate recognition and processing by subsequent module models. It should be noted that the data output by the text encoder will flow into the duration sequencer in the preset rhythm module (such as Figure 5 As shown in Figure 1, it is a neural network model for sequence modeling that can process time series data and model and predict each time step in the sequence. It can contain different types of neural layers, such as recurrent neural network layer (RNN), long short-term memory layer (LSTM), gated recurrent unit layer (GRU), etc.

[0018] Furthermore, after the output vector and the global speaker representation are predicted by a preset prosody module and before the prosody representation vector is obtained, the method further includes: The noise data to be trained is denoised by the target formula in the diffusion model to be trained to obtain the denoised original data; the target formula is: ,in, and represent the mean and variance functions parameterized by the neural network, is the output vector corresponding to the noise data to be trained, is the global speaker representation corresponding to the noise data to be trained, is the noise data to be trained, θ is the target model parameter, The raw data for the model prediction distribution.

[0019] It can be understood that this embodiment is a training and learning process of the high-pitch predictor of the diffusion model in the preset prosody module (also the denoising process mentioned in this embodiment). In the denoising process, the global speaker representation and output vector in the noise data to be trained are used as conditions and the target model parameter θ is predicted, wherein the acquisition process of the global speaker representation and output vector in the noise data to be trained is consistent with step S10 and step S20. The purpose of the data acquired in this embodiment is to be used for training and learning, and step S10 and step S20 are direct application processes; The process of this embodiment illustrates how the diffusion model to be trained predicts the data state of the previous time step given the current noise data and conditional information, and through the iterative application of the denoising process, the diffusion model to be trained can gradually recover from the high noise state to the original data state, thereby realizing the reconstruction of the original speech data. At the same time, in the application of pitch prediction, it means that the model can predict clearer fundamental frequency data from the input data containing noise data, thereby generating more natural and accurate speech output. It is also worth noting that the SALN module is introduced for information injection to ensure that the preset prosody module can observe global and fine-grained prosody information at the same time. This not only maintains the consistency of global features, but also enhances the system's adaptability to different speakers, further strengthening the model's ability in speech modeling.

[0020] Furthermore, before performing denoising on the noise data to be trained by using the target formula in the diffusion model to be trained to obtain the denoised original data, the method further includes: Acquire initial model parameters of the inverse process corresponding to the diffusion model to be trained; the initial model parameters are obtained through the inverse process corresponding to the forward formula in the diffusion processing process of the diffusion model to be trained; After estimating the target model parameters on the initial model parameters through the neural network of the diffusion model to be trained, a target formula including the target model parameters is obtained. It can be understood that the initial model parameters are the initial parameters before loss calculation in the diffusion model to be trained. The parameters can be obtained by performing the inverse process after the diffusion processing of the training diffusion model. The diffusion processing involves a forward formula. The process of obtaining the initial model parameters through the forward formula is: obtaining the model initial diffusion calculation formula corresponding to the diffusion process of the diffusion model to be trained (this process simulates the process of data being gradually contaminated by noise): ,in, Indicates that at time step The data status, is the diffusion coefficient, represents a normal distribution, is the identity matrix; in each stage of the diffusion process, the matrix corresponding to the current time t-1 Add a very small normal noise to generate , where the amount of noise is determined by a series of predetermined variance parameters Control; finally the forward formula corresponding to the entire forward process is obtained: ; The diffusion model to be trained performs the inverse process on the forward formula to obtain the formula containing the initial model parameters: ,in, and are the mean and variance functions parameterized by the neural network, represents the model parameters (by minimizing the difference between the predicted noise and the actual noise, the diffusion model to be trained can learn how to reconstruct data from the noise); the preset neural network estimates the target model parameters on the initial model parameters as the loss function corresponding to the model structure in the calculation process, where the loss function is , represents the loss function of parameter θ, represents the expectation of the data distribution q(x0), Denotes the distribution vector of the model prediction, and the variational bound of the negative log-likelihood is obtained by considering the data distribution at all time steps. and the joint distribution of the model predictions With conditional distribution The ratio between them is used for calculation; In the application of pitch prediction in this embodiment, the trained diffusion model can be used to predict the fundamental frequency (pitch) of speech, which can capture the complex rhythmic changes in speech, thereby generating a more natural and expressive speech output; during the training process, it is impossible to directly obtain the forward process corresponding to the reverse process of time t, and the neural network corresponding to the diffusion model is used to estimate the target model parameter parameters to approximate the forward process of time; the goal of the loss function is to minimize the variational bound of the negative log-likelihood, and then by minimizing this bound, the trained diffusion model can learn how to reconstruct data from noise, thereby performing well in the generation task.

[0021] Further, the preset prosody module includes a diffusion model-based pitch predictor and a hierarchical prosody adapter; the preset prosody module performs prediction processing on the output vector and the global speaker representation to obtain a prosody representation vector, including: After performing prediction processing on the output vector and the global speaker representation by the diffusion model-based pitch predictor, fundamental frequency data is obtained; After the hierarchical rhythm adapter performs generation processing on the fundamental frequency data, a sinusoidal excitation signal is obtained; After the hierarchical prosody adapter performs conversion processing on the sinusoidal excitation signal, the output vector and the global speaker representation, a prosody representation vector containing prosody and timbre is obtained.

[0022] Understandably, fundamental frequency data (pitch) is frame-level data, and the pitch information it represents is still insufficient; a sinusoidal excitation signal generally refers to a simple harmonic signal used as a system or device input, which has a fixed frequency, amplitude and phase, and is continuous, periodic and smooth in time. After the fundamental frequency data is generated and processed by a hierarchical prosody adapter, a sinusoidal excitation signal that can express pitch information more finely can be obtained; the prosody representation vector contains the speaker's prosody and timbre, and has richer information content; the hierarchical prosody adapter takes content information (output vector), frame-level pitch (sinusoidal excitation signal) and speaker embedding (global speaker representation) as input, and outputs a prosody representation vector containing prosody, timbre and content information; This embodiment can capture rhythmic information at different levels through multi-level downsampling (realized by a hierarchical rhythm adapter) of the sinusoidal excitation signal. Unlike the existing model structure, the new preset rhythmic module does not require that the duration of the sinusoidal signal strictly corresponds to the length of the content information. The scale difference between the two signals is maximized by increasing the downsampling ratio. In the process of multi-level downsampling, the signal levels can correspond to the frame level, phoneme level and word level respectively, which provides a basis for hierarchical rhythmic modeling and makes it possible to model rhythmic information at different levels. Further, after performing generation processing on the baseband data, a sinusoidal excitation signal is obtained, including: The baseband data is converted into sample-level baseband data by a preset linear interpolation method; After the baseband data at the sample level is generated using a preset high-pitched information generation formula, a sinusoidal excitation signal capable of expressing the high-pitched information is obtained.

[0023] It can be understood that the preset linear interpolation is used in combination with the preset high pitch information generation formula, and the preset linear interpolation method can be expressed as , the preset high-pitched information generation formula can be expressed as ,in, represents the baseband data at the frame level, Indicates the frame rate, represents the sampling rate, represents the sample index, represents the fundamental frequency data at the sample level, represents a sinusoidal excitation signal; This embodiment can convert the fundamental frequency information at the frame level into the fundamental frequency information at the sample level through linear interpolation, and then generate a sample-level sinusoidal excitation signal through a preset high-pitch information generation formula, wherein the sinusoidal excitation signal is used to further enhance the model's ability to capture pitch information, thereby achieving more precise and natural pitch control in the reconstructed voice assistant.

[0024] Further, the preset encoder includes a diffusion decoder and an acoustic decoder: after the preset decoder performs decoding processing on the prosody representation vector, the reconstructed speech data is obtained, including: After the diffusion decoder performs decoding processing on the prosody representation vector, the latent vector is obtained; after the acoustic decoder performs decoding processing on the latent vector, the reconstructed speech data is obtained.

[0025] Understandably, the diffusion decoder is a complex neural network model that can convert the prosodic representation vector into a latent vector; the latent vector is usually a point in a high-dimensional, continuous vector space that contains all the necessary information for reconstructing speech data; the acoustic decoder is another neural network model that can convert the latent vector into a speech waveform or other forms of speech data; Specifically, during the decoding process, the diffusion decoder gradually converts the prosody representation vector into a latent vector while retaining all the necessary information for reconstructing the speech data; the acoustic decoder applies a series of nonlinear transformations to produce speech output with specific sound quality and timbre to obtain the reconstructed speech data; In this embodiment, the timbre information can be further enhanced through the decoding process of the first decoder, and the natural speech data with high similarity to the speaker can be finally obtained through the decoding process of the second decoder.

[0026] Furthermore, after the output vector and the global speaker representation are predicted by a preset prosody module and before the prosody representation vector is obtained, the method further includes: Downsampling of a preset multiple is performed through a preset number of downsampling layers in a model structure corresponding to a preset rhythmic module, and a sampling factor of an even-numbered downsampling layer is set.

[0027] This embodiment may use four (preset number) downsampling layers and corresponding modeling units. The downsampling ratios may be set to [20, 10, 10, 2] respectively. In the data preprocessing stage, the jump size when extracting the baseband data may be set to 200. At this time, 200 times downsampling is required from the sample level to the frame level. In addition, in order to avoid the size of a single downsampling layer being too large, 200 times (preset multiple) downsampling is achieved through the first two downsampling layers.

[0028] It should be noted that the average speaking speed of Chinese is about 150 to 200 characters per minute. Considering that the phoneme set contains two phonemes per character on average, and taking into account factors such as speech pauses, the ratio between the frame level and the phoneme level is about 10 to 15. Therefore, the downsampling factor of the third level can be set to 10 and the fourth level can be set to 2 (corresponding to the phoneme level and word level, respectively), and finally the cross-attention mechanism is used at each level to fuse the prosodic information.

[0029] In summary, the above provides a method for reconstructing speech data, wherein an output vector is obtained after encoding a phoneme text through a preset text encoder; a global speaker representation is obtained after encoding a speaker prompt language through a preset speaker encoder; a prosody representation vector is obtained after predicting the output vector and the global speaker representation through a preset prosody module; and a reconstructed speech data is obtained after decoding the prosody representation vector through a preset decoder. This scheme uses a preset prosody module to process the global speaker representation of the speaker encoder and the output vector of the text encoder. Taking into account the global consistency and local changes of the rhythm, the hierarchical changes of the rhythm are captured, and finally the reconstructed speech with improved rhythm quality is obtained, which can perform better in terms of naturalness and emotional expression; a global speaker representation is used to capture the speaker's timbre characteristics, and it is used as a reference to guide the rhythm construction (preset rhythm module modeling); a pitch predictor based on a diffusion model is used to predict the pitch, and a hierarchical rhythm adapter is introduced to achieve hierarchical modeling of the rhythm, thereby significantly improving the rhythm quality of the synthesized speech; compared with directly adding rhythm information to the intermediate variable, the hierarchical modeling method significantly improves the rhythm details and jitter problems. It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0030] In one embodiment, a device for reconstructing speech data is provided, and the device for reconstructing speech data corresponds to the method for reconstructing speech data in the above embodiment. Figure 3 As shown, the speech data reconstruction device includes a first encoding processing module 11, a second encoding processing module 12, a prediction processing module 13 and a decoding processing module 14. The functional modules are described in detail as follows: A first encoding processing module 11 is used to perform encoding processing on the phoneme text through a preset text encoder to obtain an output vector; A second encoding processing module 12 is used to encode the speaker prompt language through a preset speaker encoder to obtain a global speaker representation; A prediction processing module 13, configured to obtain a prosody representation vector by performing prediction processing on the output vector and the global speaker representation through a preset prosody module; The decoding processing module 14 is used to perform decoding processing on the prosody representation vector through a preset decoder to obtain reconstructed speech data.

[0031] Furthermore, the first encoding processing module includes: A normalization processing unit, used for performing normalization processing on the phoneme text to obtain a normalized phoneme text; The encoding processing unit is used to perform encoding processing on the standard phoneme text to obtain output vectors corresponding to each phoneme.

[0032] Furthermore, the prediction processing module includes: The denoising processing module is used to perform denoising processing on the noise data to be trained by using the target formula in the diffusion model to be trained to obtain the denoised original data; the denoising processing used by the diffusion model to be trained is the inverse process of the diffusion processing process of the diffusion model to be trained, and the target formula is ,in, and represent the mean and variance functions parameterized by the neural network, is the output vector corresponding to the noise data to be trained, is the global speaker representation corresponding to the noise data to be trained, is the noise data to be trained, θ is the target model parameter, The raw data for the model prediction distribution.

[0033] Furthermore, the voice data reconstruction device also includes: An acquisition module, used to acquire initial model parameters of the inverse process corresponding to the diffusion model; the initial model parameters are obtained through the inverse process corresponding to the forward formula in the diffusion processing process of the diffusion model to be trained; The estimation module is used to estimate the target model parameters on the initial model parameters through the neural network of the diffusion model to be trained, and then obtain a target formula containing the target model parameters.

[0034] Furthermore, the voice data reconstruction device also includes: A prediction processing unit, configured to obtain fundamental frequency data by performing prediction processing on the output vector and the global speaker representation through the diffusion model-based pitch predictor; A generation processing unit, configured to obtain a sinusoidal excitation signal after performing generation processing on the fundamental frequency data through the layered rhythm adapter; The conversion processing unit is used to perform conversion processing on the sinusoidal excitation signal, the output vector and the global speaker representation through the hierarchical prosody adapter to obtain a prosody representation vector containing prosody and timbre.

[0035] Furthermore, the generation processing unit includes: A conversion subunit, used in the hierarchical prosody adapter to convert the baseband data into sample-level baseband data by a preset linear interpolation method; The generation processing subunit is used to obtain a sinusoidal excitation signal that can express the treble information after performing generation processing on the baseband data at the sample level through a preset treble information generation formula in the layered rhythm adapter.

[0036] Furthermore, the decoding processing module includes: A first decoding processing unit, configured to obtain the latent vector after performing decoding processing on the prosody representation vector through the diffusion decoder; The second decoding processing unit is used to obtain the reconstructed speech data after performing decoding processing on the latent vector through the acoustic decoder.

[0037] The specific definition of the reconstruction device of voice data can be found in the definition of the reconstruction method of voice data in the above text, which will not be repeated here. Each module in the above-mentioned reconstruction device of voice data can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0038] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method for reconstructing voice data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for reconstructing voice data is implemented.

[0039] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for reconstructing speech data in the above embodiment are implemented, such as Figure 2 Steps S10 to S40 shown include the following steps: After encoding the phoneme text through a preset text encoder, an output vector is obtained; After encoding the speaker prompt language through a preset speaker encoder, a global speaker representation is obtained; After performing prediction processing on the output vector and the global speaker representation through a preset prosody module, a prosody representation vector is obtained; After the prosody representation vector is decoded by a preset decoder, reconstructed speech data is obtained.

[0040] Alternatively, when the processor executes the computer program, the functions of the modules / units of the apparatus for reconstructing speech data in the above-mentioned embodiment are realized, for example Figure 3 The functions of modules 11 to 14 are shown in Figure 1. To avoid repetition, they will not be described here.

[0041] In one embodiment, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for reconstructing speech data in the above embodiment are implemented, for example Figure 2 Steps S10 to S40 shown include the following steps: After encoding the phoneme text through a preset text encoder, an output vector is obtained; After encoding the speaker prompt language through a preset speaker encoder, a global speaker representation is obtained; After performing prediction processing on the output vector and the global speaker representation through a preset prosody module, a prosody representation vector is obtained; After the prosody representation vector is decoded by a preset decoder, reconstructed speech data is obtained.

[0042] Alternatively, when the computer program is executed by the processor, the functions of the modules / units of the apparatus for reconstructing speech data in the above-mentioned embodiment are realized, for example Figure 3 The functions of modules 11 to 14 are shown in Figure 1. To avoid repetition, they will not be described here.

[0043] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0044] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0045] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for reconstructing speech data, characterized in that: include: After encoding the phoneme text through a preset text encoder, an output vector is obtained; After encoding the speaker prompt language through a preset speaker encoder, a global speaker representation is obtained; After performing prediction processing on the output vector and the global speaker representation through a preset prosody module, a prosody representation vector is obtained; After the prosody representation vector is decoded by a preset decoder, reconstructed speech data is obtained.

2. The method for reconstructing speech data according to claim 1, characterized in that: After encoding the phoneme text by a preset text encoder, an output vector is obtained, including: After performing normalization processing on the phoneme text, a normalized phoneme text is obtained; After encoding the standard phoneme text, output vectors corresponding to each phoneme are obtained.

3. The method for reconstructing speech data according to claim 1, characterized in that: After the output vector and the global speaker representation are predicted by the preset prosody module and before the prosody representation vector is obtained, the method further includes: The noise data to be trained is denoised by the target formula in the diffusion model to be trained to obtain the denoised original data; the denoising process used by the diffusion model to be trained is the inverse process of the diffusion process of the diffusion model to be trained, and the target formula is ,in, and represent the mean and variance functions parameterized by the neural network, is the output vector corresponding to the noise data to be trained, is the global speaker representation corresponding to the noise data to be trained, is the noise data to be trained, θ is the target model parameter, The raw data for the model prediction distribution.

4. The method for reconstructing speech data according to claim 3, characterized in that: Before performing denoising on the noise data to be trained by using the target formula in the diffusion model to be trained to obtain the denoised original data, the method further includes: Acquire initial model parameters of the inverse process corresponding to the diffusion model; the initial model parameters are obtained through the inverse process corresponding to the forward formula in the diffusion processing process of the diffusion model to be trained; After estimating the target model parameters on the initial model parameters through the neural network of the diffusion model to be trained, a target formula including the target model parameters is obtained.

5. The method for reconstructing speech data according to claim 1, characterized in that: The preset prosody module includes a diffusion model-based pitch predictor and a hierarchical prosody adapter; The step of performing prediction processing on the output vector and the global speaker representation by a preset prosody module to obtain a prosody representation vector includes: After performing prediction processing on the output vector and the global speaker representation by the diffusion model-based pitch predictor, fundamental frequency data is obtained; After the hierarchical rhythm adapter performs generation processing on the fundamental frequency data, a sinusoidal excitation signal is obtained; After the hierarchical prosody adapter performs conversion processing on the sinusoidal excitation signal, the output vector and the global speaker representation, a prosody representation vector containing prosody and timbre is obtained.

6. The method for reconstructing speech data according to claim 5, characterized in that: The step of performing generation processing on the fundamental frequency data by the layered rhythm adapter to obtain a sinusoidal excitation signal comprises: In the hierarchical prosody adapter, the baseband data is converted into sample-level baseband data by a preset linear interpolation method; In the hierarchical rhythm adapter, after the baseband data at the sample level is processed by a preset high-pitch information generation formula, a sinusoidal excitation signal capable of expressing high-pitch information is obtained.

7. The method for reconstructing speech data according to claim 1, characterized in that: The preset encoder includes a diffusion decoder and an acoustic decoder: after the preset decoder performs decoding processing on the prosody representation vector, the reconstructed speech data is obtained, including: After the diffusion decoder performs decoding processing on the prosody representation vector, the latent vector is obtained; After the acoustic decoder performs decoding processing on the latent vector, the reconstructed speech data is obtained.

8. A device for reconstructing speech data, characterized in that: include: A first encoding processing module, used for performing encoding processing on the phoneme text through a preset text encoder to obtain an output vector; A second encoding processing module, used for encoding the speaker prompt language through a preset speaker encoder to obtain a global speaker representation; A prediction processing module, configured to obtain a prosody representation vector by performing prediction processing on the output vector and the global speaker representation through a preset prosody module; The decoding processing module is used to obtain reconstructed speech data after performing decoding processing on the prosody representation vector through a preset decoder.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for reconstructing speech data according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for reconstructing speech data as claimed in any one of claims 1 to 7 is implemented.

Images