Voice generation method and device, equipment and storage medium thereof
Through the improved parameter sharing strategy and hybrid convolution method of text encoder and decoder, combined with time prediction and adversarial learning training, the problem of slow inference speed of end-to-end speech generation model is solved, and the efficiency and quality of speech generation are improved.
Patent Information
- Application Number
- CN202411375276.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-07
AI Technical Summary
The existing end-to-end speech generation model has too many codec parameters, resulting in slow inference speed and limiting speech generation efficiency.
Adopt improved text encoder and decoder to reduce the amount of encoding parameter settings through parameter sharing strategy, and use a mixed convolution method of mixing packet convolution and ordinary convolution in the decoder, combining time prediction and adversarial learning training to optimize Voice generation process.
The model parameter setting amount is reduced, the inference speed and speech generation efficiency are improved, and the generated speech is more natural and high-quality.
Smart Images
Figure CN119274536A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and is applied to speech automation generation scenarios, and in particular to a speech generation method, apparatus, device, and storage medium thereof. Background Art
[0002] Speech generation technology refers to synthesizing the pronunciation of the target speaker from a specified text. Currently, speech generation technology has been rapidly developed, and end-to-end models and large speech models have greatly promoted the development of this field.
[0003] However, in practical applications, end-to-end speech generation faces several challenges: the model is too large, the number of encoding and decoding parameters set is too large, and the traditional decoding method often uses upsampling convolution, which leads to slow model inference speed and limits the efficiency of speech generation. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a speech generation method, apparatus, device and storage medium thereof to solve the problem that when performing existing speech generation, the end-to-end generation mode sets too many encoding and decoding parameters, resulting in slow model inference speed during decoding, thereby limiting the efficiency of speech generation.
[0005] In order to solve the above technical problems, the present application provides a speech generation method, which adopts the following technical solution:
[0006] A speech generation method comprises the following steps:
[0007] Obtaining text data to be used for speech generation;
[0008] Inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information;
[0009] Inputting the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data;
[0010] Aligning the text encoding information with the generated speech duration and inputting the two together into a preset audio generation flow model to obtain audio encoding information;
[0011] The audio coding information is subjected to a latent variable adding process, and the audio coding information with the latent variable added is input into an improved decoder to obtain speech output by the improved decoder, wherein the latent variable includes timbre features.
[0012] Furthermore, the improved text encoder is transformed from an initial encoder in a transformer structure, and the encoder includes N encoding layers. Before executing the step of inputting the text data into the improved text encoder for text encoding processing to obtain text encoding information, the method further includes:
[0013] According to a preset parameter setting document, setting the encoding parameters of the first encoding layer in the initial encoder;
[0014] A parameter sharing strategy is adopted to distribute the encoding parameters of the first encoding layer to the remaining N-1 encoding layers in the initial encoder layer by layer, so as to obtain the improved text encoder.
[0015] Furthermore, before executing the step of inputting the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data, the method further includes:
[0016] Obtain batches of real reference voices;
[0017] Performing speech feature extraction on the real reference speech to extract speech features;
[0018] Inputting the speech features into a preset posterior encoder to obtain posterior distribution parameters of latent variables;
[0019] Performing sampling processing according to the posterior distribution parameters of the latent variables to obtain timbre features in the speech features;
[0020] Performing text conversion on the real reference speech to obtain text data;
[0021] Inputting the text data into the improved text encoder to obtain text encoding features;
[0022] Inputting the text encoding feature and the timbre feature into an alignment estimation component to determine a correspondence between each text and a phoneme in the text data;
[0023] The correspondence between each text and the phoneme is deployed as prediction knowledge into the duration predictor.
[0024] Furthermore, the preset duration predictor includes speech speed information of at least one real voice object that can be selected, and the step of inputting the text encoding information into the preset duration predictor to predict the generated voice duration corresponding to the text data specifically includes:
[0025] Recognize pre-selected real speech objects;
[0026] Filtering out the speaking speed information corresponding to the real voice object from the duration predictor, wherein the speaking speed information includes the corresponding relationship between each text and phoneme when the real voice object speaks;
[0027] The duration of the generated speech corresponding to the text data is predicted based on the text encoding information and the speaking speed information corresponding to the real speech object.
[0028] Further, after executing the step of obtaining the speech output by the improved decoder, the method further includes:
[0029] Acquire the real voice recorded by the real voice object according to the text data;
[0030] Inputting the real speech and the speech output by the improved decoder into a preset verification model to verify the authenticity of the speech output by the improved decoder compared with the real speech, wherein the verification model includes a wavl m model;
[0031] If the authenticity does not exceed the preset authenticity threshold, the encoding parameters of the posterior encoder and the improved text encoder are optimized respectively by using adversarial learning training until the authenticity exceeds the preset authenticity threshold, and the optimization is completed;
[0032] If the authenticity exceeds a preset authenticity threshold, the speech output by the improved decoder is output via a preset speech output device.
[0033] Furthermore, after performing the step of optimizing the encoding parameters of the posterior encoder and the improved text encoder respectively by using the adversarial learning training method until the authenticity exceeds a preset authenticity threshold and the optimization step is completed, the method further includes:
[0034] Obtaining optimized encoding parameters of the improved text encoder;
[0035] Using a parameter sharing strategy, the encoding parameters of the improved text encoder are distributed layer by layer to all encoding layers in the improved text encoder;
[0036] Obtaining optimized encoding parameters of the a posteriori encoder;
[0037] A parameter sharing strategy is adopted to distribute the encoding parameters of the a posteriori encoder layer by layer to all encoding layers in the a posteriori encoder.
[0038] Furthermore, the improved decoder is transformed from an initial decoder in a transformer structure. Before executing the step of inputting the audio coding information with added hidden variables into the improved decoder, the method further includes:
[0039] The convolution mode of the convolution layer in the initial decoder is set to a mixed convolution mode of grouped convolution and ordinary convolution to obtain the improved decoder.
[0040] In order to solve the above technical problems, the embodiment of the present application also provides a speech generation device, which adopts the following technical solution:
[0041] A speech generating device, comprising:
[0042] A text data acquisition module, used to acquire text data to be generated for speech;
[0043] A text encoding module, used for inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information;
[0044] A speech duration prediction module, used for inputting the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data;
[0045] An audio stream generation module, used to align the text encoding information and the generated speech duration and then input them into a preset audio generation stream model to obtain audio encoding information;
[0046] The audio stream decoding module is used to perform hidden variable adding processing on the audio coding information, input the audio coding information with added hidden variables into the improved decoder, and obtain the speech output by the improved decoder, wherein the hidden variables include timbre features.
[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0048] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech generation method when executing the computer-readable instructions.
[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0050] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech generation method as described above.
[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0052] The speech generation method described in the embodiment of the present application is as follows: obtaining text data to be generated for speech; inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information; inputting the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; aligning the text encoding information and the generated speech duration and inputting them into a preset audio generation flow model to obtain audio encoding information; performing hidden variable addition processing on the audio encoding information, inputting the audio encoding information with the added hidden variables into an improved decoder to obtain the speech output by the improved decoder. The speech generation method described in the present application improves the end-to-end codecs respectively, which not only reduces the amount of manual model parameter settings, but also increases the model reasoning speed, thereby improving the speech generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0055] Figure 2 is a flow chart of an embodiment of a speech generation method according to the present application;
[0056] Figure 3 It is a flowchart of a specific embodiment of improving the initial encoder in the speech generation method described in the present application;
[0057] Figure 4 It is a flowchart of a specific embodiment of acquiring duration prediction knowledge in the speech generation method described in the present application;
[0058] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;
[0059] Figure 6 is a structural schematic diagram of an embodiment of a speech generating device according to the present application;
[0060] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0066] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.
[0067] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0068] It should be noted that the speech generation method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the speech generation device is generally arranged in the terminal device.
[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech generation method according to the present application. The speech generation method comprises the following steps:
[0071] Step 201, obtaining text data to be generated into speech;
[0072] Specifically, the text data to be generated for speech includes text data directly input by the speech generation request end, and also includes text data obtained by parsing a text document. The speech generation method can be used in a scenario where intelligent customer service automatically responds based on the speech text.
[0073] Step 202, inputting the text data into the improved text encoder for text encoding processing to obtain text encoding information;
[0074] In this embodiment, the improved text encoder can be understood as being obtained by modifying the initial encoder in the transformer structure. Specifically, the traditional text encoder is composed of a transformer structure including N coding layers, which results in the need to set the coding parameters for each coding layer separately when setting the coding parameters, and a large number of parameters need to be set. The present application adopts a parameter sharing strategy to set the coding parameters of the text encoder. The parameter sharing strategy is to only set the coding parameters of the first coding layer, and the other coding layers are shared according to the coding parameters of the first coding layer. In this way, the setting of different parameter values is reduced, and the amount of manual setting is greatly reduced.
[0075] Step 203, inputting the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data;
[0076] Specifically, the duration predictor is essentially to obtain the timbre characteristics of the real reference speech according to the posterior encoder in the training stage, and then, in combination with the mapping relationship between the timbre characteristics of the real reference speech and the speech text, obtain the phoneme duration corresponding to each text; in the actual prediction stage, according to the text data to be generated for speech, and the selected speech object, combined with the phoneme duration corresponding to each text when the speech object speaks, determine the generated speech duration corresponding to the text data. In order to facilitate the subsequent alignment of the text data and the generated speech duration, it is essentially to align the text encoding length in the encoding stage and the audio decoding output in the decoding stage.
[0077] Step 204, aligning the text encoding information and the generated speech duration and inputting them into a preset audio generation flow model to obtain audio encoding information;
[0078] Specifically, by performing alignment processing in advance, it is ensured that the audio decoding output in the decoding stage is consistent with the text encoding length in the encoding stage, thereby ensuring the generation of high-quality synthesized speech.
[0079] Step 205, performing latent variable adding processing on the audio coding information, inputting the audio coding information with the latent variables added into the improved decoder, and obtaining the speech output by the improved decoder, wherein the latent variables include timbre features.
[0080] Specifically, by adding hidden variables, i.e., complex speech features such as timbre, audio, and pitch, to the audio coding information, the final output speech is made more natural. The improved decoder, because the traditional decoder uses more upsampling convolution to complete the generation of speech, the decoding process of the decoder accounts for more than 70% of the speed of the reasoning process, and the upsampling convolution involves more parameters, which will greatly reduce the decoding efficiency, resulting in low speech generation efficiency. The present application considers introducing a hybrid convolution method that mixes grouped convolution and ordinary convolution, thereby reducing the parameters involved, improving decoding efficiency, and improving speech generation efficiency.
[0081] In this embodiment, the text data to be generated for speech is obtained; the text data is input into an improved text encoder for text encoding processing to obtain text encoding information; the text encoding information is input into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; the text encoding information and the generated speech duration are aligned and input into a preset audio generation flow model to obtain audio encoding information; the audio encoding information is processed by adding hidden variables, and the audio encoding information with added hidden variables is input into an improved decoder to obtain the speech output by the improved decoder. The speech generation method described in the present application improves the end-to-end codecs respectively, which not only reduces the amount of manual model parameter settings, but also improves the model reasoning speed, thereby improving the speech generation efficiency.
[0082] In this embodiment, the improved text encoder is transformed from the initial encoder in the transformer structure, and the encoder includes N encoding layers.
[0083] Continue to refer Figure 3 In some optional implementations, the initial encoder is improved before step 202. Figure 3 It is a flowchart of a specific embodiment of improving the initial encoder in the speech generation method described in the present application, comprising the following steps:
[0084] Step 301, setting the encoding parameters of the first encoding layer in the initial encoder according to a preset parameter setting document;
[0085] Step 302: adopt a parameter sharing strategy to distribute the encoding parameters of the first encoding layer to the remaining N-1 encoding layers in the initial encoder layer by layer to obtain the improved text encoder.
[0086] A parameter sharing strategy is used to set the encoding parameters of the text encoder, which reduces the setting of different parameter values and greatly reduces the amount of manual setting, thereby reducing the computer's processing of the encoding parameters, allowing more processing resources to be used for model reasoning, that is, speech generation, thereby improving speech generation efficiency.
[0087] Continue to refer Figure 4 In some optional implementations, before step 203, the duration prediction knowledge is also acquired. Figure 4 This is a flowchart of a specific embodiment of acquiring duration prediction knowledge in the speech generation method described in the present application, comprising the following steps:
[0088] Step 401, obtaining batches of real reference speech;
[0089] Step 402, performing speech feature extraction on the real reference speech to extract speech features;
[0090] Step 403, inputting the speech feature into a preset posterior encoder to obtain the posterior distribution parameters of the latent variable;
[0091] Step 404, performing sampling processing according to the posterior distribution parameters of the latent variable to obtain the timbre features in the speech features;
[0092] Step 405, converting the real reference speech into text to obtain text data;
[0093] Step 406, inputting the text data into the improved text encoder to obtain text encoding features;
[0094] Step 407, inputting the text encoding feature and the timbre feature into an alignment estimation component to determine a correspondence between each text in the text data and a phoneme;
[0095] Step 408: deploy the correspondence between each text and the phoneme as prediction knowledge into the duration predictor.
[0096] The duration prediction knowledge is acquired through batches of real reference speech, so as to ensure that the generated speech duration predicted by the duration predictor has a higher confidence level during actual prediction.
[0097] In this embodiment, the preset duration predictor includes speaking speed information of at least one real voice object that can be selected. Specifically, the number of the real voice objects is determined by the number of speakers corresponding to the batch of real reference voices, that is, the batch of real reference voices is the speaking voices of 5 people, for example: father, mother, grandfather, grandmother and grandmother, then the real voice objects are father, mother, grandfather, grandmother and grandmother, and the number of the real voice objects is 5. When generating voice, any one of them can be selected as the voice generation object, that is, the text data is output according to the voice of the selected object.
[0098] Continue to refer Figure 5 , in some optional implementations, Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes the following steps:
[0099] Step 501, identifying a pre-selected real voice object;
[0100] Step 502, filtering out the speaking speed information corresponding to the real voice object from the duration predictor, wherein the speaking speed information includes the corresponding relationship between each text and phoneme when the real voice object speaks;
[0101] Step 503: predicting the duration of the generated speech corresponding to the text data according to the text encoding information and the speaking speed information corresponding to the real speech object.
[0102] By identifying a pre-selected real voice object, the speaking speed information corresponding to the real voice object is screened out from the duration predictor, and voice generation is performed according to the selection of the voice generation requester, which is more humane and avoids outputting only mechanical synthesized voice.
[0103] In this embodiment, after executing the step of obtaining the speech output by the improved decoder, the method further includes: obtaining the real speech recorded by the real speech object according to the text data; inputting the real speech and the speech output by the improved decoder into a preset verification model to verify the authenticity of the speech output by the improved decoder compared to the real speech, wherein the verification model includes a wav lm model; if the authenticity does not exceed a preset authenticity threshold, optimizing the encoding parameters of the posterior encoder and the improved text encoder respectively by adversarial learning training until the authenticity exceeds the preset authenticity threshold, and the optimization is completed; if the authenticity exceeds the preset authenticity threshold, outputting the speech output by the improved decoder via a preset speech output device.
[0104] Specifically, the wav lm model is a pre-trained model built on the HuBERT framework, which is specially used to process speech tasks and can perform speech recognition, speaker verification, speech sentiment analysis, etc. Here, its speaker verification function is used to verify the authenticity of the generated speech compared to the real speech object, and the adversarial learning training method is used to optimize the encoding parameters of the posterior encoder and the improved text encoder, so that the generated speech after the final training and actual output is closer to real person speech.
[0105] In this embodiment, after executing the adversarial learning training method to optimize the encoding parameters of the posterior encoder and the improved text encoder respectively until the authenticity exceeds a preset authenticity threshold and the optimization step is completed, the method also includes: obtaining the optimized encoding parameters of the improved text encoder; using a parameter sharing strategy to allocate the encoding parameters of the improved text encoder layer by layer to all encoding layers in the improved text encoder; obtaining the optimized encoding parameters of the posterior encoder; using a parameter sharing strategy to allocate the encoding parameters of the posterior encoder layer by layer to all encoding layers in the posterior encoder.
[0106] Specifically, after the optimization of the speech generation model is completed, the encoding parameters of the optimized improved text encoder are obtained, and the encoding parameters of the optimized posterior encoder are obtained, and a parameter sharing strategy is adopted to perform parameter sharing processing on the corresponding encoding layer, so that when performing subsequent speech generation, the generation model with the best speech output effect and the least encoding parameter value can be used, thereby greatly improving the speech generation efficiency.
[0107] In this embodiment, the improved decoder is transformed from the initial decoder in the transformer structure. Before executing the step of inputting the audio coding information with added hidden variables into the improved decoder, the method also includes: setting the convolution mode of the convolution layer in the initial decoder to a mixed convolution mode of grouped convolution and ordinary convolution to obtain the improved decoder.
[0108] By replacing the traditional upsampling convolution method with a hybrid convolution method that mixes group convolution and ordinary convolution in the decoding stage, compared with the previous convolution method, not only the number of decoding parameters is reduced, but also the model inference speed, that is, the decoding efficiency during speech generation is improved to a certain extent, thereby improving the speech generation efficiency.
[0109] The present application obtains text data to be generated for speech; inputs the text data into an improved text encoder for text encoding processing to obtain text encoding information; inputs the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; aligns the text encoding information and the generated speech duration and inputs them into a preset audio generation flow model to obtain audio encoding information; performs hidden variable addition processing on the audio encoding information, inputs the audio encoding information with added hidden variables into an improved decoder, and obtains the speech output by the improved decoder. The speech generation method described in the present application, by improving the end-to-end codecs respectively, not only reduces the amount of manual model parameter settings, but also increases the model reasoning speed, thereby improving the speech generation efficiency.
[0110] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0111] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0112] In an embodiment of the present application, the text data to be generated for speech is obtained; the text data is input into an improved text encoder for text encoding processing to obtain text encoding information; the text encoding information is input into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; the text encoding information and the generated speech duration are aligned and input into a preset audio generation flow model to obtain audio encoding information; the audio encoding information is processed by adding hidden variables, and the audio encoding information with added hidden variables is input into an improved decoder to obtain the speech output by the improved decoder. The speech generation method described in the present application, by improving the end-to-end codecs respectively, not only reduces the amount of manual model parameter settings, but also increases the model reasoning speed, thereby improving the speech generation efficiency.
[0113] Further references Figure 6 , as a response to the above Figure 2The present application provides an embodiment of a speech generating device, and the embodiment of the device is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0114] like Figure 6 As shown, the speech generation device 600 described in this embodiment includes: a text data acquisition module 601, a text encoding module 602, a speech duration prediction module 603, an audio stream generation module 604 and an audio stream decoding module 605. Among them:
[0115] A text data acquisition module 601 is used to acquire text data to be generated into speech;
[0116] A text encoding module 602, used for inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information;
[0117] The speech duration prediction module 603 is used to input the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data;
[0118] The audio stream generation module 604 is used to align the text encoding information and the generated speech duration and input them into a preset audio generation stream model to obtain audio encoding information;
[0119] The audio stream decoding module 605 is used to perform hidden variable adding processing on the audio coding information, input the audio coding information with hidden variables added into the improved decoder, and obtain the speech output by the improved decoder, wherein the hidden variables include timbre features.
[0120] The present application obtains text data to be generated for speech; inputs the text data into an improved text encoder for text encoding processing to obtain text encoding information; inputs the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; aligns the text encoding information and the generated speech duration and inputs them into a preset audio generation flow model to obtain audio encoding information; performs hidden variable addition processing on the audio encoding information, inputs the audio encoding information with added hidden variables into an improved decoder, and obtains the speech output by the improved decoder. The speech generation method described in the present application, by improving the end-to-end codecs respectively, not only reduces the amount of manual model parameter settings, but also increases the model reasoning speed, thereby improving the speech generation efficiency.
[0121] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0122] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0123] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.
[0124] The computer device 7 includes a memory 7a, a processor 7b, and a network interface 7c which are interconnected through a system bus. It should be noted that: Figure 7 Only a computer device 7 having a component memory 7a, a processor 7b, and a network interface 7c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0125] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0126] The memory 7a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 7a can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 7a can also be an external storage device of the computer device 7, such as a plug-in hard disk equipped on the computer device 7, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 7a can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 7a is generally used to store the operating system and various application software installed on the computer device 7, such as a computer-readable instruction of a speech generation method, etc. In addition, the memory 7a can also be used to temporarily store various types of data that have been output or are to be output.
[0127] The processor 7b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments. The processor 7b is generally used to control the overall operation of the computer device 7. In the present embodiment, the processor 7b is used to run the computer-readable instructions or process data stored in the memory 7a, such as the computer-readable instructions for running the speech generation method.
[0128] The network interface 7c may include a wireless network interface or a wired network interface. The network interface 7c is generally used to establish a communication connection between the computer device 7 and other electronic devices.
[0129] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to the scene of automatic speech generation. This application obtains text data to be generated for speech; inputs the text data into an improved text encoder for text encoding processing to obtain text encoding information; inputs the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; aligns the text encoding information and the generated speech duration and inputs them into a preset audio generation stream model to obtain audio encoding information; performs hidden variable addition processing on the audio encoding information, and inputs the audio encoding information with added hidden variables into an improved decoder to obtain the speech output by the improved decoder. The speech generation method described in this application, by improving the end-to-end codecs respectively, not only reduces the amount of manual model parameter settings, but also improves the model reasoning speed, thereby improving the speech generation efficiency.
[0130] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech generation method as described above.
[0131] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to the automatic speech generation scenario. This application obtains text data to be generated for speech; inputs the text data into an improved text encoder for text encoding processing to obtain text encoding information; inputs the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data; aligns the text encoding information and the generated speech duration and inputs them into a preset audio generation stream model to obtain audio encoding information; performs hidden variable addition processing on the audio encoding information, and inputs the audio encoding information with added hidden variables into an improved decoder to obtain the speech output by the improved decoder. The speech generation method described in this application, by improving the end-to-end codecs respectively, not only reduces the amount of manual model parameter settings, but also increases the model reasoning speed, thereby improving the speech generation efficiency.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0133] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A speech generation method, characterized in that: The steps include: Obtaining text data to be used for speech generation; Inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information; Inputting the text encoding information into a preset duration predictor to predict the duration of the generated speech corresponding to the text data; Aligning the text encoding information with the generated speech duration and inputting the two together into a preset audio generation flow model to obtain audio encoding information; The audio coding information is subjected to a latent variable adding process, and the audio coding information with the latent variable added is input into an improved decoder to obtain speech output by the improved decoder, wherein the latent variable includes timbre features.
2. The speech generation method according to claim 1, characterized in that: The improved text encoder is transformed from an initial encoder in a transformer structure, and the encoder includes N encoding layers. Before executing the step of inputting the text data into the improved text encoder for text encoding processing to obtain text encoding information, the method further includes: According to a preset parameter setting document, setting the encoding parameters of the first encoding layer in the initial encoder; A parameter sharing strategy is adopted to distribute the encoding parameters of the first encoding layer to the remaining N-1 encoding layers in the initial encoder layer by layer, so as to obtain the improved text encoder.
3. The speech generation method according to claim 1 or 2, characterized in that: Before executing the step of inputting the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data, the method further includes: Obtain batches of real reference voices; Performing speech feature extraction on the real reference speech to extract speech features; Inputting the speech features into a preset posterior encoder to obtain posterior distribution parameters of latent variables; Performing sampling processing according to the posterior distribution parameters of the latent variables to obtain timbre features in the speech features; Performing text conversion on the real reference speech to obtain text data; Inputting the text data into the improved text encoder to obtain text encoding features; Inputting the text encoding feature and the timbre feature into an alignment estimation component to determine a correspondence between each text and a phoneme in the text data; The correspondence between each text and the phoneme is deployed as prediction knowledge into the duration predictor.
4. The speech generation method according to claim 3, characterized in that: The preset duration predictor includes the speaking speed information of at least one real voice object that can be selected, and the step of inputting the text encoding information into the preset duration predictor to predict the generated voice duration corresponding to the text data specifically includes: Recognize pre-selected real speech objects; Filtering out the speaking speed information corresponding to the real voice object from the duration predictor, wherein the speaking speed information includes the corresponding relationship between each text and phoneme when the real voice object speaks; The duration of the generated speech corresponding to the text data is predicted based on the text encoding information and the speaking speed information corresponding to the real speech object.
5. The speech generation method according to claim 4, characterized in that: After executing the step of obtaining the speech output by the improved decoder, the method further comprises: Acquire the real voice recorded by the real voice object according to the text data; Inputting the real speech and the speech output by the improved decoder into a preset verification model to verify the authenticity of the speech output by the improved decoder compared with the real speech, wherein the verification model includes a wavlm model; If the authenticity does not exceed the preset authenticity threshold, the encoding parameters of the posterior encoder and the improved text encoder are optimized respectively by using adversarial learning training until the authenticity exceeds the preset authenticity threshold, and the optimization is completed; If the authenticity exceeds a preset authenticity threshold, the speech output by the improved decoder is output via a preset speech output device.
6. The speech generation method according to claim 5, characterized in that: After performing the step of optimizing the encoding parameters of the posterior encoder and the improved text encoder respectively by using the adversarial learning training method until the authenticity exceeds a preset authenticity threshold and the optimization step is completed, the method further includes: Obtaining optimized encoding parameters of the improved text encoder; Using a parameter sharing strategy, the encoding parameters of the improved text encoder are distributed layer by layer to all encoding layers in the improved text encoder; Obtaining optimized encoding parameters of the a posteriori encoder; A parameter sharing strategy is adopted to distribute the encoding parameters of the a posteriori encoder layer by layer to all encoding layers in the a posteriori encoder.
7. The speech generation method according to claim 1, characterized in that: The improved decoder is transformed from an initial decoder in a transformer structure. Before executing the step of inputting the audio coding information with hidden variables added into the improved decoder, the method further includes: The convolution mode of the convolution layer in the initial decoder is set to a mixed convolution mode of grouped convolution and ordinary convolution to obtain the improved decoder.
8. A speech generating device, characterized in that: include: A text data acquisition module, used to acquire text data to be generated for speech; A text encoding module, used for inputting the text data into an improved text encoder for text encoding processing to obtain text encoding information; A speech duration prediction module, used for inputting the text encoding information into a preset duration predictor to predict the generated speech duration corresponding to the text data; An audio stream generation module, used to align the text encoding information and the generated speech duration and then input them into a preset audio generation stream model to obtain audio encoding information; The audio stream decoding module is used to perform hidden variable adding processing on the audio coding information, input the audio coding information with added hidden variables into the improved decoder, and obtain the speech output by the improved decoder, wherein the hidden variables include timbre features.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech generation method according to any one of claims 1 to 7 are implemented.