Voice generation method and device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a voice generation method, which comprises the following steps: acquiring a to-be-converted text; performing acoustic code conversion processing on the to-be-converted text to obtain multiple pieces of speech feature information of the to-be-converted text; calculating the similarity between each piece of voice feature information and each piece of voice codebook feature information of the target voice codebook; determining multiple pieces of target voice codebook feature information corresponding to the to-be-converted text from the target voice codebook based on the similarity; and converting the multiple pieces of target voice codebook feature information into voice by adopting a target decoder to obtain target synthetic voice corresponding to the to-be-converted text. The invention further provides a voice generation device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted text of the user and multiple pieces of voice codebook feature information of the target voice codebook can be stored in a block chain. According to the invention, the naturalness and fluency of speech synthesis are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech generation method, device, equipment and medium. Background Art

[0002] Speech generation technology, namely text-to-speech (TTS) technology, is a cutting-edge artificial intelligence technology. This technology covers multiple links such as speech recognition and speech synthesis, and can realize the conversion from text to sound. Speech generation technology has shown a wide range of application value in many fields. In the field of education, it can provide voice guidance and teaching resources for students and teachers; in the intelligent customer service system, it can simulate real-person voice and provide 24 / 7 automatic customer service; in the field of entertainment and games, it can create unique voices for game characters to enhance user experience.

[0003] The core problem of speech generation technology is how to generate natural, fluent and expressive speech. The speech generated by existing speech generation systems often lacks naturalness and expressiveness, cannot accurately convey the speaker's emotions and tone, and is difficult to achieve fine control of speech style, timbre, etc., and cannot meet the needs of personalized speech synthesis. Therefore, it is urgent to develop a more intelligent and natural speech generation technology. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech generation method, device, equipment and medium to solve the problem that the speech generated by the existing speech generation system lacks naturalness and expressiveness, cannot accurately convey the speaker's emotions and tone, is difficult to achieve fine control of speech style, timbre, etc., and cannot meet the needs of personalized speech synthesis.

[0005] In order to solve the above technical problems, the present application provides a method for generating speech, which adopts the following technical solutions:

[0006] The text to be converted is obtained, and the text to be converted is used to be converted into a target synthesized speech; the text to be converted is subjected to voice coding conversion processing to obtain a plurality of speech feature information of the text to be converted; the similarity between each speech feature information and various speech codebook feature information of a target speech codebook is calculated, and the target speech codebook is a speech codebook obtained based on daily speech processing; based on the similarity, a plurality of target speech codebook feature information corresponding to the text to be converted is determined from the target speech codebook; a target decoder is used to convert the plurality of target speech codebook feature information into speech, and a target synthesized speech corresponding to the text to be converted is obtained, and the target decoder is a decoder obtained based on daily speech training.

[0007] Further, the step of performing voice coding conversion on the text to be converted to obtain a plurality of speech feature information of the text to be converted specifically includes:

[0008] Extract multiple text feature information of the text to be converted; input the multiple text feature information into a preset text decoder for decoding to obtain multiple voice feature information of the text to be converted.

[0009] Furthermore, the step of extracting multiple text feature information of the text to be converted specifically includes:

[0010] The text to be converted is preprocessed to obtain multiple text segmentations, and each text segmentation is converted into a feature vector based on a preset vocabulary; the feature vector is input into a preset text encoder to obtain multiple text feature information.

[0011] Furthermore, before the step of calculating the similarity between each speech feature information and various speech codebook feature information of the target speech codebook, the method further includes:

[0012] Acquire daily speech, input the daily speech into a preset encoder, and obtain multiple speech latent variable information; determine target codebook feature information representing multiple speech latent variable information from a preset reference codebook; input the target codebook feature information into a preset decoder, obtain synthetic speech corresponding to the daily speech, and calculate the loss function based on the daily speech and the synthetic speech; adjust the encoding parameters of the encoder and the decoding parameters of the decoder based on the loss function, return to execute the step of inputting daily speech into the preset encoder to obtain multiple speech latent variable information, until the loss function converges to a preset convergence threshold; determine the encoder and decoder when the loss function converges to the preset convergence threshold as the target encoder and target decoder, and determine the multiple speech latent variable information when the loss function converges to the preset convergence threshold as the target speech latent variable information; construct a target speech codebook based on the target speech latent variable information.

[0013] Furthermore, the step of determining target codebook feature information representing multiple speech latent variable information from a preset reference codebook specifically includes:

[0014] The information similarity between the multiple speech latent variable information and the codebook feature information of each dimension of the preset reference codebook is calculated to obtain multiple similarity results; based on the similarity results, the target codebook feature information representing the multiple speech latent variable information is determined from the reference codebook.

[0015] Furthermore, based on daily speech and synthesized speech, the steps of calculating the loss function specifically include:

[0016] The synthesized speech and daily speech are input into the preset discriminator to obtain the difference value between the synthesized speech and the daily speech; the difference value is used as the adversarial loss of the encoder and decoder to calculate the mean square error between the speech signal of the synthesized speech and the speech signal of the daily speech; the mean square error is added to the adversarial loss to obtain the loss function of the encoder and decoder.

[0017] Furthermore, based on the target speech latent variable information, the step of constructing the target speech codebook specifically includes:

[0018] The target speech latent variable information is clustered to obtain multiple speech feature clusters; the speech feature information of the center point of each speech feature cluster is obtained, and the target speech codebook is constructed based on the speech feature information.

[0019] Further, the step of calculating the similarity between each speech feature information and various speech codebook feature information of the target speech codebook specifically includes:

[0020] The cosine distance between each speech feature information and multiple speech codebook feature information of the target speech codebook is calculated; and based on the cosine distance, the similarity between each speech feature information and multiple speech codebook feature information is determined.

[0021] Further, a target decoder is used to convert multiple target speech codebook feature information into speech to obtain a target synthesized speech corresponding to the text to be converted, specifically including:

[0022] The acoustic model of the target decoder is used to perform feature mapping processing on the feature information of multiple target speech codebooks to obtain an acoustic feature sequence; the vocoder of the target decoder is used to convert the acoustic feature sequence into a speech waveform to obtain multiple speech waveform data; the target decoder is used to splice the multiple speech waveform data to generate a speech signal, and the speech corresponding to the speech signal is output to obtain the target synthesized speech corresponding to the text to be converted.

[0023] In order to solve the above technical problems, the embodiment of the present application further provides a speech generation device, which adopts the following technical solution:

[0024] An acquisition module is used to acquire the text to be converted, and the text to be converted is used to convert it into a target synthesized speech;

[0025] The voice code conversion module is used to perform voice code conversion on the text to be converted to obtain multiple voice feature information of the text to be converted;

[0026] A calculation module, used for calculating the similarity between each speech feature information and various speech codebook feature information of a target speech codebook, where the target speech codebook is a speech codebook obtained based on daily speech processing;

[0027] A first determination module is used to determine multiple target speech codebook feature information corresponding to the to-be-converted text from the target speech codebook based on the similarity;

[0028] The speech conversion module is used to convert multiple target speech codebook feature information into speech using a target decoder to obtain a target synthesized speech corresponding to the text to be converted. The target decoder is a decoder obtained based on daily speech training.

[0029] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned speech generation method are implemented.

[0030] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above-mentioned speech generation method.

[0031] Compared with the prior art, the embodiment of the present application has the following beneficial effects: by extracting multiple speech feature information of the text to be converted and finely matching it with the feature information in the target speech codebook, it can ensure that the generated synthesized speech is closer to natural speech in speech features such as pitch, length, and intensity, thereby improving the naturalness and fluency of speech synthesis. Using the target speech codebook and target decoder obtained from daily speech training, the speech features of a specific speaker can be modeled and synthesized to achieve the generation of personalized speech. This is of great significance for application scenarios that require simulating the voice of a specific person, such as virtual character dubbing, personalized customization of voice assistants, etc. By calculating the similarity between the speech feature information and the feature information of the target speech codebook, and selecting the corresponding feature information for synthesis based on the similarity, it can ensure that the generated speech is highly consistent with the original text to be converted in content, while improving the accuracy and efficiency of speech synthesis. This speech generation scheme has broad application prospects in many fields such as medical care, education, and entertainment. For example, in the medical field, it can be used to generate patient education materials and assist doctors in remote diagnosis and treatment; in the education field, it can be used to produce audiobooks and provide personalized learning guidance; in the entertainment field, it can be used for game character dubbing and virtual idol voice generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0034] Figure 2 It is a flowchart of a speech generation method provided by the present application;

[0035] Figure 3 This is a schematic diagram of the process of converting text to speech feature information of a speech generation method provided by the present application;

[0036] Figure 4 It is a schematic diagram of the training process of a speech synthesis system of a speech generation method provided in the present application;

[0037] Figure 5 It is a structural schematic diagram of a speech generating device provided by the present application;

[0038] Figure 6 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0043] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0044] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0045] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0046] It should be noted that the speech generation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the speech generation device is generally arranged in the server.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0048] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech generation method according to the present application. The speech generation method comprises the following steps:

[0049] Step S201, obtaining the text to be converted.

[0050] In this embodiment, the electronic device (eg, Figure 1The server shown in the figure) can obtain the text to be converted through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0051] The text to be converted refers to the original text content that needs to be converted into speech. These text contents can be any form of text, such as sentences, paragraphs, articles or dialogues. For example, chat conversations on social media, texts in instant messages, the body of personal or work emails, chapters or paragraphs of textbooks in various subjects (such as mathematics, science, history, literature, etc.), press releases from news websites, newspapers or TV broadcasts, detailed introductions of products on e-commerce websites, novels, biographies, and popular science books.

[0052] Step S202: Perform voice coding conversion on the text to be converted to obtain a plurality of speech feature information of the text to be converted.

[0053] The voice code conversion process refers to converting the text to be converted into voice feature information. The specific process may be: extracting multiple text feature information of the text to be converted, decoding the text feature information, and obtaining multiple voice feature information.

[0054] Among them, text feature information refers to the features extracted from the original text to be converted that can represent the content or attributes of the text. Text feature information is usually used to guide the speech synthesis process to ensure that the generated speech can accurately reflect the meaning and context of the text.

[0055] The speech feature information refers to the features that are obtained based on the text feature information and can describe the acoustic characteristics of the speech. These speech feature information are used in the speech synthesis process to generate speech output with specific sound quality, pitch and rhythm.

[0056] In one example, if Figure 3 As shown, the text to be converted can be obtained first, for example, a passage of article content that needs to be converted into speech. Then, the text to be converted is subjected to voice coding conversion processing. For example, the text to be converted can be preprocessed first to obtain multiple text segmentations, and then each text segmentation is converted into a feature vector based on a preset vocabulary, and then the feature vector is input into a preset text encoder to obtain multiple text feature information. Finally, the multiple text feature information can be input into a preset text decoder to obtain multiple speech feature information of the text to be converted.

[0057] Step S203, calculating the similarity between each speech feature information and various speech codebook feature information of a target speech codebook, where the target speech codebook is a speech codebook obtained based on daily speech processing.

[0058] The target speech codebook includes statistical information of various speech features. The similarity between each speech feature information and the speech codebook feature information in the target speech codebook is calculated. Specifically, a measurement method such as Euclidean distance and cosine similarity can be used to calculate the similarity between each speech feature information and the speech codebook feature information in the target speech codebook, so as to determine the target speech codebook feature that can best represent the multiple speech feature information based on the calculated similarity.

[0059] Among them, daily speech is the speech that people frequently use in daily life and work for daily communication. Daily speech covers many aspects such as family, friends, work, and public places, and is natural, informal, and colloquial. For example, daily communication between family members, and voice communication between people in public places such as shopping malls, supermarkets, and restaurants.

[0060] Step S204: based on the similarity, determine multiple target speech codebook feature information corresponding to the text to be converted from the target speech codebook.

[0061] The target speech codebook feature information refers to the speech codebook feature information in the target speech codebook that can best represent multiple speech feature information.

[0062] In one example, the similarity between each speech feature information and various speech codebook feature information of the target speech codebook can be calculated, for example, by calculating the cosine distance between each speech feature information and multiple speech codebook feature information of the target speech codebook, to determine the similarity between each speech feature information and multiple speech codebook feature information of the target speech codebook. The speech codebook feature information corresponding to the highest similarity is taken as the target speech codebook feature information that best represents the speech feature information.

[0063] Step S205, using a target decoder to convert multiple target speech codebook feature information into speech, and obtain a target synthesized speech corresponding to the text to be converted, and the target decoder is a decoder obtained based on daily speech training.

[0064] The target decoder refers to the decoder component used in personalized speech generation. The role of the target decoder is to convert feature information extracted from text or other forms of input into speech output with personalized features of a specific speaker.

[0065] In one example, a target decoder can be used to convert the selected target speech codebook feature information into a speech signal, and synthesize the target synthesized speech corresponding to the text content to be converted. Among them, the target encoder used is trained based on a large amount of daily speech data, and can better model the conversion relationship from speech features to speech signals. Through the above steps, any text can be converted into fluent and natural speech, which can be applied to scenarios such as speech synthesis and audio books.

[0066] The embodiment of the present application can ensure that the generated synthesized speech is closer to natural speech in speech features such as pitch, length, and intensity by extracting multiple speech feature information of the text to be converted and finely matching it with the feature information in the target speech codebook, thereby improving the naturalness and fluency of speech synthesis. Using the target speech codebook and target decoder obtained by daily speech training, the speech features of a specific speaker can be modeled and synthesized to achieve the generation of personalized speech. This is of great significance for application scenarios that require simulating the voice of a specific person, such as virtual character dubbing, personalized customization of voice assistants, etc. By calculating the similarity between the speech feature information and the feature information of the target speech codebook, and selecting the corresponding feature information for synthesis based on the similarity, it can ensure that the generated speech is highly consistent with the original text to be converted in content, while improving the accuracy and efficiency of speech synthesis. The speech generation scheme has broad application prospects in multiple fields such as medical care, education, and entertainment. For example, in the medical field, it can be used to generate patient education materials, assist doctors in remote diagnosis and treatment, etc.; in the education field, it can be used to produce audio books, provide personalized learning guidance, etc.; in the entertainment field, it can be used for game character dubbing, virtual idol voice generation, etc.

[0067] In some optional implementations of this embodiment, in step 202, the text to be converted is subjected to voice coding conversion processing to obtain a plurality of speech feature information of the text to be converted, which specifically includes the following steps:

[0068] Extracting multiple text feature information of the text to be converted;

[0069] The plurality of text feature information is input into a preset text decoder for decoding to obtain a plurality of speech feature information of the text to be converted.

[0070] In this embodiment, for a piece of text to be converted into speech, multiple feature information of the text to be converted, such as part of speech, syntactic structure, semantic information, etc., can be extracted, and natural language processing technologies such as word segmentation, part of speech tagging, syntactic analysis, etc. can be used. Then, these text feature information can be input into a preset text decoder (such as a preset 6-layer Transformer decoder), and the speech features corresponding to the text can be decoded and generated through the attention mechanism and feedforward neural network, so that multiple speech feature information can be obtained.

[0071] In the embodiments of the present application, the text feature information often contains the key elements of the content of the text to be converted, such as semantics, grammar, syntax, etc. By accurately extracting these text features and decoding them to convert them into speech feature information, it is possible to ensure that the generated speech is highly consistent with the original text to be converted in terms of content, thereby improving the accuracy of speech synthesis. Text features not only contain the literal meaning of the text, but may also imply non-literal information such as emotions and tone. By deeply analyzing these text features and decoding them into speech features, the emotions and tone of the original text can be better restored in speech synthesis, making the generated speech more expressive. Compared with direct conversion from text to speech, using text feature information as input for decoding can utilize more efficient algorithms and models to process text features, thereby optimizing the overall efficiency of speech synthesis.

[0072] In some optional implementations of this embodiment, the above step of “extracting multiple text feature information of the text to be converted” specifically includes the following steps:

[0073] Preprocess the text to be converted to obtain multiple text segmentations, and convert each text segmentation into a feature vector based on a preset vocabulary;

[0074] The feature vector is input into a preset text encoder to obtain multiple text feature information.

[0075] In the embodiment of the present application, the text to be converted can be preprocessed first, for example, a word segmentation tool can be used to segment the text, and part-of-speech tagging and named entity recognition can be performed on the text after segmentation based on a preset vocabulary to extract multiple text feature information such as part-of-speech and named entities of the text. Then the extracted text feature information is input into a preset text decoder for decoding, and finally the speech feature information corresponding to the text to be converted can be output, providing the necessary conditions for subsequent speech synthesis.

[0076] The embodiment of the present application converts the text into a feature vector by segmenting the text, and each segment is converted into a numerical and standardized representation. This representation can better capture the semantic information in the text, improve the representation ability of the text information, and provide a richer and more accurate information basis for subsequent speech synthesis. Compared with directly using the original text to be converted, converting the text to be converted into a feature vector and inputting it into a text encoder can significantly reduce the computational complexity. The feature vector usually has a fixed dimension and size, making the calculation process more efficient and controllable. By using a preset vocabulary and text encoder, texts in different fields can be uniformly processed and represented. Through the refined text feature extraction and encoding process, it can be ensured that the generated text feature information is more accurate and comprehensive. These accurate text feature information, as the input of speech synthesis, can guide the speech synthesis system to generate more accurate and original speech. Different text feature information can correspond to different speech feature information. By adjusting the extraction and encoding method of text feature information, fine control of text style, emotion, etc. can be achieved, thereby supporting the needs of personalized speech synthesis.

[0077] In some optional implementations of this embodiment, before calculating the similarity between each speech feature information and various speech codebook feature information of the target speech codebook in step S203, the method further includes the following steps:

[0078] Acquire daily speech, input the daily speech into a preset encoder, and obtain multiple speech latent variable information; determine target codebook feature information representing multiple speech latent variable information from a preset reference codebook; input the target codebook feature information into a preset decoder, obtain synthetic speech corresponding to the daily speech, and calculate the loss function based on the daily speech and the synthetic speech; adjust the encoding parameters of the encoder and the decoding parameters of the decoder based on the loss function, return to execute the step of inputting daily speech into the preset encoder to obtain multiple speech latent variable information, until the loss function converges to a preset convergence threshold; determine the encoder and decoder when the loss function converges to the preset convergence threshold as the target encoder and target decoder, and determine the multiple speech latent variable information when the loss function converges to the preset convergence threshold as the target speech latent variable information; construct a target speech codebook based on the target speech latent variable information.

[0079] In one example, if Figure 4As shown, before converting the text to be converted into the target synthesized speech, daily speech can be obtained first. For example, 100 hours of daily conversation speech data is collected. Then, the obtained daily speech is input into a preset encoder, for example, a 6-layer, 512-dimensional Transformer encoder, and multiple speech latent variable information can be output. On this basis, from a reference codebook, for example, from a reference codebook containing 10,000 speech features, the minimum Euclidean distance algorithm is used to find multiple features closest to the speech latent variable information as the target codebook feature information. Then these target codebook features are input into a preset decoder, for example, a 6-layer, 1024-dimensional Transformer decoder, and the corresponding speech waveform is decoded and output through the attention mechanism and the feedforward network, and the synthesized speech is output. During the training process, the parameters of the encoder and the decoder can be adjusted according to the difference between daily speech and synthesized speech, and the step of inputting daily speech into the preset encoder to obtain multiple speech latent variable information is returned until the loss value indicated by the loss function converges to a preset convergence threshold, for example, 50 epochs can be iterated until the loss converges to below 0.1. Among them, epoch refers to the number of times the entire training data set (daily speech) is completely traversed by the model during the training process. Each time the model is traversed, it will update the parameters according to the samples in the data set in the hope of reducing the prediction error (i.e. loss). Simply put, epoch is the number of times the model learns the complete knowledge of the data set. It can also be understood as the number of steps in which daily speech is input into the preset encoder to obtain multiple speech hidden variable information. Finally, the trained target encoder and decoder, as well as the target speech codebook are obtained.

[0080] The daily speech in the embodiment of the present application contains rich natural speech features, such as intonation, speech speed, pauses, emotions, etc. Using daily speech for training can enable the model to learn the personalized speech features of each speaker, and generate a speech that is highly similar to the speaker when converting text to speech, meeting the needs of personalized speech synthesis. By continuously adjusting the parameters of the encoder and decoder, the loss function between the generated synthetic speech and the original daily speech gradually converges until a preset convergence threshold is reached. This process ensures that the encoder and decoder can accurately capture and reproduce the key features of the speech, thereby improving the quality of speech synthesis and making the generated speech more natural and fluent. Through the target encoder, decoder and target speech codebook obtained by training, an efficient speech synthesis system (speech synthesis model) can be constructed. When the text to be converted is converted to speech, the speech synthesis system can directly use these pre-trained components for rapid synthesis without the need to re-calculate and optimize complex calculations, thereby improving the efficiency of speech synthesis.

[0081] In some optional implementations of this embodiment, the above step of “determining target codebook feature information representing multiple speech latent variable information from a preset reference codebook” specifically includes the following steps:

[0082] The information similarity between the multiple speech latent variable information and the codebook feature information of each dimension of the preset reference codebook is calculated to obtain multiple similarity results; based on the similarity results, the target codebook feature information representing the multiple speech latent variable information is determined from the reference codebook.

[0083] In an embodiment of the present application, the cosine distance between each speech latent variable information and the codebook feature information of each dimension of the reference codebook can be calculated. The information similarity between each speech latent variable information and the codebook feature information of each dimension of the reference codebook is determined based on the cosine distance. Based on the information similarity, the target codebook feature information that best represents the multiple speech latent variable information is determined from the reference codebook.

[0084] The embodiment of the present application can more accurately evaluate which codebook feature information can better characterize the current speech latent variable information by calculating the similarity between the speech latent variable information and the codebook feature information of each dimension in the reference codebook. This method makes selections based on similarity, which can ensure that the selected codebook feature information and the speech latent variable information have a high degree of correlation and consistency, thereby improving the accuracy of the codebook characterization. The reference codebook usually contains a large amount of codebook feature information, but not all codebook feature information will be used to characterize each speech latent variable information. By determining the target codebook feature information through similarity calculation, the use of unnecessary codebook feature information can be avoided, thereby optimizing the efficiency of codebook use.

[0085] In some optional implementations of this embodiment, the above step of “calculating the loss function based on daily speech and synthesized speech” specifically includes the following steps:

[0086] The synthesized speech and daily speech are input into the preset discriminator to obtain the difference value between the synthesized speech and the daily speech; the difference value is used as the adversarial loss of the encoder and decoder to calculate the mean square error between the speech signal of the synthesized speech and the speech signal of the daily speech; the mean square error is added to the adversarial loss to obtain the loss function of the encoder and decoder.

[0087] In an embodiment of the present application, for daily speech and generated synthetic speech, a pre-trained discriminator can be used to distinguish the authenticity of the generated synthetic speech. Specifically, the synthetic speech and daily speech signals can be input into a preset discriminator, and the difference value between the synthetic speech and the daily speech signal is calculated by the discriminator, such as JS divergence (i.e. Jensen-Shannon divergence, which is a metric for measuring the similarity of two probability distributions). JS divergence is used as the adversarial loss of the encoder and decoder. The mean square error between the speech signal of the synthetic speech and the speech signal of the daily speech is calculated. The mean square error is added to the adversarial loss to obtain the total loss function of the encoder and decoder. According to the total loss function, the parameters of the encoder and decoder are optimized and trained using a gradient descent algorithm. Iterate multiple epochs until the loss converges to below the convergence threshold. Finally, the trained target encoder and decoder, as well as the target speech codebook, can be obtained.

[0088] In an embodiment of the present application, the discriminator can be trained to distinguish between real speech and synthesized speech, or to distinguish between synthesized speech of different qualities. During the training process, the discriminator will learn the characteristics and patterns of real speech, and react sensitively to defects or deficiencies in the synthesized speech. Therefore, incorporating the distinction results of the discriminator into the calculation of the loss function can prompt the model to pay more attention to these defects and deficiencies during the training process, thereby improving the robustness and generalization ability of the model. By using a discriminator to distinguish synthesized speech, the performance of the synthesized speech in terms of speech characteristics, sound quality, naturalness, etc. can be evaluated more carefully, thereby improving the evaluation accuracy of the speech synthesis quality.

[0089] In some optional implementations of this embodiment, the above step of “constructing a target speech codebook based on target speech latent variable information” specifically includes the following steps:

[0090] Clustering the latent variable information of the target speech to obtain multiple speech feature clusters;

[0091] The speech feature information of the center point of each speech feature cluster is obtained, and a target speech codebook is constructed based on the speech feature information.

[0092] In an embodiment of the present application, the target speech latent variable information can be clustered into several speech feature clusters through the K-means clustering algorithm, and the center point of each speech feature cluster can be used as new speech codebook feature information, and these new speech codebook feature information can be constructed into the target speech codebook.

[0093] In the embodiment of the present application, by clustering the target speech latent variable information, similar speech features can be classified into one category, thereby reducing redundant information in the data. The speech feature information (such as the feature vector of the center point) of each speech feature cluster can represent the common features of all speech latent variable information in the cluster, and the target speech codebook is constructed according to these speech feature information, which avoids repeated storage and processing of similar information and improves the efficiency of data processing. The feature vector of the center point of each speech feature cluster after clustering is the best representative of all speech latent variable information in the cluster. These center point feature vectors (i.e., the speech feature information mentioned above in the present embodiment) can be used as codebook vectors (i.e., the new speech codebook feature information mentioned above in the present embodiment) to construct the target speech codebook, which can ensure that the target speech codebook has a high degree of representativeness and can cover and characterize most of the speech feature information in the training data.

[0094] In some optional implementations of this embodiment, the above step of “calculating the similarity between each speech feature information and various speech codebook feature information of the target speech codebook” specifically includes the following steps:

[0095] The cosine distance between each speech feature information and multiple speech codebook feature information of the target speech codebook is calculated; and based on the cosine distance, the similarity between each speech feature information and multiple speech codebook feature information is determined.

[0096] In the embodiment of the present application, for each speech feature information of the text to be converted, the cosine similarity is used to calculate the cosine distance between the speech feature vector indicated by each speech feature information and the speech codebook feature vector indicated by multiple speech codebook feature information of the target speech codebook, so as to determine the similarity between each speech feature information and multiple speech codebook feature information. Then, multiple speech codebook feature information with the highest similarity is selected as the input of the target decoder, and finally a fluent and natural speech that is highly consistent with the content and emotion of the original text to be converted is synthesized.

[0097] In the embodiment of the present application, in speech processing, speech feature information is usually represented in the form of vectors, which contain multiple features of speech (such as frequency, energy, duration, etc.). By calculating the cosine distance between two vectors, the similarity between them can be accurately measured, not just based on their size or length. In the speech synthesis process, selecting the speech codebook feature information with the highest similarity to the text is crucial for generating natural and fluent speech. Determining the similarity by cosine distance can ensure that the selected speech codebook feature information is highly consistent with the text content to be converted in terms of semantics, intonation, speaking speed, etc., thereby improving the naturalness and intelligibility of speech synthesis.

[0098] In some optional implementations of this embodiment, the above step of "using a target decoder to convert multiple target speech codebook feature information into speech to obtain a target synthesized speech corresponding to the text to be converted" specifically includes the following steps:

[0099] The acoustic model of the target decoder is used to perform feature mapping processing on the feature information of multiple target speech codebooks to obtain an acoustic feature sequence; the vocoder of the target decoder is used to convert the acoustic feature sequence into a speech waveform to obtain multiple speech waveform data; the target decoder is used to splice the multiple speech waveform data to generate a speech signal, and the speech corresponding to the speech signal is output to obtain the target synthesized speech corresponding to the text to be converted.

[0100] In an embodiment of the present application, the target decoder may include an acoustic model and a vocoder. The interior of the target decoder may generally include one or more acoustic models, which are used to map feature vectors to acoustic features. The acoustic model may be based on statistical parameters (such as a hidden Markov model HMM) or may be based on deep learning (such as a neural network). The vocoder is a core component in the decoder, which is responsible for converting an acoustic feature sequence into a speech waveform. When a plurality of target speech codebook feature information is input into the target decoder, the acoustic model of the target decoder may map these target speech codebook feature information into an acoustic feature sequence, and then the acoustic feature sequence is converted into a speech waveform by the vocoder of the target decoder. Finally, the generated multiple speech waveform data are spliced ​​to output a fluent and natural speech that is highly consistent with the original text content and emotion.

[0101] The embodiment of the present application performs feature mapping processing on the target speech codebook feature information by using the acoustic model of the target decoder to obtain an acoustic feature sequence. This step ensures the accurate conversion from codebook features to acoustic features, and provides high-quality acoustic parameters for subsequent speech synthesis. It is possible to retain the key information in the codebook features and convert them into acoustic features that can be used for speech synthesis, thereby improving the accuracy of speech synthesis. The acoustic feature sequence is input into the vocoder of the target decoder and converted into speech waveform data. The vocoder is a key component in the speech synthesis system, which is responsible for converting acoustic features into audible speech waveforms. The beneficial effect of this step is that high-quality speech waveforms can be generated, and these waveforms can be highly consistent with the original speech in terms of sound quality, intonation, speech speed, etc., thereby improving the naturalness and fluency of speech synthesis. Use the target decoder to splice multiple speech waveform data to generate a complete speech signal. In the speech synthesis process, since the text may contain multiple words, phrases or sentences, it is necessary to splice these independent speech waveforms to form a continuous speech signal. This step ensures that the speech signal maintains a smooth transition at the splicing point, avoiding abrupt breakpoints and unnatural pauses, thereby improving the coherence and expressiveness of speech synthesis. Since the target decoder has been trained with daily speech, it can capture the speaker's unique speech features, such as intonation, speaking speed, timbre, etc., and preserve and reproduce them during the synthesis process. This allows the generated synthetic speech to not only accurately convey the content of the text, but also fully demonstrate the speaker's personality and emotions, meeting the needs of personalized speech synthesis.

[0102] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned text to be converted and the multiple voice codebook feature information of the target voice codebook, the above-mentioned text to be converted and the multiple voice codebook feature information of the target voice codebook can also be stored in a node of a blockchain.

[0103] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0104] The embodiment of the present application can acquire and process the feature information of multiple speech codebooks of the text to be converted and the target speech codebook based on the relevant algorithms of the natural language processing technology (NLP) under artificial intelligence. Among them, artificial intelligence (AI) is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0105] Natural language processing technology is a key branch in the field of artificial intelligence. It focuses on giving computers the ability to understand and process human language, so that they can parse, understand and generate natural language text like humans. By simulating some core mechanisms of human language processing, NLP technology can deeply process, analyze and interpret text data, extract key information from it, understand semantic content, and perform complex tasks or generate natural language responses based on it. Natural language processing technology can be used to extract multiple voice feature information of the text to be converted, generate synthetic speech of the text to be converted, etc.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0107] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0108] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a speech generating device, and the embodiment of the device is Figure 2Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 5 As shown, the speech generation device 400 of this embodiment includes: an acquisition module 401, a voice code conversion module 402, a calculation module 403, a first determination module 404 and a speech conversion module 405. Among them:

[0110] An acquisition module 401 is used to acquire a text to be converted, and the text to be converted is used to be converted into a target synthesized speech;

[0111] The voice code conversion module 402 is used to perform voice code conversion on the text to be converted to obtain a plurality of speech feature information of the text to be converted;

[0112] A calculation module 403 is used to calculate the similarity between each speech feature information and various speech codebook feature information of a target speech codebook, where the target speech codebook is a speech codebook obtained based on daily speech processing;

[0113] A first determination module 404 is used to determine, based on the similarity, a plurality of target speech codebook feature information corresponding to the to-be-converted text from the target speech codebook;

[0114] The speech conversion module 405 is used to convert multiple target speech codebook feature information into speech using a target decoder to obtain a target synthesized speech corresponding to the text to be converted. The target decoder is a decoder obtained based on daily speech training.

[0115] The embodiment of the present application can ensure that the generated synthesized speech is closer to natural speech in speech features such as pitch, length, and intensity by extracting multiple speech feature information of the text to be converted and finely matching it with the feature information in the target speech codebook, thereby improving the naturalness and fluency of speech synthesis. Using the target speech codebook and target decoder obtained by daily speech training, the speech features of a specific speaker can be modeled and synthesized to achieve the generation of personalized speech. This is of great significance for application scenarios that require simulating the voice of a specific person, such as virtual character dubbing, personalized customization of voice assistants, etc. By calculating the similarity between the speech feature information and the feature information of the target speech codebook, and selecting the corresponding feature information for synthesis based on the similarity, it can ensure that the generated speech is highly consistent with the original text to be converted in content, while improving the accuracy and efficiency of speech synthesis. The speech generation scheme has broad application prospects in multiple fields such as medical care, education, and entertainment. For example, in the medical field, it can be used to generate patient education materials, assist doctors in remote diagnosis and treatment, etc.; in the education field, it can be used to produce audio books, provide personalized learning guidance, etc.; in the entertainment field, it can be used for game character dubbing, virtual idol voice generation, etc.

[0116] In one embodiment, the voice coding module 402 includes:

[0117] An extraction submodule, used for extracting multiple text feature information of the text to be converted;

[0118] The decoding submodule is used to input multiple text feature information into a preset text decoder for decoding to obtain multiple voice feature information of the text to be converted.

[0119] In the embodiments of the present application, the text feature information often contains the key elements of the content of the text to be converted, such as semantics, grammar, syntax, etc. By accurately extracting these text features and decoding them to convert them into speech feature information, it is possible to ensure that the generated speech is highly consistent with the original text to be converted in terms of content, thereby improving the accuracy of speech synthesis. Text features not only contain the literal meaning of the text, but may also imply non-literal information such as emotions and tone. By deeply analyzing these text features and decoding them into speech features, the emotions and tone of the original text can be better restored in speech synthesis, making the generated speech more expressive. Compared with direct conversion from text to speech, using text feature information as input for decoding can utilize more efficient algorithms and models to process text features, thereby optimizing the overall efficiency of speech synthesis.

[0120] In one embodiment, the extraction submodule is also used to preprocess the text to be converted to obtain multiple text segmentations, and convert each text segmentation into a feature vector based on a preset vocabulary; the feature vector is input into a preset text encoder to obtain multiple text feature information.

[0121] The embodiment of the present application converts the text into a feature vector by segmenting the text, and each segment is converted into a numerical and standardized representation. This representation can better capture the semantic information in the text, improve the representation ability of the text information, and provide a richer and more accurate information basis for subsequent speech synthesis. Compared with directly using the original text to be converted, converting the text to be converted into a feature vector and inputting it into a text encoder can significantly reduce the computational complexity. The feature vector usually has a fixed dimension and size, making the calculation process more efficient and controllable. By using a preset vocabulary and text encoder, texts in different fields can be uniformly processed and represented. Through the refined text feature extraction and encoding process, it can be ensured that the generated text feature information is more accurate and comprehensive. These accurate text feature information, as the input of speech synthesis, can guide the speech synthesis system to generate more accurate and original speech. Different text feature information can correspond to different speech feature information. By adjusting the extraction and encoding method of text feature information, fine control of text style, emotion, etc. can be achieved, thereby supporting the needs of personalized speech synthesis.

[0122] In one embodiment, the speech generating device may further include:

[0123] The speech acquisition module is used to acquire daily speech, input the daily speech into a preset encoder, and obtain multiple speech latent variable information;

[0124] A second determination module is used to determine target codebook feature information representing multiple speech latent variable information from a preset reference codebook;

[0125] An input module is used to input the target codebook feature information into a preset decoder to obtain a synthesized speech corresponding to the daily speech, and calculate a loss function based on the daily speech and the synthesized speech;

[0126] An adjustment module, used for adjusting the encoding parameters of the encoder and the decoding parameters of the decoder based on the loss function, and returning to execute the step of inputting daily speech into a preset encoder to obtain multiple speech latent variable information until the loss function converges to a preset convergence threshold;

[0127] A third determination module determines the encoder and decoder when the loss function converges to a preset convergence threshold as the target encoder and target decoder, and determines the multiple speech latent variable information when the loss function converges to a preset convergence threshold as the target speech latent variable information;

[0128] The construction module is used to construct a target speech codebook based on the target speech latent variable information.

[0129] The daily speech in the embodiment of the present application contains rich natural speech features, such as intonation, speech speed, pauses, emotions, etc. Using daily speech for training can enable the model to learn the personalized speech features of each speaker, and generate a speech that is highly similar to the speaker when converting text to speech, meeting the needs of personalized speech synthesis. By continuously adjusting the parameters of the encoder and decoder, the loss function between the generated synthetic speech and the original daily speech gradually converges until a preset convergence threshold is reached. This process ensures that the encoder and decoder can accurately capture and reproduce the key features of the speech, thereby improving the quality of speech synthesis and making the generated speech more natural and fluent. Through the target encoder, decoder and target speech codebook obtained by training, an efficient speech synthesis system (speech synthesis model) can be constructed. When the text to be converted is converted to speech, the speech synthesis system can directly use these pre-trained components for rapid synthesis without the need to re-calculate and optimize complex calculations, thereby improving the efficiency of speech synthesis.

[0130] In one embodiment, the second determining module includes:

[0131] A first calculation submodule is used to calculate the information similarity between the multiple speech latent variable information and the codebook feature information of each dimension of the preset reference codebook to obtain multiple similarity results;

[0132] The first determination submodule is used to determine target codebook feature information representing multiple speech latent variable information from a reference codebook based on the similarity result.

[0133] The embodiment of the present application can more accurately evaluate which codebook feature information can better characterize the current speech latent variable information by calculating the similarity between the speech latent variable information and the codebook feature information of each dimension in the reference codebook. This method makes selections based on similarity, which can ensure that the selected codebook feature information and the speech latent variable information have a high degree of correlation and consistency, thereby improving the accuracy of the codebook characterization. The reference codebook usually contains a large amount of codebook feature information, but not all codebook feature information will be used to characterize each speech latent variable information. By determining the target codebook feature information through similarity calculation, the use of unnecessary codebook feature information can be avoided, thereby optimizing the efficiency of codebook use.

[0134] In one embodiment, the input module includes:

[0135] An input submodule, used for inputting the synthesized speech and the daily speech into a preset distinguisher to obtain a difference value between the synthesized speech and the daily speech;

[0136] A second calculation submodule is used to calculate the mean square error between the speech signal of the synthesized speech and the speech signal of the daily speech by taking the difference value as the adversarial loss of the encoder and the decoder;

[0137] The addition submodule is used to add the mean square error to the adversarial loss to obtain the loss function of the encoder and decoder.

[0138] In an embodiment of the present application, the discriminator can be trained to distinguish between real speech and synthesized speech, or to distinguish between synthesized speech of different qualities. During the training process, the discriminator will learn the characteristics and patterns of real speech, and react sensitively to defects or deficiencies in the synthesized speech. Therefore, incorporating the distinction results of the discriminator into the calculation of the loss function can prompt the model to pay more attention to these defects and deficiencies during the training process, thereby improving the robustness and generalization ability of the model. By using a discriminator to distinguish synthesized speech, the performance of the synthesized speech in terms of speech characteristics, sound quality, naturalness, etc. can be evaluated more carefully, thereby improving the evaluation accuracy of the speech synthesis quality.

[0139] In one embodiment, a building block comprises:

[0140] A clustering submodule is used to cluster the latent variable information of the target speech to obtain multiple speech feature clusters;

[0141] The acquisition submodule is used to obtain the speech feature information of the center point of each speech feature cluster and construct a target speech codebook based on the speech feature information.

[0142] In the embodiment of the present application, by clustering the target speech latent variable information, similar speech features can be classified into one category, thereby reducing redundant information in the data. The speech feature information (such as the feature vector of the center point) of each speech feature cluster can represent the common features of all speech latent variable information in the cluster, and the target speech codebook is constructed according to these speech feature information, which avoids repeated storage and processing of similar information and improves the efficiency of data processing. The feature vector of the center point of each speech feature cluster after clustering is the best representative of all speech latent variable information in the cluster. These center point feature vectors (i.e., the speech feature information mentioned above in the present embodiment) can be used as codebook vectors (i.e., the new speech codebook feature information mentioned above in the present embodiment) to construct the target speech codebook, which can ensure that the target speech codebook has a high degree of representativeness and can cover and characterize most of the speech feature information in the training data.

[0143] In one embodiment, the calculation module 403 includes:

[0144] A third calculation submodule is used to calculate the cosine distance between each speech feature information and multiple speech codebook feature information of the target speech codebook;

[0145] The second determination submodule is used to determine the similarity between each speech feature information and multiple speech codebook feature information based on the cosine distance.

[0146] In the embodiment of the present application, in speech processing, speech feature information is usually represented in the form of vectors, which contain multiple features of speech (such as frequency, energy, duration, etc.). By calculating the cosine distance between two vectors, the similarity between them can be accurately measured, not just based on their size or length. In the speech synthesis process, selecting the speech codebook feature information with the highest similarity to the text is crucial for generating natural and fluent speech. Determining the similarity by cosine distance can ensure that the selected speech codebook feature information is highly consistent with the text content to be converted in terms of semantics, intonation, speaking speed, etc., thereby improving the naturalness and intelligibility of speech synthesis.

[0147] In one embodiment, the voice conversion module 405 includes:

[0148] A mapping submodule, for performing feature mapping processing on multiple target speech codebook feature information using the acoustic model of the target decoder to obtain an acoustic feature sequence;

[0149] A conversion submodule, for converting the acoustic feature sequence into a speech waveform using a vocoder of a target decoder to obtain a plurality of speech waveform data;

[0150] The splicing submodule is used to splice multiple speech waveform data using a target decoder to generate a speech signal, output the speech corresponding to the speech signal, and obtain the target synthesized speech corresponding to the text to be converted.

[0151] The embodiment of the present application performs feature mapping processing on the target speech codebook feature information by using the acoustic model of the target decoder to obtain an acoustic feature sequence. This step ensures the accurate conversion from codebook features to acoustic features, and provides high-quality acoustic parameters for subsequent speech synthesis. It is possible to retain the key information in the codebook features and convert them into acoustic features that can be used for speech synthesis, thereby improving the accuracy of speech synthesis. The acoustic feature sequence is input into the vocoder of the target decoder and converted into speech waveform data. The vocoder is a key component in the speech synthesis system, which is responsible for converting acoustic features into audible speech waveforms. The beneficial effect of this step is that high-quality speech waveforms can be generated, and these waveforms can be highly consistent with the original speech in terms of sound quality, intonation, speech speed, etc., thereby improving the naturalness and fluency of speech synthesis. Use the target decoder to splice multiple speech waveform data to generate a complete speech signal. In the speech synthesis process, since the text may contain multiple words, phrases or sentences, it is necessary to splice these independent speech waveforms to form a continuous speech signal. This step ensures that the speech signal maintains a smooth transition at the splicing point, avoiding abrupt breakpoints and unnatural pauses, thereby improving the coherence and expressiveness of speech synthesis. Since the target decoder has been trained with daily speech, it can capture the speaker's unique speech features, such as intonation, speaking speed, timbre, etc., and preserve and reproduce them during the synthesis process. This allows the generated synthetic speech to not only accurately convey the content of the text, but also fully demonstrate the speaker's personality and emotions, meeting the needs of personalized speech synthesis.

[0152] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0153] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0154] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0155] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the speech generation method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0156] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a speech generation method.

[0157] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0158] The embodiment of the present application can ensure that the generated synthesized speech is closer to natural speech in speech features such as pitch, length, and intensity by extracting multiple speech feature information of the text to be converted and finely matching it with the feature information in the target speech codebook, thereby improving the naturalness and fluency of speech synthesis. Using the target speech codebook and target decoder obtained by daily speech training, the speech features of a specific speaker can be modeled and synthesized to achieve the generation of personalized speech. This is of great significance for application scenarios that require simulating the voice of a specific person, such as virtual character dubbing, personalized customization of voice assistants, etc. By calculating the similarity between the speech feature information and the feature information of the target speech codebook, and selecting the corresponding feature information for synthesis based on the similarity, it can ensure that the generated speech is highly consistent with the original text to be converted in content, while improving the accuracy and efficiency of speech synthesis. The speech generation scheme has broad application prospects in multiple fields such as medical care, education, and entertainment. For example, in the medical field, it can be used to generate patient education materials, assist doctors in remote diagnosis and treatment, etc.; in the education field, it can be used to produce audio books, provide personalized learning guidance, etc.; in the entertainment field, it can be used for game character dubbing, virtual idol voice generation, etc.

[0159] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech generation method as described above.

[0160] The embodiment of the present application can ensure that the generated synthesized speech is closer to natural speech in speech features such as pitch, length, and intensity by extracting multiple speech feature information of the text to be converted and finely matching it with the feature information in the target speech codebook, thereby improving the naturalness and fluency of speech synthesis. Using the target speech codebook and target decoder obtained by daily speech training, the speech features of a specific speaker can be modeled and synthesized to achieve the generation of personalized speech. This is of great significance for application scenarios that require simulating the voice of a specific person, such as virtual character dubbing, personalized customization of voice assistants, etc. By calculating the similarity between the speech feature information and the feature information of the target speech codebook, and selecting the corresponding feature information for synthesis based on the similarity, it can ensure that the generated speech is highly consistent with the original text to be converted in content, while improving the accuracy and efficiency of speech synthesis. The speech generation scheme has broad application prospects in multiple fields such as medical care, education, and entertainment. For example, in the medical field, it can be used to generate patient education materials, assist doctors in remote diagnosis and treatment, etc.; in the education field, it can be used to produce audio books, provide personalized learning guidance, etc.; in the entertainment field, it can be used for game character dubbing, virtual idol voice generation, etc.

[0161] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0162] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech generation method, characterized in that: The steps include: Get the text to be converted; Performing voice coding conversion on the text to be converted to obtain a plurality of speech feature information of the text to be converted; Calculating the similarity between each speech feature information and various speech codebook feature information of a target speech codebook, wherein the target speech codebook is a speech codebook obtained based on daily speech processing; Based on the similarity, determining a plurality of target speech codebook feature information corresponding to the to-be-converted text from the target speech codebook; A target decoder is used to convert the multiple target speech codebook feature information into speech to obtain a target synthesized speech corresponding to the text to be converted, and the target decoder is a decoder obtained based on the daily speech training.

2. The method according to claim 1, characterized in that The step of performing voice coding conversion on the text to be converted to obtain a plurality of speech feature information of the text to be converted specifically includes: Extracting a plurality of text feature information of the text to be converted; The plurality of text feature information are input into a preset text decoder for decoding to obtain a plurality of speech feature information of the text to be converted.

3. The method according to claim 2, characterized in that The step of extracting a plurality of text feature information of the text to be converted specifically includes: Preprocessing the text to be converted to obtain multiple text segmentations, and converting each text segmentation into a feature vector based on a preset vocabulary; The feature vector is input into a preset text encoder to obtain a plurality of text feature information.

4. The method according to claim 1, characterized in that: Before the step of calculating the similarity between each speech feature information and various speech codebook feature information of the target speech codebook, the method further includes: Acquire daily speech, input the daily speech into a preset encoder, and obtain multiple speech latent variable information; Determining target codebook feature information representing the plurality of speech latent variable information from a preset reference codebook; Inputting the target codebook feature information into a preset decoder to obtain a synthesized speech corresponding to the daily speech, and calculating a loss function based on the daily speech and the synthesized speech; Adjusting the encoding parameters of the encoder and the decoding parameters of the decoder based on the loss function, returning to the step of inputting the daily speech into a preset encoder to obtain a plurality of speech latent variable information, until the loss function converges to a preset convergence threshold; Determine the encoder and the decoder when the loss function converges to a preset convergence threshold as a target encoder and a target decoder, and determine the plurality of speech latent variable information when the loss function converges to a preset convergence threshold as target speech latent variable information; Based on the target speech latent variable information, a target speech codebook is constructed.

5. The method according to claim 4, characterized in that The step of determining the target codebook feature information representing the plurality of speech latent variable information from the preset reference codebook specifically includes: Calculating information similarity between the plurality of speech latent variable information and codebook feature information of each dimension of a preset reference codebook to obtain a plurality of similarity results; Based on the similarity result, target codebook feature information representing the plurality of speech latent variable information is determined from the reference codebook.

6. The method according to claim 4, characterized in that The step of calculating the loss function based on the daily speech and the synthesized speech specifically includes: Inputting the synthesized speech and the daily speech into a preset distinguisher to obtain a difference value between the synthesized speech and the daily speech; Using the difference value as the adversarial loss of the encoder and the decoder, calculating the mean square error between the speech signal of the synthesized speech and the speech signal of the daily speech; The mean square error is added to the adversarial loss to obtain loss functions of the encoder and the decoder.

7. The method according to claim 4, characterized in that The step of constructing a target speech codebook based on the target speech latent variable information specifically includes: Clustering the target speech latent variable information to obtain a plurality of speech feature clusters; The speech feature information of the center point of each speech feature cluster is obtained, and a target speech codebook is constructed based on the speech feature information.

8. The method according to claim 4, characterized in that The step of using a target decoder to convert the plurality of target speech codebook feature information into speech to obtain a target synthesized speech corresponding to the text to be converted specifically includes: Using the acoustic model of the target decoder, the plurality of target speech codebook feature information are subjected to feature mapping processing to obtain an acoustic feature sequence; Using a vocoder of a target decoder, converting the acoustic feature sequence into a speech waveform to obtain a plurality of speech waveform data; The target decoder is used to splice the multiple speech waveform data to generate a speech signal, and the speech corresponding to the speech signal is output to obtain the target synthesized speech corresponding to the text to be converted.

9. A speech generating device, characterized in that: include: An acquisition module, used for acquiring a text to be converted, wherein the text to be converted is used for converting into a target synthesized speech; A voice code conversion module, used for performing voice code conversion processing on the text to be converted to obtain a plurality of speech feature information of the text to be converted; A calculation module, used for calculating the similarity between each speech feature information and various speech codebook feature information of a target speech codebook, wherein the target speech codebook is a speech codebook obtained based on daily speech processing; A first determining module is used to determine, based on the similarity, a plurality of target speech codebook feature information corresponding to the to-be-converted text from the target speech codebook; The speech conversion module is used to use a target decoder to convert the multiple target speech codebook feature information into speech to obtain the target synthesized speech corresponding to the text to be converted, and the target decoder is a decoder obtained based on the daily speech training.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech generation method according to any one of claims 1 to 8 when executing the computer-readable instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech generation method according to any one of claims 1 to 8 are implemented.

Images