Speech synthesis processing method and device and related equipment

Abstract

The invention belongs to the technical field of financial science and technology, provides a speech synthesis processing method and device and related equipment, and aims to solve the problem of low similarity between a synthesized speech and a real person speech in the traditional technology, the speech synthesis processing method comprises the following steps: obtaining a text to be converted into a speech, obtaining a prompt text, and determining a discrete semantic token; establishing an alignment relationship between the text and the discrete semantic token based on a preset text speech big language model according to the prompt text to obtain a target discrete semantic token corresponding to the text, determining a speech spectrum feature corresponding to the target discrete semantic token based on a preset condition flow matching model, and synthesizing a speech spectrum feature corresponding to the target discrete semantic token based on a preset speech synthesis decoder. And generating the voice spectrum characteristics into the voice signals to obtain the voice, so that the similarity between voice synthesis and real person voice can be improved, for example, for self-service voice service of insurance business or banking business, the voice is synthesized in the mode, and the synthesized voice and the real person voice can have relatively high similarity.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a speech synthesis processing method, apparatus, computer equipment and computer-readable storage medium. Background Art

[0002] Large Language Models (LLMs) are artificial intelligence models that are trained with large amounts of data and are designed to understand and generate natural language text. Large language models are usually based on deep learning technology and can capture the complexity and diversity of language.

[0003] Among them, the text-to-speech synthesis model based on a large language model has become a common processing method for speech synthesis due to its high naturalness and zero-sample capability. In traditional technology, the text-to-speech synthesis processing based on a large language model generally discretizes the speech signal to generate speech tags, and performs speech synthesis based on the speech tags, where the speech tags are metadata that describe the synthesized speech, such as the starting and ending positions of sentences or words in the audio stream. Speech tags play a key role in the speech synthesis model based on a large language model.

[0004] In traditional technology, speech tags generated in the text-to-speech synthesis process based on a large language model are learned in an unsupervised manner. Unsupervised learning is a method of automatically discovering the structure and pattern of speech data from unlabeled speech data for automatic learning. Since unsupervised learning cannot use labeled speech data for training, the speech tags generated in the text-to-speech synthesis process based on a large language model are inaccurate, which in turn leads to a low similarity between the synthesized speech and the real person's speech. For example, in the field of financial technology, such as in insurance companies, self-service speech services for insurance business based on text-to-speech synthesis processing based on a large language model, when generating labeled speech in an unsupervised manner, will result in a low similarity between the synthesized speech and the real person's speech for the self-service speech service for insurance business, which reduces the quality and level of the self-service voice service for insurance business, affects the self-service voice service and self-service level of insurance business, and cannot effectively reduce the human, material and financial costs required for the development of insurance business through the self-service voice service for insurance business.

[0005] Therefore, improving the similarity between speech synthesis based on large language models and real-person speech has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a speech synthesis processing method, apparatus, computer device and computer-readable storage medium, which can solve the technical problem of low similarity between synthesized speech and real-person speech in traditional technologies.

[0007] In a first aspect, the present application provides a speech synthesis processing method, comprising: obtaining text to be converted into speech, and obtaining a prompt text; determining discrete semantic tokens; based on a preset text-to-speech large language model, establishing an alignment relationship between the text and the discrete semantic tokens according to the prompt text, and obtaining a target discrete semantic token corresponding to the text; based on a preset conditional stream matching model, determining the speech spectrum features corresponding to the target discrete semantic token; based on a preset speech synthesis decoder, generating a speech signal from the speech spectrum features, and obtaining the speech corresponding to the text.

[0008] In the second aspect, the present application provides a speech synthesis processing device, comprising: a first acquisition unit, used to acquire text to be converted into speech and acquire a prompt text; a first determination unit, used to determine discrete semantic tokens; a first establishment unit, used to establish an alignment relationship between the text and the discrete semantic tokens based on the prompt text based on a preset text-to-speech large language model, and obtain a target discrete semantic token corresponding to the text; a second determination unit, used to determine the speech spectrum features corresponding to the target discrete semantic token based on a preset conditional stream matching model; a first generation unit, used to generate a speech signal from the speech spectrum features based on a preset speech synthesis decoder, and obtain the speech corresponding to the text.

[0009] In a third aspect, the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the speech synthesis processing method when executing the computer program.

[0010] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the speech synthesis processing method.

[0011] The present application provides a speech synthesis processing method, apparatus, computer device and computer-readable storage medium. The method obtains text to be converted into speech, obtains prompt text, and determines discrete semantic tokens. Then, based on a preset text-speech large language model and according to the prompt text, an alignment relationship between the text and the discrete semantic tokens is established to obtain a target discrete semantic token corresponding to the text. Then, based on a preset conditional stream matching model, speech spectrum features corresponding to the target discrete semantic token are determined. Based on a preset speech synthesis decoder, speech spectrum features are generated into a speech signal to obtain the speech corresponding to the text, thereby realizing a few-sample speech synthesis based on self-supervised semantic tokens, and being able to realize semantic self-supervision using a small amount of input speech signals, so that a large amount of unlabeled data can be used to generate text-speech pairs, so that the synthesized speech is more rich and realistic, and the similarity between the synthesized speech and the real person's speech is more matched, thereby improving the similarity between the speech synthesis and the real person's speech, and realizing that the synthesized speech is more accurate than the real person's speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a speech synthesis processing method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of a first sub-process of the speech synthesis processing method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of a second sub-process of the speech synthesis processing method provided in an embodiment of the present application;

[0016] Figure 4 A schematic block diagram of a speech synthesis processing device provided in an embodiment of the present application;

[0017] Figure 5 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] The embodiment of the present application provides a speech synthesis processing method, which can be applied to but not limited to computer devices such as smart phones, laptops, desktop computers, servers, intelligent robots, etc., and is used in the field of financial technology to perform speech synthesis processing, including but not limited to providing self-service voice services for financial services.

[0021] In the face of the technical problem that the similarity between synthesized speech and real speech in traditional technology is low, the inventor proposes a speech synthesis processing method of the embodiment of the present application. The core idea of ​​the embodiment of the present application is: to generate semantic tokens representing semantic tags corresponding to speech signals in a self-supervised manner, and these semantic tokens are obtained by recognizing a small amount of speech signals based on a speech recognition model. On the basis of obtaining the semantic tokens, when it is necessary to generate speech corresponding to the text, an alignment relationship between the text and the semantic tokens is established, and then the semantic tokens are synthesized into speech, thereby realizing a small sample speech synthesis based on self-supervised semantic tokens from speech signals to semantic tokens, and then from semantic tokens to speech corresponding to the text. It can realize self-supervision using a small amount of input speech signals, so that a large amount of unlabeled data can be used to generate text-speech pairs, so that the synthesized speech is richer and more realistic, and the similarity between the synthesized speech and the real speech is more matched, thereby improving the similarity between speech synthesis and real speech, and realizing that the synthesized speech is more accurate than the real speech.

[0022] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0023] See also Figure 1 , Figure 1 The flowchart of the speech synthesis processing method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes but is not limited to the following steps S11-S15:

[0024] S11. Obtain text to be converted into speech, and obtain prompt text.

[0025] Explanatory note, in application scenarios including but not limited to the field of financial technology, such as insurance, banking, securities and other businesses, there is usually a need to synthesize text into corresponding speech. In this case, the text to be converted into speech is obtained, that is, the text needs to be converted into speech to obtain the speech signal corresponding to the text. The speech signal can be further played, or the speech signal can be combined with an image to form a video for real-person simulation.

[0026] For example, in the field of financial technology, in one example, users can use the telephone platform or APP platform of insurance business for self-service voice services. There is a scenario where text is converted into corresponding voice, and the above-mentioned synthesized voice is played to the user to provide self-service handling of insurance business. In another example, for the bank's deposit and loan business, users can consult through the telephone platform, or use the APP to perform self-service voice handling of banking business, and can also use the self-service ATM machine to perform self-service voice handling of financial business. In these cases, there is a situation where the corresponding text is converted into synthesized voice, and the synthesized voice is played to the user as similar to the voice of a real person as possible to provide self-service handling of banking business.

[0027] It is necessary not only to obtain the text to be converted into speech, but also to obtain the prompt text. Among them, the prompt text, in English, is Prompt Text. The prompt text is a piece of text input provided to guide the speech synthesis model to generate the corresponding output. The prompt text can be a question, instruction, introduction or any other form of text. Its function is to set the context so that the speech synthesis model can understand the user's intention and generate answers or complete tasks related to the context. The prompt text can be text input in real time or pre-stored text. For example, in the field of financial technology, for insurance business, in the business scenario where users use the telephone platform or APP platform of the insurance business for self-service voice services, when the text is converted into the corresponding speech using the embodiment of the present application, the prompt text can be pre-stored to guide the speech synthesis model to generate the corresponding speech output.

[0028] S12. Determine discrete semantic tokens.

[0029] Explanatory, Tokens: In natural language processing, the text is first segmented into smaller units called tokens. These tokens can be words, phrases, or characters, and the model uses these tokens to understand and generate text.

[0030] Semantic tokens, in English, are the smallest indivisible units of semantics in speech. Semantic tokens are used to capture local dependencies and global long-term structures in speech based on semantics. The process of acquiring semantic tokens includes identifying predicates and their arguments in speech, and marking these arguments with the semantic roles they play.

[0031] Discrete semantic tokens represent speech tokens obtained by discretizing continuous speech signals, wherein the "discrete" in discrete semantic tokens is only used to distinguish different semantic tokens and is not used to limit semantic tokens, and other similar descriptions are similar.

[0032] Determining discrete semantic tokens includes but is not limited to obtaining speech signals, discretizing the speech signals to obtain initial speech tokens, and identifying the recognition text corresponding to the speech signals, and then aligning the recognition text with the initial speech tokens to obtain discrete semantic tokens corresponding to the recognition text, thereby discretizing the continuous speech signals to obtain the corresponding discrete tokens, and determining discrete semantic tokens based on speech signals. It is possible to obtain discrete semantic tokens based on a small amount of speech signals without labeling, based on a self-supervisory method. Among them, the acquired speech signals include but are not limited to training sample speech signals and user-input speech signals. The user-input speech signals include but are not limited to real-person speech corresponding to the prompt text, that is, discrete semantic tokens include but are not limited to the following two sources: 1) training sample speech signals when training speech synthesis models; 2) speech signals input by users in real time. Discrete semantic tokens can be obtained from both of these speech signal sources.

[0033] For example, in the field of financial technology, for insurance business, when performing speech synthesis for self-service voice services through the insurance business's telephone platform or APP platform, in order to generate a voice that meets the user's preferences, it can be required to input a prompt text and a prompt voice corresponding to the prompt text, and use the prompt voice as a voice signal to determine the corresponding discrete semantic token.

[0034] Compared with the traditional technology of generating speech tags based on unsupervised learning, the discrete semantic tokens used in the embodiment of the present application can improve the accuracy of synthesized speech. Compared with the traditional technology of generating speech tags based on supervised learning, it does not require a large number of labeled training samples, and can automatically generate corresponding discrete semantic tokens in time according to the instantly acquired speech signals, such as prompt speech, which can improve the efficiency, flexibility and automation of generating discrete semantic tokens.

[0035] S13. Based on a preset text-speech large language model and according to the prompt text, an alignment relationship between the text and the discrete semantic token is established to obtain a target discrete semantic token corresponding to the text.

[0036] Explanatory, a text-speech large language model is pre-set, that is, a preset text-speech large language model, which represents a large language model for converting text into speech. Large Language Models (LLMs) is an artificial intelligence model trained with a large amount of data, which is intended to understand and generate natural language text. The preset text-speech large language model is an autoregressive model, which uses an autoregressive reasoning mechanism to establish an alignment relationship between text and discrete semantic tokens. The autoregressive reasoning mechanism predicts the probability distribution of subsequent words based on the previous text, which enables LLMs to make full use of the information of the previous text to generate a coherent and reasonable text, thereby improving the accuracy of the target discrete semantic token corresponding to the text. Among them, the alignment relationship in the embodiment of the present application is to accurately match the text with the discrete semantic token in time, ensuring that when the speech corresponding to the text is generated, the text is accurately matched with the corresponding audio signal, so as to ensure that the audio signal corresponding to each character, word, sentence, etc. in the transcription process corresponds to the correct time period, and one text corresponds to several discrete speech tokens, that is, several discrete speech tokens represent one text, and the text is a single word, word, phrase or sentence.

[0037] Based on the preset text-to-speech large language model, and according to the guidance of the prompt text, autoregressive reasoning is used to establish the alignment relationship between the text and the discrete semantic tokens, and the target discrete semantic tokens corresponding to the text are obtained, so as to accurately match the text and the discrete semantic tokens in time to ensure the accuracy of the synthesized speech.

[0038] S14. Based on a preset conditional stream matching model, determine the speech spectrum features corresponding to the target discrete semantic token.

[0039] Explanatory, a conditional flow matching model is pre-set, i.e., a preset conditional flow matching model. Based on the preset conditional flow matching model, a relationship between a target discrete semantic token and speech spectrum features is established to determine the speech spectrum features corresponding to the target discrete semantic token. Since the target discrete semantic token is already the result of alignment between the text and the discrete semantic token, the speech spectrum features corresponding to the target discrete semantic token determined based on the preset conditional flow matching model are the relationship between the target discrete semantic token and the continuous acoustic features. Among them, conditional flow refers to an operation or process that determines the execution of different speech spectrum features according to given conditions. The conditional flow matching model, in English, is a model that matches the target discrete semantic token with speech spectrum features based on conditional flow. The conditional flow matching model can generate detailed waveforms. The speech spectrum features represent different components and patterns in speech. The speech spectrum features come from the training sample speech signals when training the speech synthesis model.

[0040] S15. Based on a preset speech synthesis decoder, generate a speech signal from the speech spectrum features to obtain the speech corresponding to the text.

[0041] Explanatoryly, a decoder including but not limited to HIFI-GAN or HIFTNet, i.e., a preset speech synthesis decoder, is used to restore the speech spectrum features into a speech signal to obtain the speech corresponding to the text, wherein HIFI-GAN is a neural vocoder based on a generative adversarial network, and HiFTNet is a neural vocoder that combines an inverse short-time Fourier transform (iSTFT) and a harmonic noise filter.

[0042] For example, in the field of financial technology, for insurance business, when users use the insurance business's telephone platform or APP platform for self-service voice services, the decoder based on HIFI-GAN or HIFTNet restores the speech spectrum features into a speech signal, obtains the speech corresponding to the text, and plays the speech to the user to provide self-service voice services.

[0043] In an embodiment of the present application, a text to be converted into speech is obtained, a prompt text is obtained, and discrete semantic tokens are determined. Then, based on a preset text-to-speech large language model and according to the prompt text, an alignment relationship between the text and the discrete semantic tokens is established to obtain a target discrete semantic token corresponding to the text. Then, based on a preset conditional stream matching model, speech spectrum features corresponding to the target discrete semantic tokens are determined. Based on a preset speech synthesis decoder, speech spectrum features are generated into speech signals to obtain speech corresponding to the text, thereby realizing a few-sample speech synthesis based on self-supervised semantic tokens. Semantic self-supervision can be realized using a small amount of input speech signals, so that a large amount of unlabeled data can be used to generate text-speech pairs, so that the synthesized speech is more rich and realistic, and the similarity between the synthesized speech and the real person's speech is more matched, thereby improving the similarity between the speech synthesis and the real person's speech, and realizing that the synthesized speech is more accurate than the real person's speech.

[0044] In one embodiment, see Figure 2 , Figure 2 This is a schematic diagram of the first sub-flow of the speech synthesis processing method provided in the embodiment of the present application. Figure 2 As shown, discrete semantic tokens are determined, including:

[0045] S21, obtaining an initial speech signal, and downsampling the initial speech signal based on a preset speech encoder to obtain a speech latent vector;

[0046] S22, based on a preset speech quantizer, discretizing the speech latent vector to obtain a discretized speech;

[0047] S23, based on a preset speech decoder, restoring the discretized speech into a speech signal to obtain a generated speech;

[0048] S24, based on a preset speech recognition model, recognizing the generated speech to obtain a recognized text;

[0049] S25. Using a preset text-speech large language model based on an autoregressive method, an alignment relationship between the discretized speech and the recognized text is established to obtain discrete semantic tokens.

[0050] Explanatory, the initial voice signal is obtained. In the case of training a speech synthesis model, the initial voice signal may be a training sample voice signal. In the case of using a speech synthesis model for speech synthesis inference, the initial voice signal may be a voice signal corresponding to any voice provided by the user, including but not limited to a prompt voice corresponding to a prompt text. Exemplarily, in the field of financial technology, for insurance business, a prompt voice may be provided when performing speech synthesis for self-service voice services through the telephone platform or APP platform of the insurance business.

[0051] Based on the preset speech encoder, the initial speech signal is downsampled to obtain a high-dimensional speech latent vector. For example, the 1024-dimensional speech signal is downsampled to a 128-dimensional or 256-dimensional continuous speech latent vector. The working principle of the speech encoder is to convert the speech signal into a digital signal through a compression algorithm. Downsampling means reducing the sampling rate of the speech signal, that is, reducing the number of points of the speech data.

[0052] Based on the preset speech quantizer, the speech latent vector is discretized to obtain discretized speech. In particular, based on the preset speech quantizer, the speech latent vector is discretized to quantize the continuous speech latent vector in the amplitude dimension and quantize the continuous amplitude value into discrete ones. Commonly used quantization methods include linear and nonlinear.

[0053] Then, based on the preset speech decoder, the discretized speech is restored to a speech signal to obtain the generated speech. Then, based on the preset speech recognition model, the generated speech is recognized to obtain the recognized text. The preset text-to-speech large language model based on the autoregressive method is used to establish an alignment relationship between the discretized speech and the recognized text to obtain discrete semantic tokens. Thus, the discrete semantic tokens can be obtained based on the speech signal alone and in a self-supervised manner, thereby obtaining the discrete semantic tokens based on a small amount of speech signals without the need for labeling.

[0054] The embodiments of the present application, by obtaining discrete semantic tokens based on a self-supervised manner only according to the speech signal, can realize semantic self-supervision using a small amount of input speech signals, and then can use a large amount of unlabeled data to generate text-speech pairs, so that the synthesized speech is more rich, realistic and accurate, and the similarity between the synthesized speech and the real speech is more matched, thereby improving the similarity between speech synthesis and real speech, and achieving that the synthesized speech is more accurate than the real speech.

[0055] In one embodiment, the initial speech signal includes a first language and a second language; based on a preset speech recognition model, the generated speech is recognized to obtain a recognized text, including:

[0056] Based on a preset multilingual speech recognition model, the first language included in the generated speech is recognized to obtain a first sub-recognition text, and the second language included in the generated speech is recognized to obtain a second sub-recognition text;

[0057] The first sub-recognition text and the second sub-recognition text are combined into an integral text according to the speech positions of the first language and the second language in the initial speech signal to obtain a recognized text.

[0058] Explanatoryly, the multilingual speech recognition model is pre-set, that is, the preset multilingual speech recognition model refers to a model that can recognize speech in multiple languages.

[0059] In the case where the initial voice signal includes but is not limited to the first language and the second language, based on the preset multilingual voice recognition model, the first language included in the generated voice is identified to obtain a first sub-recognition text, and the second language included in the generated voice is identified to obtain a second sub-recognition text, wherein the "first" involved in the first language and the "second" involved in the second language are only used to distinguish different languages, and are not used to limit different languages. Exemplarily, in the field of financial technology, for insurance business, when self-service voice service is performed through the telephone platform or APP platform of the insurance business for voice synthesis, the initial voice signal includes but is not limited to Chinese, English, or Chinese and French.

[0060] In the case where the initial speech signal includes but is not limited to the first language and the second language, when the speech is generated based on the preset multilingual speech recognition model recognition, the speech positions of the first language and the second language in the initial speech signal are recorded respectively, and according to the speech positions of the first language and the second language in the initial speech signal, the first sub-recognition text and the second sub-recognition text are combined into an overall text to obtain a recognized text.

[0061] In the embodiment of the present application, based on a preset multilingual speech recognition model, the first language contained in the generated speech is recognized to obtain a first sub-recognition text, and the second language contained in the generated speech is recognized to obtain a second sub-recognition text, and then according to the speech positions of the first language and the second language in the initial speech signal, the first sub-recognition text and the second sub-recognition text are combined into an overall text to obtain a recognized text, which can generate discrete semantic tokens corresponding to multiple languages, realize support for speech synthesis of multiple languages, and even support for speech generation from text of mixed languages, and can further improve the similarity between speech synthesis and real-person speech, so as to realize that the synthesized speech is more accurate than real-person speech.

[0062] In one embodiment, see Figure 3 , Figure 3 This is a schematic diagram of the second sub-flow of the speech synthesis processing method provided in the embodiment of the present application. Figure 3 As shown, in this embodiment, based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the text and the discrete semantic token is established to obtain a target discrete semantic token corresponding to the text, including:

[0063] S31, determining speaker features corresponding to the text;

[0064] S32, concatenating the speaker feature with the text to obtain a concatenated text;

[0065] S33. Based on a preset text-speech large language model and according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain a target discrete semantic token corresponding to the text.

[0066] Explanatory, determine the speaker characteristics corresponding to the text, the speaker characteristics include but are not limited to speaker information and emotional representation, the speaker information includes but is not limited to the speaker's timbre, pitch, gender and other characteristics that describe the speaker information, the emotional representation includes but is not limited to happy, sad, low, high-pitched and other characteristics that describe the speaker's emotions.

[0067] Then the speaker features are concatenated with the text to obtain a concatenated text. Exemplarily, the speaker information and emotion representation included in the speaker features are obtained, and then a sequence {discrete semantic token, speaker information, emotion representation, text} is formed to concatenate the speaker information, emotion representation and text, so that the speaker features are embedded in the text to obtain a concatenated text, so that the concatenated text corresponds to the rich content that the sound should have as much as possible, rather than being limited to text. Then, based on the preset text-to-speech large language model and the prompt text, an alignment relationship between the text and the discrete semantic token is established to obtain the target discrete semantic token corresponding to the text.

[0068] Furthermore, the speaker feature is concatenated with the text to obtain a concatenated text, including:

[0069] The speaker feature and the text are sequenced to obtain a concatenated text.

[0070] Explanatory, discrete semantic tokens TOKEN, speaker information, emotion representation, and text are combined into a sequence form, i.e., {discrete semantic tokens, speaker information, emotion representation, text}, to obtain concatenated text, wherein the concatenated text includes but is not limited to vectors and long vectors. Then, based on a preset text-to-speech large language model and according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain the target discrete semantic token corresponding to the text, and then generate synthetic speech.

[0071] In the embodiment of the present application, by embedding speaker features into text to obtain corresponding target discrete semantic tokens and then generating synthesized speech, the synthesized speech can be made richer and more realistic, and the similarity between the synthesized speech and the real speech can be more closely matched, thereby improving the similarity between the speech synthesis and the real speech and achieving more accuracy of the synthesized speech relative to the real speech.

[0072] In one embodiment, determining the speaker features corresponding to the text includes:

[0073] Determine whether a prompt voice is input;

[0074] When it is determined that a prompt voice is input, the prompt voice is acquired, and speaker features corresponding to a speaker contained in the prompt voice are extracted to obtain speaker features corresponding to the text;

[0075] When it is determined that no prompt voice is input, speaker features corresponding to the text are randomly determined.

[0076] Explanatoryly, determine whether the user has input a prompt voice. If it is determined that the user has input a prompt voice, obtain the prompt voice. The prompt voice can be the voice corresponding to the prompt text, that is, the prompt voice is the voice expression of the prompt text, and the prompt text is the text expression of the prompt voice. Methods for extracting speaker features including but not limited to speech feature extraction, speaker modeling, speaker recognition, etc. can be used to extract speaker features corresponding to the speaker contained in the prompt voice, and obtain speaker features corresponding to the text. Furthermore, if it is determined that the prompt voice is not input, randomly determine the speaker features corresponding to the text.

[0077] The embodiment of the present application can flexibly determine the speaker characteristics corresponding to the text by judging whether the prompt voice is input, so as to meet the user's sound preference requirements for the synthesized voice as much as possible, thereby further making the synthesized voice more rich and realistic, and the similarity between the synthesized voice and the real voice more closely matched, thereby improving the similarity between the speech synthesis and the real voice, and achieving that the synthesized voice is more accurate than the real voice.

[0078] In one embodiment, based on a preset text-speech large language model, an alignment relationship between the concatenated text and the discrete semantic token is established according to the prompt text to obtain a target discrete semantic token corresponding to the text, including:

[0079] Based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain an initial discrete semantic token corresponding to the text;

[0080] Determining the speech length corresponding to the concatenated text;

[0081] According to the length of the speech, information of excess length contained in the initial discrete semantic token is cut off to obtain a target discrete semantic token corresponding to the text.

[0082] Explanatory, based on the preset text-to-speech large language model, according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic tokens is established, and the initial discrete semantic token corresponding to the text is first obtained. The initial discrete semantic token is the token corresponding to the entire concatenated text, and the speech length corresponding to the concatenated text is determined based on factors including but not limited to the prompt text. Then, according to the speech length, the information of the excess length contained in the initial discrete semantic token is cut off to obtain the target discrete semantic token corresponding to the text. The target discrete semantic token is a subset of the initial discrete semantic token, and then synthetic speech is generated based on the target discrete semantic token.

[0083] The embodiments of the present application can further make the synthesized speech more accurate by controlling the length of the target discrete semantic token corresponding to the text, so that the similarity between the synthesized speech and the real speech is more closely matched, thereby improving the similarity between the speech synthesis and the real speech, and achieving that the synthesized speech is more accurate than the real speech.

[0084] It should be noted that the speech synthesis processing method described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but all of them are within the scope of protection required by this application.

[0085] See also Figure 4 , Figure 4Schematic block diagram of a speech synthesis processing device provided in an embodiment of the present application. Corresponding to the above-mentioned speech synthesis processing method, an embodiment of the present application also provides a speech synthesis processing device. Figure 4 As shown, the speech synthesis processing device includes a unit for executing the above-mentioned speech synthesis processing method, and the speech synthesis processing device can be configured in a computer device. Figure 4 The speech synthesis processing device 40 includes a first acquisition unit 41, a first determination unit 42, a first establishment unit 43, a second determination unit 44 and a first generation unit 45.

[0086] The first acquisition unit 41 is used to acquire the text to be converted into speech and acquire the prompt text;

[0087] A first determining unit 42, configured to determine a discrete semantic token;

[0088] A first establishing unit 43 is used to establish an alignment relationship between the text and the discrete semantic tokens according to the prompt text based on a preset text-speech large language model, and obtain a target discrete semantic token corresponding to the text;

[0089] A second determining unit 44 is used to determine the speech spectrum feature corresponding to the target discrete semantic token based on a preset conditional stream matching model;

[0090] The first generating unit 45 is used to generate a speech signal from the speech spectrum feature based on a preset speech synthesis decoder to obtain the speech corresponding to the text.

[0091] In one embodiment, the first determining unit 42 includes:

[0092] A sampling subunit, used for acquiring an initial speech signal, and downsampling the initial speech signal based on a preset speech encoder to obtain a speech latent vector;

[0093] A discretization subunit, configured to discretize the speech latent vector based on a preset speech quantizer to obtain a discretized speech;

[0094] A restoring subunit, used for restoring the discretized speech into a speech signal based on a preset speech decoder to obtain a generated speech;

[0095] A first recognition subunit, configured to recognize the generated speech based on a preset speech recognition model to obtain a recognized text;

[0096] The first establishing subunit is used to adopt a preset text-speech large language model based on an autoregressive method to establish an alignment relationship between the discretized speech and the recognized text to obtain discrete semantic tokens.

[0097] In one embodiment, the initial speech signal includes a first language and a second language; the first recognition subunit includes:

[0098] a second recognition subunit, configured to recognize the first language contained in the generated speech based on a preset multilingual speech recognition model to obtain a first sub-recognition text, and recognize the second language contained in the generated speech to obtain a second sub-recognition text;

[0099] The combining subunit is used to combine the first sub-recognition text and the second sub-recognition text into a whole text according to the speech positions of the first language and the second language in the initial speech signal to obtain a recognized text.

[0100] In one embodiment, the first establishing unit 43 includes:

[0101] A first determination subunit, used to determine speaker features corresponding to the text;

[0102] A concatenation subunit, used for concatenating the speaker feature with the text to obtain a concatenated text;

[0103] The second establishing subunit is used to establish an alignment relationship between the concatenated text and the discrete semantic tokens based on a preset text-speech large language model and according to the prompt text, so as to obtain a target discrete semantic token corresponding to the text.

[0104] In one embodiment, the first determining subunit includes:

[0105] A judging subunit, used to judge whether a prompt voice is input;

[0106] The first determination subunit is used to obtain the prompt voice when it is determined that the prompt voice is input, and extract the speaker features corresponding to the speaker contained in the prompt voice to obtain the speaker features corresponding to the text.

[0107] In one embodiment, the first determining subunit further includes:

[0108] The second determination subunit is used to randomly determine the speaker feature corresponding to the text when it is determined that no prompt voice is input.

[0109] In one embodiment, the second establishing subunit includes:

[0110] A third establishing subunit is used to establish an alignment relationship between the concatenated text and the discrete semantic tokens according to the prompt text based on a preset text-speech large language model, and obtain an initial discrete semantic token corresponding to the text;

[0111] A second determining subunit is used to determine the speech length corresponding to the concatenated text;

[0112] The cutting subunit is used to cut off the information of the redundant length contained in the initial discrete semantic token according to the length of the speech, so as to obtain the target discrete semantic token corresponding to the text.

[0113] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned speech synthesis processing device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0114] At the same time, the division and connection method of each unit in the above-mentioned speech synthesis processing device is only used for illustration. In other embodiments, the speech synthesis processing device can be divided into different units as needed, and the units in the speech synthesis processing device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned speech synthesis processing device.

[0115] The above-mentioned speech synthesis processing device can be implemented in the form of a computer program. The computer program can be Figure 5 Runs on the computer device shown.

[0116] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0117] See also Figure 5 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0118] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned speech synthesis processing method.

[0119] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0120] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned speech synthesis processing method.

[0121] The network interface 505 is used to communicate with other devices over the network. Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 5 The embodiments shown are consistent and will not be described again here.

[0122] The processor 502 is used to run a computer program 5032 stored in a memory to implement the following steps: obtaining text to be converted into speech and obtaining a prompt text; determining discrete semantic tokens; based on a preset text-speech large language model, establishing an alignment relationship between the text and the discrete semantic tokens according to the prompt text, and obtaining a target discrete semantic token corresponding to the text; based on a preset conditional stream matching model, determining the speech spectrum features corresponding to the target discrete semantic token; based on a preset speech synthesis decoder, generating a speech signal from the speech spectrum features to obtain the speech corresponding to the text.

[0123] In one embodiment, the processor 502 implements the following steps when determining discrete semantic tokens:

[0124] Acquire an initial speech signal, and downsample the initial speech signal based on a preset speech encoder to obtain a speech latent vector;

[0125] Based on a preset speech quantizer, the speech latent vector is discretized to obtain a discretized speech;

[0126] Based on a preset speech decoder, the discretized speech is restored into a speech signal to obtain a generated speech;

[0127] Based on a preset speech recognition model, the generated speech is recognized to obtain a recognized text;

[0128] A preset text-speech large language model based on an autoregressive method is adopted to establish an alignment relationship between the discretized speech and the recognized text to obtain discrete semantic tokens.

[0129] In one embodiment, when the processor 502 realizes that the initial speech signal includes the first language and the second language; and recognizes the generated speech based on a preset speech recognition model to obtain a recognized text, the processor 502 specifically implements the following steps:

[0130] Based on a preset multilingual speech recognition model, the first language included in the generated speech is recognized to obtain a first sub-recognition text, and the second language included in the generated speech is recognized to obtain a second sub-recognition text;

[0131] The first sub-recognition text and the second sub-recognition text are combined into an integral text according to the speech positions of the first language and the second language in the initial speech signal to obtain a recognized text.

[0132] In one embodiment, when the processor 502 establishes an alignment relationship between the text and the discrete semantic tokens according to the prompt text based on a preset text-speech large language model and obtains a target discrete semantic token corresponding to the text, the processor 502 specifically implements the following steps:

[0133] Determining speaker characteristics corresponding to the text;

[0134] splicing the speaker feature with the text to obtain a spliced ​​text;

[0135] Based on a preset text-speech large language model, an alignment relationship between the concatenated text and the discrete semantic tokens is established according to the prompt text to obtain a target discrete semantic token corresponding to the text.

[0136] In one embodiment, the processor 502 implements the following steps when determining the speaker feature corresponding to the text:

[0137] Determine whether a prompt voice is input;

[0138] When it is determined that a prompt voice is input, the prompt voice is acquired, and speaker features corresponding to a speaker contained in the prompt voice are extracted to obtain speaker features corresponding to the text.

[0139] In one embodiment, when implementing the method, the processor 502 further implements the following steps:

[0140] When it is determined that no prompt voice is input, speaker features corresponding to the text are randomly determined.

[0141] In one embodiment, when the processor 502 establishes an alignment relationship between the concatenated text and the discrete semantic tokens based on the preset text-speech large language model and obtains a target discrete semantic token corresponding to the text according to the prompt text, the processor 502 specifically implements the following steps:

[0142] Based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain an initial discrete semantic token corresponding to the text;

[0143] Determining the speech length corresponding to the concatenated text;

[0144] According to the length of the speech, information of excess length contained in the initial discrete semantic token is cut off to obtain a target discrete semantic token corresponding to the text.

[0145] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0146] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0147] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0148] A computer program product, when running on a computer, enables the computer to execute the steps of the speech synthesis processing method described in the above embodiments.

[0149] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0151] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0154] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0156] The software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech synthesis processing method, characterized in that: include: Get the text to be converted into speech and get the prompt text; Identify discrete semantic tokens; Based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the text and the discrete semantic token is established to obtain a target discrete semantic token corresponding to the text; Determine the speech spectrum features corresponding to the target discrete semantic token based on a preset conditional stream matching model; Based on a preset speech synthesis decoder, the speech spectrum features are used to generate a speech signal to obtain the speech corresponding to the text.

2. The speech synthesis processing method according to claim 1, characterized in that: Identify discrete semantic tokens, including: Acquire an initial speech signal, and downsample the initial speech signal based on a preset speech encoder to obtain a speech latent vector; Based on a preset speech quantizer, the speech latent vector is discretized to obtain a discretized speech; Based on a preset speech decoder, the discretized speech is restored into a speech signal to obtain a generated speech; Based on a preset speech recognition model, the generated speech is recognized to obtain a recognized text; A preset text-speech large language model based on an autoregressive method is adopted to establish an alignment relationship between the discretized speech and the recognized text to obtain discrete semantic tokens.

3. The speech synthesis processing method according to claim 2, characterized in that: The initial speech signal includes a first language and a second language; based on a preset speech recognition model, the generated speech is recognized to obtain a recognized text, including: Based on a preset multilingual speech recognition model, the first language included in the generated speech is recognized to obtain a first sub-recognition text, and the second language included in the generated speech is recognized to obtain a second sub-recognition text; The first sub-recognition text and the second sub-recognition text are combined into an integral text according to the speech positions of the first language and the second language in the initial speech signal to obtain a recognized text.

4. The speech synthesis processing method according to any one of claims 1 to 3, characterized in that: Based on a preset text-speech large language model, an alignment relationship between the text and the discrete semantic token is established according to the prompt text to obtain a target discrete semantic token corresponding to the text, including: Determining speaker characteristics corresponding to the text; splicing the speaker feature with the text to obtain a spliced ​​text; Based on a preset text-speech large language model, an alignment relationship between the concatenated text and the discrete semantic tokens is established according to the prompt text to obtain a target discrete semantic token corresponding to the text.

5. The speech synthesis processing method according to claim 4, characterized in that: The determining of speaker features corresponding to the text includes: Determine whether a prompt voice is input; When it is determined that a prompt voice is input, the prompt voice is acquired, and speaker features corresponding to a speaker contained in the prompt voice are extracted to obtain speaker features corresponding to the text.

6. The speech synthesis processing method according to claim 5, characterized in that: The method further comprises: When it is determined that no prompt voice is input, speaker features corresponding to the text are randomly determined.

7. The speech synthesis processing method according to claim 4, characterized in that: Based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain a target discrete semantic token corresponding to the text, including: Based on a preset text-speech large language model, according to the prompt text, an alignment relationship between the concatenated text and the discrete semantic token is established to obtain an initial discrete semantic token corresponding to the text; Determining the speech length corresponding to the concatenated text; According to the length of the speech, information of excess length contained in the initial discrete semantic token is cut off to obtain a target discrete semantic token corresponding to the text.

8. A speech synthesis processing device, characterized in that: include: A first acquisition unit, used to acquire text to be converted into speech and acquire prompt text; A first determining unit, configured to determine a discrete semantic token; A first establishing unit is used to establish an alignment relationship between the text and the discrete semantic token according to the prompt text based on a preset text-speech large language model, and obtain a target discrete semantic token corresponding to the text; A second determining unit is used to determine the speech spectrum feature corresponding to the target discrete semantic token based on a preset conditional stream matching model; The first generating unit is used to generate a speech signal from the speech spectrum feature based on a preset speech synthesis decoder to obtain the speech corresponding to the text.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images