Voice generation method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the field of artificial intelligence and the field of financial science and technology, and relates to an artificial intelligence-based voice generation method, which comprises the steps of receiving a to-be-synthesized text and an initial voice; preprocessing the to-be-synthesized text to obtain a specified text; performing text encoding on the specified text based on a text encoder to obtain a text encoding vector; extracting a speaker embedding vector from the initial voice; processing the text coding vector and the speaker embedding vector based on a large language model to generate a target voice marking sequence; processing the target voice marking sequence and the speaker embedding vector based on a target condition flow matching model to obtain a Mel spectrogram; and performing conversion processing on the Mel-frequency spectrogram based on a vocoder to obtain synthetic speech. The invention further provides a voice generation device based on artificial intelligence, computer equipment and a storage medium. In addition, the synthetic voice can be stored in the block chain. According to the invention, the voice quality of the generated synthetic voice is improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence development technology and the field of financial technology, and in particular to artificial intelligence-based speech generation methods, devices, computer equipment and storage media. Background Art

[0002] Speech synthesis methods can synthesize text into corresponding speech, and are widely used in many fields such as the Internet, finance, medical care, and education. In financial business scenarios, in order to answer customers' questions about financial products, transaction processes, and other business content, customer service personnel are usually deployed to communicate with customers. Due to the complexity and diversity of financial business, a large number of simple tasks such as consulting services and after-sales services will seriously occupy the energy and time of business personnel, reducing their work efficiency and work quality. The use of intelligent conversation methods based on automatic speech synthesis can save a lot of labor costs, and at the same time, the quality of service to customers can be improved by controlling the synthesized speech. Therefore, speech synthesis technology plays an important auxiliary role in financial business scenarios.

[0003] At present, financial enterprises usually adopt speech synthesis based on large language models, which work by converting speech signals into sequences of tokens. Large language models use text as conditions to simulate these sequences of tokens. Then, a token vocoder is used to reconstruct the original waveform from the tokenized speech to complete the generation of synthesized speech. However, the synthesized speech generated by this processing method often has problems such as unnaturalness and poor quality, which leads to poor user experience. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech generation method, apparatus, computer equipment and storage medium based on artificial intelligence to solve the technical problem that the speech synthesis method based on large language models adopted by existing financial enterprises has poor quality of generated synthesized speech.

[0005] In order to solve the above technical problems, the present application provides a method for generating speech based on artificial intelligence, which adopts the following technical solutions:

[0006] Receive the text to be synthesized and the initial speech input by the user;

[0007] Preprocessing the text to be synthesized to obtain a corresponding specified text;

[0008] Performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector;

[0009] Extracting a corresponding speaker embedding vector from the initial speech;

[0010] Processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence;

[0011] Processing the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram;

[0012] The mel-spectrogram is converted based on a preset vocoder to obtain corresponding synthesized speech.

[0013] Furthermore, the step of processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence specifically includes:

[0014] Calling the large language model;

[0015] Processing the text encoding vector and the speaker embedding vector based on the large language model to obtain a corresponding first speech tag sequence;

[0016] performing a process of removing duplicate items from the first speech mark sequence to obtain a corresponding second speech mark sequence;

[0017] Performing sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence;

[0018] Performing sequence smoothing processing on the third speech mark sequence to obtain a corresponding fourth speech mark sequence;

[0019] The fourth speech mark sequence is used as the target speech mark sequence.

[0020] Furthermore, the step of performing sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence specifically includes:

[0021] Obtaining a sequence length of the second speech mark sequence;

[0022] Get the preset target sequence length;

[0023] Determining whether the sequence length is less than the target sequence length;

[0024] If yes, perform element addition processing on the second speech mark sequence based on a preset element addition strategy to obtain a second speech mark sequence after element addition, and use the second speech mark sequence after element addition as the third speech mark sequence;

[0025] If not, perform element deletion processing on the second speech mark sequence based on a preset element deletion strategy to obtain a second speech mark sequence after element deletion, and use the second speech mark sequence after element deletion as the third speech mark sequence.

[0026] Furthermore, the step of extracting the corresponding speaker embedding vector from the initial speech specifically includes:

[0027] Call the preset voiceprint model;

[0028] Inputting the initial speech into the voiceprint model;

[0029] The initial speech is subjected to vector extraction processing based on the voiceprint model to obtain the speaker embedding vector corresponding to the initial speech.

[0030] Furthermore, before the step of processing the target speech tag sequence and the speaker embedding vector based on the preset target conditional stream matching model to obtain a corresponding mel-spectrogram, the method further includes:

[0031] Get the initial condition flow matching model;

[0032] Get the preset cosine scheduler;

[0033] Adjusting the initial conditional flow matching model based on the cosine scheduler to obtain a corresponding first conditional flow matching model;

[0034] Optimizing the first conditional flow matching model based on a preset classifier free guidance strategy to obtain a corresponding second conditional flow matching model;

[0035] The target conditional flow matching model is constructed based on the second conditional flow matching model.

[0036] Furthermore, after the step of converting the Mel-spectrogram based on the preset vocoder to obtain the corresponding synthesized speech, the method further includes:

[0037] Optimizing the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech;

[0038] Generate a corresponding target audio file based on the target synthesized speech;

[0039] Get the preset push method;

[0040] Based on the pushing method, the target audio file is pushed to the user.

[0041] Furthermore, the step of optimizing the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech specifically includes:

[0042] Performing denoising processing on the synthesized speech to obtain a corresponding first synthesized speech;

[0043] Performing volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech;

[0044] performing range compression processing on the second synthesized speech to obtain a corresponding third synthesized speech;

[0045] The third synthesized speech is used as the target synthesized speech.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a speech generation device based on artificial intelligence, which adopts the following technical solution:

[0047] A receiving module, used for receiving the text to be synthesized and the initial speech input by the user;

[0048] A preprocessing module, used for preprocessing the text to be synthesized to obtain a corresponding specified text;

[0049] An encoding module, used for performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector;

[0050] An extraction module, configured to extract a corresponding speaker embedding vector from the initial speech;

[0051] A first processing module, configured to process the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence;

[0052] A second processing module is used to process the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram;

[0053] The conversion module is used to convert the Mel-spectrogram based on a preset vocoder to obtain corresponding synthesized speech.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0055] Receive the text to be synthesized and the initial speech input by the user;

[0056] Preprocessing the text to be synthesized to obtain a corresponding specified text;

[0057] Performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector;

[0058] Extracting a corresponding speaker embedding vector from the initial speech;

[0059] Processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence;

[0060] Processing the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram;

[0061] The mel-spectrogram is converted based on a preset vocoder to obtain corresponding synthesized speech.

[0062] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0063] Receive the text to be synthesized and the initial speech input by the user;

[0064] Preprocessing the text to be synthesized to obtain a corresponding specified text;

[0065] Performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector;

[0066] Extracting a corresponding speaker embedding vector from the initial speech;

[0067] Processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence;

[0068] Processing the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram;

[0069] The mel-spectrogram is converted based on a preset vocoder to obtain corresponding synthesized speech.

[0070] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0071] The present application first receives the text to be synthesized and the initial speech input by the user; and pre-processes the text to be synthesized to obtain the corresponding specified text; then performs text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector; extracts the corresponding speaker embedding vector from the initial speech; then processes the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence; subsequently, processes the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding Mel-spectrogram; finally, converts the Mel-spectrogram based on a preset vocoder to obtain the corresponding synthesized speech. The present application receives the text to be synthesized and the initial speech input by the user, then encodes the preprocessed designated text based on the use of a text encoder to obtain a text encoding vector, and extracts the corresponding speaker embedding vector from the initial speech, and then processes the text encoding vector and the speaker embedding vector based on the use of a large language model to generate a target speech tag sequence, and then processes the target speech tag sequence and the speaker embedding vector based on a target conditional stream matching model to obtain a Mel-spectrogram, and subsequently converts the Mel-spectrogram based on the use of a vocoder, thereby realizing rapid synthesis from text to high-quality speech, effectively improving the speech quality of the generated synthesized speech, and helping to improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0074] Figure 2 A flowchart of an embodiment of a method for speech generation based on artificial intelligence according to the present application;

[0075] Figure 3 is a structural schematic diagram of an embodiment of an artificial intelligence-based speech generation device according to the present application;

[0076] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0078] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0079] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0080] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0081] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0082] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc.

[0083] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0084] It should be noted that the artificial intelligence-based speech generation method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based speech generation device is generally arranged in the server / terminal device.

[0085] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0086] Continue to refer Figure 2 , showing a flow chart of an embodiment of the speech generation method based on artificial intelligence according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The speech generation method based on artificial intelligence provided in the embodiment of the present application can be applied to any scenario that requires speech synthesis, and the speech generation method based on artificial intelligence can be applied to products in these scenarios, for example, speech synthesis in the field of finance and insurance. The speech generation method based on artificial intelligence comprises the following steps:

[0087] Step S201, receiving the text to be synthesized and the initial speech input by the user.

[0088] In this embodiment, the electronic device (e.g. Figure 1The server / terminal device shown in the figure) can obtain the text to be synthesized and the initial voice input by the user through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (ultra wi deband) connection, and other wireless connection methods currently known or to be developed in the future. The execution subject of this application is specifically a speech synthesis system, or simply referred to as a system. The user can input the text to be synthesized through the interface or API of the voice system, and can also input the initial voice. Among them, in the business scenario of product push of financial insurance, the above-mentioned text to be synthesized may be a business text related to transaction data, payment data, business data, etc. The above-mentioned initial voice may be a voice containing useful acoustic features, and the acoustic features may include pitch, sound intensity, sound length, timbre, etc., as well as a more advanced speaker embedding vector. The speaker embedding vector is used to characterize the identity characteristics of the speaker.

[0089] The speech synthesis system of the present application consists of four components, including a text encoder, a speech segmenter, a large language model and a target conditional stream matching model. Specifically, the text encoder is used to align the semantic space of text and speech tags, while the speech segmenter is used for semantic tagging. By using a large language model to learn the sequence mapping relationship between the entire text encoding and speech tags, the TTS (text to speech) task is restated as an autoregressive sequence generation problem for a given text prompt. The target conditional stream matching model is used to convert speech tags into mel-spectrograms through a denoising process on an optimal path. In order to obtain a perceptible signal, a vocoder is subsequently used to synthesize a waveform using the generated mel-spectrogram as input.

[0090] Step S202, preprocessing the text to be synthesized to obtain the corresponding specified text.

[0091] In this embodiment, the text to be synthesized is preprocessed, specifically including removing unnecessary spaces and punctuation marks, performing lowercase processing, etc., to ensure the consistency of the text format, thereby obtaining the corresponding designated text.

[0092] Step S203: performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector.

[0093] In this embodiment, the above-mentioned text encoder can specifically adopt a pre-trained text encoder, such as an encoder based on Transformer. The pre-processed specified text is converted into a token sequence through a tokenizer, and then the token sequence is input into the text encoder. The text encoder outputs a text encoding vector aligned with the semantic space of the speech tag. This text encoding vector captures the semantic information of the specified text and is used to provide input for subsequent steps.

[0094] Step S204: extracting a corresponding speaker embedding vector from the initial speech.

[0095] In this embodiment, the specific implementation process of extracting the corresponding speaker embedding vector from the initial speech will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0096] Step S205: Process the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence.

[0097] In this embodiment, the specific implementation process of processing the text encoding vector and the speaker embedding vector based on the preset large language model to generate the corresponding target speech tag sequence will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated here.

[0098] Step S206: Process the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram.

[0099] In this embodiment, the target conditional stream matching model is a model generated by adjusting and optimizing the initial conditional stream matching model based on a preset cosine scheduler and classifier free guidance strategy. And the target conditional stream matching model can generate a matching mel-spectrogram according to the generated speech tag sequence and speaker embedding vector.

[0100] Step S207: converting the mel-spectrogram based on a preset vocoder to obtain corresponding synthesized speech.

[0101] In this embodiment, the vocoder has the function of synthesizing an audio waveform based on the input mel-spectrogram. By inputting the generated mel-spectrogram into the vocoder, the vocoder converts the mel-spectrogram into a final audio waveform, that is, obtaining the synthesized speech.

[0102] The present application first receives the text to be synthesized and the initial speech input by the user; and pre-processes the text to be synthesized to obtain the corresponding specified text; then performs text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector; extracts the corresponding speaker embedding vector from the initial speech; then processes the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence; subsequently, processes the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding Mel-spectrogram; finally, converts the Mel-spectrogram based on a preset vocoder to obtain the corresponding synthesized speech. The present application receives the text to be synthesized and the initial speech input by the user, then encodes the preprocessed designated text based on the use of a text encoder to obtain a text encoding vector, and extracts the corresponding speaker embedding vector from the initial speech, and then processes the text encoding vector and the speaker embedding vector based on the use of a large language model to generate a target speech tag sequence, and then processes the target speech tag sequence and the speaker embedding vector based on a target conditional stream matching model to obtain a Mel-spectrogram, and subsequently converts the Mel-spectrogram based on the use of a vocoder, thereby realizing rapid synthesis from text to high-quality speech, effectively improving the speech quality of the generated synthesized speech, and helping to improve the user experience.

[0103] In some optional implementations, step S205 includes the following steps:

[0104] The large language model is called.

[0105] In this embodiment, the above-mentioned large language model is a large language model architecture suitable for sequence generation, such as GPT, BERT, etc. Specifically, the TTS (text to speech) task is expressed as an autoregressive speech tag generation problem of a large language model (LLM). For a large language model, sequence construction is the most important issue, and its construction method is as follows: in, and Represent the beginning and end of the sequence respectively. v is the speaker embedding vector extracted from the initial speech X, obtained through the pre-trained voiceprint model. The text encoding is obtained by passing the input text to be synthesized through the byte pair encoding (BPE) tokenizer and text encoder. Since text and speech tags are at different semantic levels, a text encoder is used to align their semantic spaces and facilitate the modeling of large language models. A start identifier is inserted between the text encoding and the speech tag sequence extracted by the supervised semantic tagger. During the training phase, we adopted a teacher forcing scheme, where the left-shifted sequence was used as the model input and the original sequence as the expected output. During training, only speech tokens and end-of-speech tokens were considered. The cross entropy loss is used to optimize the performance of the large language model. In addition, the training process of the large language model specifically includes: taking the text encoding and the speaker embedding vector as input. The large language model generates a corresponding speech tag sequence based on the input. Subsequently, the cross entropy loss is used to evaluate the performance of the large language model, and the parameters of the large language model are optimized through the back propagation algorithm, so as to construct a large language model that can generate a corresponding speech tag sequence based on the text encoding.

[0106] The text encoding vector and the speaker embedding vector are processed based on the large language model to obtain a corresponding first speech tag sequence.

[0107] In this embodiment, the text encoding vector and the speaker embedding vector are input into the large language model, so that the text encoding vector and the speaker embedding vector are processed by the large language model, and a corresponding first speech tag sequence is output. The first speech tag sequence may include phonemes, syllables or other speech units.

[0108] The first speech mark sequence is processed by removing duplicate items to obtain a corresponding second speech mark sequence.

[0109] In this embodiment, by removing duplicates from the first speech mark sequence, unnecessary duplicate elements are avoided in the generated sequence, which may be caused by model errors or coding problems. Specifically, starting from the first element of the sequence, each element in the sequence is traversed one by one. Then for each element in the sequence, check whether it is the same as the next element. If it is found that the adjacent elements are the same, it is decided to delete one of them (usually the second one) according to the needs. After deleting or modifying the duplicates, the sequence is updated to reflect the corresponding changes.

[0110] The second speech mark sequence is subjected to sequence length adjustment processing to obtain a corresponding third speech mark sequence.

[0111] In this embodiment, the specific implementation process of adjusting the sequence length of the second speech mark sequence to obtain the corresponding third speech mark sequence will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0112] Perform sequence smoothing processing on the third speech mark sequence to obtain a corresponding fourth speech mark sequence.

[0113] In this embodiment, the sequence smoothing process includes: analyzing the generated third speech mark sequence to determine the pronunciation duration of each mark. Then, adjusting the pronunciation duration of each mark according to linguistic rules, statistical models or user preferences. For example, some phonemes may tend to have a longer pronunciation time, while others may have a shorter pronunciation time. Then, the adjusted pronunciation duration information is integrated into the sequence for use in the subsequent speech synthesis step.

[0114] The fourth speech mark sequence is used as the target speech mark sequence.

[0115] The present application processes the text encoding vector and the speaker embedding vector by calling the large language model to obtain the corresponding first speech mark sequence, and then automatically and intelligently removes duplicate items, adjusts the sequence length, and smoothes the sequence of the first speech mark sequence, thereby completing the automatic optimization processing of the speech mark sequence output by the large language model and generating a corresponding target speech mark sequence, effectively ensuring that the generated target speech mark sequence can meet the requirements of the subsequent speech synthesis processing steps, which is conducive to ensuring the smooth progress of the speech synthesis processing, and then helps to improve the speech quality of the generated synthesized speech.

[0116] In some optional implementations of this embodiment, performing sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence includes the following steps:

[0117] The sequence length of the second speech mark sequence is obtained.

[0118] In this embodiment, the length of the second speech mark sequence is counted to obtain the corresponding sequence length.

[0119] Get the preset target sequence length.

[0120] In this embodiment, there is no specific limitation on the selection of the above-mentioned target sequence length, which can be set according to actual business usage requirements. The target sequence length can be a fixed length or dynamically determined based on a certain rule (such as being the same as the reference sequence length).

[0121] It is determined whether the sequence length is less than the target sequence length.

[0122] In this embodiment, the sequence length of the generated second speech mark sequence can be numerically compared with the target sequence length to obtain a corresponding numerical comparison result, wherein the numerical comparison result includes that the sequence length is less than the target sequence length, or the sequence length is greater than the target sequence length.

[0123] If so, element addition processing is performed on the second speech mark sequence based on a preset element addition strategy to obtain a second speech mark sequence after element addition, and the second speech mark sequence after element addition is used as the third speech mark sequence.

[0124] In this embodiment, the strategy content of the element adding strategy may include: if the sequence length of the generated second speech mark sequence is less than the target length, then adding a padding element to the second speech mark sequence to reach the target length. Exemplarily, the padding element may be a specific placeholder or the last element of the repeated sequence.

[0125] If not, perform element deletion processing on the second speech mark sequence based on a preset element deletion strategy to obtain a second speech mark sequence after element deletion, and use the second speech mark sequence after element deletion as the third speech mark sequence.

[0126] In this embodiment, the strategy content of the above-mentioned element deletion strategy may include: if the sequence length of the generated second voice marker sequence is greater than the above-mentioned target length, then deleting the element at the end of the second voice marker sequence, or selectively deleting the elements in the second voice marker sequence based on probability or importance so that the length of the second voice marker sequence is equal to the above-mentioned target length.

[0127] The present application obtains the sequence length of the second speech marker sequence and the preset target sequence length, and then, based on the comparison result of the sequence length and the target sequence length, intelligently and accurately adopts an element addition strategy or an element deletion strategy to adjust the sequence length of the second speech marker sequence corresponding to the target length, thereby effectively ensuring that the generated second speech marker sequence meets specific requirements and ensuring the accuracy and standardization of the obtained second speech marker sequence.

[0128] In some optional implementations, step S204 includes the following steps:

[0129] Call the preset voiceprint model.

[0130] In this embodiment, the above is a pre-trained voiceprint model with acoustic feature extraction function.

[0131] The initial speech is input into the voiceprint model.

[0132] In this embodiment, the initial speech can be used as a model input and input into the voiceprint model. The voiceprint model can extract useful acoustic features from the initial speech, and these acoustic features may include pitch, intensity, duration, timbre, etc., as well as more advanced speaker embedding vectors.

[0133] The initial speech is subjected to vector extraction processing based on the voiceprint model to obtain the speaker embedding vector corresponding to the initial speech.

[0134] In this embodiment, the above-mentioned speaker embedding vector is used to characterize the identity feature of the speaker. Among them, through the speaker embedding vector, the model can extract the unique voice features of the speaker, such as timbre, intonation, speaking speed, etc. In addition, in the applications of speech synthesis and speech replication, the speaker embedding vector can be input into the model as a condition to control the speaker style of the generated speech. For example, in the task of text to speech (TTS), the text encoding vector and the speaker embedding vector can be used as input to generate speech output with a specific speaker style through the model. In the task of speech replication, a new speaker embedding vector can be inferred from a small number of speech samples, and then applied to a multi-speaker generation model to generate speech with the speaker style.

[0135] The present application calls a preset voiceprint model; then inputs the initial speech into the voiceprint model; and then performs vector extraction processing on the initial speech based on the voiceprint model to obtain the speaker embedding vector corresponding to the initial speech. The present application performs vector extraction processing on the initial speech based on the use of the voiceprint model, and can achieve fast and accurate extraction of the speaker embedding vector corresponding to the initial speech, thereby improving the efficiency and intelligence of acquiring the speaker embedding vector.

[0136] In some optional implementations, before step S206, the electronic device may further perform the following steps:

[0137] Get the initial condition flow matching model.

[0138] In this embodiment, according to the actual processing requirements, an initial conditional flow matching model is designed based on optimal transport (OT) and continuous normalized flow (CNF). The training process of the initial conditional flow matching model includes: taking the generated speech tag sequence and speaker embedding vector as conditional input. Then, the target Mel spectrogram distribution is gradually generated from the prior distribution by solving the ordinary differential equation (ODE). Subsequently, the optimal transport loss (such as Wasser stein distance) is used to measure the similarity between the generated Mel spectrogram and the target distribution, and the OT-CFM parameters are optimized by the back propagation algorithm to obtain an initial conditional flow matching model that meets the requirements. The distribution of the Mel spectrogram is learned by adopting the optimal transport conditional flow matching model (OT-CFM), and samples are generated from the distribution using the generated speech tags as conditions. Compared with diffusion probability models (DPMs), OT-CFM has simpler gradients, is easier to train, and has a faster generation speed, so it can achieve better performance. In the continuous time normalized flow (CNFs), a probability density path is constructed from the prior distribution p0(X) to the Mel spectrogram data distribution q(X).

[0139] Get the preset cosine scheduler.

[0140] In this embodiment, OT-CFM gradually generates the target mel-spectrogram distribution from the prior distribution by solving ordinary differential equations (ODE). In this process, a cosine scheduler can be used to control the time step, that is, the time step t starts from 0 and gradually increases with the iteration until it reaches a maximum value of 1, so as to ensure the smoothness of the mel-spectrogram generation process.

[0141] The initial conditional flow matching model is adjusted based on the cosine scheduler to obtain a corresponding first conditional flow matching model.

[0142] In this embodiment, by adapting the cosine scheduler into the initial conditional flow matching model, it helps to provide a smoother transition in the early stage of the mel-spectrogram generation process, thereby making it easier for the model to learn the transition from the initial state to the target state.

[0143] The first conditional flow matching model is optimized based on a preset classifier free guidance strategy to obtain a corresponding second conditional flow matching model.

[0144] In this embodiment, the above-mentioned classifier free guidance strategy is specifically classifier free guidance (CFG), which adapts CFG to the first conditional flow matching model. During the training phase, the conditions are randomly discarded with a fixed probability of 0.2, so that conditional and unconditional flows can be learned, and the context learning ability can be enhanced by shielding 70% to 100% of the pre-feature conditions.

[0145] The target conditional flow matching model is constructed based on the second conditional flow matching model.

[0146] In this embodiment, the second conditional stream matching model can be used as the final target conditional stream matching model. Alternatively, the second conditional stream matching model can also be masked and optimized to obtain the target conditional stream matching model. Specifically, by using masked speech to train the model to learn the intrinsic structure and characteristics of the speech signal, by randomly masking part of the speech frames, the model needs to predict or reconstruct the masked frames based on the remaining unmasked frames, thereby forcing the model to learn a more robust and comprehensive speech representation. This self-supervised learning method does not require a large amount of labeled data, can make full use of unlabeled speech resources, and reduce the cost of data collection and labeling.

[0147] The present application obtains an initial conditional stream matching model, and then adjusts and optimizes the initial conditional stream matching model based on the use of a cosine scheduler and a classifier free-guiding strategy, so as to realize a rapid and intelligent construction of a corresponding target conditional stream matching model, thereby improving the model processing efficiency of the target conditional stream matching model. The target speech tag sequence and the speaker embedding vector are subsequently processed based on the target conditional stream matching model, so as to improve the quality of the obtained mel-spectrogram.

[0148] In some optional implementations of this embodiment, after step S207, the electronic device may further perform the following steps:

[0149] The synthesized speech is optimized based on a preset quality optimization strategy to obtain a corresponding target synthesized speech.

[0150] In this embodiment, the above-mentioned specific implementation process of optimizing the synthesized speech based on the preset quality optimization strategy to obtain the corresponding target synthesized speech will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated in detail here.

[0151] A corresponding target audio file is generated based on the target synthesized speech.

[0152] In this embodiment, the target synthesized speech can be saved as an audio file (such as WAV, MP3, etc. format), so as to obtain a corresponding target audio file.

[0153] Get the preset push method.

[0154] In this embodiment, there is no specific limitation on the selection of the above push method, for example, methods such as sending by email, sending by interface, and sending by application may be used.

[0155] Based on the pushing method, the target audio file is pushed to the user.

[0156] In this embodiment, the target audio file is pushed to the user by using the push method, so that the user can hear the speech generated by converting the input text to be synthesized.

[0157] The present application optimizes the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech; then generates a corresponding target audio file based on the target synthesized speech; then obtains a preset push method; and subsequently pushes the target audio file to the user based on the push method. The present application converts the Mel-spectrogram based on a preset vocoder to obtain a synthesized speech, and then optimizes the synthesized speech based on the use of a quality optimization strategy, effectively improving the speech quality of the generated target synthesized speech. Subsequently, a corresponding target audio file is generated based on the target synthesized speech, and based on the obtained push method, the target audio file is pushed to the user, so that the user can hear the high-quality speech generated by the conversion of the input text to be synthesized, which is conducive to improving the user experience.

[0158] In some optional implementations of this embodiment, the optimizing process on the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech includes the following steps:

[0159] The synthesized speech is subjected to denoising processing to obtain a corresponding first synthesized speech.

[0160] In this embodiment, the denoising process refers to the process of removing unnecessary background noise or interference sound in the synthesized speech, which can be performed by analyzing the spectrum characteristics of the noise in the synthesized speech and then subtracting the estimated noise spectrum from the spectrum of the noisy speech.

[0161] Volume adjustment processing is performed on the first synthesized speech to obtain a corresponding second synthesized speech.

[0162] In this embodiment, the volume adjustment process refers to a process of changing the overall loudness of the first synthesized speech. Specifically, the amplitude of the first synthesized speech may be scaled to increase or decrease the volume of the first synthesized speech. Specifically, this may be achieved by multiplying the first synthesized speech by a constant (gain factor). Alternatively, the gain of the audio signal may be dynamically adjusted using an automatic gain control (AGC) to maintain a consistent volume level.

[0163] Perform range compression processing on the second synthesized speech to obtain a corresponding third synthesized speech.

[0164] In this embodiment, by performing dynamic range compression processing on the second synthesized speech, the difference between the highest and lowest levels in the second synthesized speech can be effectively reduced to make it sound smoother and more consistent, thereby improving the auditory quality of the target synthesized speech.

[0165] The third synthesized speech is used as the target synthesized speech.

[0166] The present application performs denoising, volume adjustment and range compression on the synthesized speech, thereby achieving intelligent and accurate optimization of the synthesized speech, and effectively improving the speech quality of the generated target synthesized speech.

[0167] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0168] In addition, the non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0169] The codec-based speech synthesis scheme proposed in this application consists of a large language model for text-to-token generation and a conditional stream matching model for token-to-speech synthesis. The model adopts a conditional stream matching method, which can accelerate training and reasoning compared with traditional diffusion models. The language synthesis system of this application is a scalable zero-shot TTS synthesis system that combines a large language model for text-to-token generation and a conditional stream matching model for token-to-speech synthesis, eliminating the need for additional phonemizers and forced aligners. In order to further improve the quality of generated speech, speech modeling is divided into three components: semantics, speaker, and prosody by integrating x-vector into the large language model. The large language is responsible for modeling semantic content and prosody, while the conditional stream matching model captures timbre and environmental information. In addition, the stream matching process is further optimized using techniques such as classifier-free guidance, cosine scheduler, and mask conditions.

[0170] In addition, the scalable multilingual speech generation system proposed in this application supports zero-sample context learning, cross-language sound cloning, command generation, and fine control of emotional and paralinguistic features. The model can be expanded through enhanced command following capabilities. Specifically, it supports control of various aspects, such as speaker identity (i.e., speaker characteristics), speaking style (including emotion, gender, speaking rate, and pitch), and fine paralinguistic features. These features include the ability to insert laughter, breathing, talking while laughing, and emphasizing certain words. At the same time, the ability of zero-sample context learning is demonstrated, allowing any sound to be copied with only a short reference voice sample. In addition, speaker embedding and mel-spectrograms of prompt speech are integrated to further enhance sound quality and environmental consistency.

[0171] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0172] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned synthesized speech, the above-mentioned synthesized speech can also be stored in a node of a blockchain.

[0173] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0174] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0175] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0177] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0178] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech generation device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0179] like Figure 3 As shown, the speech generation device 300 based on artificial intelligence described in this embodiment includes: a receiving module 301, a preprocessing module 302, an encoding module 303, an extraction module 304, a first processing module 305, a second processing module 306 and a conversion module 307. Among them:

[0180] The receiving module 301 is used to receive the text to be synthesized and the initial speech input by the user;

[0181] A preprocessing module 302 is used to preprocess the text to be synthesized to obtain a corresponding specified text;

[0182] The encoding module 303 is used to perform text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector;

[0183] An extraction module 304, configured to extract a corresponding speaker embedding vector from the initial speech;

[0184] A first processing module 305 is used to process the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence;

[0185] A second processing module 306 is used to process the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram;

[0186] The conversion module 307 is used to convert the Mel-spectrogram based on a preset vocoder to obtain corresponding synthesized speech.

[0187] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0188] In some optional implementations of this embodiment, the first processing module 305 includes:

[0189] A first calling submodule, used for calling the large language model;

[0190] A first processing submodule, configured to process the text encoding vector and the speaker embedding vector based on the large language model to obtain a corresponding first speech tag sequence;

[0191] A second processing submodule is used to remove duplicate items from the first speech mark sequence to obtain a corresponding second speech mark sequence;

[0192] A third processing submodule, configured to perform sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence;

[0193] A fourth processing submodule, configured to perform sequence smoothing processing on the third speech mark sequence to obtain a corresponding fourth speech mark sequence;

[0194] The first determining submodule is configured to use the fourth speech mark sequence as the target speech mark sequence.

[0195] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0196] In some optional implementations of this embodiment, the third processing submodule includes:

[0197] A first acquiring unit, configured to acquire a sequence length of the second speech mark sequence;

[0198] A second acquisition unit, used to acquire a preset target sequence length;

[0199] A judging unit, used to judge whether the sequence length is less than the target sequence length;

[0200] A first processing unit is configured to, if yes, perform element addition processing on the second speech mark sequence based on a preset element addition strategy to obtain a second speech mark sequence after element addition, and use the second speech mark sequence after element addition as the third speech mark sequence;

[0201] The second processing unit is used to, if not, perform element deletion processing on the second voice marker sequence based on a preset element deletion strategy to obtain a second voice marker sequence after element deletion, and use the second voice marker sequence after element deletion as the third voice marker sequence.

[0202] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0203] In some optional implementations of this embodiment, the extraction module 304 includes:

[0204] The second calling submodule is used to call a preset voiceprint model;

[0205] An input submodule, used for inputting the initial speech into the voiceprint model;

[0206] The extraction submodule is used to perform vector extraction processing on the initial speech based on the voiceprint model to obtain the speaker embedding vector corresponding to the initial speech.

[0207] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0208] In some optional implementations of this embodiment, the artificial intelligence-based speech generation device further includes:

[0209] A first acquisition module is used to acquire an initial condition flow matching model;

[0210] A second acquisition module, used to acquire a preset cosine scheduler;

[0211] An adjustment module, configured to adjust the initial conditional flow matching model based on the cosine scheduler to obtain a corresponding first conditional flow matching model;

[0212] A first optimization module, used for optimizing the first conditional flow matching model based on a preset classifier free guidance strategy to obtain a corresponding second conditional flow matching model;

[0213] A construction module is used to construct the target conditional flow matching model based on the second conditional flow matching model.

[0214] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0215] In some optional implementations of this embodiment, the artificial intelligence-based speech generation device further includes:

[0216] A second optimization module is used to optimize the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech;

[0217] A generating module, used for generating a corresponding target audio file based on the target synthesized speech;

[0218] The second acquisition module is used to acquire a preset push method;

[0219] A push module is used to push the target audio file to the user based on the push method.

[0220] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0221] In some optional implementations of this embodiment, the second optimization module includes:

[0222] A denoising submodule, used for performing denoising processing on the synthesized speech to obtain a corresponding first synthesized speech;

[0223] An adjustment submodule, used for performing volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech;

[0224] A compression submodule, configured to perform range compression processing on the second synthesized speech to obtain a corresponding third synthesized speech;

[0225] The second determining submodule is used to use the third synthesized speech as the target synthesized speech.

[0226] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech generation method in the aforementioned implementation mode, and will not be repeated here.

[0227] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0228] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0229] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0230] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a speech generation method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0231] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the artificial intelligence-based speech generation method.

[0232] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0233] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based speech generation method.

[0234] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0235] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech generation method based on artificial intelligence, characterized in that: The steps include: Receive the text to be synthesized and the initial speech input by the user; Preprocessing the text to be synthesized to obtain a corresponding specified text; Performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector; Extracting a corresponding speaker embedding vector from the initial speech; Processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence; Processing the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram; The mel-spectrogram is converted based on a preset vocoder to obtain corresponding synthesized speech.

2. The method for generating speech based on artificial intelligence according to claim 1, characterized in that: The step of processing the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence specifically includes: Calling the large language model; Processing the text encoding vector and the speaker embedding vector based on the large language model to obtain a corresponding first speech tag sequence; performing a process of removing duplicate items from the first speech mark sequence to obtain a corresponding second speech mark sequence; Performing sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence; Performing sequence smoothing processing on the third speech mark sequence to obtain a corresponding fourth speech mark sequence; The fourth speech mark sequence is used as the target speech mark sequence.

3. The method for generating speech based on artificial intelligence according to claim 2, characterized in that: The step of performing sequence length adjustment processing on the second speech mark sequence to obtain a corresponding third speech mark sequence specifically includes: Obtaining a sequence length of the second speech mark sequence; Get the preset target sequence length; Determining whether the sequence length is less than the target sequence length; If yes, perform element addition processing on the second speech mark sequence based on a preset element addition strategy to obtain a second speech mark sequence after element addition, and use the second speech mark sequence after element addition as the third speech mark sequence; If not, perform element deletion processing on the second speech mark sequence based on a preset element deletion strategy to obtain a second speech mark sequence after element deletion, and use the second speech mark sequence after element deletion as the third speech mark sequence.

4. The method for generating speech based on artificial intelligence according to claim 1, characterized in that: The step of extracting the corresponding speaker embedding vector from the initial speech specifically includes: Call the preset voiceprint model; Inputting the initial speech into the voiceprint model; The initial speech is subjected to vector extraction processing based on the voiceprint model to obtain the speaker embedding vector corresponding to the initial speech.

5. The method for generating speech based on artificial intelligence according to claim 1, characterized in that: Before the step of processing the target speech tag sequence and the speaker embedding vector based on the preset target conditional stream matching model to obtain a corresponding mel-spectrogram, the method further includes: Get the initial condition flow matching model; Get the preset cosine scheduler; Adjusting the initial conditional flow matching model based on the cosine scheduler to obtain a corresponding first conditional flow matching model; Optimizing the first conditional flow matching model based on a preset classifier free guidance strategy to obtain a corresponding second conditional flow matching model; The target conditional flow matching model is constructed based on the second conditional flow matching model.

6. The method for generating speech based on artificial intelligence according to claim 1, characterized in that: After the step of converting the Mel-spectrogram based on the preset vocoder to obtain the corresponding synthesized speech, the method further includes: Optimizing the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech; Generate a corresponding target audio file based on the target synthesized speech; Get the preset push method; Based on the pushing method, the target audio file is pushed to the user.

7. The method for generating speech based on artificial intelligence according to claim 6, characterized in that: The step of optimizing the synthesized speech based on a preset quality optimization strategy to obtain a corresponding target synthesized speech specifically includes: Performing denoising processing on the synthesized speech to obtain a corresponding first synthesized speech; Performing volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech; performing range compression processing on the second synthesized speech to obtain a corresponding third synthesized speech; The third synthesized speech is used as the target synthesized speech.

8. A speech generation device based on artificial intelligence, characterized in that: include: A receiving module, used for receiving the text to be synthesized and the initial speech input by the user; A preprocessing module, used for preprocessing the text to be synthesized to obtain a corresponding specified text; An encoding module, used for performing text encoding processing on the specified text based on a preset text encoder to obtain a corresponding text encoding vector; An extraction module, configured to extract a corresponding speaker embedding vector from the initial speech; A first processing module, configured to process the text encoding vector and the speaker embedding vector based on a preset large language model to generate a corresponding target speech tag sequence; A second processing module is used to process the target speech tag sequence and the speaker embedding vector based on a preset target conditional stream matching model to obtain a corresponding mel-spectrogram; The conversion module is used to convert the Mel-spectrogram based on a preset vocoder to obtain corresponding synthesized speech.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7 are implemented.

Images