Voice processing method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an artificial intelligence-based voice processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the preprocessing of an obtained to-be-converted text, and obtaining a target text; extracting speaker embedding of a target speaker from the reference audio; calling a target voice conversion model; wherein the target voice conversion model at least comprises an acoustic model and a discriminator; performing feature extraction on the to-be-converted text and speaker embedding based on the acoustic model to obtain initial acoustic features and initial rhythm features; evaluating the initial acoustic features and the initial rhythm features based on a discriminator to obtain an evaluation result; optimizing the initial acoustic feature and the initial rhythm feature based on an evaluation result to obtain a target acoustic feature and a target rhythm feature; performing conversion processing on the target acoustic features and the target rhythm features based on a vocoder to obtain synthetic speech; and performing output processing on the synthesized voice. According to the invention, the speech generation quality of speech synthesis processing is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence development technology, financial technology, and medical health, and in particular to artificial intelligence-based speech processing methods, devices, computer equipment, and storage media. Background Art

[0002] With the rapid development of information technology, text-to-speech (TTS) systems have received more and more attention and applications in the financial and healthcare fields. When it comes to zero-shot voice cloning technology, traditional TTS models used in TTS systems usually rely on reconstruction loss for training. This reconstruction loss mainly measures the error between the generated acoustic features and the real acoustic features. By minimizing this error, the model can gradually learn how to generate the corresponding speech signal from the text. However, there is a significant problem with using reconstruction loss during training, that is, the model tends to predict the average value. This phenomenon is called "mean regression" or "mean shift". Specifically, when the model tries to minimize the reconstruction error of all samples during training, it tends to find a solution that can achieve good average performance on all samples, rather than finding the most accurate solution for each sample.

[0003] In practical applications of TTS systems, this mean regression problem causes the generated speech to be too smooth and lack the diversity and variation found in natural speech. This is because human speech is highly complex and diverse, containing many subtle changes in timbre, intonation, and speech rate, which play a vital role in voice communication. However, when the TTS model relies solely on reconstruction loss for training, it cannot fully capture these subtle changes, so the generated speech is often closer to the average behavior of the dataset rather than the unique speech characteristics of a specific speaker.

[0004] This averaged speech generation method not only reduces the naturalness and realism of the speech, but also limits the performance of the TTS system in advanced applications such as zero-shot speech cloning. Zero-shot speech cloning refers to the generation of speech that is highly similar to the target speaker's voice using only a small number or no speech samples of the target speaker. In this application scenario, the model needs to have a high degree of generalization and fine speech feature capture capabilities, but the traditional TTS model that relies on reconstruction loss obviously cannot meet these requirements, resulting in low speech generation quality in speech synthesis processing. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a speech processing method, apparatus, computer device and storage medium based on artificial intelligence to solve the technical problem that the existing TTS model that relies on reconstruction loss has low speech generation quality in speech synthesis processing.

[0006] In order to solve the above technical problems, the present application provides a speech processing method based on artificial intelligence, which adopts the following technical solutions:

[0007] Obtaining input text to be converted, and preprocessing the text to be converted to obtain a target text;

[0008] Extract the speaker embedding of the target speaker from the preset reference audio;

[0009] Calling a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator;

[0010] Extracting features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features;

[0011] evaluating the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results;

[0012] Optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features;

[0013] The target acoustic feature and the target prosodic feature are converted based on a preset vocoder to obtain corresponding synthesized speech;

[0014] The synthesized speech is outputted.

[0015] Furthermore, before the step of calling the preset target speech conversion model, the method further includes:

[0016] Get a pre-built speaker dataset;

[0017] Calling a preset initial conversion model; wherein the initial conversion model includes an acoustic model constructed by a preset speech synthesis model, and a discriminator constructed by a preset encoder structure;

[0018] determining a first overall loss corresponding to the acoustic model;

[0019] determining a second overall loss corresponding to the discriminator;

[0020] Based on the first overall loss and the second overall loss, using the speaker data set, perform adversarial training on the acoustic model and the discriminator in the initial conversion model to obtain a trained designated model;

[0021] The designated model is used as the target speech conversion model.

[0022] Furthermore, the step of determining a first overall loss corresponding to the acoustic model specifically includes:

[0023] Acquiring preset acoustic feature generation loss, prosodic feature generation loss, and prosodic feature discrimination loss corresponding to the acoustic model;

[0024] Get preset integration strategies;

[0025] Based on the integration strategy, the acoustic feature generation loss, the prosodic feature generation loss and the acoustic feature generation loss are integrated to obtain a corresponding integration loss;

[0026] The first overall loss is generated based on the integrated loss.

[0027] Furthermore, the step of optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features specifically includes:

[0028] Get the preset adjustment strategy;

[0029] Based on the evaluation result, the acoustic model is adjusted using the adjustment strategy until a preset iteration condition is met and an adjusted acoustic model is obtained;

[0030] Based on the adjusted acoustic model, feature extraction is performed on the text to be converted and the speaker embedding to obtain corresponding specified acoustic features and specified prosodic features;

[0031] The designated acoustic feature is used as the target acoustic feature, and the designated prosodic feature is used as the target prosodic feature.

[0032] Furthermore, the step of outputting the synthesized speech specifically includes:

[0033] Performing noise removal processing on the synthesized speech to obtain a corresponding first synthesized speech;

[0034] Adjusting the volume of the first synthesized speech to obtain a corresponding second synthesized speech;

[0035] adjusting the speech rate of the second synthesized speech to obtain a corresponding third synthesized speech;

[0036] The third synthesized speech is output.

[0037] Furthermore, the step of extracting the speaker embedding of the target speaker from the preset reference audio specifically includes:

[0038] Acquire reference audio corresponding to the target speaker;

[0039] Call the preset speaker recognition model;

[0040] Performing acoustic characteristic analysis on the reference audio based on the speaker recognition model, and outputting a corresponding specified vector;

[0041] A speaker embedding of the target speaker is generated based on the specified vector.

[0042] Furthermore, the step of preprocessing the text to be converted to obtain the target text specifically includes:

[0043] Performing word segmentation processing on the text to be converted to obtain a corresponding first text;

[0044] Performing part-of-speech tagging on the first text to obtain a corresponding second text;

[0045] Normalizing the second text to obtain a corresponding third text;

[0046] The third text is used as the target text.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a speech processing device based on artificial intelligence, which adopts the following technical solution:

[0048] A preprocessing module, used to obtain input text to be converted, and preprocess the text to be converted to obtain a target text;

[0049] A first extraction module, configured to extract a speaker embedding of a target speaker from a preset reference audio;

[0050] A first calling module, used to call a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator;

[0051] A second extraction module is used to extract features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features;

[0052] An evaluation module, configured to evaluate the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results;

[0053] An optimization module, configured to optimize the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features;

[0054] A conversion module, configured to convert the target acoustic features and the target prosodic features based on a preset vocoder to obtain corresponding synthesized speech;

[0055] An output module is used to perform output processing on the synthesized speech.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0057] Obtaining input text to be converted, and preprocessing the text to be converted to obtain a target text;

[0058] Extract the speaker embedding of the target speaker from the preset reference audio;

[0059] Calling a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator;

[0060] Extracting features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features;

[0061] evaluating the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results;

[0062] Optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features;

[0063] The target acoustic feature and the target prosodic feature are converted based on a preset vocoder to obtain corresponding synthesized speech;

[0064] The synthesized speech is outputted.

[0065] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0066] Obtaining input text to be converted, and preprocessing the text to be converted to obtain a target text;

[0067] Extract the speaker embedding of the target speaker from the preset reference audio;

[0068] Calling a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator;

[0069] Extracting features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features;

[0070] evaluating the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results;

[0071] Optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features;

[0072] The target acoustic feature and the target prosodic feature are converted based on a preset vocoder to obtain corresponding synthesized speech;

[0073] The synthesized speech is outputted.

[0074] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0075] The present application first obtains an input text to be converted, and preprocesses the text to be converted to obtain a target text; then extracts a speaker embedding of a target speaker from a preset reference audio; then calls a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator; and based on the acoustic model, feature extraction is performed on the text to be converted and the speaker embedding to obtain corresponding initial acoustic features and initial prosodic features; subsequently, the initial acoustic features and the initial prosodic features are evaluated based on the discriminator to obtain corresponding evaluation results; further, based on the evaluation results, the initial acoustic features and the initial prosodic features are optimized to obtain optimized target acoustic features and target prosodic features; subsequently, the target acoustic features and the target prosodic features are converted based on a preset vocoder to obtain corresponding synthesized speech; and finally, the synthesized speech is output. The present application obtains the target text by preprocessing the input text to be converted, and extracts the speaker embedding of the target speaker from the reference audio, and then extracts the features of the text to be converted and the speaker embedding based on the acoustic model in the target speech conversion model to obtain the corresponding initial acoustic features and initial prosodic features, and then evaluates the initial acoustic features and initial prosodic features based on the discriminator in the target speech conversion model to obtain the evaluation results, and then optimizes the initial acoustic features and initial prosodic features based on the evaluation results to obtain the target acoustic features and target prosodic features, and then converts the target acoustic features and target prosodic features based on the vocoder to obtain the synthesized speech similar to the target speaker's voice and outputs it. Through the use of the discriminator, the feature differences between the real speech and the generated speech can be conditionally identified, and then the acoustic characteristics and prosodic characteristics of the target speech conversion model can be improved, so that the processing intelligence of speech synthesis can be effectively improved, and the quality of the generated synthesized speech can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0078] Figure 2 A flowchart of an embodiment of a speech processing method based on artificial intelligence according to the present application;

[0079] Figure 3 is a structural schematic diagram of an embodiment of an artificial intelligence-based speech processing device according to the present application;

[0080] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0082] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0083] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0084] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0085] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0086] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0087] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0088] It should be noted that the artificial intelligence-based speech processing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based speech processing device is generally arranged in the server / terminal device.

[0089] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0090] Continue to refer Figure 2, showing a flow chart of an embodiment of an artificial intelligence-based speech processing method according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The artificial intelligence-based speech processing method provided in the embodiment of the present application can be applied to any scenario that requires speech processing, and the artificial intelligence-based speech processing method can be applied to products in these scenarios, for example, speech processing in the financial insurance field. The artificial intelligence-based speech processing method comprises the following steps:

[0091] Step S201: obtaining input text to be converted, and preprocessing the text to be converted to obtain a target text.

[0092] In this embodiment, the electronic device (e.g. Figure 1 The server / terminal device shown in the figure) can obtain the text to be converted through a wired connection or a wireless connection. It should be pointed out that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The executor of the present application may be a speech processing system, which can be referred to as the system for short. The above-mentioned text to be converted is the text that needs to be converted, which can be a single sentence, a paragraph or a complete document. Among them, the specific implementation process of preprocessing the text to be converted to obtain the target text will be further described in detail in the subsequent specific embodiments of the present application, and will not be elaborated on here.

[0093] Step S202: extracting the speaker embedding of the target speaker from the preset reference audio.

[0094] In this embodiment, the specific implementation process of extracting the speaker embedding of the target speaker from the preset reference audio will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0095] Step S203, calling a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator.

[0096] In this embodiment, the model process of the above-mentioned target speech conversion model will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0097] Step S204: extracting features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features.

[0098] In this embodiment, by inputting the text to be converted and the speaker embedding into the acoustic model, the acoustic model will generate corresponding initial acoustic features (such as mel-spectrogram) and initial prosodic features (such as pitch, speech rate, etc.) according to the input text to be converted and the speaker embedding. During the generation process, by adjusting the parameters of the acoustic model (such as conditional embedding, attention mechanism, etc.), the generated acoustic features and prosodic features are made similar to those of the target speaker.

[0099] Step S205: evaluating the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results.

[0100] In this embodiment, the generated initial acoustic features and initial prosodic features are input into a trained discriminator for evaluation, and the discriminator outputs a score or probability indicating the possibility that the input feature is a true feature.

[0101] Step S206: Optimize the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features.

[0102] In this embodiment, the specific implementation process of optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain the optimized target acoustic features and target prosodic features will be described in further detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0103] Step S207: converting the target acoustic features and the target prosodic features based on a preset vocoder to obtain corresponding synthesized speech.

[0104] In this embodiment, during the speech synthesis stage, the generated target acoustic features and target prosodic features are sent to the vocoder for conversion. The vocoder reconstructs the speech signal based on these features and generates the final speech waveform, i.e., the synthesized speech. In the process of speech synthesis, the prosodic features play an important role in controlling the rhythm, intonation, and volume of the speech. For example, by adjusting the pitch and duration of the prosodic features, the intonation and rhythm of the speech can be changed to make it more in line with the style of the target speaker or the expression requirements in a specific situation.

[0105] Step S208: output the synthesized speech.

[0106] In this embodiment, the specific implementation process of the above-mentioned output processing of the synthesized speech will be further described in detail in the subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0107] The present application obtains the target text by preprocessing the input text to be converted, and extracts the speaker embedding of the target speaker from the reference audio, and then extracts the features of the text to be converted and the speaker embedding based on the acoustic model in the target speech conversion model to obtain the corresponding initial acoustic features and initial prosodic features, and then evaluates the initial acoustic features and initial prosodic features based on the discriminator in the target speech conversion model to obtain the evaluation results, and then optimizes the initial acoustic features and initial prosodic features based on the evaluation results to obtain the target acoustic features and target prosodic features, and then converts the target acoustic features and target prosodic features based on the vocoder to obtain the synthesized speech similar to the target speaker's voice and outputs it. Through the use of the discriminator, the feature differences between the real speech and the generated speech can be conditionally identified, and then the acoustic characteristics and prosodic characteristics of the target speech conversion model can be improved, so that the processing intelligence of speech synthesis can be effectively improved, and the quality of the generated synthesized speech can be improved.

[0108] In some optional implementations, before step S203, the electronic device may further perform the following steps:

[0109] Get a pre-built speaker dataset.

[0110] In this embodiment, the speaker dataset is a pre-constructed large-scale dataset of multiple speakers, including text and corresponding speech data.

[0111] Calling a preset initial conversion model; wherein the initial conversion model includes an acoustic model constructed by a preset speech synthesis model, and a discriminator constructed by a preset encoder structure.

[0112] In this embodiment, the above-mentioned speech synthesis model can specifically adopt FastSpeech2. FastSpeech2 is a non-autoregressive speech synthesis model based on the Transformer structure, which can efficiently generate high-quality speech. It solves the problems of slow speed and inconsistent speech quality in the autoregressive model through the length regulator and variance adapter.

[0113] The above encoder structure can specifically adopt an encoder-decoder structure based on Transformer. Among them, the encoder includes: processing input text and speaker information (such as speaker embedding). The encoder converts the text into a series of vector representations and combines speaker embedding to capture specific features of the speaker. The decoder includes: processing acoustic features or prosodic features that need to be evaluated. The decoder converts the acoustic features into a series of vector representations and compares them with the output of the encoder. The objective function includes: The goal of the discriminator is to distinguish between real features and generated features, and output a value close to 1 (real) or -1 (generated). It is specifically achieved by applying a fully connected layer and a sigmoid activation function.

[0114] A key challenge in achieving natural and expressive TTS models is the one-to-many mapping problem between text and speech. TTS models trained with reconstruction losses tend to predict averages, which can lead to TTS systems generating overly smooth predictions that are closer to the average behavior of the dataset rather than capturing the diversity of human speech. The quality gap in zero-shot voice cloning is partly due to the large multi-speaker datasets used to train these models, which increase the variability in this one-to-many mapping due to the variety of voices and speaking styles contained in them.

[0115] This application overcomes the phenomenon of over-smoothing of acoustic features caused by the difficulty of one-to-many mapping in the TTS model by applying a generative adversarial network (GAN). The acoustic model of TTS is used as a generator to generate acoustic features and prosodic features. At the same time, an independent discriminator is trained to distinguish the acoustic features predicted by the acoustic model from the real acoustic features, and adversarial loss is used to optimize the acoustic feature parameters. As the discriminator gradually learns to distinguish between real and synthetic data, it can guide the optimization of synthetic data through adversarial loss, so that the generated data distribution is closer to the real data, thereby solving the quality difference problem in zero-sample voice cloning TTS. Specifically, the Transformer architecture was selected because it performs well in processing large-scale data and parameters. An innovative Transformer encoder-decoder is designed, which conditionally identifies TTS features through multimodal fusion technology. And a multi-feature generative adversarial training process is constructed, which uses the construction of a discriminator to improve acoustic features and prosodic features, thereby achieving a more natural and expressive TTS model.

[0116] A first overall loss corresponding to the acoustic model is determined.

[0117] In this embodiment, the specific implementation process of determining the first overall loss corresponding to the acoustic model will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0118] A second overall loss corresponding to the discriminator is determined.

[0119] In this embodiment, the acoustic feature discrimination loss and the prosodic feature discrimination loss corresponding to the discriminator can be obtained, and then the acoustic feature discrimination loss and the prosodic feature discrimination loss can be added to obtain a second overall loss corresponding to the discriminator. Among them, the above-mentioned acoustic feature discrimination loss refers to the ability of the discriminator to distinguish between real acoustic features and generated acoustic features. Conditional hinge loss can be used as the above-mentioned prosodic feature discrimination loss. In practical applications, the discriminator predicts each frame, and the conditional hinge loss is calculated for each output. Hinge loss is a commonly used loss function used in support vector machines and other classifiers. In the conditional hinge loss, other conditional information can be further considered, such as speaker identity or text content. By minimizing this acoustic feature discrimination loss, the discriminator is trained to better distinguish between real acoustic features and generated acoustic features, while taking into account additional contextual information. In addition, the above-mentioned prosodic feature discrimination loss is used to guide the discriminator to distinguish between real prosodic features and generated prosodic features. Similarly, conditional hinge loss can be used. By minimizing the second overall loss, the discriminator is trained to more accurately identify real and generated acoustic and prosodic features.

[0120] Based on the first overall loss and the second overall loss, adversarial training is performed on the acoustic model and the discriminator in the initial conversion model using the speaker data set to obtain a trained designated model.

[0121] In this embodiment, the training process of the above-mentioned initial conversion model includes: in each iteration, the acoustic model is first used to generate acoustic features and prosodic features, and then they are input into the discriminator together with the real features for evaluation. Then the generation loss of the acoustic model, the discriminant loss of the discriminator, and the generation loss and discriminant loss of the prosodic features are calculated, and the first overall loss and the second overall loss are further calculated. After that, the back propagation algorithm is used to update the parameters of the acoustic model and the discriminator to minimize the loss function, thereby obtaining a trained specified model. Among them, during the training process, the generator attempts to minimize the generation loss to deceive the discriminator, while the discriminator attempts to minimize the discriminant loss to correctly distinguish between real and generated data. Adversarial training enables the generator to learn more realistic and diverse acoustic feature representations.

[0122] This application proposes a multimodal fusion discriminator network and a multi-feature generative adversarial training method to enhance the performance of the discriminator. This helps to improve the voice quality and migration ability of the acoustic model in the zero-sample voice cloning task. The multimodal fusion discriminator network includes: the encoder of the Transformer structure is responsible for processing the input text and speaker information, while the decoder processes the feature information that needs to be evaluated and predicts the authenticity of this information. This method enables the discriminator to have a more comprehensive perspective when evaluating acoustic features, thereby improving the diversity and richness of acoustic features. Multi-feature generative adversarial training includes: through adversarial training, it aims to enable the acoustic model to generate more realistic and diverse acoustic features, while the addition of rhythmic features helps to capture the rhythm and intonation changes in the speech, thereby generating more natural and expressive speech output.

[0123] In adversarial training, the generator and discriminator compete by minimizing their respective loss functions. The generator tries to minimize the generation loss to deceive the discriminator, while the discriminator tries to minimize the discrimination loss to correctly distinguish between real and generated data. Through this adversarial process, the generator is able to learn more realistic and diverse acoustic feature representations.

[0124] The designated model is used as the target speech conversion model.

[0125] In this embodiment, the target speech conversion model may also be referred to as a TTS model. The loss value and the quality of the generated speech during the training process may also be monitored. The learning rate, batch size and other hyperparameters may be adjusted as needed. The validation set may also be used for early stopping to avoid overfitting.

[0126] The present application uses a pre-constructed speaker dataset to perform adversarial training on the acoustic model and the discriminator in the initial conversion model according to the determined first overall loss corresponding to the acoustic model and the second overall loss corresponding to the discriminator, so as to quickly and accurately construct a target speech conversion model that can recognize the characteristic differences between real speech and generated speech, and can generate high-quality speech and capture the specific characteristics of the target speaker, thereby improving the construction efficiency of the target speech conversion model and improving the model processing effect of the target speech conversion model.

[0127] In some optional implementations of this embodiment, determining the first overall loss corresponding to the acoustic model includes the following steps:

[0128] Acquire preset acoustic feature generation loss, prosodic feature generation loss, and prosodic feature discrimination loss corresponding to the acoustic model.

[0129] In this embodiment, the acoustic feature generation loss is a measure of the difference between the acoustic features predicted by the acoustic model and the real acoustic features, and is usually calculated using the mean square error (MSE). The prosodic feature generation loss evaluates the difference between the prosodic features generated by the acoustic model and the real prosodic features, and also uses the mean square error as the loss function. In addition, in order to enhance the fidelity and diversity of the prosodic features generated by the acoustic model, an adversarial training mechanism is also introduced, in which the prosodic feature discrimination loss is used to guide the discriminator to distinguish between real and generated prosodic features.

[0130] Get the preset integration strategy.

[0131] In this embodiment, the formula corresponding to the above integration strategy includes: L = L1 + L2 + wL3. Among them, L is the first overall loss, L1 is the acoustic feature generation loss, L2 is the prosodic feature generation loss, L3 is the acoustic feature generation loss, and w is a hyperparameter used to balance the generation loss and the discrimination loss. By minimizing the first overall loss to train the acoustic model, the acoustic model is trained to generate speech that is both realistic and has rich prosodic changes.

[0132] Based on the integration strategy, the acoustic feature generation loss, the prosodic feature generation loss and the acoustic feature generation loss are integrated to obtain a corresponding integration loss.

[0133] In this embodiment, the acoustic feature generation loss, the prosodic feature generation loss, and the acoustic feature generation loss can be integrated using the formula corresponding to the above integration strategy to generate a corresponding integration loss.

[0134] The first overall loss is generated based on the integrated loss.

[0135] In this embodiment, the integrated loss can be directly used as the final first overall loss.

[0136] The present application obtains the preset acoustic feature generation loss, rhythmic feature generation loss and rhythmic feature discrimination loss corresponding to the acoustic model; then obtains a preset integration strategy; then integrates the acoustic feature generation loss, the rhythmic feature generation loss and the acoustic feature generation loss based on the integration strategy to obtain the corresponding integration loss; and subsequently generates the first overall loss based on the integration loss. The present application obtains the acoustic feature generation loss, the rhythmic feature generation loss and the rhythmic feature discrimination loss corresponding to the acoustic model, and then integrates the acoustic feature generation loss, the rhythmic feature generation loss and the acoustic feature generation loss based on the use of the integration strategy, thereby efficiently and accurately constructing the first overall loss corresponding to the acoustic model, and ensuring the accuracy and intelligence of the obtained acoustic model.

[0137] In some optional implementations, step S206 includes the following steps:

[0138] Get the preset adjustment strategy.

[0139] In this embodiment, the strategy content of the above adjustment strategy includes: according to a score or probability of the output of the discriminator (indicating the possibility that the input feature is a real feature), the back propagation algorithm is used to adjust the parameters of the acoustic model. And the goal is to minimize the probability that the generated feature is identified as false (i.e., not real) by the discriminator, thereby improving the authenticity of the generated feature and the similarity with the target speaker. The adjustment process is repeated until a predetermined number of iterations is reached or the quality of the generated feature is no longer significantly improved.

[0140] Based on the evaluation result, the adjustment strategy is used to adjust the parameters of the acoustic model until the preset iteration conditions are met and an adjusted acoustic model is obtained.

[0141] In this embodiment, according to the strategy content of the above adjustment strategy and the evaluation result, the parameters of the acoustic model can be adjusted until the iteration conditions are met and the adjusted acoustic model is obtained.

[0142] Based on the adjusted acoustic model, feature extraction is performed on the text to be converted and the speaker embedding to obtain corresponding specified acoustic features and specified prosodic features.

[0143] In this embodiment, the above-mentioned processing process of extracting features from the text to be converted and the speaker embedding based on the adjusted acoustic model can refer to the above-mentioned processing process of extracting features from the text to be converted and the speaker embedding based on the acoustic model, and will not be described in detail here.

[0144] The designated acoustic feature is used as the target acoustic feature, and the designated prosodic feature is used as the target prosodic feature.

[0145] The present application obtains a preset adjustment strategy; then based on the evaluation result, uses the adjustment strategy to adjust the parameters of the acoustic model until the preset iteration conditions are met and an adjusted acoustic model is obtained; then based on the adjusted acoustic model, feature extraction is performed on the text to be converted and the speaker embedding to obtain corresponding specified acoustic features and specified prosodic features; subsequently, the specified acoustic features are used as the target acoustic features, and the specified prosodic features are used as the target prosodic features. The present application uses an adjustment strategy to adjust the parameters of the acoustic model according to the evaluation results until the preset iteration conditions are met and an adjusted acoustic model is obtained, and then uses the adjusted acoustic model to extract features from the text to be converted and the speaker embedding, thereby achieving fast and accurate generation of optimized target acoustic features and target prosodic features, so that the target acoustic features and target prosodic features can be highly similar to the target speaker, thereby achieving high-quality speech cloning task processing.

[0146] In some optional implementations, step S208 includes the following steps:

[0147] The synthesized speech is subjected to noise removal processing to obtain a corresponding first synthesized speech.

[0148] In this embodiment, the synthesized speech can be subjected to noise removal processing by using a noise suppression algorithm, such as spectral subtraction, Wiener filtering, or a denoising model based on deep learning. These algorithms analyze the spectral characteristics of the waveform and attempt to remove noise components that are not related to the speech, thereby obtaining the corresponding first synthesized speech.

[0149] The volume of the first synthesized speech is adjusted to obtain a corresponding second synthesized speech.

[0150] In this embodiment, the root mean square (RMS) value or peak volume of the waveform is calculated to determine its overall volume level, and then the volume of the first synthesized speech is adjusted according to actual needs to ensure that the speech has appropriate loudness when played, thereby obtaining the corresponding second synthesized speech.

[0151] The speech rate of the second synthesized speech is adjusted to obtain a corresponding third synthesized speech.

[0152] In this embodiment, the duration and rhythm of the waveform may be analyzed to determine the speech rate of the speech, and then a time stretching or compression algorithm (such as linear interpolation, nonlinear interpolation, or a time adjustment model based on deep learning) may be used to adjust the speech rate of the second synthesized speech to obtain a third synthesized speech that meets the requirements.

[0153] The third synthesized speech is output.

[0154] In this embodiment, the speech may be further processed to improve the quality by removing silent segments, equalizing, compressing, and limiting. Removing silent segments includes: detecting and deleting silent parts in the waveform to reduce unnecessary pauses. Equalizing includes: adjusting the spectrum balance of the waveform to ensure that different frequency components have appropriate relative strengths. Compression and limiting include: applying dynamic range compression and limiting algorithms to prevent the waveform from generating excessive peaks or distortion during playback.

[0155] The present application performs noise removal processing on the synthesized speech to obtain a corresponding first synthesized speech; then adjusts the volume of the first synthesized speech to obtain a corresponding second synthesized speech; then adjusts the speech rate of the second synthesized speech to obtain a corresponding third synthesized speech; and subsequently outputs the third synthesized speech. Before outputting the synthesized speech, the present application also intelligently performs noise removal processing, volume adjustment, and speech rate adjustment on the synthesized speech, thereby achieving efficient and accurate optimization processing of the synthesized speech, effectively improving the quality and intelligibility of the generated third synthesized speech, and thus achieving the best speech synthesis effect.

[0156] In some optional implementations of this embodiment, step S202 includes the following steps:

[0157] Obtain reference audio corresponding to the target speaker.

[0158] In this embodiment, the reference audio refers to a speech sample of a target speaker.

[0159] Call the preset speaker recognition model.

[0160] In this embodiment, the speaker recognition model is a model that has been trained on a large-scale multi-speaker dataset in advance and has the function of accurately identifying the identities of different speakers and generating corresponding speaker embeddings.

[0161] The reference audio is subjected to acoustic characteristic analysis based on the speaker recognition model, and a corresponding designated vector is output.

[0162] In this embodiment, by inputting the reference audio into the speaker recognition model, the speaker recognition model analyzes the acoustic characteristics in the reference audio and outputs a specified vector of fixed dimension, which is the speaker embedding. Speaker embedding refers to a low-dimensional vector representation extracted from the reference audio that can represent the identity of the speaker. This vector representation captures the unique acoustic characteristics of the speaker, such as timbre, intonation, etc.

[0163] A speaker embedding of the target speaker is generated based on the specified vector.

[0164] The present application obtains reference audio corresponding to the target speaker; then calls a preset speaker recognition model; then performs acoustic characteristic analysis on the reference audio based on the speaker recognition model and outputs a corresponding specified vector; and subsequently generates a speaker embedding of the target speaker based on the specified vector. The present application obtains reference audio corresponding to the target speaker, and then performs acoustic characteristic analysis on the reference audio based on the use of a speaker recognition model, so that the speaker embedding of the target speaker can be extracted quickly and accurately from the reference audio, thereby improving the extraction efficiency of the speaker embedding and ensuring the accuracy of the obtained speaker embedding.

[0165] In some optional implementations of this embodiment, step S201 includes the following steps:

[0166] Perform word segmentation processing on the text to be converted to obtain a corresponding first text.

[0167] In this embodiment, a word segmentation tool may be used to perform word segmentation processing on the text to be converted, so as to divide the text into a first text including words or sub-word units.

[0168] Part-of-speech tagging is performed on the first text to obtain a corresponding second text.

[0169] In this embodiment, a part-of-speech tag (such as noun, verb, adjective, etc.) may be assigned to each word in the first text to help improve text-to-speech mapping.

[0170] The second text is normalized to obtain a corresponding third text.

[0171] In this embodiment, the above-mentioned normalization processing refers to processing special characters, numbers, abbreviations, etc. in the text to make it meet the input requirements of the target speech conversion model.

[0172] The third text is used as the target text.

[0173] The present application performs word segmentation processing on the text to be converted to obtain a corresponding first text; then performs part-of-speech tagging on the first text to obtain a corresponding second text; then performs normalization processing on the second text to obtain a corresponding third text; and subsequently uses the third text as the target text. The present application performs word segmentation processing, part-of-speech tagging and normalization processing on the text to be converted, thereby achieving efficient and accurate preprocessing of the text to be converted, and ensuring the accuracy of the generated text to be converted.

[0174] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0175] In addition, the non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0176] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0177] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned synthesized speech, the above-mentioned synthesized speech can also be stored in a node of a blockchain.

[0178] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0179] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0180] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0181] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0182] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0183] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech processing device based on artificial intelligence, and the embodiment of the device is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0184] like Figure 3 As shown, the speech processing device 300 based on artificial intelligence described in this embodiment includes: a preprocessing module 301, a first extraction module 302, a first calling module 303, a second extraction module 304, an evaluation module 305, an optimization module 306, a conversion module 307 and an output module 308. Among them:

[0185] The preprocessing module 301 is used to obtain the input text to be converted and preprocess the text to be converted to obtain the target text;

[0186] A first extraction module 302, configured to extract a speaker embedding of a target speaker from a preset reference audio;

[0187] The first calling module 303 is used to call a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator;

[0188] A second extraction module 304 is used to extract features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features;

[0189] An evaluation module 305, configured to evaluate the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results;

[0190] An optimization module 306, configured to optimize the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features;

[0191] A conversion module 307, configured to convert the target acoustic features and the target prosodic features based on a preset vocoder to obtain corresponding synthesized speech;

[0192] The output module 308 is used to perform output processing on the synthesized speech.

[0193] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based speech processing method in the aforementioned implementation mode, and will not be repeated here.

[0194] In some optional implementations of this embodiment, the speech processing device based on artificial intelligence further includes:

[0195] The acquisition module is used to obtain a pre-built speaker dataset;

[0196] A second calling module, used to call a preset initial conversion model; wherein the initial conversion model includes an acoustic model constructed by a preset speech synthesis model, and a discriminator constructed by a preset encoder structure;

[0197] A first determination module, configured to determine a first overall loss corresponding to the acoustic model;

[0198] A second determination module, used to determine a second overall loss corresponding to the discriminator;

[0199] A training module, configured to perform adversarial training on the acoustic model and the discriminator in the initial conversion model based on the first overall loss and the second overall loss using the speaker data set to obtain a trained designated model;

[0200] The third determination module is used to use the specified model as the target speech conversion model.

[0201] In some optional implementations of this embodiment, the first determining module includes:

[0202] A first acquisition submodule, used for acquiring preset acoustic feature generation loss, prosodic feature generation loss and prosodic feature discrimination loss corresponding to the acoustic model;

[0203] The second acquisition submodule is used to acquire a preset integration strategy;

[0204] An integration submodule, used for integrating the acoustic feature generation loss, the prosodic feature generation loss and the acoustic feature generation loss based on the integration strategy to obtain a corresponding integration loss;

[0205] The first generating submodule is used to generate the first overall loss based on the integrated loss.

[0206] In some optional implementations of this embodiment, the optimization module 306 includes:

[0207] The third acquisition submodule is used to acquire a preset adjustment strategy;

[0208] An adjustment submodule, configured to adjust parameters of the acoustic model using the adjustment strategy based on the evaluation result until a preset iteration condition is met and an adjusted acoustic model is obtained;

[0209] An extraction submodule, configured to perform feature extraction on the text to be converted and the speaker embedding based on the adjusted acoustic model to obtain corresponding specified acoustic features and specified prosodic features;

[0210] The first determination submodule is configured to use the designated acoustic feature as the target acoustic feature and the designated prosodic feature as the target prosodic feature.

[0211] In some optional implementations of this embodiment, the output module 308 includes:

[0212] A first processing submodule, configured to perform noise removal processing on the synthesized speech to obtain a corresponding first synthesized speech;

[0213] A second processing submodule, configured to adjust the volume of the first synthesized speech to obtain a corresponding second synthesized speech;

[0214] A third processing submodule, configured to adjust the speech rate of the second synthesized speech to obtain a corresponding third synthesized speech;

[0215] An output submodule is used to output the third synthesized speech.

[0216] In some optional implementations of this embodiment, the first extraction module 302 includes:

[0217] A fourth acquisition submodule, used to acquire reference audio corresponding to the target speaker;

[0218] A calling submodule is used to call a preset speaker recognition model;

[0219] An analysis submodule, configured to perform acoustic characteristic analysis on the reference audio based on the speaker recognition model, and output a corresponding specified vector;

[0220] The second generating submodule is configured to generate a speaker embedding of the target speaker based on the specified vector.

[0221] In some optional implementations of this embodiment, the preprocessing module 301 includes:

[0222] A fourth processing submodule is used to perform word segmentation processing on the text to be converted to obtain a corresponding first text;

[0223] A fifth processing submodule, configured to perform part-of-speech tagging on the first text to obtain a corresponding second text;

[0224] A sixth processing submodule, configured to perform normalization processing on the second text to obtain a corresponding third text;

[0225] The second determining submodule is used to use the third text as the target text.

[0226] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0227] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0228] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0229] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a speech processing method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0230] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the artificial intelligence-based speech processing method.

[0231] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0232] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based speech processing method.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0234] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech processing method based on artificial intelligence, characterized in that: The steps include: Obtaining input text to be converted, and preprocessing the text to be converted to obtain a target text; Extract the speaker embedding of the target speaker from the preset reference audio; Calling a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator; Extracting features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features; evaluating the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results; Optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features; The target acoustic feature and the target prosodic feature are converted based on a preset vocoder to obtain corresponding synthesized speech; The synthesized speech is outputted.

2. The speech processing method based on artificial intelligence according to claim 1, characterized in that: Before the step of calling the preset target voice conversion model, the method further includes: Get a pre-built speaker dataset; Calling a preset initial conversion model; wherein the initial conversion model includes an acoustic model constructed by a preset speech synthesis model, and a discriminator constructed by a preset encoder structure; determining a first overall loss corresponding to the acoustic model; determining a second overall loss corresponding to the discriminator; Based on the first overall loss and the second overall loss, using the speaker data set, perform adversarial training on the acoustic model and the discriminator in the initial conversion model to obtain a trained designated model; The designated model is used as the target speech conversion model.

3. The artificial intelligence-based speech processing method according to claim 2, characterized in that: The step of determining a first overall loss corresponding to the acoustic model specifically comprises: Acquiring preset acoustic feature generation loss, prosodic feature generation loss, and prosodic feature discrimination loss corresponding to the acoustic model; Get preset integration strategies; Based on the integration strategy, the acoustic feature generation loss, the prosodic feature generation loss and the acoustic feature generation loss are integrated to obtain a corresponding integration loss; The first overall loss is generated based on the integrated loss.

4. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of optimizing the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features specifically includes: Get the preset adjustment strategy; Based on the evaluation result, the acoustic model is adjusted using the adjustment strategy until a preset iteration condition is met and an adjusted acoustic model is obtained; Based on the adjusted acoustic model, feature extraction is performed on the text to be converted and the speaker embedding to obtain corresponding specified acoustic features and specified prosodic features; The designated acoustic feature is used as the target acoustic feature, and the designated prosodic feature is used as the target prosodic feature.

5. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of outputting the synthesized speech specifically includes: Performing noise removal processing on the synthesized speech to obtain a corresponding first synthesized speech; Adjusting the volume of the first synthesized speech to obtain a corresponding second synthesized speech; adjusting the speech rate of the second synthesized speech to obtain a corresponding third synthesized speech; The third synthesized speech is output.

6. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of extracting the speaker embedding of the target speaker from the preset reference audio specifically includes: Acquire reference audio corresponding to the target speaker; Call the preset speaker recognition model; Performing acoustic characteristic analysis on the reference audio based on the speaker recognition model, and outputting a corresponding specified vector; A speaker embedding of the target speaker is generated based on the specified vector.

7. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of preprocessing the text to be converted to obtain the target text specifically includes: Performing word segmentation processing on the text to be converted to obtain a corresponding first text; Performing part-of-speech tagging on the first text to obtain a corresponding second text; Normalizing the second text to obtain a corresponding third text; The third text is used as the target text.

8. A speech processing device based on artificial intelligence, characterized in that: include: A preprocessing module, used to obtain input text to be converted, and preprocess the text to be converted to obtain a target text; A first extraction module, configured to extract a speaker embedding of a target speaker from a preset reference audio; A first calling module, used to call a preset target speech conversion model; wherein the target speech conversion model at least includes an acoustic model and a discriminator; A second extraction module is used to extract features of the text to be converted and the speaker embedding based on the acoustic model to obtain corresponding initial acoustic features and initial prosodic features; An evaluation module, configured to evaluate the initial acoustic features and the initial prosodic features based on the discriminator to obtain corresponding evaluation results; An optimization module, configured to optimize the initial acoustic features and the initial prosodic features based on the evaluation results to obtain optimized target acoustic features and target prosodic features; A conversion module, configured to convert the target acoustic features and the target prosodic features based on a preset vocoder to obtain corresponding synthesized speech; An output module is used to perform output processing on the synthesized speech.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7 are implemented.

Images