Speech synthesis method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an artificial intelligence-based speech synthesis method and device, computer equipment and a storage medium, and the method comprises the steps: receiving input reference speech and text data; performing accent intensity extraction on the reference voice based on an accent intensity modeling module to obtain accent intensity; processing the accent intensity based on an accent embedding module to obtain accent intensity embedding; performing rhythm prediction processing on the text data to obtain rhythm features; based on a stream matching diffusion model, performing speech synthesis according to accent intensity embedding, rhythm features and time step embedding to obtain an acoustic feature sequence; processing the acoustic feature sequence based on a vocoder to obtain a voice waveform; and performing format conversion on the voice waveform to obtain target voice data and outputting the target voice data. In addition, the target voice data can be stored in the block chain. The method can be applied to voice synthesis scenes in the fields of financial science and technology and medical science and technology, and the intelligence and flexibility of voice synthesis are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to fields such as fintech and digital healthcare, particularly to artificial intelligence-based speech synthesis methods, devices, computer equipment, and storage media. Background Technology

[0002] In the field of speech synthesis, various deep learning-based text-to-speech (TTS) systems are currently available, with zero-shot speech synthesis becoming a research hotspot. Existing methods have made significant progress in naturalness and timbre transfer, enabling personalized speech synthesis with only a short reference audio input. However, current technologies have significant shortcomings in accent controllability. Most existing research focuses on the transfer of timbre, prosody, and emotion, relying primarily on accent features carried by the cue audio for accent control. This lack of explicit, adjustable accent intensity modeling makes it difficult to achieve controllable accent intensity, thus reducing the intelligence and flexibility of speech synthesis.

[0003] In the financial insurance sector, taking the voice introduction of insurance products as an example, traditional speech synthesis systems generate fixed accents, unable to flexibly adjust to the accent preferences of customers in different regions. For instance, when introducing insurance products to customers in Guangdong, if the customer is more accustomed to Cantonese, but the system can only output standard Mandarin, it may affect the customer's acceptance of the product introduction. In the medical field, such as in medical consultation voice services, when doctors explain conditions to patients, patients from different regions have different accent preferences. If a patient is from Sichuan and is more accustomed to Sichuan accented explanations, but existing speech synthesis systems cannot meet this need, potentially affecting the patient's understanding of the condition and the effectiveness of communication.

[0004] Therefore, there is an urgent need for a speech synthesis method with controllable accents to improve the intelligence and flexibility of speech synthesis and meet the diverse accent needs of users in different scenarios. Summary of the Invention

[0005] The purpose of this application is to propose a speech synthesis method, apparatus, computer device, and storage medium based on artificial intelligence, so as to solve the technical problem that existing speech synthesis methods cannot achieve controllable accents, resulting in low intelligence and flexibility of speech synthesis.

[0006] Firstly, an artificial intelligence-based speech synthesis method is provided, including: Receive input reference speech and text data corresponding to the reference speech; The reference speech is processed by an accent intensity extraction module based on a preset accent intensity modeling module to obtain the corresponding accent intensity. The accent intensity is processed based on a preset accent embedding module to obtain the corresponding accent intensity embedding. The text data is subjected to prosodic prediction processing to obtain the corresponding prosodic features; Based on a preset flow matching diffusion model, speech synthesis processing is performed according to the accent intensity embedding, the prosodic features and the preset time step embedding to obtain the corresponding acoustic feature sequence. The acoustic feature sequence is processed based on a preset vocoder to obtain the corresponding speech waveform; The speech waveform is converted to obtain the corresponding target speech data, and the target speech data is then output.

[0007] Secondly, an artificial intelligence-based speech synthesis device is provided, comprising: A receiving module is used to receive input reference speech and text data corresponding to the reference speech; The extraction module is used to perform accent intensity extraction processing on the reference speech based on the preset accent intensity modeling module to obtain the corresponding accent intensity; The first processing module is used to process the accent intensity based on a preset accent embedding module to obtain the corresponding accent intensity embedding. The prediction module is used to perform prosodic prediction processing on the text data to obtain the corresponding prosodic features; The synthesis module is used to perform speech synthesis processing based on a preset flow matching diffusion model, according to the accent intensity embedding, the prosodic features and the preset time step embedding, to obtain the corresponding acoustic feature sequence. The second processing module is used to process the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform. The third processing module is used to convert the speech waveform into a format to obtain the corresponding target speech data, and to output the target speech data.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based speech synthesis method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned artificial intelligence-based speech synthesis method.

[0010] In the aforementioned scheme implemented by the artificial intelligence-based speech synthesis method, apparatus, computer equipment, and storage medium, the following steps are taken: First, input reference speech and corresponding text data are received. Then, the reference speech is processed by an accent intensity extraction module based on a preset accent intensity modeling module to obtain the corresponding accent intensity. The accent intensity is then processed by an accent embedding module based on a preset accent embedding module to obtain the corresponding accent intensity embedding. Subsequently, the text data is processed by prosodic prediction to obtain the corresponding prosodic features. Subsequently, based on a preset flow matching diffusion model, speech synthesis is performed according to the accent intensity embedding, the prosodic features, and a preset time step embedding to obtain the corresponding acoustic feature sequence. The acoustic feature sequence is further processed by a preset vocoder to obtain the corresponding speech waveform. Finally, the speech waveform is format-converted to obtain the corresponding target speech data, and the target speech data is output. Based on the above automated processing flow, this application extracts the accent intensity from the input reference speech using an accent intensity modeling module, and processes the accent intensity using an accent embedding module to obtain accent intensity embedding. Then, it performs prosodic prediction processing on the input text data to obtain prosodic features. Subsequently, based on the use of a stream matching diffusion model, it performs speech synthesis processing based on the accent intensity embedding, prosodic features, and preset time step embedding to obtain an acoustic feature sequence. Then, it processes the acoustic feature sequence using a vocoder to obtain a speech waveform. Finally, it performs format conversion on the speech waveform to obtain the target speech data and outputs it. Thus, this application utilizes a multi-step, multi-module collaborative approach. Text analysis and prosodic prediction provide the basic information and prosodic features of the text for speech synthesis; reference speech feature extraction and accent intensity processing provide a reference for pronunciation style and accent features; speech feature generation and speech waveform synthesis transform the text and feature information into actual speech waveforms; finally, the output synthesized speech allows users to access and use the generated speech, converting text into natural, fluent speech with controllable accent intensity. This achieves high-quality speech synthesis, improves the intelligence and flexibility of speech synthesis, and ensures the quality of the generated target speech data. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2This is a flowchart of an embodiment of the artificial intelligence-based speech synthesis method according to this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the artificial intelligence-based speech synthesis device according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal device 101.

[0020] It should be noted that the AI-based speech synthesis method provided in this application is generally executed by a server / terminal device, and correspondingly, the AI-based speech synthesis device is generally located in the server / terminal device.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Continue to refer to Figure 2 This document illustrates a flowchart of an embodiment of the AI-based speech synthesis method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The AI-based speech synthesis method provided in this application can be applied to any scenario requiring product recommendations, and thus can be applied to products in these scenarios, such as product recommendations in the financial insurance field. The AI-based speech synthesis method includes the following steps: Step S201: Receive the input reference speech and the text data corresponding to the reference speech.

[0023] In this embodiment, the artificial intelligence-based speech synthesis method runs on an electronic device (e.g., Figure 1The server / terminal device shown can acquire the image to be checked via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods. The executing entity of this application is specifically a speech synthesis system, which can be simply referred to as the system. The system consists of two modules: an accent intensity modeling module and a non-autoregressive speech synthesis model based on a flow matching diffusion model.

[0024] The aforementioned reference speech can be input by the user, with or without an accent, according to their personal needs. In certain specific scenarios, users may want to synthesize the same text data based on the accent and style of the reference speech. For example, a user might have a reference speech they read aloud and want to synthesize the same text again using the same accent for better consistency. In this case, the input text data and the text corresponding to the reference speech have the same content. Alternatively, the input text data can differ from the reference speech. The reference speech primarily provides the system with accent information; for example, the system extracts accent features from a user-provided speech with a certain accent and uses this feature to assign the corresponding accent when synthesizing other text content. The input text is the specific content the user actually wants to synthesize the speech for; it can be any text, not limited by the reference speech text.

[0025] This application can be applied to speech synthesis scenarios in the fintech and medical technology fields. For example, in an insurance product introduction scenario within the financial insurance sector, the input text content could include: Unaccented text: "Dear customer, our company has launched a new critical illness insurance product. This product covers 120 critical illnesses, including common cancers, heart disease, and stroke. Once a critical illness covered by the contract is diagnosed, we will pay a lump sum of high insurance benefits to help you and your family cope with high medical expenses and subsequent rehabilitation costs, giving you peace of mind." Text with an accent (let's assume it's a southern accent): "Dear valued customer, our company has just launched a new critical illness insurance product. This product covers 120 critical illnesses, including common diseases like cancer, heart disease, and stroke. Once diagnosed with a critical illness specified in the contract, we will pay a lump sum of high insurance money to help you and your family cope with the high medical expenses and subsequent rehabilitation costs, giving you peace of mind."

[0026] In the context of disease diagnosis in digital healthcare, the input text can include: Unaccented text: "After a thorough examination and diagnosis, you have type 2 diabetes. This is a common chronic disease, primarily related to insufficient insulin secretion or insulin resistance. Currently, your blood sugar control is not ideal. You need to adjust your diet, reduce your intake of high-sugar and high-fat foods, increase your physical activity, and take your blood sugar medication on time to control your blood sugar levels." Accented text (assuming a slight Northwestern accent): "After a thorough examination and diagnosis, you have type 2 diabetes. This is a common chronic disease, primarily related to insufficient insulin secretion or insulin resistance. Currently, your blood sugar control is not ideal. You need to adjust your diet, eat less high-sugar and high-fat foods, increase your physical activity, and take your blood sugar medication on time to control your blood sugar levels."

[0027] Step S202: Based on the preset accent intensity modeling module, the reference speech is processed to extract the accent intensity to obtain the corresponding accent intensity.

[0028] In this embodiment, the specific implementation process of extracting the accent intensity of the reference speech based on the preset accent intensity modeling module to obtain the corresponding accent intensity will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0029] Step S203: Process the accent intensity based on the preset accent embedding module to obtain the corresponding accent intensity embedding.

[0030] In this embodiment, the specific implementation process of processing the accent intensity based on the preset accent embedding module to obtain the corresponding accent intensity embedding will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0031] Step S204: Perform prosodic prediction processing on the text data to obtain the corresponding prosodic features.

[0032] In this embodiment, the specific implementation process of performing prosodic prediction processing on the text data to obtain the corresponding prosodic features will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0033] Step S205: Based on the preset flow matching diffusion model, speech synthesis processing is performed according to the accent intensity embedding, the prosodic features and the preset time step embedding to obtain the corresponding acoustic feature sequence.

[0034] In this embodiment, the specific implementation process of speech synthesis processing based on the preset flow matching diffusion model, according to the accent intensity embedding, the prosodic features and the preset time step embedding, to obtain the corresponding acoustic feature sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0035] Step S206: Process the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform.

[0036] In this embodiment, the specific implementation process of processing the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0037] Step S207: Convert the format of the speech waveform to obtain the corresponding target speech data, and output the target speech data.

[0038] In this embodiment, the generated speech waveform can be converted into common audio formats such as WAV and MP3 for easy playback and saving by the user. During the conversion process, parameters such as the audio sampling rate and quantization bit depth can be set according to the user's needs to obtain the corresponding target speech data. The converted audio file (target speech data) is then output to the user, who can play and download it through the system interface. The system can also provide additional functions such as speech speed adjustment and volume control to meet the user's personalized needs.

[0039] This application first receives input reference speech and corresponding text data; then, based on a preset accent intensity modeling module, it extracts the accent intensity of the reference speech to obtain the corresponding accent intensity; and processes the accent intensity based on a preset accent embedding module to obtain the corresponding accent intensity embedding; subsequently, it performs prosodic prediction processing on the text data to obtain the corresponding prosodic features; subsequently, based on a preset stream matching diffusion model, it performs speech synthesis processing according to the accent intensity embedding, the prosodic features, and a preset time step embedding to obtain the corresponding acoustic feature sequence; further, it processes the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform; finally, it performs format conversion on the speech waveform to obtain the corresponding target speech data, and outputs the target speech data. Based on the above automated processing flow, this application extracts the accent intensity from the input reference speech using an accent intensity modeling module, and processes the accent intensity using an accent embedding module to obtain accent intensity embedding. Then, it performs prosodic prediction processing on the input text data to obtain prosodic features. Subsequently, based on the use of a stream matching diffusion model, it performs speech synthesis processing based on the accent intensity embedding, prosodic features, and preset time step embedding to obtain an acoustic feature sequence. Then, it processes the acoustic feature sequence using a vocoder to obtain a speech waveform. Finally, it performs format conversion on the speech waveform to obtain the target speech data and outputs it. Thus, this application utilizes a multi-step, multi-module collaborative approach. Text analysis and prosodic prediction provide the basic information and prosodic features of the text for speech synthesis; reference speech feature extraction and accent intensity processing provide a reference for pronunciation style and accent features; speech feature generation and speech waveform synthesis transform the text and feature information into actual speech waveforms; finally, the output synthesized speech allows users to access and use the generated speech, converting text into natural, fluent speech with controllable accent intensity. This achieves high-quality speech synthesis, improves the intelligence and flexibility of speech synthesis, and ensures the quality of the generated target speech data.

[0040] In some alternative implementations, the system proposed in this application mainly consists of two modules: an accent intensity modeling module and a non-autoregressive speech synthesis model based on a flow matching diffusion model.

[0041] 1. The accent intensity modeling module includes: This system uses a pre-trained accent intensity modeling module to extract phoneme-level accent intensity from reference audio. Accent intensity is used to quantify the degree of pronunciation deviation of a speaker in the target language. This module models this by incorporating the concept of Goodness of Pronunciation (GOP): when the pronunciation of corresponding phonemes in the native language (L1) and the target language (L2) is highly similar in acoustic features, the system determines that the L2 accent intensity of that phoneme is low; when the acoustic differences are significant, the system determines that the L2 accent intensity of that phoneme is high. By evaluating accent intensity at the phoneme level, the system can achieve fine-grained quantification of speaker pronunciation deviation, thereby providing accurate input for subsequent speech synthesis.

[0042] The accent intensity modeling module used in this system specifically includes: an acoustic model for extracting speaker-independent feature embeddings from the input audio, and an accent intensity calculation module.

[0043] (1) For the acoustic model: This model adopts the architecture of a Time-Delay Neural Network (TDNN) with Mel-Frequency Cepstrum Coefficients (MFCCs) of the reference audio as input. The acoustic model needs to be trained on an L1 corpus first to ensure that the model learns the benchmark representation of standard pronunciation, so that the pronunciation of different speakers can be compared under the same benchmark. During the training process, a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) forced aligner is first used to perform phoneme-level alignment of the speech, thereby obtaining the phoneme state label corresponding to each frame; then these alignment labels are used to train the deep learning-based acoustic model.

[0044] (2) For the accent intensity calculation module, after training the acoustic model on the L1 corpus, the trained acoustic model is used to embed and extract the L2 data. Here, the GOP calculation method in the toolkit kaldi is used.

[0045] 2. Non-autoregressive speech synthesis model based on flow-matching diffusion model. The speech synthesis model of this system is a modification of F5 TTS. F5 TTS is an open-source TTS model that achieves fast training, fast inference, and high-quality speech generation by combining diffusion model and flow-matching technology. In this system, F5 TTS is further extended to achieve controllable accent intensity in the synthesized audio.

[0046] (1) The accent embedding module includes: In order to control the accent intensity of the generated speech during the inference stage, this system introduces the AccentEmbed module based on F5 TTS. This module consists of sinusoidal positional encoding and multilayer perceptron (MLP), and the input is the accent intensity parameter. The high-dimensional vector representation of the output is used for subsequent conditional control.

[0047] (2) Combination of accent embedding and time step embedding: In the F5 TTS stream matching training and inference process, the computation at each step depends on the time step embedding. This system combines accent intensity embedding with time step embedding, and further incorporates prosodic features to define control conditions. In this way, the model can simultaneously consider the temporal evolution of speech generation and the user-defined accent intensity during synthesis, achieving continuous control over accent intensity.

[0048] (3) Training and Inference: Training phase: The model is trained on mixed data of L1 and L2 corpora. The input is provided by the accent intensity modeling module, ensuring the model learns the correspondence between speech features and accent intensity. Inference phase: The system input is a reference speech (with or without accent) and its corresponding text content, plus an optional accent intensity control parameter. If the user provides the accent intensity parameter, it is directly used as the input. The system takes the input as reference speech; if the user does not provide the accent intensity, the system calls the pre-trained accent intensity modeling module to calculate the accent intensity value.

[0049] In some alternative implementations, step S202 includes the following steps: The reference speech is preprocessed to obtain the corresponding processed speech.

[0050] In this embodiment, speech preprocessing includes steps such as noise reduction, framing, and windowing. Noise reduction removes background noise from the speech signal and improves its quality. Framing divides the continuous speech signal into short frames, typically 20-30 milliseconds long, with some overlap between frames. Windowing reduces potential spectral leakage during framing and can be achieved using window functions such as Hamming and Hanning windows.

[0051] Feature extraction is performed on the processed speech to obtain the corresponding speech features.

[0052] In this embodiment, MFCC features (i.e., speech features) can be extracted from the preprocessed speech. MFCC is a commonly used speech feature that can simulate the human ear's perception of sound. The specific extraction process is as follows: Pre-emphasis: The speech signal is pre-emphasized to boost the high-frequency components, making the signal spectrum flatter for subsequent spectrum analysis. Fast Fourier Transform (FFT): The time-domain signal is converted into a frequency-domain signal to obtain the speech signal's spectrum. Mel Filter Bank: A set of Mel filter banks is designed to filter the spectrum, obtaining the output energy of each filter. The frequency spacing of the Mel filter bank is narrower in the low-frequency range and wider in the high-frequency range, which matches the auditory characteristics of the human ear. Logarithmic Operation: The logarithm of the output energy of the Mel filter bank is taken to simulate the nonlinear perception of sound intensity by the human ear. Discrete Cosine Transform (DCT): The logarithmic energy is subjected to a DCT to obtain the MFCC coefficients. Typically, the first 12-13 coefficients are retained as the speech feature vector.

[0053] The speech features are aligned at the phoneme level using a preset alignment tool to obtain the corresponding phoneme status labels.

[0054] In this embodiment, the alignment tool can employ a Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) forced aligner. This alignment tool can be used to perform phoneme-level alignment of speech with extracted MFCC features. The alignment tool determines the phoneme state label corresponding to each frame of speech based on the speech features and a pre-trained acoustic model. The specific process is as follows: Initialization: The MFCC feature sequence of the speech is input into the GMM-HMM model to initialize the phoneme state sequence. Forward-Backward Algorithm: The forward-backward algorithm is used to calculate the probability of each phoneme state in each frame, thereby determining the most likely phoneme state sequence. Viterbi Algorithm: The Viterbi algorithm is used to find the optimal phoneme state path, i.e., the phoneme state label corresponding to each frame of speech.

[0055] The accent intensity modeling module calculates the accent intensity based on the speech features and the phoneme status labels to obtain the corresponding accent intensity.

[0056] In this embodiment, the specific implementation process of calculating the accent intensity of the speech features and the phoneme state labels based on the accent intensity modeling module to obtain the corresponding accent intensity will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0057] This application preprocesses the reference speech to obtain processed speech; then extracts features from the processed speech to obtain corresponding speech features; subsequently, it performs phoneme-level alignment processing on the speech features using a preset alignment tool to obtain corresponding phoneme state labels; and finally, it calculates the accent intensity of the speech features and phoneme state labels using the accent intensity modeling module to obtain the corresponding accent intensity. Based on the above processing flow, this application obtains speech features by preprocessing and extracting features from the reference speech, and obtains phoneme state labels by performing phoneme-level alignment of the speech features using an alignment tool. Furthermore, the use of the accent intensity modeling module enables efficient and accurate calculation of accent intensity for speech features and phoneme state labels, improving the calculation efficiency of accent intensity and ensuring the accuracy of the obtained accent intensity data.

[0058] In some optional implementations of this embodiment, the step of calculating the accent intensity based on the accent intensity modeling module on the speech features and the phoneme state labels to obtain the corresponding accent intensity includes the following steps: The preset accent intensity modeling module is invoked; wherein, the accent intensity modeling module includes an acoustic model and an accent intensity calculation module.

[0059] In this embodiment, the accent intensity modeling module consists of an acoustic model and an accent intensity calculation module. The acoustic model can be a deep learning-based acoustic model.

[0060] Based on the acoustic model, the probability of the speech features and the phoneme state labels is calculated to obtain the corresponding posterior probability of the phonemes.

[0061] In this embodiment, the posterior probability of each phoneme in each frame can be calculated by inputting the phoneme-level aligned phoneme state labels and speech features (i.e., MFCC features of the speech) into the acoustic model. The acoustic model has already learned the baseline representation of the standard pronunciation, and therefore can output the corresponding phoneme posterior probability.

[0062] Obtain the preset accent intensity calculation strategy.

[0063] In this embodiment, the above-mentioned accent intensity calculation strategy includes: calculating Log PhonemePosterior (LPP): taking the logarithm of the posterior probability of the output phoneme to obtain the LPP value. According to the formula... ,in, The input observations are the MFCC features of the speech, also known as speech features. It is a standard pronunciation phoneme. Frame indices representing the start and end points, Let GOP be the posterior probability of a phoneme. Calculate GOP using the formula: Calculate the GOP value for each phoneme, where, Let q be the set of all phoneme categories. For any phoneme in the set of phoneme categories It includes all possible phoneme categories. The GOP value reflects the degree of difference between the pronunciation of the current phoneme and the standard pronunciation. Normalization yields accent intensity: The GOP value is normalized and mapped to the [0,1] interval to obtain the accent intensity value i. The closer i is to 0, the closer the pronunciation of the phoneme is to the standard native language pronunciation; the closer i is to 1, the stronger the accent. Overall accent intensity is calculated: The accent intensity values ​​of all phonemes are calculated comprehensively, for example, by taking the average or other statistical measures, to obtain the accent intensity value of the entire reference speech.

[0064] Based on the accent intensity calculation strategy, the accent intensity calculation module is used to calculate the posterior probability of the phoneme and the speech features to obtain the corresponding calculation results.

[0065] In this embodiment, based on the above-mentioned accent intensity calculation strategy, the above-mentioned accent intensity calculation module can be used to calculate the posterior probability of phonemes and speech features, and the calculation result can be used as the corresponding accent intensity.

[0066] The calculation result is used as the accent intensity.

[0067] This application utilizes a pre-defined accent intensity modeling module, which includes an acoustic model and an accent intensity calculation module. Based on the acoustic model, it performs probability calculations on the speech features and phoneme state labels to obtain the corresponding phoneme posterior probabilities. Then, it acquires a pre-defined accent intensity calculation strategy. Subsequently, based on the accent intensity calculation strategy, it uses the accent intensity calculation module to calculate the phoneme posterior probabilities and speech features to obtain the corresponding calculation results. Finally, it uses the calculation results as the accent intensity. Based on this processing flow, this application obtains phoneme posterior probabilities by performing probability calculations on speech features and phoneme state labels using the acoustic model in the accent intensity modeling module. Then, based on the use of the accent intensity calculation strategy, it uses the accent intensity calculation module in the accent intensity modeling module to calculate the phoneme posterior probabilities and speech features, and uses the obtained calculation results as the corresponding accent intensity. This allows for the automatic and accurate extraction of phoneme-level accent intensity from reference speech, improving the intelligence of accent intensity extraction and ensuring the accuracy of the obtained accent intensity.

[0068] In some alternative implementations, step S203 includes the following steps: A preset accent embedding module is invoked; wherein the accent embedding module includes a sinusoidal position code and a multilayer perceptron.

[0069] In this embodiment, the above-mentioned accent embedding module may also be referred to as the AccentEmbed module, which is composed of a sinusoidal position code and a multilayer perceptron.

[0070] The accent intensity is encoded based on the sinusoidal positional encoding to obtain the corresponding specified vector.

[0071] In this embodiment, the accent embedding module first encodes the input accent intensity using sinusoidal position coding. Sinusoidal position coding maps the accent intensity to a high-dimensional space by combining sinusoidal functions of different frequencies, generating a high-dimensional vector with positional information, i.e., the specified vector mentioned above. This encoding method preserves the relative relationship of accent intensity, facilitating subsequent multilayer perceptron (MLP) processing.

[0072] The specified vector is transformed based on the multilayer perceptron to obtain the corresponding transformed vector.

[0073] In this embodiment, a high-dimensional vector generated by sinusoidal position encoding is input into a multilayer perceptron. The multilayer perceptron consists of multiple fully connected layers, which further process and extract features from the high-dimensional vector through nonlinear transformation. After processing by the multilayer perceptron, the accent intensity parameter is converted into a high-dimensional accent embedding vector, i.e., the aforementioned transformed vector. This accent embedding vector can more comprehensively represent the feature information of accent intensity for subsequent conditional control.

[0074] The transformation vector is embedded as the accent intensity.

[0075] This application utilizes a pre-defined accent embedding module, which includes a sinusoidal positional encoder and a multilayer perceptron. The accent intensity is then encoded using the sinusoidal positional encoder to obtain a corresponding specified vector. This specified vector is then transformed using the multilayer perceptron to obtain a corresponding transformed vector. Finally, the transformed vector is used as the accent intensity embedding. Based on this process, this application achieves accurate accent intensity embedding by encoding the accent intensity using the sinusoidal positional encoder in the accent embedding module, transforming the specified vector using the multilayer perceptron, and then embedding the transformed vector as the corresponding accent intensity.

[0076] In some alternative implementations, step S204 includes the following steps: The text data is preprocessed to obtain the corresponding first processed text.

[0077] In this embodiment, the text preprocessing includes: removing redundant spaces, special characters, etc., to ensure proper text formatting. Simultaneously, it checks for uncommon characters or words not found in the system dictionary; if found, it performs appropriate processing, such as replacing them with common characters or providing a prompt.

[0078] The first processed text is segmented and part-of-speech tagging is performed to obtain the corresponding second processed text.

[0079] In this embodiment, a trained word segmentation model and part-of-speech tagging model can be used to perform word segmentation and part-of-speech tagging on the preprocessed first processed text. Word segmentation is the process of dividing continuous text into individual words, while part-of-speech tagging is the process of labeling the part of speech of each word, such as noun, verb, adjective, etc.

[0080] Invoke the preset prosody prediction model.

[0081] In this embodiment, the prosody prediction model described above can be a general prosody prediction model, such as a pre-trained prosody prediction model.

[0082] Based on the prosody prediction model, the second processed text is subjected to prosody prediction processing to obtain the corresponding speech prosody.

[0083] In this embodiment, based on the results of word segmentation and part-of-speech tagging (the second processed text), and combined with a pre-trained prosodic prediction model, the prosodic features such as stress, pauses, and intonation of each word in the second processed text are predicted. The prosodic prediction model considers factors such as the part of speech of the words, their position in the sentence, and their semantics to generate a natural and fluent speech prosody.

[0084] The phonological prosody is used as the prosodic feature.

[0085] This application preprocesses the text data to obtain a first processed text; then, it performs word segmentation and part-of-speech tagging on the first processed text to obtain a second processed text; subsequently, it calls a preset prosodic prediction model; and based on the prosodic prediction model, it performs prosodic prediction processing on the second processed text to obtain the corresponding speech prosody; finally, it uses the speech prosody as the prosodic feature. Based on the above processing flow, this application obtains a second processed text by preprocessing, segmenting, and tagging the text data, and then performs prosodic prediction processing on the second processed text based on the use of a prosodic prediction model, using the obtained speech prosody as the corresponding prosodic feature. This allows for efficient and accurate prosodic prediction processing of text data, ensuring the accuracy of the obtained prosodic features.

[0086] In some optional implementations of this embodiment, step S205 includes the following steps: Get the preset time step embedding.

[0087] In this embodiment, during the training and inference process of the flow matching diffusion model, the computation at each step relies on the time step embedding (TimeEmbed(t)). The time step embedding (TimeEmbed(t)) encodes discrete time step information into a continuous vector representation to be integrated into the model training and inference process, enabling the model to perceive the current processing time stage. Its acquisition typically involves the following steps: Time step encoding: Discrete to continuous mapping: The time step t in the flow matching diffusion model is originally a discrete integer (e.g., from 0 to the total number of steps T), but the model needs continuous input to handle the subtle differences between different time steps. Therefore, the discrete time step t is first mapped to a continuous numerical range. A common practice is to normalize t to the interval [0, 1], i.e., t_normalized = t / T, where T is the total number of steps in the diffusion process. Embedding layer initialization: An embedding layer is created, similar to the embedding layer used in natural language processing to map word indices to word vectors. This embedding layer is a learnable parameter matrix, typically with dimensions (T + 1, embedding_dim), where T + 1 corresponds to all possible time steps (from 0 to T), and embedding_dim is the dimension of the embedding vectors. Embedding vector generation: Querying the embedding matrix: For a given time step t, use it as an index to query the corresponding embedding vector from the embedding matrix. For example, if t = 3, then the corresponding vector v_t is obtained from the 4th row of the embedding matrix (index starts from 0), with dimension embedding_dim.

[0088] The accent intensity embedding, the prosodic features, and the time step embedding are combined to obtain the corresponding control conditions.

[0089] In this embodiment, the control condition is defined as c = TimeEmbed(t) + AccentEmbed(i) + rhythm, where TimeEmbed(t) is the time step embedding, AccentEmbed(i) is the accent intensity embedding, rhythm is the prosodic feature, t represents the time step of the flow matching process, and i represents the accent intensity value. The control condition emphasizes both the time step embedding and the accent embedding, enabling the flow matching diffusion model to simultaneously consider the temporal evolution of speech generation and the set accent intensity during synthesis, achieving continuous control over accent intensity. Furthermore, the prosodic feature is also integrated into the overall control of the model in some way. The flow matching diffusion model adjusts the changes in acoustic features based on information such as stress and pauses obtained from prosodic prediction. For example, when predicting an stressed syllable, the flow matching diffusion model may increase the energy and pitch of the syllable to make it more prominent in the speech; when encountering a pause, the model will generate a corresponding silent segment or reduce the energy to simulate a pause in natural language.

[0090] Furthermore, time-step embedding reflects the temporal evolution of speech generation, accent embedding reflects the user-defined accent intensity, and prosodic features ensure the natural fluency of the speech. These three elements work together to control the speech synthesis process. For example, at different time steps, the model gradually generates appropriate acoustic features based on the current progress and accent intensity, combined with prosodic features. At the beginning of the speech, it may be necessary to determine a suitable intonation starting point based on prosodic features; in the middle, acoustic features need to be adjusted according to stress and pause information; and at the end, it is necessary to ensure that the speech ends in a natural prosodic pattern.

[0091] Based on the control conditions, the flow matching diffusion model is used for speech synthesis processing to generate corresponding acoustic feature data.

[0092] In this embodiment, based on the predefined control condition c, the stream matching diffusion model begins speech synthesis. During the stream matching process, the stream matching diffusion model gradually generates the acoustic features of the speech through a series of iterative steps. At each time step, the model predicts the changes in acoustic features for the next step based on the current control condition c and the generation result of the previous step. Through continuous iteration, the stream matching diffusion model gradually constructs a complete sequence of speech acoustic features, i.e., the aforementioned acoustic feature data.

[0093] The acoustic feature data is used as the acoustic feature sequence.

[0094] This application obtains a preset time-step embedding; then combines the accent intensity embedding, prosodic features, and time-step embedding to obtain corresponding control conditions; subsequently, based on the control conditions, it uses the flow matching diffusion model for speech synthesis to generate corresponding acoustic feature data; and finally, it uses the acoustic feature data as the acoustic feature sequence. Based on the above processing flow, this application obtains control conditions by combining accent intensity embedding, prosodic features, and the obtained time-step embedding, and then uses the flow matching diffusion model for speech synthesis based on the control conditions, using the generated acoustic feature data as the corresponding acoustic feature sequence. This allows for efficient and accurate speech synthesis using the flow matching diffusion model based on accent intensity embedding, prosodic features, and the preset time-step embedding, ensuring the accuracy of the generated acoustic feature sequence.

[0095] In some optional implementations of this embodiment, step S206 includes the following steps: The acoustic feature sequence is input into the vocoder.

[0096] In this embodiment, the selection of the vocoder is not specifically limited and can be determined according to actual business needs. For example, WaveNet or WaveRNN can be used.

[0097] The acoustic feature sequence is decoded and reconstructed using the vocoder to obtain the corresponding initial speech waveform.

[0098] In this embodiment, after receiving the acoustic feature sequence, the selected vocoder uses its learned knowledge of a large amount of speech data to decode and reconstruct these features, converting them into an initial speech waveform.

[0099] Obtain the preset optimization strategy.

[0100] In this embodiment, the optimization strategy includes post-processing the generated speech waveform, including volume adjustment and timbre optimization. Volume adjustment ensures that the speech volume is moderate, neither too loud nor too soft, to adapt to different playback environments and user needs. Timbre optimization further improves the speech quality, making it clearer and more natural, and removing any possible noise or distortion.

[0101] The initial speech waveform is optimized based on the optimization strategy to obtain the corresponding specified speech waveform.

[0102] In this embodiment, optimization processing of the initial speech waveform can be performed based on the strategy content of the optimization strategy, and the resulting specified speech waveform can be used as the desired speech waveform.

[0103] The specified speech waveform is used as the speech waveform.

[0104] This application inputs the acoustic feature sequence into a vocoder; then, the vocoder decodes and reconstructs the acoustic feature sequence to obtain a corresponding initial speech waveform; subsequently, a preset optimization strategy is obtained; and the initial speech waveform is optimized based on the optimization strategy to obtain a corresponding specified speech waveform; finally, the specified speech waveform is used as the desired speech waveform. Based on the above processing flow, this application obtains an initial speech waveform by decoding and reconstructing the acoustic feature sequence using a vocoder, then performs optimization processing on the initial speech waveform based on the optimization strategy, and uses the obtained specified speech waveform as the desired speech waveform. This enables efficient and accurate waveform conversion processing of the acoustic feature sequence while ensuring the accuracy and quality of the generated speech waveform.

[0105] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0106] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0107] Furthermore, this application offers the following benefits: Building upon the efficient training and high-fidelity generation advantages of F5 TTS, this system further achieves controllable accent intensity, significantly enhancing the model's flexibility in cross-language scenarios and personalized speech synthesis applications. Moreover, the system can continuously adjust accent intensity based on the input accent intensity parameters, rather than simply controlling it as a binary "with / without accent." Compared to the limitations of traditional zero-shot TTS, which can only "reproduce the accent of the reference audio," this system enables continuous adjustment and personalized control. Users can choose a personalized accent or maintain their native language characteristics according to their actual needs, thereby enhancing the system's application value in education, voice-over, and personalized voice assistants.

[0108] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] It should be emphasized that, in order to further ensure the privacy and security of the aforementioned target voice data, the target voice data can also be stored in a blockchain node.

[0110] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0111] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0113] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0114] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an artificial intelligence-based speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0115] like Figure 3 As shown, the artificial intelligence-based speech synthesis device 300 described in this embodiment includes: a receiving module 301, an extraction module 302, a first processing module 303, a prediction module 304, a synthesis module 305, a second processing module 306, and a third processing module 307. Wherein: The receiving module 301 is used to receive the input reference speech and the text data corresponding to the reference speech; Extraction module 302 is used to perform accent intensity extraction processing on the reference speech based on a preset accent intensity modeling module to obtain the corresponding accent intensity; The first processing module 303 is used to process the accent intensity based on a preset accent embedding module to obtain the corresponding accent intensity embedding. Prediction module 304 is used to perform prosodic prediction processing on the text data to obtain the corresponding prosodic features; The synthesis module 305 is used to perform speech synthesis processing based on a preset flow matching diffusion model, according to the accent intensity embedding, the prosodic features and the preset time step embedding, to obtain the corresponding acoustic feature sequence. The second processing module 306 is used to process the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform. The third processing module 307 is used to convert the speech waveform into a format to obtain the corresponding target speech data, and to output the target speech data.

[0116] In some optional implementations of this embodiment, the extraction module 302 includes: The first preprocessing submodule is used to perform speech preprocessing on the reference speech to obtain the corresponding processed speech. The extraction submodule is used to extract features from the processed speech to obtain the corresponding speech features; The alignment submodule is used to perform phoneme-level alignment processing on the speech features based on a preset alignment tool to obtain the corresponding phoneme status labels; The calculation submodule is used to calculate the accent intensity based on the speech features and the phoneme state labels of the accent intensity modeling module, and obtain the corresponding accent intensity.

[0117] In some optional implementations of this embodiment, the computation submodule includes: The calling unit is used to call a preset accent intensity modeling module; wherein, the accent intensity modeling module includes an acoustic model and an accent intensity calculation module; The first calculation unit is used to perform probability calculation on the speech features and the phoneme state labels based on the acoustic model to obtain the corresponding phoneme posterior probability. The acquisition unit is used to acquire the preset accent intensity calculation strategy; The second calculation unit is used to calculate the phoneme posterior probability and the speech features based on the accent intensity calculation strategy and the accent intensity calculation module to obtain the corresponding calculation results. A determining unit is used to take the calculation result as the accent intensity.

[0118] In some optional implementations of this embodiment, the first processing module 303 includes: The first calling submodule is used to call a preset accent embedding module; wherein, the accent embedding module includes a sinusoidal position code and a multilayer perceptron; The encoding submodule is used to encode the accent intensity based on the sinusoidal position encoding to obtain the corresponding specified vector; The transformation submodule is used to perform transformation processing on the specified vector based on the multilayer perceptron to obtain the corresponding transformed vector; A first determining submodule is used to embed the transformation vector as the accent intensity.

[0119] In some optional implementations of this embodiment, the prediction module 304 includes: The second preprocessing submodule is used to perform text preprocessing on the text data to obtain the corresponding first processed text; The first processing submodule is used to perform word segmentation and part-of-speech tagging on the first processed text to obtain the corresponding second processed text. The second calling submodule is used to call the preset prosody prediction model; The prediction submodule is used to perform prosodic prediction processing on the second processed text based on the prosodic prediction model to obtain the corresponding speech prosodic. The second determining submodule is used to use the speech prosody as the prosodic feature.

[0120] In some optional implementations of this embodiment, the synthesis module 305 includes: The first acquisition submodule is used to acquire the preset time step embedding; The combined submodule is used to combine the accent intensity embedding, the prosodic features, and the time step embedding to obtain the corresponding control conditions. The synthesis submodule is used to perform speech synthesis processing using the flow matching diffusion model based on the control conditions to generate corresponding acoustic feature data; The third determining submodule is used to use the acoustic feature data as the acoustic feature sequence.

[0121] In some optional implementations of this embodiment, the second processing module 306 includes: An input submodule is used to input the acoustic feature sequence into the vocoder; The second processing submodule is used to decode and reconstruct the acoustic feature sequence through the vocoder to obtain the corresponding initial speech waveform; The second acquisition submodule is used to acquire preset optimization strategies; An optimization submodule is used to optimize the initial speech waveform based on the optimization strategy to obtain the corresponding specified speech waveform; The fourth determining submodule is used to use the specified speech waveform as the speech waveform.

[0122] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0123] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0124] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0125] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions based on artificial intelligence speech synthesis methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0126] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based speech synthesis method.

[0127] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0128] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based speech synthesis method described above.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that, Includes the following steps: Receive input reference speech and text data corresponding to the reference speech; The reference speech is processed by an accent intensity extraction module based on a preset accent intensity modeling module to obtain the corresponding accent intensity. The accent intensity is processed based on a preset accent embedding module to obtain the corresponding accent intensity embedding. The text data is subjected to prosodic prediction processing to obtain the corresponding prosodic features; Based on a preset flow matching diffusion model, speech synthesis processing is performed according to the accent intensity embedding, the prosodic features and the preset time step embedding to obtain the corresponding acoustic feature sequence. The acoustic feature sequence is processed based on a preset vocoder to obtain the corresponding speech waveform; The speech waveform is converted to obtain the corresponding target speech data, and the target speech data is then output.

2. The artificial intelligence-based speech synthesis method according to claim 1, characterized in that, The step of extracting the accent intensity of the reference speech based on the preset accent intensity modeling module to obtain the corresponding accent intensity specifically includes: The reference speech is preprocessed to obtain the corresponding processed speech; The processed speech is subjected to feature extraction to obtain the corresponding speech features; The speech features are aligned at the phoneme level using a preset alignment tool to obtain the corresponding phoneme status labels. The accent intensity modeling module calculates the accent intensity based on the speech features and the phoneme status labels to obtain the corresponding accent intensity.

3. The artificial intelligence-based speech synthesis method according to claim 2, characterized in that, The step of calculating the accent intensity based on the accent intensity modeling module for the speech features and the phoneme state labels to obtain the corresponding accent intensity specifically includes: Invoke the preset accent intensity modeling module; wherein, the accent intensity modeling module includes an acoustic model and an accent intensity calculation module; Based on the acoustic model, the probability of the speech features and the phoneme state labels is calculated to obtain the corresponding posterior probability of the phonemes. Obtain the preset accent intensity calculation strategy; Based on the accent intensity calculation strategy, the accent intensity calculation module is used to calculate and process the posterior probability of the phoneme and the speech features to obtain the corresponding calculation results; The calculation result is used as the accent intensity.

4. The artificial intelligence-based speech synthesis method according to claim 1, characterized in that, The step of processing the accent intensity based on the preset accent embedding module to obtain the corresponding accent intensity embedding specifically includes: A preset accent embedding module is invoked; wherein, the accent embedding module includes a sinusoidal position code and a multilayer perceptron; The accent intensity is encoded based on the sinusoidal position encoding to obtain the corresponding specified vector; The specified vector is transformed based on the multilayer perceptron to obtain the corresponding transformed vector. The transformation vector is embedded as the accent intensity.

5. The artificial intelligence-based speech synthesis method according to claim 1, characterized in that, The step of performing prosodic prediction processing on the text data to obtain the corresponding prosodic features specifically includes: The text data is preprocessed to obtain the corresponding first processed text; The first processed text is segmented and part-of-speech tagging is performed to obtain the corresponding second processed text; Invoke the preset prosody prediction model; Based on the prosody prediction model, the second processed text is subjected to prosody prediction processing to obtain the corresponding speech prosody. The phonological prosody is used as the prosodic feature.

6. The artificial intelligence-based speech synthesis method according to claim 1, characterized in that, The steps of obtaining the corresponding acoustic feature sequence by performing speech synthesis processing based on the preset flow matching diffusion model, according to the accent intensity embedding, the prosodic features, and the preset time step embedding, specifically include: Obtain the preset time step embedding; The accent intensity embedding, the prosodic features, and the time step embedding are combined to obtain the corresponding control conditions; Based on the control conditions, the flow matching diffusion model is used to perform speech synthesis processing to generate corresponding acoustic feature data; The acoustic feature data is used as the acoustic feature sequence.

7. The artificial intelligence-based speech synthesis method according to claim 1, characterized in that, The step of processing the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform specifically includes: The acoustic feature sequence is input into the vocoder; The acoustic feature sequence is decoded and reconstructed using the vocoder to obtain the corresponding initial speech waveform; Obtain the preset optimization strategy; The initial speech waveform is optimized based on the optimization strategy to obtain the corresponding specified speech waveform; The specified speech waveform is used as the speech waveform.

8. A speech synthesis device based on artificial intelligence, characterized in that, include: A receiving module is used to receive input reference speech and text data corresponding to the reference speech; The extraction module is used to perform accent intensity extraction processing on the reference speech based on the preset accent intensity modeling module to obtain the corresponding accent intensity; The first processing module is used to process the accent intensity based on a preset accent embedding module to obtain the corresponding accent intensity embedding. The prediction module is used to perform prosodic prediction processing on the text data to obtain the corresponding prosodic features; The synthesis module is used to perform speech synthesis processing based on a preset flow matching diffusion model, according to the accent intensity embedding, the prosodic features and the preset time step embedding, to obtain the corresponding acoustic feature sequence. The second processing module is used to process the acoustic feature sequence based on a preset vocoder to obtain the corresponding speech waveform. The third processing module is used to convert the speech waveform into a format to obtain the corresponding target speech data, and to output the target speech data.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the artificial intelligence-based speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the artificial intelligence-based speech synthesis method as described in any one of claims 1 to 7.

Images