Artificial intelligence-based speech generation method and apparatus, computer device, and medium

Abstract

The application belongs to the technical field of artificial intelligence, and relates to a speech generation method and device based on artificial intelligence, computer equipment and a storage medium, which comprises the following steps: receiving a text to be synthesized, performing feature extraction on received speaker prompt audio to obtain speaker features; calling a speech generation model; converting the text to be synthesized into target semantic representation by using a text encoder based on the speaker features; constructing a hard alignment matrix based on a prediction result obtained by predicting the target semantic representation by a time length predictor; adjusting the hard alignment matrix by a soft alignment generation module to generate a soft alignment matrix, generating a context vector based on the soft alignment matrix; decoding the residual variable and the context vector by a decoder to generate a mel spectrum; and performing speech synthesis on the mel spectrum by a vocoder to obtain target speech data and outputting the target speech data. The application can be applied to the speech synthesis scene in the financial technology field and the medical and health field, and can improve the speech generation efficiency and quality.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to fields such as fintech and healthcare, particularly to artificial intelligence-based speech generation methods, devices, computer equipment, and storage media. Background Technology

[0002] In recent years, text-to-speech (TTS) technology has made significant progress, enabling machines to automatically synthesize natural and fluent speech based on text. Most mainstream TTS models employ a generative architecture based on neural networks, with autoregressive models being a typical example.

[0003] While autoregressive models can generate speech with high naturalness, they have significant drawbacks. Their generation process involves frame-by-frame inference, which is slow and inefficient, making it difficult to meet the real-time requirements of applications. Furthermore, the generated speech quality is poor, potentially lacking in clarity and coherence, thus impacting user experience.

[0004] For example, in the financial insurance sector, customer service systems often need to convert textual information such as insurance terms and business descriptions into speech for customers during insurance transactions. If an autoregressive model is used, the slow inference speed causes customers to wait a long time, reducing service efficiency; moreover, poor voice quality may lead to misunderstandings of key information and cause business disputes. In the healthcare sector, in scenarios where medical knowledge is explained via voice, the poor coherence of the speech generated by the autoregressive model affects patients' reception and understanding of medical knowledge, hindering their correct disease prevention and treatment.

[0005] Therefore, there is an urgent need for a new speech generation technology to improve reasoning speed and speech quality. Summary of the Invention

[0006] The purpose of this application is to propose a speech generation method, apparatus, computer device, and storage medium based on artificial intelligence, so as to solve the technical problems of low speech generation efficiency and poor speech quality in existing speech generation technologies based on autoregressive models.

[0007] Firstly, an artificial intelligence-based speech generation method is provided, including: The system receives the input text to be synthesized and the speaker prompt audio of a specified duration, and performs feature extraction on the speaker prompt audio to obtain the corresponding speaker features; A preset speech generation model is invoked; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; Based on the speaker features, the text encoder is used to transform the text to be synthesized to obtain the corresponding target semantic representation; The phoneme duration is predicted based on the duration predictor, and a corresponding hard alignment matrix is ​​constructed based on the prediction results. The hard alignment matrix is ​​adjusted based on the soft alignment generation module to generate a soft alignment matrix, and a corresponding context vector is generated based on the soft alignment matrix. Obtain the preset residual variables; The residual variable and the context vector are decoded based on the decoder to generate the corresponding Mel spectrum; Based on the vocoder, speech synthesis is performed on the Mel spectrum to obtain the corresponding target speech data, and the target speech data is then output.

[0008] Secondly, an artificial intelligence-based speech generation device is provided, comprising: The first processing module is used to receive the input text to be synthesized and the speaker prompt audio of a specified duration, and to extract the corresponding speaker features from the speaker prompt audio. The calling module is used to call a preset speech generation model; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; The conversion module is used to convert the text to be synthesized based on the speaker features using the text encoder to obtain the corresponding target semantic representation; The second processing module is used to predict the phoneme duration of the target semantic representation based on the duration predictor, and to construct a corresponding hard alignment matrix based on the obtained prediction results. The generation module is used to adjust the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generate a corresponding context vector based on the soft alignment matrix; The acquisition module is used to acquire preset residual variables; The decoding module is used to decode the residual variable and the context vector based on the decoder to generate the corresponding Mel spectrum; The third processing module is used to perform speech synthesis on the Mel spectrum based on the vocoder to obtain the corresponding target speech data, and to output the target speech data.

[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based speech generation method.

[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described artificial intelligence-based speech generation method.

[0011] In the aforementioned AI-based speech generation method, apparatus, computer device, and storage medium, the following steps are taken: First, the input text to be synthesized and a speaker prompt audio of a specified duration are received. Feature extraction is performed on the speaker prompt audio to obtain corresponding speaker features. Then, a preset speech generation model is invoked. This speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder. Next, based on the speaker features, the text encoder is used to convert the text to be synthesized, obtaining a corresponding target semantic representation. Subsequently, the duration predictor performs phoneme duration prediction on the target semantic representation, and a corresponding hard alignment matrix is ​​constructed based on the prediction results. Then, the soft alignment generation module adjusts the hard alignment matrix to generate a soft alignment matrix, and a corresponding context vector is generated based on the soft alignment matrix. Further, preset residual variables are obtained. The decoder decodes the residual variables and the context vector to generate a corresponding Mel spectrum. Finally, the vocoder performs speech synthesis on the Mel spectrum to obtain the corresponding target speech data, and the target speech data is output. Unlike existing autoregressive model-based speech generation methods, this application provides a novel and intelligent speech generation technology solution based on the above automated processing flow. This solution utilizes a collaborative speech generation model comprising multiple components, including a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder. By starting from text input and progressively performing semantic representation, duration prediction, alignment adjustment, introduction of residual variables, Mel spectrum generation, and waveform synthesis, it can automatically and accurately convert text into natural and diverse speech. This achieves high-quality text-to-speech synthesis, effectively improving speech generation efficiency and enhancing the quality of synthesized speech. Attached Figure Description

[0012] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2This is a flowchart of an embodiment of the AI-based speech generation method according to this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the artificial intelligence-based speech generation device according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0017] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0018] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0019] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0020] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0021] It should be noted that the AI-based speech generation method provided in this application is generally executed by a server / terminal device, and correspondingly, the AI-based speech generation device is generally located in the server / terminal device.

[0022] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0023] Continue to refer to Figure 2 This document illustrates a flowchart of an embodiment of the AI-based speech generation method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The AI-based speech generation method provided in this application can be applied to any scenario requiring speech generation, and thus can be applied to products in these scenarios, such as speech generation products in the financial insurance and healthcare fields. The AI-based speech generation method includes the following steps: Step S201: Receive the input text to be synthesized and the speaker prompt audio of a specified duration, and extract features from the speaker prompt audio to obtain the corresponding speaker features.

[0024] In this embodiment, the AI-based speech generation method runs on an electronic device (e.g., Figure 1The server / terminal device shown can acquire the text to be synthesized and speaker prompt audio of a specified duration via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future known wireless connection methods. The implementing entity of this application is specifically a speech generation system, which can be simply referred to as the system.

[0025] The aforementioned short-duration speaker prompt can be text data input by the user based on actual business needs, which requires synthesis processing. The specified duration can be determined according to the actual short-term requirements, for example, 3 seconds. The speaker prompt audio (or short-duration speaker prompt) contains the speaker's unique characteristics, such as timbre and pronunciation habits. By providing a small amount of key speaker feature information, combined with speaker conditionalization processing, the short-duration speaker prompt enables zero-sample speech synthesis.

[0026] A speaker embedding vector (i.e., speaker features) s can be extracted from a 3-second speaker prompt audio clip using a speaker encoder. Typically, a speaker encoder uses convolutional networks or statistical pooling to generate this vector. A convolutional network acts as a feature extractor, performing layer-by-layer convolution operations on the audio signal. Each convolutional kernel captures different levels of features in the audio; lower-level kernels might capture basic features like frequency and amplitude, while higher-level kernels capture more complex features such as timbre and intonation. Through multiple convolutional operations, the audio signal is converted into a series of feature maps. Statistical pooling performs statistical operations on these feature maps, such as calculating the mean and variance. Combining these statistics forms the speaker embedding vector s. This vector contains features such as the speaker's timbre and intonation; different speakers' audio clips will produce different speaker embedding vectors after processing by the speaker encoder.

[0027] Furthermore, this application can be applied to speech generation scenarios in the financial insurance and healthcare sectors. For example, in an insurance product introduction scenario within the financial insurance sector, the text to be synthesized could be: "This whole life insurance product provides comprehensive death benefits. Whether death is due to accident or illness, you will receive a high payout, providing solid financial support for your and your family's future." The corresponding speaker prompt audio could be: an explanation from a professional, composed, and moderately paced insurance salesperson. In the audio, the salesperson introduces the features of the insurance product clearly and systematically, with a friendly voice that conveys professionalism and reliability to the listener. For instance, the salesperson might first briefly explain the product type, then elaborate on the coverage, and finally emphasize the importance of family financial security, maintaining a steady tone with appropriate emphasis on key points.

[0028] Alternatively, in a financial investment advice scenario within the financial and insurance sector, the text to be synthesized could be: "In the current market environment, it is recommended that investors allocate some funds to bond funds to reduce portfolio risk. Bond funds offer relatively stable returns and can effectively balance the high volatility of stock funds." The corresponding audio prompt could be: an audio message from an experienced financial analyst. The analyst's voice is confident and logically clear, using some professional terminology but providing appropriate explanations. In the audio, the analyst first analyzes the current market situation, points out the risks, then introduces the advantages of bond funds, such as stable returns and lower risk, and finally gives specific allocation suggestions, with varied intonation to highlight key information.

[0029] Furthermore, in the context of disease prevention outreach in the healthcare field, the text to be synthesized could be: "To prevent influenza, cultivate good personal hygiene habits, wash your hands frequently, and ventilate your home often. It is also recommended that susceptible individuals get vaccinated against influenza before the peak season to boost their immunity." The corresponding audio prompt could be: an audio recording of a kind and approachable community doctor. The doctor explains disease prevention knowledge in a gentle and patient tone, as if communicating face-to-face with residents. In the audio, the doctor first emphasizes the importance of influenza prevention, then details specific practices for personal hygiene, and finally introduces the benefits of vaccination and suitable groups, speaking at a moderate pace to ensure easy understanding and acceptance by the audience.

[0030] Alternatively, in a health advocacy scenario within the healthcare field, the text to be synthesized could be: "Maintaining a healthy lifestyle is crucial for physical health. Ensure sufficient sleep daily, eat a balanced diet, consume plenty of fruits and vegetables, and engage in moderate exercise, such as at least three aerobic exercise sessions per week." The corresponding speaker prompt audio could be: an audio recording of an energetic fitness coach. The coach's voice is loud and passionate, and their explanations are highly engaging. In the audio, the coach first explains the importance of a healthy lifestyle, then provides specific advice on sleep, diet, and exercise, sharing their own experiences and tips in a cheerful tone that inspires listeners to aspire to a healthy life.

[0031] Step S202: Invoke a preset speech generation model; wherein the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder.

[0032] In this embodiment, the model construction process of the above-mentioned speech generation model will be described in more detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0033] Step S203: Based on the speaker features, the text encoder is used to convert the text to be synthesized to obtain the corresponding target semantic representation.

[0034] In this embodiment, the specific implementation process of using the text encoder to convert the text to be synthesized based on the speaker features to obtain the corresponding target semantic representation will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0035] Step S204: Based on the duration predictor, the target semantic representation is processed to predict the phoneme duration, and a corresponding hard alignment matrix is ​​constructed based on the obtained prediction results.

[0036] In this embodiment, the duration predictor begins to function after the speaker-conditional semantic representation `hs` is input. The duration predictor used here can be a Transformer-based regression model. The self-attention mechanism in the Transformer structure is a core component. For each phoneme representation in the semantic representation `hs`, the self-attention mechanism calculates the degree of association between that phoneme and all other phonemes. Through this calculation, the model can capture the temporal dependencies between phonemes, understanding that the duration of a phoneme may be influenced by preceding and following phonemes. For example, in a word, if the preceding phoneme is a plosive, it may have a certain influence on the duration of the following vowel; the self-attention mechanism helps the model learn this complex dependency. After completing the self-attention calculation, the model outputs a scalar duration for each phoneme. This prediction process uses a non-autoregressive approach, meaning that the duration of all phonemes is generated independently, without relying on the output of the previous phoneme to generate the duration of the current phoneme, as in an autoregressive model. For example, for the phoneme sequence / h / , / The model will independently predict / h / for 3 frames, / , / l / , / oʊ / , / ... Duration information such as / for 2 frames, / l for 4 frames, / oʊ for 3 frames, etc.

[0037] After obtaining the predicted duration for each phoneme, a hard alignment matrix Ahard is constructed based on these durations. The hard alignment matrix has dimensions T×N, where T is the total number of frames (the sum of the predicted durations of all phonemes), and N is the number of phonemes. The matrix is ​​constructed using an "expansion" operation; specifically, each column of the matrix corresponds to one phoneme. For the i-th phoneme with a predicted duration of di, the values ​​in the first di rows of the column corresponding to that phoneme are set to 1, and the values ​​in the remaining rows are set to 0. For example, if the predicted duration of / h / is 3 frames, then in the hard alignment matrix, the first 3 rows of the column corresponding to / h / are set to 1, and the remaining rows are set to 0. In this way, the hard alignment matrix clearly identifies the active frames of each phoneme on the timeline, that is, which time frames correspond to which phoneme. For example, in a simple phoneme sequence, the total number of frames T=9 and the number of phonemes N=3. If the duration of the first phoneme is 3 frames, the duration of the second phoneme is 3 frames, and the duration of the third phoneme is 3 frames, then in the first 3 columns of the hard alignment matrix, the first 3 rows of the first column are 1 and the rest are 0; the middle 3 rows of the second column are 1 and the rest are 0; and the last 3 rows of the third column are 1 and the rest are 0.

[0038] This step maps the semantic representation `hs` to the temporal dimension, specifying the duration of each phoneme in the synthesized speech. The hard alignment matrix provides an initial framework for subsequent soft alignment, ensuring coarse-grained alignment of phonemes with acoustic features (e.g., ` / h / ` corresponds to the acoustic features of the previous 3 frames). If the duration prediction is inaccurate (e.g., ` / h / ` is predicted as 2 frames), hard alignment can lead to truncation or redundancy of phonemes, but subsequent soft alignment can correct such errors.

[0039] Step S205: Adjust the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generate a corresponding context vector based on the soft alignment matrix.

[0040] In this embodiment, the specific implementation process of adjusting the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generating a corresponding context vector based on the soft alignment matrix will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0041] Step S206: Obtain the preset residual variables.

[0042] In this embodiment, the specific implementation process of obtaining the preset residual variable will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0043] Step S207: Based on the decoder, the residual variable and the context vector are decoded to generate the corresponding Mel spectrum.

[0044] In this embodiment, the specific implementation process of decoding the residual variable and the context vector based on the decoder to generate the corresponding Mel spectrum will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0045] Step S208: Based on the vocoder, perform speech synthesis on the Mel spectrum to obtain the corresponding target speech data, and then output the target speech data.

[0046] In this embodiment, the specific implementation process of obtaining the corresponding target speech data by synthesizing the Mel spectrum based on the vocoder will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0047] In addition, the generated target voice data can be sent to relevant users via email, SMS, or other means to complete the output processing of the target voice data.

[0048] This application first receives the input text to be synthesized and a speaker prompt audio of a specified duration, and extracts features from the speaker prompt audio to obtain corresponding speaker features; then, it calls a preset speech generation model; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; then, based on the speaker features, the text encoder is used to convert the text to be synthesized to obtain the corresponding target semantic representation; subsequently, the duration predictor performs phoneme duration prediction processing on the target semantic representation, and constructs a corresponding hard alignment matrix based on the obtained prediction results; then, the soft alignment generation module adjusts the hard alignment matrix to generate a soft alignment matrix, and generates a corresponding context vector based on the soft alignment matrix; further, a preset residual variable is obtained; and the decoder decodes the residual variable and the context vector to generate the corresponding Mel spectrum; finally, the vocoder performs speech synthesis on the Mel spectrum to obtain the corresponding target speech data, and outputs the target speech data. Unlike existing autoregressive model-based speech generation methods, this application provides a novel and intelligent speech generation technology solution based on the above automated processing flow. This solution utilizes a collaborative speech generation model comprising multiple components, including a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder. By starting from text input and progressively performing semantic representation, duration prediction, alignment adjustment, introduction of residual variables, Mel spectrum generation, and waveform synthesis, it can automatically and accurately convert text into natural and diverse speech. This achieves high-quality text-to-speech synthesis, effectively improving speech generation efficiency and enhancing the quality of synthesized speech.

[0049] In some alternative implementations, step S203 includes the following steps: The text to be synthesized is subjected to phonetic conversion processing to obtain the corresponding phoneme sequence.

[0050] In this embodiment, the above-mentioned phoneme-to-phoneme conversion process includes: converting the input text to be synthesized into a phoneme sequence (such as / h) through a grapheme-to-phoneme (G2P) module. l oʊ / . This module is based on a pre-trained statistical model or neural network (such as LSTM or Transformer) and analyzes the mapping relationship between character combinations and phonemes (e.g., "h" corresponds to / h / , "e" in open syllables corresponds to / l oʊ / ). / ) generates phoneme sequences.

[0051] The phoneme sequence is encoded using the text encoder to obtain the corresponding initial speech representation.

[0052] In this embodiment, the text encoder can employ either Transformer or BiLSTM. After obtaining the phoneme sequence, the text encoder encodes the phoneme sequence: each phoneme is mapped to a learnable embedding vector, and contextual dependencies are captured through a self-attention mechanism or a recurrent structure to generate the corresponding initial semantic representation.

[0053] Based on a preset fusion method, the initial speech representation is fused using the speaker features to obtain a fused semantic representation.

[0054] In this embodiment, after obtaining the speaker features, they are incorporated into the initial semantic representation generated by the text encoder to generate a speaker-conditional semantic representation hs.

[0055] The aforementioned fusion method can employ two approaches. One is concatenation, where the speaker embedding vector *s* is directly concatenated with the phoneme representation output from each layer of the text encoder. For example, if the phoneme *pi* output by a certain layer of the text encoder is represented as *hpi*, concatenation results in [*hpi;s*], thus the new representation simultaneously incorporates the semantic information of the phoneme and speaker features. The other approach is layer-wise injection, which is more refined. It transforms the speaker embedding vector *s* (e.g., a linear transformation) and injects it into the input of each layer of the text encoder. Taking the Transformer as an example, before calculating the self-attention mechanism at each layer, the transformed speaker embedding vector is added to the phoneme embedding vector, ensuring that the influence of speaker features is considered when calculating the self-attention weights and updating the phoneme representation. Through this operation, the final generated speaker-conditional semantic representation hs contains both the semantic information of the text and the speaker's personalized features. Its dimension is T×D (T is the number of frames in the phoneme sequence, D is the feature dimension, and D will increase the dimension of the speaker embedding vector in the concatenation method).

[0056] Furthermore, based on the selected fusion method, the aforementioned speaker features can be used to fuse the initial speech representation, and the resulting fused semantic representation can be used as the corresponding target semantic representation.

[0057] The fused semantic representation is used as the target semantic representation.

[0058] This application performs phonetic conversion on the text to be synthesized to obtain the corresponding phoneme sequence; then, it encodes the phoneme sequence using a text encoder to obtain the initial speech representation; subsequently, based on a preset fusion method, it fuses the initial speech representation using speaker features to obtain the fused semantic representation; finally, it uses the fused semantic representation as the target semantic representation. Based on the above processing flow, this application uses a text encoder to convert the text to be synthesized into a numerical semantic representation that the model can process, while incorporating speaker features. This speaker-conditional processing method allows the synthesized speech to carry the characteristics of a specific speaker, making the speech more personalized. Phonetic conversion ensures accurate conversion from text to phonemes, avoiding errors caused by irregular spelling; the introduction of speaker embedding enables the model to have style transfer capabilities; and the final generated target semantic representation, as input to subsequent steps, contains both the semantic information of the text and implicitly includes the speaker's personalized features, ensuring the accuracy of the generated target semantic representation.

[0059] In some optional implementations of this embodiment, step S205 includes the following steps: Based on a preset flow matching strategy, the soft alignment generation module dynamically adjusts the hard alignment matrix to generate a corresponding soft alignment matrix.

[0060] In this embodiment, after inputting the semantic representation hs and the hard alignment matrix Ahard, the soft alignment generation (SAG) module begins to function. Its core is the use of flow-matching technology to dynamically adjust the hard alignment. Flow-matching is an iterative optimization method that aims to minimize the difference between hard and soft alignment, which can be measured using Kullback-Leibler Divergence (KL Divergence). KL Divergence can be understood as a measure of the "distance" between two probability distributions; a smaller value indicates greater similarity between the two distributions. During the iteration process, the model continuously adjusts the parameters of the soft alignment, causing the soft alignment matrix Asoft (initially possibly a randomly initialized or simple rule-based matrix) to gradually approach the hard alignment matrix Ahard, but not completely identical to it. This is because hard alignment is discrete (elements take values ​​of 0 or 1), while soft alignment is a continuous probability distribution (elements take values ​​between [0,1]). Through this iterative optimization, the discrete constraints of hard alignment are gradually softened, eventually generating a soft alignment matrix Asoft that meets the requirements. Asoft has dimensions T×N, where T is the total number of frames and N is the number of phonemes. For example, in the initial stage, the elements of the soft alignment matrix may be randomly distributed. As iterations proceed, it gradually exhibits a pattern similar to the hard alignment matrix, but each element's value is a probability value between 0 and 1, representing the activation probability of the i-th phoneme in frame t.

[0061] Call the preset target function.

[0062] In this embodiment, the objective function is specifically the Softmax function.

[0063] The soft alignment matrix is ​​normalized based on the objective function to obtain the normalized target soft alignment matrix.

[0064] In this embodiment, after obtaining the soft alignment matrix Asoft, a Softmax normalization operation is performed on each row. The Softmax function transforms a set of values ​​into a probability distribution such that the sum of all values ​​is 1. After Softmax normalization, the element values ​​of each row in the soft alignment matrix represent the relative magnitudes of the activation probabilities of each phoneme in the corresponding time frame. Then, the normalized soft alignment matrix is ​​used to generate the context vector. Its dimension is N×D (assuming the feature dimension of the semantic representation hs is D).

[0065] The target soft alignment matrix is ​​processed based on a preset context generation method to obtain the corresponding generated data.

[0066] In this embodiment, the above-mentioned context generation method includes: for the context vector of the i-th phoneme, it is a weighted sum of the features of the semantic representation hs across all time frames, and the weights are determined by the element values ​​of the corresponding columns in the normalized soft alignment matrix. That is, for the i-th phoneme, it comprehensively considers the semantic features across all time frames and performs a weighted sum of these features based on the activation probability of the phoneme in each time frame.

[0067] The target soft alignment matrix can be processed based on the above context generation method, and the generated data can be used as the corresponding context vector.

[0068] The generated data is used as the context vector.

[0069] This application uses a soft alignment generation module to dynamically adjust the hard alignment matrix based on a preset stream matching strategy, generating a corresponding soft alignment matrix. Then, a preset objective function is called, and the soft alignment matrix is ​​normalized based on the objective function to obtain a normalized target soft alignment matrix. Subsequently, the target soft alignment matrix is ​​processed based on a preset context generation method to obtain corresponding generated data. This generated data is then used as a context vector. Based on this processing flow, the soft alignment processing in this application, by generating a soft alignment matrix with a continuous probability distribution, can more flexibly capture the dynamic relationship between phonemes and acoustic features. The generated context vector provides a more robust semantic-temporal joint representation for the subsequent decoding process. In hard alignment, errors in duration prediction can lead to misalignment between phonemes and acoustic features. Soft alignment, through its flexible probability distribution and weighted summation, can solve this alignment inaccuracy problem to some extent, thereby improving the quality and accuracy of speech synthesis.

[0070] In some alternative implementations, step S206 includes the following steps: The noise is obtained by sampling from a pre-defined standard normal distribution.

[0071] In this embodiment, the residual sampling step is independent, not dependent on the input text or speaker features. In implementation, sampling is performed directly from the standard Gaussian distribution N(0,1) to obtain the residual variable z. Here, it is assumed that the dimension of z is set to 128 (the specific dimension is determined during training and kept consistent to ensure model stability). The sampling process varies depending on the model's architecture. If the model uses a variational autoencoder (VAE) architecture, reparameterization techniques are employed for sampling. Specifically, the model learns the learnable mean μ and standard deviation σ, while simultaneously sampling noise from the standard normal distribution N(0,1). .

[0072] Call the preset residual calculation formula.

[0073] In this embodiment, the above residual calculation formula may specifically include z=μ+ σ. Where μ represents the mean, which determines the central location of the sampling and reflects the model's estimate of the data distribution center; σ is the standard deviation, which controls the dispersion of the sampling, i.e. the range of data fluctuation around the mean; The noise is sampled from the standard normal distribution N(0,1), and its randomness ensures that each sampled z is different. In this way, the random sampling process, which is originally difficult to backpropagate directly, is transformed into a deterministic computation process that can differentiate and update μ and σ. Specifically, the residual calculation formula allows the model to perform random sampling while simultaneously updating the mean μ and standard deviation σ through the backpropagation algorithm, thereby achieving model training optimization.

[0074] In addition, if the model uses a simple sampling method, the residual variable z is directly generated from the standard Gaussian distribution N(0,I). This method is more direct and simple, but it lacks the flexible control over the sampling process that the VAE structure has.

[0075] The noise is calculated and processed based on the residual calculation formula to obtain the corresponding first calculation result.

[0076] In this embodiment, the noise can be substituted into the corresponding position in the residual calculation formula for calculation, and the first calculation result obtained can be used as the residual variable.

[0077] The first calculation result is used as the residual variable.

[0078] In this embodiment, the residual variable plays a crucial role, primarily in capturing unpredictable variations in the data. Real-world speech data contains many factors that are difficult to accurately describe using only the input text and speaker features. These include background noise, the intensity and frequency of which vary depending on the recording environment; and subtle intonation fluctuations, where even with the same text, a speaker's intonation may exhibit slight variations during pronunciation. Maintaining the same dimensionality as during training ensures model stability during both training and inference. Since the model learns the data distribution based on a specific dimensional residual variable during training, changing the dimensionality during inference could prevent the model from correctly processing residual information, thus affecting the quality of the generated speech.

[0079] This application obtains noise by sampling from a pre-defined standard normal distribution; then, it calls a pre-defined residual calculation formula; subsequently, it calculates the noise based on the residual calculation formula to obtain a first calculation result; and finally, it uses the first calculation result as a residual variable. Based on this processing flow, this application improves the intelligence of residual variable generation by sampling from a standard normal distribution to obtain noise, then calculating the noise based on the residual calculation formula, and using the first calculation result as a residual variable. Residual sampling introduces randomness into the model, a characteristic that enables the model to generate diverse speech waveforms.

[0080] In some alternative implementations, step S207 includes the following steps: Obtain the preset fusion strategy.

[0081] In this embodiment, the above-mentioned fusion strategy can adopt two approaches. One is a concatenation method, which involves directly concatenating the context vector and the residual variable along a certain dimension to form a new vector. This new vector contains the semantic information of the text (carried by the context vector) and the random variation information of the residual (carried by the residual variable). The other approach is a cross-attention mechanism. Under this mechanism, the decoder dynamically calculates the attention weights between the context vector and the residual variable based on their relationship, and then performs weighted fusion of the information from the two vectors based on these weights.

[0082] Based on the fusion strategy, the decoder is used to fuse the residual variable and the context vector to generate the corresponding initial Mel spectrum.

[0083] In this embodiment, the decoder described above can be a Mel decoder, which can employ an autoregressive or non-autoregressive Transformer structure. When the input context vector and residual variables are input, the decoder begins operation, fusing the residual variables and context vector according to the selected fusion strategy, and generating an initial Mel spectrogram x_pred (or initial Mel spectrum), with dimensions T′×80, where T′ represents the number of time frames, reflecting the information of the speech at different time points; and 80 represents the number of Mel bands, indicating the number of different frequency bands into which the speech spectrum is divided. Although this initial Mel spectrogram has captured the semantic information of the text and the random variations of the residuals, due to the characteristics of the decoder itself, there may be a problem of blurred high-frequency details (oversmoothing), meaning that the spectral information in the high-frequency part is not clear and accurate enough.

[0084] The initial Mel spectrum is optimized based on a preset post-processing network to obtain the corresponding target Mel spectrum.

[0085] In this embodiment, the specific implementation process of optimizing the initial Mel spectrum based on the preset post-processing network to obtain the corresponding target Mel spectrum will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0086] The target Mel spectrum is taken as the Mel spectrum.

[0087] This application obtains a preset fusion strategy; then, based on the fusion strategy, a decoder is used to fuse the residual variables and context vectors to generate the corresponding initial Mel spectrum; subsequently, a preset post-processing network is used to optimize the initial Mel spectrum to obtain the corresponding target Mel spectrum; finally, the target Mel spectrum is used as the Mel spectrum. Based on the above processing flow, the Mel decoding and post-processing steps provided in this application convert semantic and residual information into acoustic features (Mel spectrum). The initial Mel spectrum generated by the decoder is essentially a coarse-grained spectrum, which mainly captures basic information such as the low-frequency energy distribution of speech, reflecting the approximate semantic content of the text and the random variation trend brought about by the residuals. The post-processing network, through its local modeling capabilities, sharpens the high-frequency details in the initial Mel spectrum, improving the quality of the spectrum. The final generated Mel spectrum is closer to the spectrum of real speech, reducing the mechanical feel of synthesized speech, making the final generated speech more natural, realistic, and more in line with the characteristics of human speech.

[0088] In some optional implementations of this embodiment, the optimization of the initial Mel spectrum based on a preset post-processing network to obtain the corresponding target Mel spectrum includes the following steps: Invoke the preset post-processing network.

[0089] In this embodiment, the post-processing network can specifically adopt a CNN structure. Since the initial Mel spectrogram x_pred suffers from blurred high-frequency details, the post-processing network optimizes x_pred. The post-processing network has local modeling capabilities, allowing it to focus on local features in the Mel spectrogram. During optimization, residual connections are used, adding the initial Mel spectrogram xpred to the output of the post-processing network. Specifically, the post-processing network performs a series of convolutional operations on xpred to obtain the output PostNet(xpred). This output PostNet(xpred) primarily learns and corrects high-frequency details, such as formant positions (formants are regions of concentrated energy in the speech spectrum, and their positions significantly influence timbre and other characteristics of speech) and spectral slope (the spectral slope reflects the trend of speech spectral energy change with frequency).

[0090] The initial Mel spectrum is convolved using the post-processing network to obtain the corresponding output result.

[0091] In this embodiment, the post-processing network performs a series of convolution operations on the initial Mel spectrum x_pred to obtain the output PostNet(x_pred). This output mainly learns and corrects high-frequency details, such as formant positions (formants are regions of concentrated energy in the speech spectrum, and their positions have an important impact on the timbre and other features of speech) and spectral slope (the spectral slope reflects the trend of speech spectrum energy changing with frequency).

[0092] The initial Mel spectrum and the output result are calculated based on the preset residual connection formula to obtain the corresponding second calculation result.

[0093] In this embodiment, the residual connection formula may specifically include: x_final = x_pred + PostNet(x_pred). This reflects the design philosophy of the post-processing network. This connection method ensures that the main task of the post-processing network is to correct the errors in x_pred, rather than completely changing the main features of x_pred. This is because directly outputting the final Mel spectrogram by the post-processing network could lead to overfitting, where the post-processing network focuses too much on the details of the training data and fails to generalize well to new data. However, through residual connections, the post-processing network only needs to learn the differences between x_pred and the true Mel spectrogram, thereby improving the spectral quality while maintaining the main features such as textual semantics and residual information accurately captured in x_pred.

[0094] Specifically, the initial Mel spectrum and the output result can be substituted into the corresponding positions in the residual connection formula for calculation, and the obtained second calculation result can be used as the corresponding target Mel spectrum.

[0095] The second calculation result is used as the target Mel spectrum.

[0096] This application calls a pre-defined post-processing network; then, based on the post-processing network, it performs a convolution operation on the initial Mel spectrum to obtain the corresponding output result; subsequently, it calculates and processes the initial Mel spectrum and the output result based on a pre-defined residual connection formula to obtain the corresponding second calculation result; finally, the second calculation result is used as the target Mel spectrum. Based on the above processing flow, this application uses the local modeling capability of the post-processing network to sharpen the high-frequency details in the initial Mel spectrum, improving the quality of the spectrum, making the final generated Mel spectrum closer to the spectrum of real speech, reducing the mechanical feel of synthesized speech, and making the final generated speech more natural, realistic, and more in line with the characteristics of human speech.

[0097] In some optional implementations of this embodiment, step S208 includes the following steps: Based on the vocoder, the Mel spectrum is processed by waveform synthesis to obtain the corresponding initial speech data.

[0098] In this embodiment, the vocoder specifically employs a vocoder such as HiFi-GAN. After inputting an 80-dimensional Mel spectrum x_final, HiFi-GAN and similar technologies convert it into a speech waveform using a Generative Adversarial Network (GAN). The generator consists of multiple layers of transposed convolutions and residual blocks: the transposed convolutions progressively upsample the spectrum (e.g., from 80 dimensions to 1024 dimensions), and the residual blocks capture local features of the audio (e.g., periodicity, harmonic structure). The discriminator distinguishes between real and synthesized audio through multi-scale discrimination: lower-level discriminators focus on local details (e.g., sampling point continuity), while higher-level discriminators focus on global structure (e.g., intonation coherence). The generator minimizes the discriminator's loss through adversarial training, making the output waveform closer to the real audio. The sampling rate of the final generated waveform (i.e., the initial speech data) is consistent with the original audio (e.g., 22.05 kHz).

[0099] The initial speech data is subjected to spectral dynamic range adjustment processing to obtain the corresponding first speech data.

[0100] In this embodiment, the speech spectrum generated by the vocoder may differ from that of real speech in terms of dynamic range. Dynamic range refers to the amplitude difference between the strongest and weakest parts of an audio signal. Real speech typically has a rich dynamic range, encompassing a variety of amplitude variations from faint consonants to loud vowels. If the dynamic range of the generated speech is too narrow, the speech will sound flat and lack expressiveness; if it is too wide, distortion or noise problems may occur. By adjusting the dynamic range of the spectrum, the amplitude variations of the generated speech can be made more consistent with the characteristics of real speech, thereby improving the quality and naturalness of the speech.

[0101] The implementation process of the aforementioned dynamic range adjustment includes: First, analyzing the speech spectrum generated by the vocoder to calculate its dynamic range. Using statistical methods, the maximum and minimum amplitude values ​​in the spectrum are identified; the difference between these two values ​​is the dynamic range. Then, a target dynamic range is set based on the distribution of the dynamic range of real speech. Generally, a suitable dynamic range reference value is obtained by analyzing a large number of real speech samples. Finally, dynamic range compression or expansion operations are performed on the generated speech spectrum. If the dynamic range of the generated speech is too wide, a compression algorithm, such as logarithmic compression, is used to appropriately reduce the larger amplitude values, bringing the dynamic range closer to the target value. If the dynamic range is too narrow, an expansion algorithm, such as linear expansion, is used to appropriately amplify the amplitude values, enhancing the amplitude variation of the speech.

[0102] The first speech data is subjected to harmonic enhancement processing to obtain the corresponding second speech data.

[0103] In this embodiment, the speech signal contains rich harmonic structures, especially the vowel components. Harmonics are frequency components that are integer multiples of the fundamental frequency, and they collectively constitute the timbre characteristics of speech. When generating speech, the vocoder may not be able to perfectly reproduce these harmonic structures, resulting in a less rich and natural timbre. Through harmonic enhancement processing, the harmonic components in the speech can be highlighted, improving the timbre and making it closer to real speech.

[0104] The implementation process of the aforementioned harmonic enhancement processing includes: performing fundamental frequency detection on the generated speech to determine the fundamental frequency F0. Fundamental frequency detection methods such as autocorrelation algorithms and cepstral algorithms can be used to accurately find the fundamental frequency of the speech signal. Based on the fundamental frequency F0, the frequencies of each harmonic in the speech are calculated, i.e., nF0 (n=1,2,3,...). Design a harmonic enhancement filter that has high gain at each harmonic frequency and low gain at other frequencies. This can be achieved using a bandpass filter bank, where the center frequency of each bandpass filter corresponds to a harmonic frequency, and the bandwidth is adjusted according to the specific requirements. Passing the generated speech through the harmonic enhancement filter bank amplifies the harmonic components, thereby improving the timbre of the speech.

[0105] The second speech data is subjected to noise suppression and smoothing to obtain the corresponding third speech data.

[0106] In this embodiment, the vocoder may introduce some noise during the speech generation process. This noise may originate from factors such as model imperfections or noise in the training data. Furthermore, the speech signal may contain discontinuities or abrupt changes in the frequency or time domains, resulting in a grainy or uneven sound. Noise suppression and smoothing processes can remove or reduce the impact of noise, making the speech signal smoother and more natural.

[0107] The noise suppression and smoothing process described above includes: Noise suppression: Noise suppression is performed using spectral subtraction. First, the noise spectrum in the generated speech is estimated. This can be done by sampling noise in silent segments of the speech (if any) or by using noise estimation algorithms to obtain an approximation of the noise spectrum. Then, the estimated noise spectrum is subtracted from the spectrum of the generated speech to obtain the noise-suppressed spectrum. During spectral subtraction, it is important to control the magnitude of the subtraction to avoid excessive subtraction that could lead to speech distortion.

[0108] Smoothing: In the time domain, the speech signal is smoothed using a moving average filter. The moving average filter reduces abrupt changes and noise by averaging samples of the speech signal within a certain time window. In the frequency domain, windowing techniques, such as Hanning or Hamming windows, are applied to the speech spectrum to smooth out spectral edges and reduce spectral leakage. Through smoothing in both the time and frequency domains, the generated speech becomes more fluent and natural.

[0109] The third voice data is used as the target voice data.

[0110] Based on the above processing flow, this application uses a vocoder to perform waveform synthesis processing on the Mel spectrum to generate high-fidelity speech. Furthermore, it intelligently adjusts the dynamic range of the spectrum, enhances harmonics, and suppresses and smooths noise in the initial speech data, thereby further improving the naturalness and quality of the synthesized speech.

[0111] In one optional implementation, the core objective of the model training phase of the aforementioned speech generation model is to learn the mapping relationship from phoneme sequences to speech spectra using text-speech alignment data, while simultaneously modeling the residual distribution of speech. The specific model construction process includes: 1. Training data input preparation (data preparation). (1) Text processing (phonetic conversion). The input text is converted into a phoneme sequence using a phonetic conversion tool. For example: "hello" → / h loʊ / . The purpose of this is to convert discrete characters into basic units of speech, which facilitates subsequent processing by the model. (2) Preparation of target Mel spectrogram. The real speech is converted into a Mel spectrogram (80-dimensional spectrogram) through short-time Fourier transform (STFT) and Mel filter bank. Its function is to serve as a supervisory signal to guide the model to learn the mapping relationship from text to spectrum. (3) Optional: short-term speaker cue. A 3-second Mel segment of speech is extracted as a speaker cue (speaker cue), and the speaker embedding (speaker embeddings) is extracted through the speaker encoder (speaker encoder). The purpose is to model the speaker style (such as timbre, intonation).

[0112] 2. Training Process Data Processing (Training Data Process). (1) Semantic Representation Extraction. Input: Phoneme sequence. Processing: The text encoder (e.g., Transformer) encodes the phoneme sequence into a semantic representation h_s, which has a dimension of T×D, where T represents the phoneme duration and D represents the hidden layer dimension. For example: / h loʊ / →h_s=[h1,h2,h3,h4]. The output h_s contains semantic information and coarse-grained duration information.

[0113] (2) Soft Alignment Matrix Generation. Input: h_s + hard alignment matrix A_hard (from duration predictor). Processing: Duration Predictor: Predicts the number of Mel frames corresponding to each phoneme, for example, predicting / h / corresponds to 3 frames. Monotonic Alignment Search: Generates the initial hard alignment matrix A_hard, with dimensions T×N, where N is the number of Mel frames. Soft Alignment Generation (SAG): Learns the vector field through flow-matching, mapping A_hard to the target attention logarithm matrix A_log (calculated from the real Mel). Define the target vector field: u_t^SAG = A_logarithm - (1 - σ_minimum)A_hard, where σ_minimum is a noise coefficient, its role is to introduce a certain degree of randomness during training, prevent the model from overfitting, and enable the model to better generalize to different situations. Minimize Mean Squared Error (MSE) loss: .in This represents the expectation over all time steps t. The model is based on the current hard alignment matrix. Semantic representation The model learns a vector field at time step t using parameters θ. By minimizing this loss, the learned vector field is made as close as possible to the target vector field. The output is a soft alignment matrix A_soft, which reflects the soft alignment relationship between phonemes and mermaid frames. That is, one phoneme may correspond to multiple mermaid frames with varying degrees of correspondence.

[0114] (3) Residual Variable Extraction and Gaussianization. Input: Target Mel vector x + context vector c_s (obtained by soft upsampling A_). Processing: Reversible encoder: Maps the Mel residuals to latent variables z using a conditional normalizing flow, i.e., z = f(x; c_s). Simultaneously optimizes the KL divergence: KL(p(z|c_s)∥q(z)), where q(z) is a standard Gaussian distribution. The purpose of optimizing the KL divergence is to make the distribution p(z|c_s) of the latent variable z under the given context vector c_s as close as possible to the standard Gaussian distribution q(z), so that z can be easily sampled from the standard Gaussian distribution during inference (speech generation). The output is the residual variable z.

[0115] (4) Mel Reconstruction. Input: c_s + z. Processing: The Mel decoder reconstructs the Mel spectrum x_predict from the context vector c_s and the residual variable z. The loss function used is L1 loss, i.e., L_recon = ||x - x_predict||_1, where x is the true target Mel spectrum. By minimizing this loss, the reconstructed Mel spectrum x_predict is made as close as possible to the true target Mel spectrum x. The output is the reconstructed Mel x_predict.

[0116] (5) Post-processing network post-processing. Input: x_prediction. Processing procedure: The post-processing network based on flow-matching (PostNet based on Flow-Matching) improves the quality of the Mel spectrum with fewer optimization steps, such as reducing spectral blur. The loss function used is also L1 loss, i.e., L_post=||x-PostNet(x_prediction)||_1. By minimizing this loss, the Mel spectrum after processing by the post-processing network is made closer to the true target Mel spectrum x. The output is the final training target Mel x_final.

[0117] 3. Joint Training Objective (Joint Loss). The total loss function is the weighted sum of the losses at each stage, i.e., L_total = λ1L_SAG + λ2L_recon + λ3L_post + λ4KL. Here, λ1, λ2, λ3, and λ4 are hyperparameters. Their role is to balance the weight of each loss in the total loss. By adjusting these hyperparameters, the learning focus of the model in different aspects can be controlled. For example, if you want the model to focus more on learning the soft alignment matrix, you can appropriately increase the value of λ1.

[0118] During model training, training data is first prepared according to the methods described above, including phonetic conversion, generation of target Mel spectrograms, and extraction of speaker embeddings (if applicable). Next, the training data flow proceeds sequentially, including semantic representation extraction, soft alignment matrix generation, residual variable extraction and Gaussianization, Mel reconstruction, and post-processing by the post-processing network. Finally, based on the joint loss function, the parameters of each component in the model (such as the text encoder, duration predictor, reversible encoder, Mel decoder, and post-processing network) are continuously adjusted using the backpropagation algorithm to minimize the total loss. After multiple rounds of iterative training, a speech generation model capable of generating high-quality speech spectrograms from phoneme sequences is finally generated.

[0119] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0120] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0121] Furthermore, the technical solution of this application has the following advantages: Zero-sample speech synthesis: It can generate natural, fluent, and intonation-rich speech without requiring target speaker audio, significantly improving the system's flexibility in real-world applications. Controllable and interpretable prosody: Through explicit prosodic vector decomposition and contrastive constraints, the model can independently adjust prosodic features such as speech rate, intonation, and emotion, making the generated speech more expressive. Improved inference speed: The use of a non-autoregressive structure for parallel generation significantly reduces inference latency, making it suitable for real-time voice interaction systems. Balance of sound quality and stability: The decoupled structure avoids timbre aliasing, ensuring that the generated speech maintains speaker consistency and clarity. Integration of theory and engineering: The model structure combines interpretability and scalability, making it easily portable to other speech tasks (such as speech conversion, speech editing, and speech emotion control).

[0122] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0123] It should be emphasized that, in order to further ensure the privacy and security of the aforementioned target voice data, the target voice data can also be stored in a blockchain node.

[0124] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0125] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0126] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an artificial intelligence-based speech generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0127] like Figure 3 As shown, the AI-based speech generation device 300 described in this embodiment includes: a first processing module 301, a calling module 302, a conversion module 303, a second processing module 304, a generation module 305, an acquisition module 306, a decoding module 307, and a third processing module 308. Wherein: The first processing module 301 is used to receive the input text to be synthesized and the speaker prompt audio of a specified duration, and to extract the corresponding speaker features from the speaker prompt audio. The calling module 302 is used to call a preset speech generation model; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; The conversion module 303 is used to convert the text to be synthesized based on the speaker features using the text encoder to obtain the corresponding target semantic representation; The second processing module 304 is used to perform phoneme duration prediction processing on the target semantic representation based on the duration predictor, and construct a corresponding hard alignment matrix based on the obtained prediction result. The generation module 305 is used to adjust the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generate a corresponding context vector based on the soft alignment matrix. The acquisition module 306 is used to acquire preset residual variables; Decoding module 307 is used to perform decoding processing on the residual variable and the context vector based on the decoder to generate the corresponding Mel spectrum; The third processing module 308 is used to perform speech synthesis on the Mel spectrum based on the vocoder to obtain corresponding target speech data, and to output the target speech data.

[0128] In some optional implementations of this embodiment, the conversion module 303 includes: The conversion submodule is used to perform phonetic conversion processing on the text to be synthesized to obtain the corresponding phoneme sequence; The encoding submodule is used to encode the phoneme sequence based on the text encoder to obtain the corresponding initial speech representation; The first fusion submodule is used to perform fusion processing on the initial speech representation using the speaker features based on a preset fusion method to obtain the fused semantic representation; The first determining submodule is used to use the fused semantic representation as the target semantic representation.

[0129] In some optional implementations of this embodiment, the generation module 305 includes: The adjustment submodule is used to dynamically adjust the hard alignment matrix based on a preset flow matching strategy using the soft alignment generation module to generate a corresponding soft alignment matrix. The first calling submodule is used to call the preset target function; The operation submodule is used to perform a normalization operation on the soft alignment matrix based on the objective function to obtain the normalized target soft alignment matrix; The first processing submodule is used to process the target soft alignment matrix based on a preset context generation method to obtain the corresponding generated data; The second determining submodule is used to use the generated data as the context vector.

[0130] In some optional implementations of this embodiment, the acquisition module 306 includes: The sampling submodule is used to sample from a preset standard normal distribution to obtain the corresponding noise. The second calling submodule is used to call the preset residual calculation formula; The calculation submodule is used to perform calculations on the noise based on the residual calculation formula to obtain the corresponding first calculation result; The third determining submodule is used to use the first calculation result as the residual variable.

[0131] In some optional implementations of this embodiment, the decoding module 307 includes: The `get` submodule is used to retrieve preset fusion strategies; The second fusion submodule is used to perform fusion processing on the residual variable and the context vector using the decoder based on the fusion strategy to generate the corresponding initial Mel spectrum; An optimization submodule is used to optimize the initial Mel spectrum based on a preset post-processing network to obtain the corresponding target Mel spectrum; The fourth determining submodule is used to use the target Mel spectrum as the Mel spectrum.

[0132] In some optional implementations of this embodiment, the optimized submodule includes: The calling unit is used to invoke a preset post-processing network; A convolutional unit is used to perform a convolution operation on the initial Mel spectrum based on the post-processing network to obtain the corresponding output result; The calculation unit is used to perform calculations on the initial Mel spectrum and the output result based on a preset residual connection formula to obtain the corresponding second calculation result; A determining unit is used to take the second calculation result as the target Mel spectrum.

[0133] In some optional implementations of this embodiment, the third processing module 308 includes: The second processing submodule is used to perform waveform synthesis processing on the Mel spectrum based on the vocoder to obtain the corresponding initial speech data; The third processing submodule is used to perform spectral dynamic range adjustment processing on the initial speech data to obtain the corresponding first speech data. The fourth processing submodule is used to perform harmonic enhancement processing on the first speech data to obtain the corresponding second speech data; The fifth processing submodule is used to perform noise suppression and smoothing processing on the second speech data to obtain the corresponding third speech data; The fifth determining submodule is used to use the third voice data as the target voice data.

[0134] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0135] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0136] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0137] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for speech generation methods based on artificial intelligence. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0138] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based speech generation method.

[0139] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0140] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based speech generation method described above.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0142] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0143] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A speech generation method based on artificial intelligence, characterized in that, Includes the following steps: The system receives the input text to be synthesized and the speaker prompt audio of a specified duration, and performs feature extraction on the speaker prompt audio to obtain the corresponding speaker features; A preset speech generation model is invoked; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; Based on the speaker features, the text encoder is used to transform the text to be synthesized to obtain the corresponding target semantic representation; The phoneme duration is predicted based on the duration predictor, and a corresponding hard alignment matrix is ​​constructed based on the prediction results. The hard alignment matrix is ​​adjusted based on the soft alignment generation module to generate a soft alignment matrix, and a corresponding context vector is generated based on the soft alignment matrix. Obtain the preset residual variables; The residual variable and the context vector are decoded based on the decoder to generate the corresponding Mel spectrum; Based on the vocoder, speech synthesis is performed on the Mel spectrum to obtain the corresponding target speech data, and the target speech data is then output.

2. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of converting the text to be synthesized using the text encoder based on the speaker features to obtain the corresponding target semantic representation specifically includes: The text to be synthesized is subjected to phonetic conversion processing to obtain the corresponding phoneme sequence; The phoneme sequence is encoded based on the text encoder to obtain the corresponding initial speech representation; Based on a preset fusion method, the initial speech representation is fused using the speaker features to obtain a fused semantic representation. The fused semantic representation is used as the target semantic representation.

3. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of adjusting the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generating a corresponding context vector based on the soft alignment matrix, specifically includes: Based on a preset flow matching strategy, the soft alignment generation module is used to dynamically adjust the hard alignment matrix to generate a corresponding soft alignment matrix. Call the preset target function; The soft alignment matrix is ​​normalized based on the objective function to obtain the normalized target soft alignment matrix. The target soft alignment matrix is ​​processed based on a preset context generation method to obtain the corresponding generated data; The generated data is used as the context vector.

4. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of obtaining the preset residual variable specifically includes: The noise is obtained by sampling from a pre-defined standard normal distribution; Call the preset residual calculation formula; The noise is calculated and processed based on the residual calculation formula to obtain the corresponding first calculation result; The first calculation result is used as the residual variable.

5. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of decoding the residual variable and the context vector based on the decoder to generate the corresponding Mel spectrum specifically includes: Obtain the preset fusion strategy; Based on the fusion strategy, the decoder is used to fuse the residual variables and the context vector to generate the corresponding initial Mel spectrum; The initial Mel spectrum is optimized based on a preset post-processing network to obtain the corresponding target Mel spectrum; The target Mel spectrum is taken as the Mel spectrum.

6. The artificial intelligence-based speech generation method according to claim 5, characterized in that, The step of optimizing the initial Mel spectrum based on a preset post-processing network to obtain the corresponding target Mel spectrum specifically includes: Invoke the preset post-processing network; The initial Mel spectrum is convolved based on the post-processing network to obtain the corresponding output result; The initial Mel spectrum and the output result are calculated based on the preset residual connection formula to obtain the corresponding second calculation result; The second calculation result is used as the target Mel spectrum.

7. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of synthesizing the corresponding target speech data based on the vocoder using the Mel spectrum specifically includes: Based on the vocoder, the Mel spectrum is processed by waveform synthesis to obtain the corresponding initial speech data; The initial speech data is subjected to spectral dynamic range adjustment processing to obtain the corresponding first speech data; The first speech data is subjected to harmonic enhancement processing to obtain the corresponding second speech data; The second speech data is subjected to noise suppression and smoothing to obtain the corresponding third speech data; The third voice data is used as the target voice data.

8. A speech generation device based on artificial intelligence, characterized in that, include: The first processing module is used to receive the input text to be synthesized and the speaker prompt audio of a specified duration, and to extract the corresponding speaker features from the speaker prompt audio. The calling module is used to call a preset speech generation model; wherein, the speech generation model includes a text encoder, a duration predictor, a soft alignment generation module, a decoder, and a vocoder; The conversion module is used to convert the text to be synthesized based on the speaker features using the text encoder to obtain the corresponding target semantic representation; The second processing module is used to predict the phoneme duration of the target semantic representation based on the duration predictor, and to construct a corresponding hard alignment matrix based on the obtained prediction results. The generation module is used to adjust the hard alignment matrix based on the soft alignment generation module to generate a soft alignment matrix, and generate a corresponding context vector based on the soft alignment matrix; The acquisition module is used to acquire preset residual variables; The decoding module is used to decode the residual variable and the context vector based on the decoder to generate the corresponding Mel spectrum; The third processing module is used to perform speech synthesis on the Mel spectrum based on the vocoder to obtain the corresponding target speech data, and to output the target speech data.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7.

Images