Speech generation method and device

Abstract

The present application relates to a method and device for speech generation, wherein the speech generation method comprises: extracting features from first speech data corresponding to an input text to obtain first and second breath features of the first speech data; fusing the first and second breath features to obtain breath fusion features; processing the breath fusion features through a pre-trained breath prediction model to obtain breath feature information of the first speech data; determining acoustic feature information corresponding to the input text, and generating second speech data corresponding to the input text based on the breath feature information and acoustic feature information. Through this application, the problem of being unable to generate speech that accurately restores breath features, resulting in insufficient fluency and naturalness of the generated speech, is solved, and the ability to generate speech that accurately restores breath features is achieved, significantly improving the anthropomorphism of speech generation.

Description

Technical Field

[0001] The present application relates to the field of speech generation technology, and in particular to a speech generation method and device. Background Art

[0002] Speech generation technology converts text into natural, fluent speech and is widely used in diverse scenarios, such as audiobooks and voice translation. However, existing speech generation methods typically focus on improving the authenticity of timbre to enhance the anthropomorphic nature of pronunciation, failing to accurately reproduce the characteristics of breath patterns, resulting in insufficient fluency and naturalness in the generated speech.

[0003] There is currently no effective solution to the problem that related technologies cannot generate speech that accurately restores the characteristics of the air outlet, resulting in insufficient fluency and naturalness of the generated speech. Summary of the Invention

[0004] In this embodiment, a speech generation method and device are provided to solve the problem in related technologies that speech cannot be generated that accurately restores the air vent position, duration and other air vent-related features, resulting in insufficient fluency and naturalness of the generated speech.

[0005] In a first aspect, this embodiment provides a speech generation method, including:

[0006] Performing feature extraction on first speech data corresponding to the input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data;

[0007] fusing the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0008] Processing the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0009] Acoustic feature information corresponding to the input text is determined, and second voice data corresponding to the input text is generated based on the air port feature information and the acoustic feature information.

[0010] In some embodiments, the training process of the gas outlet prediction model includes:

[0011] Separating the human voice from each original audio to obtain the human voice audio in the original audio;

[0012] Identifying the text content and word-level timestamp sequence corresponding to each of the human voice audios, and performing audio segmentation on the human voice audios based on a matching result between the text content and the word-level timestamp sequence;

[0013] Annotate each audio segment obtained by segmentation to obtain the corresponding sample audio;

[0014] The breath prediction model is trained based on the breath fusion features of each of the sample audios.

[0015] In some embodiments, segmenting the human voice audio based on the matching result between the text content and the word-level timestamp sequence includes:

[0016] Optimizing the text content by using a transformer-based bidirectional encoder representation model to obtain multiple sentences of the text content;

[0017] Each of the sentences is matched with the word-level timestamp sequence, and the human voice audio is segmented based on the matching results; the matching results are used to indicate the continuous word segments in the word-level timestamp sequence that match each of the sentences.

[0018] In some embodiments, the training of the breath prediction model based on the breath fusion feature of each of the sample audios includes:

[0019] Performing feature extraction on each of the sample audios to obtain a breath fusion feature of each of the sample audios;

[0020] Performing random masking on the breath fusion features of each sample audio;

[0021] Each of the air vent fusion features after random mask processing is input into the air vent prediction model, and the air vent prediction model is trained based on a focal loss function.

[0022] In some embodiments, extracting features from the first voice data corresponding to the input text to obtain first and second breath features of the first voice data includes:

[0023] Determine a static Mel-frequency cepstral coefficient feature vector of the first speech data, and a first-order difference coefficient and a second-order difference coefficient corresponding to the static Mel-frequency cepstral coefficient feature vector;

[0024] Determining the first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficient, and the second-order difference coefficient;

[0025] Multiple audio segments of the first speech data are processed using a self-supervised speech representation learning model, and the second breath feature of the first speech data is determined based on the processing results.

[0026] In some embodiments, determining a static Mel-frequency cepstral coefficient feature vector of the first speech data includes:

[0027] Performing frame processing on the first voice data to obtain multiple frames of audio signals;

[0028] Performing a fast Fourier transform on each frame of the windowed audio signal to obtain a corresponding amplitude spectrum;

[0029] Performing frequency domain filtering on the amplitude spectrum through a Mel filter bank, and determining the energy of each filter channel of the Mel filter bank after filtering;

[0030] Performing discrete cosine transform on the energy of the filter channel, and determining a plurality of static Mel-frequency cepstral coefficient feature vectors of the first speech data based on the transform result.

[0031] In some embodiments, processing the plurality of audio segments of the first speech data using a self-supervised speech representation learning model, and determining the second breath feature of the first speech data based on the processing results, includes:

[0032] dividing the preprocessed first voice data into a plurality of audio segments;

[0033] Processing each of the audio clips using a self-supervised speech representation learning model to obtain an output result of each target output layer in the self-supervised speech representation learning model; the target output layer is the output layer that is most sensitive to non-speech events;

[0034] The output results of each target output layer are weightedly fused to obtain the second breath feature of the first speech data.

[0035] In some embodiments, the processing of the gas outlet prediction model includes:

[0036] Processing the breath fusion features of the first speech data through a bidirectional long short-term memory network to obtain corresponding time series features;

[0037] Input the temporal features into the multi-head attention module to obtain the corresponding attention weights;

[0038] The fusion result of the time series feature and the attention weight is input into the fully connected layer, and the breath feature information of the first speech data is output.

[0039] In some embodiments, determining the acoustic feature information corresponding to the input text, and generating second speech data corresponding to the input text based on the airway feature information and the acoustic feature information, includes:

[0040] Determining the acoustic feature information based on a phoneme sequence corresponding to the input text;

[0041] generating a corresponding mel-spectrogram based on the air port feature information and the acoustic feature information;

[0042] The mel-spectrogram is processed by a vocoder to obtain the second speech data corresponding to the input text.

[0043] In a second aspect, this embodiment provides a speech generation device, including:

[0044] an extraction module, configured to perform feature extraction on first speech data corresponding to an input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data;

[0045] a fusion module, configured to fuse the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0046] a prediction module, configured to process the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0047] A generation module is used to determine the acoustic feature information corresponding to the input text, and generate second voice data corresponding to the input text based on the air outlet feature information and the acoustic feature information.

[0048] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech generation method described in the first aspect when executing the computer program.

[0049] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the speech generation method described in the first aspect is implemented.

[0050] Compared with the related art, the speech generation method and device provided in this embodiment obtain the first breath feature and the second breath feature of the first speech data by extracting features from the first speech data corresponding to the input text; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing the breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing the breath feature information in the first speech data; the first breath feature and the second breath feature are fused to obtain the breath fusion feature of the first speech data; the breath fusion feature of the first speech data is processed by a pre-trained breath prediction model to obtain the breath feature information of the first speech data; the breath feature information includes the probability value of each audio frame in the first speech data being the breath position; the acoustic feature information corresponding to the input text is determined, and based on the breath feature information and the acoustic feature information, the second speech data corresponding to the input text is generated, which solves the problem that speech that accurately restores the breath feature cannot be generated, resulting in insufficient fluency and naturalness of the generated speech, and realizes the ability to generate speech that accurately restores the breath feature, significantly improving the anthropomorphism of speech generation.

[0051] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 This is a hardware structure block diagram of a terminal device for the speech generation method provided in one embodiment of the present application;

[0054] Figure 2 This is a flowchart of a speech generation method provided by an embodiment of the present application;

[0055] Figure 3 This is a flowchart of a method for preprocessing raw audio provided by an embodiment of the present application;

[0056] Figure 4 This is a flowchart of an audio segmentation method provided by an embodiment of the present application;

[0057] Figure 5 This is a flow chart of a gas port feature fusion method provided in one embodiment of the present application;

[0058] Figure 6 This is a flow chart of a method for extracting gas port features provided in one embodiment of the present application;

[0059] Figure 7 is a flow chart of a gas port feature extraction method provided by another embodiment of the present application;

[0060] Figure 8 This is a flow chart of a gas outlet prediction model training method provided in one embodiment of the present application;

[0061] Figure 9 This is a flow chart of a speech generation method provided by a preferred embodiment of the present application;

[0062] Figure 10 This is a structural block diagram of a speech generation device provided in one embodiment of the present application.

[0063] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, extraction module; 20, fusion module; 30, prediction module; 40, generation module. DETAILED DESCRIPTION

[0064] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0065] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0066] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a block diagram of the hardware structure of the terminal of the speech generation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0067] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the speech generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0068] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0069] In this embodiment, a speech generation method is provided. Figure 2 is a flow chart of the speech generation method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0070] Step S210: Feature extraction is performed on the first speech data corresponding to the input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data;

[0071] Step S220, fusing the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0072] Step S230: Processing the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0073] Step S240: Determine the acoustic feature information corresponding to the input text, and generate second voice data corresponding to the input text based on the airway feature information and the acoustic feature information.

[0074] Specifically, live audio data from the broadcasting and television industry is collected as raw audio. The human voice is then separated from each raw audio source to obtain the human voice audio within the original audio. For example, the human voice separation tool UVR5 is used to separate the human voice from the accompaniment. The UVR5 tool uses a deep learning algorithm based on a spectral mask model to convert the audio signal into a time-frequency graph. Using a neural network to predict the mask, it separates the human voice and accompaniment from the mixed spectrum, ultimately preserving the human voice audio within the original audio.

[0075] Furthermore, speech recognition is performed on each vocal audio to obtain the text content and word-level timestamp sequence corresponding to each vocal audio, the text content and word-level timestamp sequence are matched, the vocal audio is segmented according to the matching results, and each audio segment obtained by segmentation is annotated to obtain the corresponding sample audio.

[0076] Feature extraction is performed on each audio sample to obtain a first and second breath-in feature for each audio sample. The first breath-in feature is a dynamic Mel-Frequency Cepstral Coefficient (MFCC) feature vector containing breath-in feature information in the audio sample. The second breath-in feature is a self-supervised speech representation learning feature vector containing breath-in feature information in the audio sample. The first and second breath-in features of the audio sample are fused to obtain a breath-in fusion feature for the audio sample. A breath-in prediction model is trained based on the breath-in fusion features of each audio sample to obtain a pre-trained breath-in prediction model.

[0077] During the actual speech generation process, first speech data corresponding to the input text is pre-generated. The first speech data is the initial speech generated based on the input text. Feature extraction is performed on the first speech data to obtain first and second breath features of the first speech data. The first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data, namely, a HuBERT feature vector.

[0078] The first breath feature dimension and the second breath feature dimension of the first speech data are unified, and the first breath feature and the second breath feature after the dimension unification are time-series aligned. Specifically, cubic spline interpolation is used to ensure that the 10ms frame rate of the second breath feature is strictly synchronized with the first breath feature. The first breath feature and the second breath feature that have undergone the above preprocessing are spliced ​​together to fuse the first breath feature and the second breath feature to obtain the breath fusion feature of the first speech data, forming a cross-modal representation. Then, the breath fusion feature of the first speech data is predicted and analyzed by the pre-trained breath prediction model to obtain the breath feature information of the first speech data, which includes the probability value of each audio frame in the first speech data being the breath position.

[0079] The input text is then converted into a corresponding phoneme sequence. Based on the phoneme sequence, acoustic feature information corresponding to the input text is predicted. This acoustic feature information includes phoneme duration information and prosody information such as stress. Based on the breath feature information and acoustic feature information, a corresponding acoustic feature vector is generated. This acoustic feature vector is converted into a mel-spectrogram. This mel-spectrogram is processed by a vocoder to obtain the second speech data corresponding to the input text. This second speech data is the final synthesized target speech and can be saved in formats such as MP3 and WAV.

[0080] Speech generation technology converts text into natural, fluent speech and is widely used in diverse scenarios, such as audiobooks and voice translation. However, existing speech generation methods typically focus on improving the authenticity of timbre to enhance the anthropomorphic nature of pronunciation, failing to accurately reproduce the characteristics of breath patterns, resulting in insufficient fluency and naturalness in the generated speech.

[0081] The speech generation method and device provided in this embodiment extract features of the first speech data corresponding to the input text to obtain the first breath feature and the second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing the breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing the breath feature information in the first speech data; the first breath feature and the second breath feature are fused to obtain the breath fusion feature of the first speech data; the breath fusion feature of the first speech data is processed by a pre-trained breath prediction model to obtain the breath feature information of the first speech data; the breath feature information includes the probability value of each audio frame in the first speech data being the breath position; the acoustic feature information corresponding to the input text is determined, and based on the breath feature information and the acoustic feature information, the second speech data corresponding to the input text is generated. Based on this, by generating initial speech data corresponding to the input text, extracting the dynamic MFCC feature vector and HuBERT feature vector containing the breath feature information of the initial speech data, and based on the fusion result of the two feature vectors, accurately predicting the breath position through the breath prediction model, it is possible to combine the breath prediction results to synthesize speech that accurately restores the breath features. This solves the problem of being unable to generate speech that accurately restores the breath features, resulting in insufficient fluency and naturalness of the generated speech. It achieves the ability to generate speech that accurately restores the breath features, significantly improving the anthropomorphism of the speech generation. Among them, the breath feature extraction and fusion technology can significantly increase the naturalness of pronunciation in speech generation, making the breath position and duration of the subsequently generated speech between short sentences, long sentences and paragraphs more in line with the pronunciation habits of real people.

[0082] In some embodiments, the training process of the gas outlet prediction model includes the following steps:

[0083] Separate the human voice from each original audio to obtain the human voice audio in the original audio;

[0084] Identify the text content and word-level timestamp sequence corresponding to each voice audio, and segment the voice audio based on the matching results of the text content and word-level timestamp sequence;

[0085] Annotate each audio segment obtained by segmentation to obtain the corresponding sample audio;

[0086] The breath prediction model is trained based on the breath fusion features of each sample audio.

[0087] Specifically, such as Figure 3As shown, each original audio is subjected to vocal separation to obtain the vocal audio in the original audio. For example, the vocal separation tool UVR5 is used to separate the vocals and accompaniment. The vocal separation tool UVR5 uses a deep learning algorithm based on a spectrum mask model to convert the audio signal into a time-frequency graph. It then uses a neural network to predict the mask, separating the vocals and accompaniment from the mixed spectrum, ultimately retaining the vocal audio in the original audio.

[0088] Each voice audio is subjected to noise reduction. For example, a Frequency Recurrence Convolutional Recurrent Network (FRCRN) model based on a Continuous Recurrent Network (CRN) is used to jointly model the harmonic structure and non-stationary noise characteristics of speech signals in both the frequency and time domains. This significantly improves speech quality and intelligibility in complex acoustic scenarios while preserving the clean human voice to the greatest extent possible. Speech recognition models such as Whisper are used to perform speech recognition on each denoised voice audio. The corresponding text content and word-level timestamp sequence are identified. For example, the word-level timestamp sequence includes the word "today," with the start and end times of the word "today" appearing in the audio at 0.5 and 0.8, respectively. The Whisper model extracts and integrates the text from the recognition results through the encoder and decoder of a Transformer model, merging them into JSON-formatted text content.

[0089] Furthermore, the text content is segmented and optimized to obtain multiple sentences. Each sentence is then matched with a word-level timestamp sequence. Based on the matching results, the human voice audio is segmented, with the matching results used to indicate the continuous word segments in the word-level timestamp sequence that match each sentence. Each segmented audio segment is then annotated to obtain sample audio. For example, preliminary detection methods such as silence detection, fundamental frequency detection, unvoiced sound detection, and breathing detection are used to annotate the status of each frame in the sample audio. The automatically annotated results are then manually reviewed to correct incorrect labels (such as misidentifying unvoiced sounds as breaths) to optimize labeling accuracy.

[0090] Afterwards, the breath prediction model is trained based on the breath fusion features of each sample audio, so that the trained breath prediction model can accurately predict the breath position. Among them, each sample audio is subjected to feature extraction to obtain the first sample breath feature and the second sample breath feature of each sample audio. The first sample breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the sample audio, and the second sample breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the sample audio. The first sample breath feature and the second sample breath feature of the sample audio are fused to obtain the breath fusion features of the sample audio, and the breath prediction model is trained based on the breath fusion features of each sample audio to obtain a pre-trained breath prediction model. It should be noted that the first sample breath feature extraction method, the second sample breath feature extraction method and the breath feature fusion method of the sample audio are the same as the corresponding processing method of the first speech data.

[0091] Through this embodiment, the human voice is separated from each original audio to obtain the human voice audio in the original audio, the text content and word-level timestamp sequence corresponding to each human voice audio are identified, and based on the matching results of the text content and the word-level timestamp sequence, the human voice audio is segmented, and each audio segment obtained by segmentation is labeled to obtain the corresponding sample audio, and the breath fusion feature of each sample audio is used to train the breath prediction model, thereby realizing the pre-training of the breath prediction model, so that the trained breath prediction model can accurately predict the breath position, which is helpful for the subsequent generation of speech that accurately restores the breath features.

[0092] In some embodiments, audio segmentation of human voice audio based on the matching results of text content and word-level timestamp sequence includes the following steps:

[0093] The text content is segmented and optimized through a transformer-based bidirectional encoder representation model to obtain multiple sentences of the text content;

[0094] Each sentence is matched with the word-level timestamp sequence, and the human voice audio is segmented based on the matching results; the matching results are used to indicate the continuous word segments in the word-level timestamp sequence that match each sentence.

[0095] Specifically, such as Figure 4 As shown in the figure, the Bidirectional Encoder Representations from Transformers (BERT) model optimizes sentence segmentation of text content, thereby segmenting the continuous text content obtained by speech recognition into multiple grammatically correct sentences. The output of the BERT model is several sentences with punctuation (such as periods, commas, and question marks).

[0096] It should be noted that in order to enhance the semantic coherence of the text, each sentence can be merged and split to obtain multiple optimized sentences. The embedding vector of each sentence is generated in advance using the Sentence Bidirectional Encoder Representation (SBERT) model. The cosine similarity between the embedding vectors is calculated to measure the relevance of different sentences. It is also determined whether the similarity of adjacent sentences is greater than a preset threshold (such as 0.85) and whether the length of the merged sentences is less than a preset maximum sentence length (such as 50 words). If the similarity of adjacent sentences is greater than the preset threshold and the length of the merged sentences is less than the preset maximum sentence length, the adjacent sentences are merged. In addition, if the length of the sentence exceeds the preset maximum sentence length, the sentence is split based on conjunctions.

[0097] Furthermore, each sentence is matched against the word-level timestamp sequence to determine the continuous word segments in the word-level timestamp sequence that match each sentence. Each sentence is aligned with the word-level timestamp, and the human voice audio is segmented based on the matching results. For example, the word-level timestamp sequence includes the word "today", and the sentence to be matched includes "today's weather". The minimum edit distance between "today" and "today's weather" is calculated as the insertion of two words.

[0098] In this embodiment, word-level timestamps are used to locate sentences, and words are used as the window values ​​of the sliding window. The minimum edit distance (MED) algorithm is used to quantify the similarity between the word segments in the sliding window and the text sentences to find the minimum edit distance alignment, and obtain continuous word segments that match each sentence in the word-level timestamp sequence, thereby determining the precise start and end time of the sentence in the audio, fully preserving the breath information in the sentence, and helping to accurately segment the audio.

[0099] Through this embodiment, the text content is optimized for sentence segmentation through a transformer-based bidirectional encoder representation model to obtain multiple sentences of the text content, each sentence is matched with a word-level timestamp sequence, and the human voice audio is segmented based on the matching results. The matching results are used to indicate the continuous word segments that match each sentence in the word-level timestamp sequence, thereby achieving accurate audio segmentation. Compared with setting parameters such as volume threshold and minimum segmentation time as the basis for segmenting audio, it can avoid ignoring the breath signals between sentences, and can retain complete breath information while accurately segmenting, which helps to improve the training effect of the breath prediction model and enable the trained model to more accurately predict the breath position. In addition, audio segmentation processing helps to improve the training efficiency of the model.

[0100] In some embodiments, training a breath prediction model based on breath fusion features of each audio sample includes the following steps:

[0101] Extract features from each audio sample to obtain the breath fusion features of each audio sample;

[0102] Perform random masking on the breath fusion features of each sample audio;

[0103] The fusion features of each air vent after random mask processing are input into the air vent prediction model, and the air vent prediction model is trained based on the focal loss function.

[0104] Specifically, such as Figure 5 As shown, each sample audio is passed through a pre-emphasis filter to compensate for high-frequency attenuation. By attenuating low frequencies and enhancing high frequencies, the high-frequency loss of the signal is compensated. The specific formula of the transfer function H(z) of the pre-emphasis filter is as follows:

[0105] (1)

[0106] In formula (1), α represents the pre-emphasis coefficient; z represents the time index of the discrete time signal, z -1 Represents the unit delay operator. The sample audio is framed based on the preset frame length and the preset frame shift to obtain multiple frames of audio signals. A Hamming window function is applied to each frame of audio signal to reduce the spectrum leakage effect. A fast Fourier transform (FFT) is performed on each frame of audio signal after windowing to obtain the corresponding amplitude spectrum. The amplitude spectrum is then filtered in the frequency domain using a Mel-scale triangular filter bank with a center frequency of Mel(f)=2595log 10 (1+f / 700). Determine the energy of each filter channel of the filtered Mel filter bank, and perform discrete cosine transform (DCT) on the energy of each filter channel after taking the logarithm to remove frequency domain correlation and compress information. The first 13 coefficients of the transformation result are intercepted to form a static Mel frequency cepstral coefficient feature vector, which can be expressed as the static Mel frequency cepstral coefficient feature vector c=[c0,c1,……,c 12 ].

[0107] In order to enhance the temporal dynamic information, the corresponding first-order difference coefficients are calculated according to the static Mel-frequency cepstral coefficient eigenvector of the sample audio and the second-order difference coefficient , the first-order difference coefficient is used to reflect the rate of change between adjacent frames, and the second-order difference coefficient is used to characterize the acceleration change of the MFCC feature. Based on the static Mel-frequency cepstral coefficient feature vector, first-order difference coefficient and second-order difference coefficient of the sample audio, the first sample air-mouth feature of the sample audio is determined. The first sample air-mouth feature is 39-dimensional, which refers to the dynamic Mel-frequency cepstral coefficient feature vector containing the air-mouth feature information in the sample audio, which can be expressed as This feature set extracts the local mean and standard deviation through sliding window statistics (the preset window length can be 100ms and the preset step size can be 10ms) to construct contextual statistical features, thereby fully preserving the spectral envelope characteristics and short-term energy fluctuation pattern of the gas outlet signal, providing an accurate basis for subsequent gas outlet feature prediction.

[0108] It should be noted that although the spectral shapes of breath marks and consonants in pronunciation are similar, they have significant differences in dynamic characteristics. The MFCC coefficients of breath marks change slowly, while the MFCC coefficients of pronunciation change rapidly over time, such as the transition from consonants to vowels. Based on this, in addition to extracting pronunciation features, this embodiment enhances the extraction of breath mark features by calculating temporal dynamic information to achieve accurate extraction of breath mark features based on MFCC.

[0109] Further, if Figure 6 As shown in the figure, each sample audio is resampled, the sampling rate is uniformly adjusted to a preset frequency (such as 16kHz), and the amplitude of the audio signal is normalized to the range of [-1,1]. The preprocessed sample audio is divided into multiple 60ms audio segments, each audio segment is input into the HuBERT-base model, and the initial acoustic features with a 50Hz frame rate (20ms per frame) are generated through the convolutional subsampling layer. The features of each frame can be expressed as H t .

[0110] To capture the low-frequency respiratory resonance and energy gradient characteristics unique to the breath point, the hierarchical selection strategy prioritizes aggregating the output results of the target output layer that is most sensitive to non-speech events, that is, aggregating the outputs of the 7th to 9th layers of the Transformer. Through cross-layer weighted averaging, the ability of the middle-layer features to represent the fuzzy area of ​​the breath point boundary is enhanced. Finally, the output results of each target output layer are weightedly fused to obtain the second sample breath point feature of the sample audio. The second sample breath point feature is a HuBERT vector containing the breath point feature information in the sample audio. The specific formula for the second sample breath point feature H is as follows:

[0111] (2)

[0112] In formula (2), H7, H8, and H9 represent the outputs of the 7th, 8th, and 9th layers of Transformer, respectively; 0.3, 0.5, and 0.2 are the weight coefficients corresponding to the outputs of the 7th, 8th, and 9th layers of Transformer, respectively.

[0113] Afterwards, if Figure 7As shown in the figure, the first sample breath feature dimension and the second sample breath feature dimension of the sample audio are unified. Among them, the first sample breath feature (39 dimensions) is dimensionally expanded to map the feature to a 40-dimensional space, while the original dimension feature of the second sample breath feature is 768 dimensions. Through the trainable linear projection layer (weight matrix ) compresses high-dimensional semantic features to 40 dimensions and applies layer normalization to eliminate distribution shift. A temporal alignment check is performed on the unified first and second sample air outlet features. Cubic spline interpolation is used to ensure that the 10ms frame rate of the second sample air outlet features is strictly synchronized with the first sample air outlet features.

[0114] The first sample air port feature and the second sample air port feature that have been preprocessed are time-series aligned and spectrum-sensitively projected, and then spliced ​​to fuse the first sample air port feature and the second sample air port feature to obtain the air port fusion feature F of the sample audio. M+H The breath fusion feature is an 80-dimensional joint feature vector, which forms a cross-modal representation. The breath prediction model is then trained based on the breath fusion features of each sample audio to obtain a pre-trained breath prediction model. It can be understood that the above-mentioned first sample breath feature and the second sample breath feature complement each other. The first sample breath feature provides millisecond-level breath state information, while the second sample breath feature is used to capture the breath position and length across the sentence level. After the two are fused, they can provide a more discriminative input signal for the downstream breath prediction model.

[0115] like Figure 8 As shown in the figure, the breath fusion features of each sample audio are randomly masked, and each breath fusion feature after random masking is used as the input of the Transformer encoder in the breath prediction model. The long-term acoustic dependency is captured through the bidirectional long short-term memory network, and a multi-head self-attention mechanism is introduced to focus on the breath boundary key frames. After the temporal features and attention weights output by the bidirectional long short-term memory network are fused through the residual connection, the fully connected layer outputs the probability value of each audio frame being the breath position.

[0116] Among them, in order to solve the serious imbalance problem of positive and negative samples in breath detection, the focus loss function is used to train the breath prediction model to increase the penalty for difficult samples, forcing the model to focus on the breath boundary frames that are difficult to classify (such as the weak energy rise at the beginning of breathing) and the easily confused unvoiced frames, thereby enhancing the model's ability to extract the key acoustic fingerprints of the breath in massive data. The focus loss function The specific formula is as follows:

[0117] (3)

[0118] In formula (3), Represents the output probability of the model; Represents the balance coefficient, which is used to balance the loss weight of the t-th class sample; Represents the focus parameter.

[0119] In addition, during the verification phase, the frame-level F1 score and boundary error were used to comprehensively evaluate the generalization performance of the air vent prediction model. Compared with single-modal feature extraction, the multimodal feature F1 score was higher than 0.9 and the boundary error was less than 30ms. The high precision and recall rate can convert the probability sequence into an accurate air vent interval, thereby ensuring the practicality and reliability of the model in complex acoustic environments.

[0120] Through this embodiment, features are extracted for each sample audio to obtain the breath fusion features of each sample audio, the breath fusion features of each sample audio are randomly masked, and each breath fusion feature after random masking is input into the breath prediction model, and the breath prediction model is trained based on the focal loss function, thereby realizing the training of the breath prediction model, so that the trained model can predict the breath position more accurately.

[0121] In some embodiments, step S210 of extracting features from the first voice data corresponding to the input text to obtain first and second breath features of the first voice data includes the following steps:

[0122] Step S211, determining a static Mel-frequency cepstral coefficient feature vector of the first speech data, and a first-order difference coefficient and a second-order difference coefficient corresponding to the static Mel-frequency cepstral coefficient feature vector;

[0123] Step S212, determining a first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficient, and the second-order difference coefficient;

[0124] Step S213: Process multiple audio segments of the first speech data using a self-supervised speech representation learning model, and determine a second breath feature of the first speech data based on the processing result.

[0125] Specifically, the first voice data is pre-processed through a pre-emphasis filter to compensate for high-frequency attenuation, and the first voice data after pre-emphasis filtering is framed to obtain multiple frames of audio signals. A Hamming window function is applied to each frame of the audio signal to reduce the spectrum leakage effect, and each frame of the windowed audio signal is subjected to a fast Fourier transform to obtain a corresponding amplitude spectrum. The amplitude spectrum is frequency-domain filtered through a Mel filter bank to determine the energy of each filter channel of the filtered Mel filter bank. The energy of the filter channel is discrete cosine transformed, and the static Mel-frequency cepstral coefficient eigenvector of the first voice data is determined based on the transformation result.

[0126] Furthermore, in order to enhance the temporal dynamic information, the corresponding first-order difference coefficients and second-order difference coefficients are calculated based on the static Mel-frequency cepstral coefficient feature vector of the first speech data. The first-order difference coefficients are used to reflect the rate of change between adjacent frames, and the second-order difference coefficients are used to characterize the acceleration change of the MFCC feature. Based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficients and the second-order difference coefficients of the first speech data, the first breath feature of the first speech data is determined. The first breath feature is 39-dimensional and refers to the dynamic Mel-frequency cepstral coefficient feature vector containing the breath feature information in the first speech data. This feature set extracts the local mean and standard deviation through sliding window statistics (the preset window length can be 100ms, and the preset step size can be 10ms) to construct contextual statistical features, thereby fully retaining the spectral envelope characteristics and short-term energy fluctuation pattern of the breath signal, providing an accurate basis for subsequent breath feature prediction.

[0127] It should be noted that although the spectral shapes of breath marks and consonants in pronunciation are similar, they have significant differences in dynamic characteristics. The MFCC coefficients of breath marks change slowly, while the MFCC coefficients of pronunciation change rapidly over time, such as the transition from consonants to vowels. Based on this, in addition to extracting pronunciation features, this embodiment enhances the extraction of breath mark features by calculating temporal dynamic information to achieve accurate extraction of breath mark features based on MFCC.

[0128] Afterwards, the resampled and standardized first speech data is divided into multiple audio clips, and each audio clip is processed by the self-supervised speech representation learning model to obtain the output result of each target output layer in the self-supervised speech representation learning model. The target output layer refers to the output layer with the highest sensitivity to non-speech events. Finally, the output results of each target output layer are weightedly fused to obtain the second breath feature of the first speech data. The second breath feature is a HuBERT feature vector containing the breath feature information in the first speech data.

[0129] Through this embodiment, the static Mel-frequency cepstral coefficient feature vector of the first speech data, as well as the first-order differential coefficients and second-order differential coefficients corresponding to the static Mel-frequency cepstral coefficient feature vector are determined; based on the static Mel-frequency cepstral coefficient feature vector, the first-order differential coefficients and the second-order differential coefficients, the first breath feature of the first speech data is determined; multiple audio clips of the first speech data are processed through a self-supervised speech representation learning model; the second breath feature of the first speech data is determined based on the processing results, thereby achieving accurate extraction of the first breath feature and the second breath feature.

[0130] In some embodiments, determining the static Mel-frequency cepstral coefficient feature vector of the first speech data in step S211 includes the following steps:

[0131] Performing frame processing on the first voice data to obtain multiple frames of audio signals;

[0132] Perform fast Fourier transform on each frame of audio signal after windowing to obtain the corresponding amplitude spectrum;

[0133] Perform frequency domain filtering on the magnitude spectrum through a Mel filter bank, and determine the energy of each filter channel of the filtered Mel filter bank;

[0134] Performing discrete cosine transform on the energy of the filter channel, and determining a static Mel-frequency cepstral coefficient feature vector of the first speech data based on the transform result.

[0135] Specifically, each first voice data is subjected to a pre-emphasis filter to compensate for high-frequency attenuation. By attenuating low frequencies and boosting high frequencies, the high-frequency loss of the signal is compensated. The specific formula of the transfer function of the pre-emphasis filter is referred to above equation (1). Based on a preset frame length and a preset frame shift, the first voice data is framed to obtain multiple frames of audio signals. A Hamming window function is applied to each frame of the audio signal to reduce spectral leakage effects. A fast Fourier transform is performed on each windowed frame of the audio signal to obtain a corresponding amplitude spectrum.

[0136] Furthermore, the amplitude spectrum is filtered in the frequency domain by a Mel-scale triangular filter bank with a center frequency of Mel(f)=2595log 10 (1 + f / 700) to determine the energy of each filter channel of the filtered Mel filter bank. The logarithmic energy of each filter channel is then discrete cosine transformed to remove frequency domain correlation and compress information. The first 13 coefficients of the transformed result are then truncated to form the static Mel-frequency cepstral coefficient feature vector.

[0137] Through this embodiment, the first speech data is framed to obtain multiple frames of audio signals, and each frame of the windowed audio signal is fast Fourier transformed to obtain a corresponding amplitude spectrum. The amplitude spectrum is frequency-domain filtered using a Mel filter bank to determine the energy of each filter channel of the filtered Mel filter bank, the energy of the filter channel is discrete cosine transformed, and the static Mel-frequency cepstral coefficient eigenvector of the first speech data is determined based on the transformation result, thereby accurately extracting the static MFCC eigenvector.

[0138] In some embodiments, the processing of the plurality of audio segments of the first speech data by the self-supervised speech representation learning model in step S213, and determining the second breath feature of the first speech data based on the processing results, includes the following steps:

[0139] dividing the preprocessed first voice data into a plurality of audio segments;

[0140] Each audio clip is processed by the self-supervised speech representation learning model to obtain the output of each target output layer in the self-supervised speech representation learning model; the target output layer refers to the output layer with the highest sensitivity to non-speech events;

[0141] The output results of each target output layer are weightedly fused to obtain the second breath feature of the first speech data.

[0142] Specifically, each first speech data is resampled, the sampling rate is uniformly adjusted to a preset frequency (such as 16kHz), and the amplitude of the audio signal is normalized to the range of [-1, 1]. The first speech data after the above preprocessing is divided into multiple 60ms audio segments, each audio segment is input into the HuBERT-base model, and the initial acoustic features with a 50Hz frame rate (20ms per frame) are generated through the convolutional subsampling layer. The features of each frame can be expressed as H t .

[0143] In order to capture the low-frequency respiratory resonance and energy gradient characteristics unique to the breath point, the hierarchical selection strategy prioritizes aggregating the output results of the target output layer that is most sensitive to non-speech events, that is, aggregating the outputs of the 7th to 9th layers of Transformer. Through cross-layer weighted averaging, the ability of the middle-layer features to represent the fuzzy area of ​​the breath point boundary is enhanced. Finally, the output results of each target output layer are weightedly fused to obtain the second breath point feature of the first speech data. The second breath point feature is a HuBERT feature vector containing the breath point feature information in the first speech data.

[0144] Through this embodiment, the preprocessed first speech data is divided into multiple audio segments, and each audio segment is processed by the self-supervised speech representation learning model to obtain the output results of each target output layer in the self-supervised speech representation learning model. The target output layer refers to the output layer with the highest sensitivity to non-speech events. The output results of each target output layer are then weightedly fused to obtain the second breath feature of the first speech data, thereby realizing the accurate extraction of the second breath feature.

[0145] In some embodiments, the processing of the gas outlet prediction model includes the following steps:

[0146] Processing the breath fusion features of the first speech data through a bidirectional long short-term memory network to obtain corresponding time series features;

[0147] Input the temporal features into the multi-head attention module to obtain the corresponding attention weights;

[0148] The fusion result of the time series feature and the attention weight is input into the fully connected layer, and the breath feature information of the first speech data is output.

[0149] Specifically, the breath fusion features of each first speech data are randomly masked, and each breath fusion feature after random masking is used as the input of the Transformer encoder in the breath prediction model, and the long-term acoustic dependency is captured through the bidirectional long short-term memory network.

[0150] Furthermore, a multi-head self-attention mechanism is introduced to focus on the key frames of the breath boundary to determine the corresponding attention weights. After the temporal features and attention weights output by the bidirectional long-short term memory (LSTM) network are fused through residual connection, the fully connected layer outputs the breath feature information of the first speech data. The breath feature information of the first speech data includes the probability value of each audio frame being the breath position.

[0151] Through this embodiment, the breath fusion features of the first speech data are processed through a bidirectional long short-term memory network to obtain corresponding time series features, the time series features are input into the multi-head attention module to obtain corresponding attention weights, and the fusion results of the time series features and the attention weights are input into the fully connected layer to output the breath feature information of the first speech data, thereby achieving accurate prediction of the breath features of the first speech data.

[0152] In some embodiments, determining acoustic feature information corresponding to the input text in step S240 and generating second speech data corresponding to the input text based on the airway feature information and the acoustic feature information includes the following steps:

[0153] Step S241, determining acoustic feature information based on the phoneme sequence corresponding to the input text;

[0154] Step S242: generating a corresponding mel-spectrogram based on the air port feature information and the acoustic feature information;

[0155] Step S243: Process the mel-spectrogram through a vocoder to obtain second speech data corresponding to the input text.

[0156] Specifically, the input text is converted into a corresponding phoneme sequence, and acoustic feature information corresponding to the input text is predicted based on the phoneme sequence. The acoustic feature information includes phoneme duration information and prosody information such as stress.

[0157] Furthermore, based on the air port feature information and acoustic feature information, a corresponding acoustic feature vector is generated, and the acoustic feature vector is converted into a Mel-spectrogram. The Mel-spectrogram is processed by a vocoder to obtain second speech data corresponding to the input text. The second speech data is the final synthesized target speech and can be saved in MP3, WAV and other formats.

[0158] It should be noted that the speech generation method of this embodiment can be further combined with attributes such as timbre and emotion to be widely applied to other scalable scenarios.

[0159] Through this embodiment, the acoustic feature information is determined based on the phoneme sequence corresponding to the input text, and the corresponding mel-spectrogram is generated based on the breath feature information and the acoustic feature information. The mel-spectrogram is processed by the vocoder to obtain the second speech data corresponding to the input text, thereby generating a speech that accurately restores the breath features and significantly improves the anthropomorphism of the speech generation.

[0160] The present embodiment is described and illustrated below through preferred embodiments.

[0161] Figure 9 Flowchart of the speech generation method of the preferred embodiment is as follows: Figure 9 As shown, the speech generation method includes the following steps:

[0162] Step S910, determining a static Mel-frequency cepstral coefficient feature vector of the first speech data, and a first-order difference coefficient and a second-order difference coefficient corresponding to the static Mel-frequency cepstral coefficient feature vector;

[0163] Step S920, determining a first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficient, and the second-order difference coefficient;

[0164] Step S930, processing the plurality of audio segments of the first speech data using a self-supervised speech representation learning model, and determining a second breath feature of the first speech data based on the processing result;

[0165] Step S940: Fusing the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0166] Step S950: Processing the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0167] Step S960: determining acoustic feature information based on the phoneme sequence corresponding to the input text; generating a corresponding mel-spectrogram based on the breath feature information and the acoustic feature information;

[0168] Step S970: Process the mel-spectrogram through a vocoder to obtain second speech data corresponding to the input text.

[0169] Through this embodiment, a static Mel-frequency cepstral coefficient feature vector of the first speech data, as well as first-order difference coefficients and second-order difference coefficients corresponding to the static Mel-frequency cepstral coefficient feature vector, are determined. Based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficients, and the second-order difference coefficients, a first breath feature of the first speech data is determined. Multiple audio clips of the first speech data are processed using a self-supervised speech representation learning model. Based on the processing results, a second breath feature of the first speech data is determined. The first breath feature and the second breath feature are fused to obtain a breath fusion feature of the first speech data.

[0170] Furthermore, the breath fusion features of the first speech data are processed using a pre-trained breath prediction model to obtain breath feature information of the first speech data, which includes the probability value of each audio frame in the first speech data being a breath position. Based on the phoneme sequence corresponding to the input text, the acoustic feature information is determined, and based on the breath feature information and the acoustic feature information, a corresponding mel-spectrogram is generated. The mel-spectrogram is then processed by a vocoder to obtain second speech data corresponding to the input text. This solves the problem of being unable to generate speech that accurately restores breath features, resulting in insufficient fluency and naturalness in the generated speech, and achieves the ability to generate speech that accurately restores breath features, significantly improving the anthropomorphism of the generated speech.

[0171] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0172] In this embodiment, a speech generating device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described are not repeated here. The terms "module," "unit," "subunit," etc. used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0173] Figure 10 This is a structural block diagram of the speech generating device of this embodiment. Figure 10 As shown, the device includes:

[0174] Extraction module 10, configured to perform feature extraction on first speech data corresponding to the input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data;

[0175] A fusion module 20 is configured to fuse the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0176] The prediction module 30 is configured to process the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0177] The generation module 40 is used to determine the acoustic feature information corresponding to the input text, and generate second speech data corresponding to the input text based on the air hole feature information and the acoustic feature information.

[0178] Through the device provided in this embodiment, the first voice data corresponding to the input text is feature extracted to obtain the first breath feature and the second breath feature of the first voice data; wherein, the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing the breath feature information in the first voice data, and the second breath feature is a self-supervised speech representation learning feature vector containing the breath feature information in the first voice data; the first breath feature and the second breath feature are fused to obtain the breath fusion feature of the first voice data; the breath fusion feature of the first voice data is processed by a pre-trained breath prediction model to obtain the breath feature information of the first voice data; the breath feature information includes the probability value of each audio frame in the first voice data being the breath position; the acoustic feature information corresponding to the input text is determined, and based on the breath feature information and the acoustic feature information, the second voice data corresponding to the input text is generated, which solves the problem that the voice that accurately restores the breath feature cannot be generated, resulting in insufficient fluency and naturalness of the generated voice, and realizes the ability to generate voice that accurately restores the breath feature, significantly improving the anthropomorphism of the voice generation.

[0179] In some of these embodiments, Figure 10 On the basis of , the device also includes a training module for separating the human voice from each original audio to obtain the human voice audio in the original audio; identifying the text content and word-level timestamp sequence corresponding to each human voice audio, and performing audio segmentation on the human voice audio based on the matching results of the text content and the word-level timestamp sequence; annotating each audio segment obtained by segmentation to obtain the corresponding sample audio; and training the breath prediction model based on the breath fusion features of each sample audio.

[0180] In some of the embodiments, the training module is further used to optimize the sentence segmentation of the text content through a transformer-based bidirectional encoder representation model to obtain multiple sentences of the text content; match each sentence with a word-level timestamp sequence, and perform audio segmentation on the human voice audio based on the matching results; the matching results are used to indicate the continuous word segments in the word-level timestamp sequence that match each sentence.

[0181] In some of the embodiments, the training module is also used to extract features from each sample audio to obtain the breath fusion features of each sample audio; perform random masking on the breath fusion features of each sample audio; input each breath fusion feature after random masking into the breath prediction model, and train the breath prediction model based on the focal loss function.

[0182] In some embodiments, the extraction module 10 is further used to determine a static Mel-frequency cepstral coefficient feature vector of the first speech data, and first-order differential coefficients and second-order differential coefficients corresponding to the static Mel-frequency cepstral coefficient feature vector; determine a first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order differential coefficients and the second-order differential coefficients; process multiple audio clips of the first speech data through a self-supervised speech representation learning model, and determine a second breath feature of the first speech data based on the processing results.

[0183] In some embodiments, the extraction module 10 is further used to perform frame processing on the first speech data to obtain multiple frames of audio signals; perform fast Fourier transform on each frame of the windowed audio signal to obtain a corresponding amplitude spectrum; perform frequency domain filtering on the amplitude spectrum through a Mel filter group to determine the energy of each filter channel of the Mel filter group after filtering; perform discrete cosine transform on the energy of the filter channel, and determine the static Mel-frequency cepstral coefficient eigenvector of the first speech data based on the transformation result.

[0184] In some embodiments, the extraction module 10 is further used to divide the preprocessed first speech data into multiple audio segments; process each audio segment through a self-supervised speech representation learning model to obtain the output result of each target output layer in the self-supervised speech representation learning model; the target output layer refers to the output layer with the highest sensitivity to non-speech events; and perform weighted fusion on the output results of each target output layer to obtain the second breath feature of the first speech data.

[0185] In some of the embodiments, the prediction module 30 is also used to process the breath fusion features of the first speech data through a bidirectional long short-term memory network to obtain corresponding time series features; input the time series features into the multi-head attention module to obtain corresponding attention weights; input the fusion result of the time series features and the attention weights into the fully connected layer to output the breath feature information of the first speech data.

[0186] In some embodiments, the generation module 40 is further used to determine acoustic feature information based on a phoneme sequence corresponding to the input text; generate a corresponding Mel-spectrogram based on the breath feature information and the acoustic feature information; and process the Mel-spectrogram through a vocoder to obtain second speech data corresponding to the input text.

[0187] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0188] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0189] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0190] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0191] S1, performing feature extraction on first speech data corresponding to the input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data;

[0192] S2, fusing the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data;

[0193] S3, processing the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position;

[0194] S4, determining acoustic feature information corresponding to the input text, and generating second voice data corresponding to the input text based on the air vent feature information and the acoustic feature information.

[0195] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0196] In addition, in combination with the speech generation method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the speech generation method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the speech generation methods in the above embodiments is implemented.

[0197] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0198] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0199] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0200] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A speech generation method, characterized in that: include: Performing feature extraction on first speech data corresponding to the input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data; The step of extracting features from the first speech data corresponding to the input text to obtain the first breath feature and the second breath feature of the first speech data includes: determining a static Mel-frequency cepstral coefficient feature vector of the first speech data, and first-order differential coefficients and second-order differential coefficients corresponding to the static Mel-frequency cepstral coefficient feature vector; determining the first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order differential coefficients, and the second-order differential coefficients; and processing multiple audio clips of the first speech data through a self-supervised speech representation learning model, and determining the second breath feature of the first speech data based on the processing results. The method of processing the plurality of audio segments of the first speech data by the self-supervised speech representation learning model and determining the second breath feature of the first speech data based on the processing results includes: dividing the preprocessed first speech data into the plurality of audio segments; processing each of the audio segments by the self-supervised speech representation learning model to obtain an output result of each target output layer in the self-supervised speech representation learning model; the target output layer refers to the output layer with the highest sensitivity to non-speech events; and performing weighted fusion on the output results of each target output layer to obtain the second breath feature of the first speech data. fusing the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data; Processing the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position; Acoustic feature information corresponding to the input text is determined, and second voice data corresponding to the input text is generated based on the air port feature information and the acoustic feature information.

2. The speech generation method according to claim 1, wherein: The training process of the gas outlet prediction model includes: Separating the human voice from each original audio to obtain the human voice audio in the original audio; Identifying the text content and word-level timestamp sequence corresponding to each of the human voice audios, and performing audio segmentation on the human voice audios based on a matching result between the text content and the word-level timestamp sequence; Annotate each audio segment obtained by segmentation to obtain the corresponding sample audio; The breath prediction model is trained based on the breath fusion features of each of the sample audios.

3. The speech generation method according to claim 2, wherein: The step of segmenting the human voice audio based on the matching result between the text content and the word-level timestamp sequence includes: Optimizing the text content by using a transformer-based bidirectional encoder representation model to obtain multiple sentences of the text content; Each of the sentences is matched with the word-level timestamp sequence, and the human voice audio is segmented based on the matching results; the matching results are used to indicate the continuous word segments in the word-level timestamp sequence that match each of the sentences.

4. The speech generation method according to claim 2, wherein: The training of the breath prediction model based on the breath fusion feature of each of the sample audios includes: Performing feature extraction on each of the sample audios to obtain a breath fusion feature of each of the sample audios; Performing random masking on the breath fusion features of each sample audio; Each of the air vent fusion features after random mask processing is input into the air vent prediction model, and the air vent prediction model is trained based on a focal loss function.

5. The speech generation method according to claim 1, wherein: The determining of the static Mel-frequency cepstral coefficient feature vector of the first speech data includes: Performing frame processing on the first voice data to obtain multiple frames of audio signals; Performing a fast Fourier transform on each frame of the windowed audio signal to obtain a corresponding amplitude spectrum; Performing frequency domain filtering on the amplitude spectrum through a Mel filter bank, and determining the energy of each filter channel of the Mel filter bank after filtering; Performing discrete cosine transform on the energy of the filter channel, and determining a plurality of static Mel-frequency cepstral coefficient feature vectors of the first speech data based on the transform result.

6. The speech generation method according to claim 1, wherein: The processing process of the gas outlet prediction model includes: Processing the breath fusion features of the first speech data through a bidirectional long short-term memory network to obtain corresponding time series features; Input the temporal features into the multi-head attention module to obtain the corresponding attention weights; The fusion result of the time series feature and the attention weight is input into the fully connected layer, and the breath feature information of the first speech data is output.

7. The speech generation method according to claim 1, wherein: The determining of acoustic feature information corresponding to the input text, and generating second speech data corresponding to the input text based on the airway feature information and the acoustic feature information, includes: Determining the acoustic feature information based on a phoneme sequence corresponding to the input text; generating a corresponding mel-spectrogram based on the air port feature information and the acoustic feature information; The mel-spectrogram is processed by a vocoder to obtain the second speech data corresponding to the input text.

8. A speech generating device, characterized in that: include: an extraction module, configured to perform feature extraction on first speech data corresponding to an input text to obtain a first breath feature and a second breath feature of the first speech data; wherein the first breath feature is a dynamic Mel-frequency cepstral coefficient feature vector containing breath feature information in the first speech data, and the second breath feature is a self-supervised speech representation learning feature vector containing breath feature information in the first speech data; The extraction module is further configured to determine a static Mel-frequency cepstral coefficient feature vector of the first speech data, and first-order difference coefficients and second-order difference coefficients corresponding to the static Mel-frequency cepstral coefficient feature vector; determine the first breath feature of the first speech data based on the static Mel-frequency cepstral coefficient feature vector, the first-order difference coefficients, and the second-order difference coefficients; and process multiple audio clips of the first speech data using a self-supervised speech representation learning model, and determine the second breath feature of the first speech data based on the processing results. The extraction module is further configured to divide the preprocessed first speech data into a plurality of audio segments; process each of the audio segments using a self-supervised speech representation learning model to obtain an output result of each target output layer in the self-supervised speech representation learning model; the target output layer is the output layer with the highest sensitivity to non-speech events; and perform weighted fusion on the output results of each target output layer to obtain the second breath feature of the first speech data; a fusion module, configured to fuse the first breath feature and the second breath feature to obtain a breath fusion feature of the first speech data; a prediction module, configured to process the breath fusion features of the first speech data using a pre-trained breath prediction model to obtain breath feature information of the first speech data; the breath feature information includes a probability value of each audio frame in the first speech data being a breath position; A generation module is used to determine the acoustic feature information corresponding to the input text, and generate second voice data corresponding to the input text based on the air outlet feature information and the acoustic feature information.

Images