Data augmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the present application discloses a data enhancement method, device, equipment, system and computer storage medium, the method comprises: obtaining a trained speech conversion model; the speech conversion model is used for converting speech according to the label information of the speaker and the prosody information of the speech data, obtaining the speech conversion result; the speech data of the target field and the label information of the speaker are input into the trained speech conversion model, and the trained speech conversion model is used for processing the speech data of the target field and the label information of the speaker, obtaining the speech conversion result corresponding to the speech data of the target field.

Description

Technical Field

[0001] This application relates to speech processing technology, and more particularly to a data enhancement method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] Speech recognition technology is the technology that automatically converts speech into text. With the development of deep learning technology, deep learning frameworks for sequence-to-sequence modeling have gradually matured, and end-to-end speech recognition models have become a popular technology. These include mainstream end-to-end speech recognition models such as Connectionist Temporal Classification (CTC) models, Recurrent Neural Network-Transducer (RNN-T) models, and Attention Encoder Decoder models.

[0003] End-to-end speech recognition models accept audio feature input and directly output text, solving the problem of the mismatch between the lengths of the input and output sequences. While end-to-end speech recognition models offer improved performance and a relatively simple structure, they are entirely data-driven, thus increasing the amount of data required for training. Domain-specific end-to-end speech recognition models require training with domain-specific speech data. Therefore, how to expand domain-specific speech data to facilitate the training of domain-specific end-to-end speech recognition models is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a data augmentation method that can expand speech data in the target domain, which is beneficial for training a speech recognition model suitable for the target domain.

[0005] This application provides a data augmentation method, the method comprising:

[0006] Obtain the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data, and obtain the speech conversion result;

[0007] The speech data of the target domain and the speaker's label information are input into the trained speech conversion model. The trained speech conversion model is then used to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain.

[0008] In some embodiments, the step of processing the speech data in the target domain and the speaker's label information using the trained speech conversion model to obtain the speech conversion result corresponding to the speech data in the target domain includes:

[0009] Extract explicit prosodic features from the speech data in the target domain;

[0010] Based on the Mel spectrum and bottleneck features of the speech data in the target domain, the implicit prosodic representation of the speech data in the target domain is determined;

[0011] The speaker's label information, explicit prosodic features, and implicit prosodic representation are processed to obtain the speech conversion result corresponding to the speech data of the target domain.

[0012] In some embodiments, determining the implicit prosodic representation of the speech data in the target domain based on the Mel spectrum and bottleneck features of the speech data in the target domain includes:

[0013] A first prosodic representation of the speech data of the target domain is extracted from the Mel spectrum using a variational autoencoder (VAE).

[0014] A second prosodic representation of the speech data in the target domain is extracted from the bottleneck features using a reference encoder.

[0015] The implicit prosodic representation of the speech data in the target domain is obtained by superimposing the first prosodic representation and the second prosodic representation.

[0016] In some embodiments, extracting explicit prosodic features from the speech data of the target domain includes:

[0017] At least one of the endpoint detection results (voice / unvoice flag feature, VUV), number domain fundamental frequency, and short-term average amplitude of the speech data in the target domain is used as the explicit prosodic feature of the speech data in the target domain.

[0018] In some embodiments, processing the speaker's label information, the explicit prosodic features, and the implicit prosodic representation to obtain the speech conversion result corresponding to the speech data in the target domain includes:

[0019] Sentence representations of speech data in the target domain are determined from the bottleneck features of the weighted aggregation self-attention (SA-WA) encoder.

[0020] The speaker's label information, explicit prosodic features, implicit prosodic representation, and sentence representation are processed to obtain the speech conversion result corresponding to the speech data in the target domain.

[0021] In some embodiments, the training process of the speech conversion model includes:

[0022] Obtain a training dataset, which includes speech data from a preset domain and speaker labeling information; the preset domain and the target domain are different domains.

[0023] Based on the training dataset, the speech conversion model is trained using an adversarial training method to obtain the trained speech conversion model.

[0024] In some embodiments, training the speech conversion model using an adversarial training method based on the training dataset to obtain the trained speech conversion model includes:

[0025] The speech data of the preset domain and the speaker's label information are input into the speech conversion model. The speech conversion model is then used to process the speech data of the preset domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the preset domain.

[0026] The first prosodic representation of the speech data in the preset domain is classified using a speaker classifier to obtain a classification result; the first prosodic representation of the speech data in the preset domain is information extracted from the Mel spectrum of the speech data in the speech conversion model by the VAE.

[0027] The speaker classifier is used as a discriminator, and the speech conversion model is used as a generator; the loss of the discriminator and the loss of the generator are determined based on the classification results.

[0028] Based on the loss of the discriminator and the loss of the generator, and using an adversarial training method, the speech conversion model is trained to obtain the trained speech conversion model.

[0029] In some embodiments, determining the loss of the generator based on the classification result includes:

[0030] The generator loss is determined based on the KL divergence of the VAE, the Mel spectrum reconstruction loss of the speech conversion model, and the adversarial loss of the speech conversion model.

[0031] This application embodiment also provides a data enhancement device, the device comprising: an acquisition module and a processing module, wherein...

[0032] The acquisition module is used to acquire the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data to obtain the speech conversion result.

[0033] The processing module is used to input the speech data of the target domain and the speaker's label information into the trained speech conversion model, and use the trained speech conversion model to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain.

[0034] This application also provides an electronic device, including a processor and a memory for storing a computer program capable of running on the processor; wherein,

[0035] The processor is used to run the computer program to perform any of the above data augmentation methods.

[0036] This application also provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described data augmentation methods.

[0037] As can be seen, the embodiments of this application can input the speech data of the target domain and the speaker's label information into the trained speech conversion model based on the training of the speech conversion model, thereby obtaining the speech conversion result corresponding to the speech data of the target domain. Since the speech conversion result corresponding to the speech data of the target domain can be used as new speech data of the target domain, the training data of the end-to-end speech recognition model of the target domain is expanded. Attached Figure Description

[0038] Figure 1 A flowchart illustrating the data augmentation method of this application embodiment. Figure 1 ;

[0039] Figure 2 A flowchart illustrating the data augmentation method of this application embodiment. Figure 2 ;

[0040] Figure 3 This is a schematic diagram of the architecture of the speech conversion model and speaker classifier in the embodiments of this application;

[0041] Figure 4 This is a flowchart illustrating the implicit prosodic representation of speech data extracted in this embodiment of the application;

[0042] Figure 5 This is a flowchart illustrating the data augmentation method in an embodiment of this application. Figure 3 ;

[0043] Figure 6 This is a schematic diagram of the SA-WA encoder architecture in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of the training process of the speech conversion model in the embodiments of this application. Figure 1 ;

[0045] Figure 8 This is a schematic diagram of the training process of the speech conversion model in the embodiments of this application. Figure 2 ;

[0046] Figure 9 This is a schematic diagram of the data enhancement device in the embodiments of this application;

[0047] Figure 10 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0048] In related technologies, mature speech recognition models can include Hidden Markov Models (HMMs). It mainly consists of three parts: an acoustic module, a pronunciation dictionary, and a language model. The acoustic module is responsible for recognizing speech as HMM states; the pronunciation dictionary maps phonemes to characters or words; and the language model statistically analyzes language information to make the entire sentence sound closer to normal human language.

[0049] In related technologies, end-to-end speech recognition models based on deep learning frameworks can also be used to achieve speech recognition. The biggest feature of end-to-end speech recognition models is that they integrate the three parts of an Hidden Markov Model (HMM) into a single model. End-to-end speech recognition models are entirely data-driven, thus increasing the amount of data required for training. The labeled data used in training end-to-end speech recognition models includes pairs of speech and text. However, these pairs of speech and text are often difficult to obtain, requiring significant human, material, and financial resources. If general-domain speech data is used to train the speech recognition model, the resulting model cannot accurately and reliably achieve speech recognition in a specific domain. Therefore, how to expand domain-specific speech data to facilitate the training of domain-specific end-to-end speech recognition models is a pressing technical problem that needs to be solved.

[0050] In related technologies, data augmentation techniques for speech data can include the following: 1) Pitch and speed shifting, i.e., changing the pitch and speech rate of audio to augment new data; 2) Speech distortion, i.e., artificially designing losses to damage the spectrum to augment new data; 3) Spectral augmentation, i.e., randomly removing some rows and columns from the spectrum during training to effectively prevent overfitting; 4) Adding noise to the data, i.e., adding noise to the training data or convolutional room impulse response to add noise to the data, turning clean audio data into data augmentation model with noise interference to improve robustness. However, the above data augmentation techniques that only target speech data cannot substantially generate new data; they only perform certain transformations on the existing data.

[0051] In related technologies, speech conversion techniques can be used to augment domain-specific speech data to facilitate the training of domain-specific end-to-end speech recognition models. The goal of speech conversion is to modify the speech of a source speaker into that of a target speaker without altering the language content. Speech conversion methods in related technologies include: methods based on Gaussian Mixed Models (GMMs), speech conversion methods based on frequency distortion, and demonstration methods. These methods typically require paired speech data from both the source and target speakers, which is costly to collect. Unpaired data can also be used for speech conversion. Leveraging the powerful feature extraction and nonlinear regression capabilities of deep learning, methods based on deep learning frameworks for unpaired data speech conversion can be proposed. For example, a method based on Phonetic Posterior Grams (PPG) can be implemented through speech recognition and speech synthesis stages. In PPG-based methods, PPG or neural network bottleneck features (BN) serve as "information bridges." In the speech recognition stage, an ASR acoustic model is used to extract speaker-independent PPG or BN features, representing the language content. The synthesis stage is a conversion model that maps the features extracted by the Automatic Speech Recognition (ASR) acoustic model to the target speaker's acoustic features. While the speech recognition and speech synthesis stages can be integrated into a unified encoder-decoder neural architecture and trained end-to-end, the speech recognition-synthesis framework remains a more general framework for speech conversion using unpaired data. This is because it allows for flexible training of the two stages separately, improving the robustness of speech recognition and synthesis, and training the ASR acoustic model can utilize large amounts of training data. How to use it?

[0052] In related technologies, speech conversion methods focus on converting timbre to the target timbre and preserving the language content of the source speech. However, the prosody of the source speech also needs to be correctly converted to the target prosody. Prosodic information includes at least emotion, pitch, duration, and loudness, which are crucial for scenarios requiring expressive speech, such as dubbing, live streaming, and data augmentation. Therefore, how to convert the prosodic information of the source language in speech conversion methods is a technical problem to be solved.

[0053] In view of the above-mentioned technical problems, the technical solutions of the embodiments of this application are proposed.

[0054] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the embodiments of the present application can be implemented in any combination.

[0055] This application provides a data augmentation method that can be applied to electronic devices. The electronic devices may include terminals and / or servers. Here, the terminal may be a thin client, a thick client, a handheld or laptop device, a microprocessor-based system, a programmable consumer electronics product, a network PC, a minicomputer system, etc. The server may be a minicomputer system, a mainframe computer system, or a distributed cloud computing environment including any of the above systems, etc.

[0056] Servers and other electronic devices may include program modules that execute computer instructions. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0057] Figure 1 A flowchart illustrating the data augmentation method of this application embodiment. Figure 1 ,like Figure 1 As shown, the process may include:

[0058] Step 101: Obtain the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data, and obtain the speech conversion result.

[0059] In this embodiment of the application, the speaker's label information is used to identify the speaker. For example, the speaker's label information may be information input by the user.

[0060] In this embodiment of the application, the prosodic information of the speech data may include explicit prosodic features and implicit prosodic representations. Explicit prosodic features are explicit features extracted from the speech data. For example, explicit prosodic features may include duration, pitch, energy, etc. In order to finely describe and control the prosodic, explicit prosodic features can be extracted from the speech data of the source speaker.

[0061] For example, implicit prosodic representations can be extracted from speech data using global style tokens (GST), reference encoders, and VAEs.

[0062] For example, in a speech conversion model, after obtaining the speaker's label information and the prosodic information of the speech data, the speaker's label information and the prosodic information of the speech data can be concatenated to obtain concatenated information; the concatenated information is then processed to obtain the speech conversion result.

[0063] Step 102: Input the speech data of the target domain and the speaker's label information into the trained speech conversion model, and use the trained speech conversion model to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain.

[0064] In this embodiment, the target domain can be any domain specified by the user. When training the speech conversion model, speech data from domains other than the target domain can be used for training. Thus, after obtaining the trained speech conversion model, new target domain speech data and speaker label information can be input into the trained model to accurately obtain the speech conversion result corresponding to the target domain speech data. It can be seen that because the speech conversion model can extract prosodic information from the speech data, the speech conversion result corresponding to the target domain speech data is a speech conversion result that includes the prosodic information of the target domain speech data.

[0065] In practical applications, steps 101 to 102 can be implemented based on a processor in an electronic device. This processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, or microprocessor. It is understood that other electronic devices can also implement the above processor functions, and this application embodiment does not impose any limitations.

[0066] As can be seen, the embodiments of this application can input the speech data of the target domain and the speaker's label information into the trained speech conversion model based on the training of the speech conversion model, thereby obtaining the speech conversion result corresponding to the speech data of the target domain. Since the speech conversion result corresponding to the speech data of the target domain can be used as new speech data of the target domain, the training data of the end-to-end speech recognition model of the target domain is expanded.

[0067] In some embodiments, refer to Figure 2 The process of using a trained speech conversion model to process speech data in the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data in the target domain can include:

[0068] Step 201: Extract explicit prosodic features from the speech data of the target domain.

[0069] For example, at least one of the vuv, logarithmic domain fundamental frequency (lf0, log f0), and short-term average amplitude of the speech data in the target domain can be used as an explicit prosodic feature of the speech data in the target domain. Here, lf0 can be used to simulate pitch, and the short-term average amplitude can be used to simulate energy.

[0070] In practical applications, the fundamental frequency, lf0, or short-term average amplitude of each frame of speech data in the target domain can be extracted separately; the frame-level speech conversion process can keep the duration information consistent before and after the speech conversion.

[0071] For example, lf0 can be extracted using the feature extractor in the speech synthesis system world; for example, lf0 and the short-term average amplitude can be normalized so that the normalized data is in the range [0,1], which helps to reduce the performance degradation caused by unseen style in the application stage of the speech conversion model.

[0072] Step 202: Determine the implicit prosodic representation of the speech data in the target domain based on the Mel Spectrum and Bottleneck Feature.

[0073] In this embodiment, the Mel spectrum is a nonlinear transformation of the spectrum, used to make human perception of frequency more linear; the bottleneck feature is a new feature generated by a bottleneck-shaped neural network containing a small number of explicit layers and a large number of hidden layer nodes.

[0074] In practical applications, refer to Figure 3 Mel spectrum and bottleneck features can be extracted for each frame of speech data in the target domain. For example, to extract the bottleneck features of each frame of data, the ASR acoustic model can be used to extract the features of each frame of data. Then, the bottleneck-shaped neural network described above can be used to process the features extracted by the ASR acoustic model to obtain the bottleneck features.

[0075] It should be noted that the embodiments of this application do not limit the execution order of steps 201 and 202. Step 201 can be executed before step 202 or after step 202, and steps 201 and 202 can also be executed simultaneously.

[0076] Step 203: Process the speaker's label information, explicit prosodic features, and implicit prosodic representation to obtain the speech conversion results corresponding to the speech data in the target domain.

[0077] In some embodiments, the speaker's label information, explicit prosodic features, and implicit prosodic representation can be concatenated to obtain a concatenated result; then, the concatenated result can be decoded or otherwise processed to obtain the parenting conversion result corresponding to the speech data in the target domain.

[0078] Understandably, explicit prosodic features are intuitive, and their impact on prosodicity is easily understood. Prosodicity can be independently controlled and interpreted through different explicit features. However, prosodicity involves many factors, and manually crafted explicit acoustic features are insufficient to perfectly describe prosodic information. For implicit prosodic representations, those extracted through prosodic embedding layers struggle to interpret and control different aspects of prosodicity. Furthermore, the extracted implicit prosodic representations may contain other information unrelated to prosodicity, such as timbre—a phenomenon known as timbre leakage. To address these issues, in this embodiment, explicit and implicit prosodic features of speech data from the target domain can be combined to extract more comprehensive prosodic information.

[0079] In some embodiments, refer to Figure 4 The implementation of step 202 above can include:

[0080] Step 2021: Use VAE to extract the first prosodic representation of the speech data in the target domain from the Mel spectrum.

[0081] Here, VAE, as a form of deep generative model, is a generative network structure based on Variational Bayes (VB) inference.

[0082] Step 2022: Use a reference encoder to extract the second prosodic representation of the speech data in the target domain from the bottleneck features.

[0083] In the embodiments of this application, reference is made to Figure 3 Using a reference encoder, a second prosodic representation that is independent of the speaker can be directly extracted from the bottleneck features; based on the extraction of the first prosodic representation, the content of the implicit prosodic representation can be enhanced by extracting the second prosodic representation.

[0084] It should be noted that the embodiments of this application do not limit the execution order of steps 2021 and 2022. Step 2021 can be executed before step 2022 or after step 2022. Steps 2021 and 2022 can also be executed simultaneously.

[0085] Step 2023: Superimpose the first prosodic representation and the second prosodic representation of the speech data in the target domain to obtain the implicit prosodic representation of the speech data in the target domain.

[0086] Reference Figure 3 The first prosodic representation extracted by the VAE and the second prosodic representation extracted by the reference encoder can be concatenated to obtain the implicit prosodic representation of the speech data in the target domain.

[0087] Understandably, based on the characteristics that Mel spectra depend on the speaker and bottleneck features are independent of the speaker but depend on prosody, we use VAE and reference encoder to implicitly model the prosody with Mel spectra and BN as inputs, respectively. Then, by superimposing the first and second prosodic representations of the speech data in the target domain, we can obtain more comprehensive information about the implicit prosodic representation of the speech data in the target domain.

[0088] In some embodiments, refer to Figure 5 Step 203 can be implemented in the following ways:

[0089] Step 2031: Use the SA-WA encoder to determine the sentence representation of the speech data in the target domain from the bottleneck features.

[0090] In this embodiment, the sentence representation determined by the SA-WA encoder is aggregated from prosodic contextual information at different levels of self-attention; refer to Figure 6 The SA-WA encoder includes: an encoder pre-module, a self-attention module, and a weighted aggregation module. The architecture of the encoder pre-module is the same as that of the Tacotron speech synthesis model. The self-attention module consists of N self-attention blocks, each composed of two sub-networks: a multi-head attention network and a feedforward network. Each sub-network also employs layer normalization units and residual connection units. When n ranges from 2 to N, for the nth self-attention block, the output f of the (n-1)th self-attention block... n-1 The output m of the first subnetwork (i.e., the multi-head attention network) of the nth self-attention block. n The output f of the second subnetwork (i.e., the feedforward network) of the nth self-attention block n The relationship between them can be explained by formulas (1) and (2).

[0091] m n =LN(MultiHead(f n-1 )+f n-1 (1)

[0092] f n =LN(FFN(m n )+m n (2)

[0093] Where MultiHead(·) represents the data processing operation performed by the multi-head attention network, FFN(·) represents the data processing operation performed by the feedforward network, and LN(·) represents the layer normalization operation.

[0094] Similar to the self-attention module, the weighted aggregation module consists of two sub-networks. The output g of the weighted aggregation is the sentence representation of the speech data in the target domain, and the output g can be calculated using formulas (3), (4), and (5).

[0095] g n =MeanPool(Conv1d(f n (3)

[0096] c = LN(MultiHead(g) 1 ,…,g N )+g N (4)

[0097] g=LN(FFN(c)+c) (5)

[0098] Where Conv1d(·) represents a one-dimensional convolution operation, and MeanPool(·) represents an average pooling operation.

[0099] Step 2032: Process the speaker's label information, explicit prosodic features, implicit prosodic representation, and sentence representation to obtain the speech conversion results corresponding to the speech data in the target domain.

[0100] In some embodiments, refer to Figure 3 The embedding layer can be used to convert the speaker's label information into a format, and the format conversion result can be obtained. The first prosodic representation extracted by VAE, the second prosodic representation extracted by the reference encoder, and the sentence representation output by SA-WA encoder can be superimposed to obtain a superposition result. Then, the prosodic features, the superposition result, and the format conversion result corresponding to the speaker's label information can be concatenated to obtain a concatenated result. Finally, the speech conversion result corresponding to the speech data in the target domain can be obtained based on the concatenated result.

[0101] In some embodiments, refer to Figure 3 After obtaining the concatenated result, the concatenated result can be decoded to obtain the decoding result; then, the Mel spectrum is extracted from the decoding result corresponding to each frame of speech data to obtain the Mel spectrum extraction result. WaveRNN is used to process the Mel spectrum extraction result to obtain the speech conversion result corresponding to the speech data in the target domain.

[0102] It can be seen that the SA-WA encoder can extract sentence context from bottleneck features. The sentence representation determined by the SA-WA encoder can implicitly aggregate the prosodic information of the source speech from the hierarchical representation. Therefore, by combining explicit prosodic features, implicit prosodic representation and sentence representation, the prosodic information of the source speech can be extracted more accurately and comprehensively, which is conducive to accurately obtaining the speech conversion results corresponding to the speech data in the target domain.

[0103] In some embodiments, refer to Figure 7 The training process for a speech conversion model may include:

[0104] Step 701: Obtain the training dataset, which includes speech data from a preset domain and speaker label information; the preset domain and the target domain are different domains.

[0105] Step 702: Based on the training dataset, use an adversarial training method to train the speech conversion model and obtain the trained speech conversion model.

[0106] Understandably, training a speech conversion model using an adversarial training method can improve the data processing accuracy of the trained speech conversion model. Furthermore, after training the speech conversion model with speech data from a predefined domain, the trained speech conversion model can be directly used to convert speech data from the target domain. Since the speech conversion results corresponding to the speech data from the target domain can be used as new speech data from the target domain, the training data for the end-to-end speech recognition model in the target domain is expanded.

[0107] In this embodiment, the first prosodic representation z extracted from the Mel spectrum by VAE is related to the speaker and will affect the timbre of the speech after speech conversion. Therefore, based on the speaker classifier C, an adversarial training method can be used to train the speech conversion model to remove speaker-related information.

[0108] In some embodiments, refer to Figure 8 The implementation of step 702 may include:

[0109] Step 7021: Input the speech data of the preset domain and the speaker's label information into the speech conversion model, and use the speech conversion model to process the speech data of the preset domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the preset domain.

[0110] In this embodiment, the method for obtaining the speech conversion result corresponding to the speech data in the preset domain is the same as the method for obtaining the speech conversion result corresponding to the speech data in the target domain, and will not be described again here.

[0111] Step 7022: Use the speaker classifier to classify the first prosodic representation of the speech data in the preset domain to obtain the classification result; the first prosodic representation of the speech data in the preset domain is the information extracted by the VAE in the speech conversion model from the Mel spectrum of the speech data in the preset domain.

[0112] Reference Figure 3The speaker classifier C can be used to classify the first prosodic representation extracted by the VAE, and the classification result p can be obtained. s Here, the classification result p can be described according to formula (6). s .

[0113] p s =C(z) (6)

[0114] Where z represents the first prosodic representation of the speech data in the preset domain, and C(·) represents the data processing operation of the speaker classifier.

[0115] Step 7023: Use the speaker classifier as the discriminator and the speech conversion model as the generator; determine the loss of the discriminator and the loss of the generator based on the classification results.

[0116] It should be noted that the speaker classifier is a discriminator used when training the speech conversion model. The trained speech conversion model does not include a speaker classifier. That is, when using the trained speech conversion model to process speech data in the target domain, the speaker classifier is not used.

[0117] In some embodiments, the loss of the discriminator can be calculated according to formulas (7) and (8). D .

[0118] L CE =CE(p s ,l s (7)

[0119] Loss D =L CE (8)

[0120] Among them, l s It represents the label information of an independent speaker, and CE(·) represents the cross-entropy loss function.

[0121] In some embodiments, the generator loss can be determined based on the KL divergence of the VAE, the Mel spectrum reconstruction loss of the speech conversion model, and the adversarial loss of the speech conversion model.

[0122] For example, the generator loss can be calculated according to formulas (9) and (10). G .

[0123]

[0124] Loss G =L recons +βL adv +γL kl (10)

[0125] Where e represents a uniform distribution, e = [1 / S, ..., 1 / S], and S is the number of speakers; ||·||2 represents the 2-norm, L recons L represents the Mel spectrum reconstruction loss. recons It is the loss calculated based on the Mel spectrum extraction results of the speech data corresponding to the preset neighborhood of each frame after extracting the Mel spectrum; L kl L represents the KL divergence of the VAE. adv Represents the adversarial loss, where β and γ represent L, respectively. adv and L kl The weight.

[0126] Understandably, the generator loss can be accurately derived from the KL divergence of the VAE, the Mel spectrum reconstruction loss of the speech conversion model, and the adversarial loss of the speech conversion model.

[0127] Step 7024: Based on the discriminator's loss and the generator's loss, and using an adversarial training method, train the speech conversion model to obtain the trained speech conversion model.

[0128] In some embodiments, to prevent the speaker classifier from accurately distinguishing the first prosodic representation z, the classification result p can be minimized by aiming to reduce the adversarial loss. s It follows a uniform distribution e.

[0129] As can be seen, the embodiments of this application use an adversarial training method to train the speech conversion model, which is beneficial for optimizing modules such as VAE of the speech conversion model, thereby improving the data processing accuracy of the trained speech conversion model.

[0130] In some embodiments, in order to improve the generalization ability of the speech conversion model and the speaker similarity of the converted speech, the entire speech conversion model can be trained on a multi-speaker dataset, and then the target speaker's information can be input for adaptive processing. Here, since the speaker's information is the input of the decoder, the decoder's processing is related to the speaker's information. Furthermore, the speaker's information is independent of VAE, reference encoder, SA-WA encoder, etc., and thus, the decoder can achieve adaptive optimization during the adaptive process.

[0131] In some embodiments, the speech conversion model is trained using an adversarial training method, which can successfully remove speaker-related information from the information in the first prosodic representation z, but may also remove some prosodic information. To address this issue, the implicit prosodic representation of the speech data can be accurately and comprehensively obtained by superimposing the second prosodic representation extracted by the reference encoder.

[0132] The training and application processes of a speech conversion model will be illustrated below through an example.

[0133] The training dataset consists of general-domain speech data A and speaker label information for speech data A; the target-domain speech data is a smaller dataset B. First, a speech conversion model can be trained based on the training dataset, enabling the model to output target speech based on prosody, content, and speaker label information. After obtaining the trained speech conversion model, dataset B can be input into it, using the speaker label information from speech data A as input. By processing dataset B through the trained speech conversion model, speech conversion results corresponding to the target-domain speech data can be obtained, thereby expanding the data to include the style of dataset B and achieving speech data augmentation. Using the data augmentation method of this embodiment, the data volume of a smaller target dataset can be effectively expanded, enabling the speech recognition model to achieve better recognition results on the target dataset.

[0134] Based on the data augmentation method proposed in the foregoing embodiments, this application also proposes a data augmentation device; Figure 9 This is a schematic diagram of the composition structure of a data enhancement device according to an embodiment of this application, as shown below. Figure 9 As shown, the device may include an acquisition module 900 and a processing module 901, wherein,

[0135] The acquisition module 900 is used to acquire the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data to obtain the speech conversion result;

[0136] The processing module 901 is used to input the speech data of the target domain and the speaker's label information into the trained speech conversion model, and use the trained speech conversion model to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain.

[0137] In some embodiments of this application, the processing module 901 is used to process the speech data of the target domain and the speaker's label information using the trained speech conversion model to obtain the speech conversion result corresponding to the speech data of the target domain, including:

[0138] Extract explicit prosodic features from the speech data in the target domain;

[0139] Based on the Mel spectrum and bottleneck features of the speech data in the target domain, the implicit prosodic representation of the speech data in the target domain is determined;

[0140] The speaker's label information, explicit prosodic features, and implicit prosodic representation are processed to obtain the speech conversion result corresponding to the speech data of the target domain.

[0141] In some embodiments of this application, the processing module 901 is configured to determine the implicit prosodic representation of the speech data in the target domain based on the Mel spectrum and bottleneck features of the speech data in the target domain, including:

[0142] A first prosodic representation of the speech data in the target domain is extracted from the Mel spectrum using a VAE.

[0143] A second prosodic representation of the speech data in the target domain is extracted from the bottleneck features using a reference encoder.

[0144] The implicit prosodic representation of the speech data in the target domain is obtained by superimposing the first prosodic representation and the second prosodic representation.

[0145] In some embodiments of this application, the processing module 901 is used to extract explicit prosodic features of the speech data in the target domain, including:

[0146] At least one of the vuv, number domain fundamental frequency, and short-term average amplitude of the speech data in the target domain is used as the explicit prosodic feature of the speech data in the target domain.

[0147] In some embodiments of this application, the processing module 901 is used to process the speaker's label information, the explicit prosodic features, and the implicit prosodic representation to obtain the speech conversion result corresponding to the speech data of the target domain, including:

[0148] A weighted aggregation self-attention SA-WA encoder is used to determine the sentence representation of the speech data in the target domain from the bottleneck features;

[0149] The speaker's label information, explicit prosodic features, implicit prosodic representation, and sentence representation are processed to obtain the speech conversion result corresponding to the speech data in the target domain.

[0150] In some embodiments of this application, the apparatus further includes a training module for training the speech conversion model according to the following steps:

[0151] Obtain a training dataset, which includes speech data from a preset domain and speaker labeling information; the preset domain and the target domain are different domains.

[0152] Based on the training dataset, the speech conversion model is trained using an adversarial training method to obtain the trained speech conversion model.

[0153] In some embodiments of this application, the training module is used to train the speech conversion model using an adversarial training method based on the training dataset, to obtain the trained speech conversion model, including:

[0154] The speech data of the preset domain and the speaker's label information are input into the speech conversion model. The speech conversion model is then used to process the speech data of the preset domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the preset domain.

[0155] The first prosodic representation of the speech data in the preset domain is classified using a speaker classifier to obtain a classification result; the first prosodic representation of the speech data in the preset domain is information extracted from the Mel spectrum of the speech data in the speech conversion model by the VAE.

[0156] The speaker classifier is used as a discriminator, and the speech conversion model is used as a generator; the loss of the discriminator and the loss of the generator are determined based on the classification results.

[0157] Based on the loss of the discriminator and the loss of the generator, and using an adversarial training method, the speech conversion model is trained to obtain the trained speech conversion model.

[0158] In some embodiments of this application, the training module is used to determine the loss of the generator based on the classification result, including:

[0159] The generator loss is determined based on the KL divergence of the VAE, the Mel spectrum reconstruction loss of the speech conversion model, and the adversarial loss of the speech conversion model.

[0160] In practical applications, both the acquisition module 900 and the processing module 901 can be implemented using the processor of the electronic device.

[0161] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0162] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0163] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the data augmentation methods provided in this application embodiment.

[0164] Accordingly, this application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the data augmentation methods provided in the above embodiments.

[0165] This application also provides an electronic device. Figure 10 A schematic diagram of an optional component structure of the electronic device provided in an embodiment of this application is shown below. Figure 10 As shown, the electronic device 1000 includes:

[0166] Memory 1001 is used to store executable instructions;

[0167] The processor 1002 is configured to implement any of the above-described data augmentation methods when executing executable instructions stored in the memory 1001.

[0168] The processor 1002 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0169] The aforementioned computer-readable storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0170] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0171] It should be understood that the phrase "some embodiments" mentioned throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "some embodiments" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0172] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0174] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0175] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0176] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause the device automatic test line to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0177] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0178] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0179] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data augmentation method, characterized in that, The method includes: Obtain the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data, and obtain the speech conversion result; The speech data of the target domain and the speaker's label information are input into the trained speech conversion model. The trained speech conversion model is then used to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain. The step of processing the speech data in the target domain and the speaker's label information using the trained speech conversion model to obtain the speech conversion result corresponding to the speech data in the target domain includes: Extract explicit prosodic features from the speech data in the target domain; Based on the Mel spectrum and bottleneck features of the speech data in the target domain, the implicit prosodic representation of the speech data in the target domain is determined; The speaker's label information, explicit prosodic features, and implicit prosodic representation are processed to obtain the speech conversion result corresponding to the speech data in the target domain; wherein, the speech conversion result corresponding to the speech data in the target domain is used as new speech data in the target domain; Specifically, determining the implicit prosodic representation of the speech data in the target domain based on the Mel spectrum and bottleneck features includes: A first prosodic representation of the speech data in the target domain is extracted from the Mel spectrum using a variational autoencoder (VAE). A second prosodic representation of the speech data in the target domain is extracted from the bottleneck features using a reference encoder. The first prosodic representation and the second prosodic representation of the speech data in the target domain are superimposed to obtain the implicit prosodic representation of the speech data in the target domain. The extraction of explicit prosodic features from the speech data in the target domain includes: using at least one of the endpoint detection results vuv, number domain fundamental frequency, and short-term average amplitude of the speech data in the target domain as explicit prosodic features of the speech data in the target domain.

2. The method according to claim 1, characterized in that, The process of processing the speaker's label information, the explicit prosodic features, and the implicit prosodic representation to obtain the speech conversion result corresponding to the speech data in the target domain includes: A weighted aggregation self-attention SA-WA encoder is used to determine the sentence representation of the speech data in the target domain from the bottleneck features; The speaker's label information, explicit prosodic features, implicit prosodic representation, and sentence representation are processed to obtain the speech conversion result corresponding to the speech data in the target domain.

3. The method according to claim 1, characterized in that, The training process of the speech conversion model includes: Obtain a training dataset, which includes speech data from a preset domain and speaker labeling information; the preset domain and the target domain are different domains. Based on the training dataset, the speech conversion model is trained using an adversarial training method to obtain the trained speech conversion model.

4. The method according to claim 3, characterized in that, The step of training the speech conversion model using an adversarial training method based on the training dataset to obtain the trained speech conversion model includes: The speech data of the preset domain and the speaker's label information are input into the speech conversion model. The speech conversion model is then used to process the speech data of the preset domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the preset domain. The first prosodic representation of the speech data in the preset domain is classified using a speaker classifier to obtain a classification result; the first prosodic representation of the speech data in the preset domain is information extracted from the Mel spectrum of the speech data in the speech conversion model by the VAE. The speaker classifier is used as a discriminator, and the speech conversion model is used as a generator; the loss of the discriminator and the loss of the generator are determined based on the classification results. Based on the loss of the discriminator and the loss of the generator, and using an adversarial training method, the speech conversion model is trained to obtain the trained speech conversion model.

5. The method according to claim 4, characterized in that, The loss of the generator is determined based on the classification results, including: The generator loss is determined based on the KL divergence of the VAE, the Mel spectrum reconstruction loss of the speech conversion model, and the adversarial loss of the speech conversion model.

6. A data augmentation device, characterized in that, The device includes an acquisition module and a processing module, wherein, The acquisition module is used to acquire the trained speech conversion model; the speech conversion model is used to convert the speech data into speech based on the speaker's label information and the prosodic information of the speech data to obtain the speech conversion result. The processing module is used to input the speech data of the target domain and the speaker's label information into the trained speech conversion model, and use the trained speech conversion model to process the speech data of the target domain and the speaker's label information to obtain the speech conversion result corresponding to the speech data of the target domain. The step of processing the speech data in the target domain and the speaker's label information using the trained speech conversion model to obtain the speech conversion result corresponding to the speech data in the target domain includes: Extract explicit prosodic features from the speech data in the target domain; Based on the Mel spectrum and bottleneck features of the speech data in the target domain, the implicit prosodic representation of the speech data in the target domain is determined; The speaker's label information, explicit prosodic features, and implicit prosodic representation are processed to obtain the speech conversion result corresponding to the speech data in the target domain; wherein, the speech conversion result corresponding to the speech data in the target domain is used as new speech data in the target domain; Specifically, determining the implicit prosodic representation of the speech data in the target domain based on the Mel spectrum and bottleneck features includes: A first prosodic representation of the speech data in the target domain is extracted from the Mel spectrum using a variational autoencoder (VAE). A second prosodic representation of the speech data in the target domain is extracted from the bottleneck features using a reference encoder. The first prosodic representation and the second prosodic representation of the speech data in the target domain are superimposed to obtain the implicit prosodic representation of the speech data in the target domain. The extraction of explicit prosodic features from the speech data in the target domain includes: using at least one of the endpoint detection results vuv, number domain fundamental frequency, and short-term average amplitude of the speech data in the target domain as explicit prosodic features of the speech data in the target domain.

7. An electronic device, characterized in that, Includes a processor and memory for storing computer programs that can run on the processor; wherein, The processor is used to run the computer program to perform the data augmentation method according to any one of claims 1 to 5.

8. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data augmentation method according to any one of claims 1 to 5.

Images