Data augmentation method, device and electronic equipment

Abstract

The embodiment of the present application provides a data augmentation method, device and electronic device. The method includes: obtaining a Mongolian text word sequence containing an unregistered word and visual text features of a text slice in the text word sequence; performing speech synthesis processing on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slice in the text word sequence to obtain spectrogram information of the text word sequence; and obtaining an augmented pronunciation of the unregistered word based on the spectrogram information. The method is used to achieve the effect of improving data augmentation.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and image processing technology, and in particular to a data augmentation method, device and electronic equipment. Background Art

[0002] Data augmentation, as an effective strategy to improve the generalization ability and robustness of machine learning models, has been widely used in many fields. Whether it is speech recognition, natural language processing (NLP), or computer vision (CV), data augmentation is one of the key means to overcome the problem of data scarcity or imbalance. By increasing the variation factors of training data, this method can not only enhance the learning ability of the model, but also prevent the occurrence of overfitting to a certain extent. Especially in the context of increasingly complex deep learning models and growing demand for data volume, the role of data augmentation is particularly important.

[0003] In the prior art, for speech recognition, data augmentation can be achieved by adjusting the volume and speed of the audio, adding background noise, or performing perturbations at the phoneme level, so as to simulate the changes in sound in the real environment and improve the model's adaptability to different environments.

[0004] However, existing data augmentation methods have the problem of poor data augmentation effect when performing speech recognition. Summary of the invention

[0005] The embodiments of the present application provide a data augmentation method, device and electronic device to solve the problem that the existing data augmentation method has poor data augmentation effect when performing speech recognition.

[0006] In a first aspect, an embodiment of the present application provides a data augmentation method, comprising:

[0007] Acquire a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence;

[0008] Performing speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence;

[0009] According to the spectrum information, the augmented pronunciation of the unregistered word is obtained.

[0010] In a possible implementation, obtaining a Mongolian text word sequence containing an unregistered word and visual text features of a text slice in the text word sequence includes:

[0011] Obtaining a Mongolian text word sequence containing an unregistered word;

[0012] According to the text word sequence, generate a visual text representation of the text word sequence;

[0013] Slicing the visual text representation of the text word sequence to obtain text slices of the visual text representation;

[0014] The text slices are encoded to obtain the visual text features of the text slices.

[0015] In a possible implementation, slicing the visual text representation of the text word sequence to obtain text slices of the visual text representation includes:

[0016] According to a sliding window that meets size requirements and word formation relationships in a text word sequence, a visual text representation of the text word sequence is sliced ​​to obtain text slices of the visual text representation.

[0017] In a possible implementation, encoding processing is performed on the text slice to obtain visual text features of the text slice, including:

[0018] The text slice is input into the convolution layer module in the feature processing model for convolution processing to obtain visual feature information; the convolution layer module includes a convolution layer, an activation function layer, a pooling layer and a fully connected layer arranged in sequence;

[0019] Inputting the visual feature information into a reshaping module in a feature processing model for reshaping to obtain reshaping feature information;

[0020] The reshaped feature information is input into the linear layer module to obtain visual text features.

[0021] In a possible implementation, performing speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence includes:

[0022] Performing feature fusion on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slice in the text word sequence to obtain fused features;

[0023] Encoding the fused features to obtain feature information of the fused features;

[0024] Decode the feature information of the fused features to obtain the prediction information of the spectrum graph;

[0025] The predicted information of the spectrogram is subjected to residual optimization processing to obtain the spectrogram information of the text word sequence.

[0026] In a possible implementation, decoding the feature information of the fused feature to obtain prediction information of the spectrum graph includes:

[0027] The feature information of the fused features is input into the preprocessing network in the decoder for processing to obtain the feature vector after dimensionality reduction;

[0028] Input the feature information of the fused features into the attention mechanism module in the decoder to obtain the context vector;

[0029] Concatenate the feature vector after dimension reduction and the context vector to obtain a concatenated feature vector;

[0030] Inputting the concatenated feature vector into a stack module in a decoder for processing to obtain a feature representation representing the time step, wherein the stack module includes at least two unidirectional LSTM layers;

[0031] The feature representation representing the time step and the context vector are linearly transformed to obtain the prediction information of the spectrogram.

[0032] In a possible implementation, residual optimization processing is performed on the prediction information of the spectrogram to obtain the spectrogram information of the text word sequence, including:

[0033] Input the prediction information of the spectrum graph into the convolutional network for residual prediction processing to obtain the residual prediction result;

[0034] The residual prediction results and the prediction information of the spectrogram are fused to obtain the spectrogram information of the text word sequence.

[0035] In a possible implementation, obtaining the augmented pronunciation of the unregistered word according to the spectrogram information includes:

[0036] The spectrogram information is input into the vocoder to obtain the augmented pronunciation of the unregistered word.

[0037] In a second aspect, an embodiment of the present application provides a data augmentation device, including:

[0038] An acquisition module, used for acquiring a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence;

[0039] A speech synthesis module is used to perform speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain the spectrogram information of the text word sequence;

[0040] The module is used to obtain the augmented pronunciation of the unregistered word according to the spectrum information.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of a scenario of the data augmentation method provided in this application;

[0048] Figure 2 Schematic diagram of the data augmentation method provided in this application Figure 1 ;

[0049] Figure 3 Schematic diagram of the data augmentation method provided in this application Figure 2 ;

[0050] Figure 3a A schematic diagram of the structure of the feature processing model provided for this application;

[0051] Figure 4 Schematic diagram of the data augmentation method provided in this application Figure 3 ;

[0052] Figure 4a A schematic diagram of the structure of the speech synthesis model provided for this application;

[0053] Figure 5 A schematic diagram of the structure of the data augmentation device provided in this application;

[0054] Figure 6 A schematic diagram of the structure of the electronic device provided in this application.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] First, the terms involved in this application are explained:

[0058] Natural Language Processing (NLP) is an interdisciplinary subject in the fields of computer science, artificial intelligence and linguistics. It mainly studies how to enable computers to understand, process, generate and simulate human language, so as to achieve the goal of having natural conversations with humans. NLP technology has a wide range of applications, including but not limited to machine translation, sentiment analysis, text summarization, question-answering systems, speech recognition and speech synthesis.

[0059] Speech recognition technology can refer to the process of converting human voice signals into corresponding text or other useful forms. This technology enables computers to "understand" human language and is an important branch of natural language processing. It is widely used in automatic subtitle generation, voice assistants, telephone customer service systems and other fields.

[0060] Computer Vision can refer to the science of studying how to enable computers to "see" and "understand" digital images or videos. Its goal is to imitate the capabilities of the human visual system so that computers can recognize, understand and interpret visual information.

[0061] In the prior art, existing speech recognition technology faces many challenges when processing high-quality pronunciation data. The main problem is that it is difficult to obtain high-quality pronunciation data. This is mainly because the production of such data requires professional recording equipment, a standard recording environment, and professional annotation personnel, which is costly and time-consuming. In addition, pronunciation data is often scarce and unbalanced. For example, the amount of data for certain dialects or minority languages ​​is far less than that of mainstream languages, and the speech data in specific scenarios is relatively insufficient. These problems result in insufficient training data to cover all possible variants, which limits the learning ability of the model and affects the generalization ability and robustness of the model. Therefore, it is particularly important to develop effective data augmentation technology, which can not only improve the diversity and representativeness of pronunciation data, but also alleviate the problems of data scarcity and imbalance to a certain extent, and ultimately help optimize model performance.

[0062] The data augmentation method, device, and electronic device provided by the present application have their own advantages and limitations in expressing and understanding the target object, as visual information and text information have their own advantages and limitations. The former directly reflects the appearance characteristics of the object, while the latter provides a rich semantic background. Therefore, by integrating these two features, additional context clues are provided for the speech generation model, so that the reasonable pronunciation of the unregistered words can be generated even in the absence of direct pronunciation samples, effectively solving the problems of difficulty in obtaining high-quality pronunciation data, data scarcity, and imbalance, improving the generalization ability and robustness of the model, and making the data augmentation effect of the unregistered words outputted in the end better.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] Figure 1 A schematic diagram of a scenario of the data augmentation method provided in this application, such as Figure 1 As shown, the specific application scenario of the present application is applied to a data augmentation system, which includes a feature processing model, a speech synthesis model, a Mongolian speech corpus and a vocoder, wherein a text sequence T containing an unregistered Mongolian word is input into the feature processing model, a corresponding visual text representation Vt is first generated in the feature processing model, and then the visual text representation Vt is sliced ​​to form a visual text slice V, the visual feature encoder extracts the visual feature d of the visual text slice V, the visual feature d is input into the speech synthesis model, and combined with the Mongolian speech corpus, a Mel spectrogram y is obtained, the Mel spectrogram y is input into the vocoder, and the vocoder generates an augmented pronunciation Sa of the unregistered word according to the Mel spectrogram y.

[0065] Figure 2 Schematic diagram of the data augmentation method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0066] S201, obtaining a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence.

[0067] Among them, Out-of-Vocabulary (OOV) can refer to words that have not appeared in the training data set. These words do not exist in the vocabulary of the model, so the model cannot directly process them. For example, the unregistered words can be words that are not registered in the Mongolian speech corpus, where the Mongolian speech corpus refers to a Mongolian speech data set collected and organized, which is usually used in research fields such as speech recognition, speech synthesis, and language learning.

[0068] A text word sequence may refer to an ordered list formed by splitting a text into individual words or vocabulary units according to certain rules (usually based on spaces or punctuation marks) and arranging them in the order of the original text. A text slice may refer to an image of a portion of a continuous text segment extracted from a text word sequence. These segments may be single words, phrases, sentences, or paragraphs, depending on the application scenario and requirements. For example, when the text word sequence is a Mongolian text word sequence, a text slice may refer to a Mongolian morpheme, which may be a complete word or a part of a word.

[0069] Visual text features may refer to text features extracted from text slices after encoding the images of the text slices.

[0070] In some embodiments, the user can enable the data augmentation system to obtain a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence by inputting, or after the text word sequence is input to the data augmentation system, the data augmentation system can process the text word sequence to obtain visual text features.

[0071] S202: Perform speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence.

[0072] The text slice group may refer to a text slice group obtained by any combination of text slices, and the text slice group may include a single text slice or a slice composed of multiple text slices.

[0073] The phoneme feature may refer to the phoneme information corresponding to the text slice group, wherein when the text word sequence is a Mongolian sequence, the factor information may obtain the phonemes describing the text slice group according to an existing Mongolian speech corpus.

[0074] Speech synthesis processing may refer to the process of converting text into sound. Speech synthesis processing may be performed by a speech synthesis model. By inputting phoneme features and visual text features into the speech synthesis model, a corresponding audio waveform or spectrogram may be obtained. In some embodiments, the speech synthesis model may be a deep learning model, such as WaveNet, Tacotron, etc.

[0075] Spectrogram information may refer to a two-dimensional image obtained by converting a sound signal from the time domain to the frequency domain through methods such as Fourier transform in speech signal processing. In this image, the horizontal axis may represent time, the vertical axis may represent frequency, and the change in color or grayscale may reflect the intensity or energy distribution of different frequency components at each time point. The spectrogram can intuitively display the frequency domain characteristics of the sound signal, which is of great value for applications such as speech recognition, speech synthesis, and audio analysis. In an embodiment of the present application, the spectrogram information may be information obtained after speech synthesis processing of phoneme features and visual text features.

[0076] S203, obtaining the augmented pronunciation of the unregistered word according to the spectrum information.

[0077] Among them, by analyzing the time-frequency-intensity distribution in the spectrogram information, key features such as syllable boundaries, fundamental frequency (tone), and resonance peak positions of the unregistered words can be extracted to obtain the augmented pronunciation of the unregistered words.

[0078] Among them, in the embodiment of the present application, the augmented pronunciation of the unregistered word is obtained according to the spectrum information, including:

[0079] The spectrogram information is input into the vocoder to obtain the augmented pronunciation of the unregistered word.

[0080] Among them, the vocoder (vocoder, a speech signal analysis and synthesis system) can analyze and synthesize sound, and is mainly used to synthesize human speech.

[0081] In the embodiment of the present application, after the spectrogram information is input into the vocoder, the vocoder can extract acoustic features from the spectrogram, such as fundamental frequency (F0), formant frequency, harmonic structure, energy distribution, etc. This feature can characterize the main attributes of the sound signal, such as pitch, timbre and speech rate.

[0082] After the acoustic features are extracted, the extracted acoustic features are transformed to obtain parameters that can characterize, for example, Mel Frequency Cepstral Coefficients (MFCC), Linear Predictive Coding (LPC) coefficients, and the like.

[0083] The vocoder uses these parameters to generate the corresponding audio waveform through a pre-trained model (such as WaveNet, Tacotron 2, MelGAN, etc.).

[0084] According to the generated audio waveform, it is converted into the augmented pronunciation of the unregistered word.

[0085] The data augmentation method provided in the embodiment of the present application obtains a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence; performs speech synthesis processing on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence; and obtains means for obtaining an augmented pronunciation of the unregistered words based on the spectrogram information, thereby improving the data augmentation effect.

[0086] Figure 3 Schematic diagram of the data augmentation method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, a method for obtaining a Mongolian text word sequence containing an unregistered word and a visual text feature of a text slice in the text word sequence is described in detail. The method includes:

[0087] S301, obtaining a Mongolian text word sequence containing an unregistered word;

[0088] S302, generating a visual text representation of the text word sequence according to the text word sequence;

[0089] S303: Slice the visual text representation of the text word sequence to obtain text slices of the visual text representation.

[0090] Among them, the visual text representation can refer to the image of the generated text word sequence.

[0091] Slicing can refer to the process of dividing a continuous text or visual text representation into multiple smaller segments or units for subsequent processing and analysis. Each slice can include a portion of visual text features, which can be morphological information of a single character, word, or phrase.

[0092] In the embodiment of the present application, the visual text representation of the text word sequence is sliced ​​to obtain text slices of the visual text representation, including:

[0093] According to a sliding window that meets size requirements and word formation relationships in a text word sequence, a visual text representation of the text word sequence is sliced ​​to obtain text slices of the visual text representation.

[0094] The sliding window may refer to a data processing technique that generates a series of subsequences or segments by moving a fixed-size window over a data sequence one small step (usually 1 unit) at a time. In the embodiment of the present application, the sliding window may divide a text word sequence into multiple short segments for the extraction and analysis of local features.

[0095] The size requirement may refer to the size or width of the sliding window, i.e., the amount of data contained in each window. In an embodiment of the present application, selecting an appropriate window size can ensure that each slice contains enough information while avoiding information overload or too little. For example, the window size can be set to contain 1 morpheme.

[0096] The word-forming relationship can characterize the writing characteristics of Mongolian, namely vertical writing and the way of forming new words by connecting roots and affixes.

[0097] According to the size requirements and word formation relationships, the image can be sliced ​​using the vertical sliding window slicing method to extract text slices represented by visual text.

[0098] S304: Encode the text slice to obtain visual text features of the text slice.

[0099] Among them, encoding processing can refer to the process of converting characters or words in a text slice into a digital or vector form that can be processed by a computer, while extracting and retaining its visual features.

[0100] Among them, in the embodiment of the present application, the text slice is encoded to obtain the visual text features of the text slice, including:

[0101] The text slice is input into the convolution layer module in the feature processing model for convolution processing to obtain visual feature information; the convolution layer module includes a convolution layer, an activation function layer, a pooling layer and a fully connected layer arranged in sequence;

[0102] Inputting the visual feature information into a reshaping module in a feature processing model for reshaping to obtain reshaping feature information;

[0103] The reshaped feature information is input into the linear layer module to obtain visual text features.

[0104] in, Figure 3a The structural diagram of the feature processing model provided for this application is as follows: Figure 3aAs shown, the feature processing model includes: CNN layer (convolution layer module), Reshape (reshaping module) and Linear (linear layer module), among which CNNLayer includes 2*Convolution (convolution layer), relu (activation function layer), Pooling layer (pooling layer) and FullyConnected Layer (fully connected layer).

[0105] In the embodiments of the present application, Figure 3a As shown, the text sequence processing process for Mongolian unregistered words can include: 1. Convert the input text sequence into grayscale slices. In order to ensure the consistency of Mongolian fonts, the width of each character is w, the height is h, and the font size is fs. After conversion, the Mongolian text is processed into an image with a width of nw and a height of h. 2. Due to the writing characteristics of Mongolian, that is, vertical writing and the way of forming new words through roots and affixes, the image can be sliced ​​by vertical sliding window slicing method, so as to extract the corresponding image feature information and construct a slice image set of text visual features.

[0106] In the embodiments of the present application, Figure 3a As shown in the figure, after obtaining the slice image set of the text visual features, the slice image set is input into the CNN layer. Through the two two-dimensional convolution operations of 2*Convolution in the CNN layer, the feature information describing the basic characteristics of Mongolian characters, such as the edge, corner, contour and texture of the image, is extracted; then relu is used to increase nonlinearity so that the model can fit more complex functions, and Pooling layer is used to reduce the spatial dimension of the feature map through downsampling operations, reducing the computational complexity while retaining important features, and Fully Connected Layer is used to connect all nodes to extract visual feature information representing high-level abstract features; finally, Reshape is used to reshape the visual feature information into a one-dimensional vector, and the reshaped features are linearly transformed using Linear to finally obtain the visual text features. Therefore, by integrating these visual features, the acoustic frame can be predicted more accurately, and the pronunciation of unregistered Mongolian words can be effectively predicted.

[0107] Figure 4 Schematic diagram of the data augmentation method provided in this application Figure 3 ,like Figure 4 As shown, in this embodiment Figure 2 Based on the embodiment, a method for performing speech synthesis processing on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slice in the text word sequence to obtain the spectrogram information of the text word sequence is described in detail, and the method includes:

[0108] S401, performing feature fusion on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slice in the text word sequence to obtain fused features;

[0109] S402, encoding the fused features to obtain feature information of the fused features;

[0110] S403, decoding the feature information of the fused feature to obtain prediction information of the spectrum graph;

[0111] S404: Perform residual optimization processing on the prediction information of the spectrogram to obtain the spectrogram information of the text word sequence.

[0112] Among them, in the embodiment of the present application, the feature information of the fusion feature is decoded to obtain the prediction information of the spectrum graph, including:

[0113] The feature information of the fused features is input into the preprocessing network in the decoder for processing to obtain a feature vector after dimensionality reduction; the feature information of the fused features is input into the attention mechanism module in the decoder to obtain a context vector; the feature vector after dimensionality reduction and the context vector are spliced ​​to obtain a spliced ​​feature vector; the spliced ​​feature vector is input into the stack module in the decoder for processing to obtain a feature representation representing the time step, wherein the stack module includes at least two unidirectional LSTM layers; the feature representation representing the time step and the context vector are linearly transformed to obtain prediction information of the spectrum graph.

[0114] In the embodiment of the present application, residual optimization processing is performed on the prediction information of the spectrogram to obtain the spectrogram information of the text word sequence, including:

[0115] The prediction information of the spectrogram is input into the convolutional network for residual prediction processing to obtain the residual prediction result; the residual prediction result and the prediction information of the spectrogram are feature fused to obtain the spectrogram information of the text word sequence.

[0116] in, Figure 4a The structural diagram of the speech synthesis model provided for this application is as follows: Figure 4a As shown, the speech synthesis model includes: Encoder, Decoder and residual processing module, wherein the Encoder includes a feature encoding unit, a 3*convolutional layer unit, and a bidirectional LSTM unit connected in sequence.

[0117] The Decoder includes an attention mechanism unit, a 2*LSTM unit, a 2*Pre-Net unit, a linear layer + Sigmoid unit, a linear projection unit and a Stop token unit, wherein the first input end of the attention mechanism unit is connected to the Encoder, the second input end and the output end are connected to the 2*LSTM unit, the 2*LSTM unit is also connected to the linear layer + Sigmoid unit and the linear projection unit, the linear layer + Sigmoid unit is connected to the Stop token unit, the linear projection unit is also connected to the 2*LSTM unit through the 2*Pre-Net unit, and is connected to the residual processing module.

[0118] In the embodiments of the present application, Figure 4a As shown, the speech synthesis model can be a Tacotron2 model. Therefore, compared with the non-autoregressive model Fast Speech2, the Tacotron2 model can more accurately capture the details and changes of speech when processing Mongolian unregistered word pronunciation data.

[0119] The phoneme features and visual text features are fused to obtain fused features, and the fused features are input into the Encoder. Through the feature encoding unit, 3*convolutional layer unit, and bidirectional LSTM unit in the Encoder, the encoder not only processes the visual text features, but also combines the phoneme information in the text sequence for joint encoding. Richer feature information is obtained through the embedded representation of the encoded visual text sequence and the phoneme sequence, and the extraction of the context vector by the three-layer CNN.

[0120] The feature information of the fused features is then input into the Decoder to predict the Mel spectrogram frame by frame using the autoregressive neural network structure. Among them, the 2*Pre-Net unit can combine the encoded input sequence with the context vector of the attention mechanism. The 2*LSTM unit can perform stack processing to process the forward and reverse time series information respectively. The linear projection unit can concatenate the output of the 2*LSTM unit with the context vector of the attention mechanism for linear transformation. The attention mechanism unit can solve the alignment problem and ensure that the decoder does not repeat or ignore certain subsequences. The Stop token unit can control when the generation process stops.

[0121] Therefore, the position-sensitive attention mechanism is adopted, and the accumulated attention weight of the time step in the decoder is used as an additional feature to effectively solve the alignment problem caused by the decoder repeating or ignoring certain subsequences. The decoder part adopts an autoregressive neural network structure to predict the Mel spectrogram frame by frame through the encoded input sequence. In the prediction process, the 2*Pre-Net unit is combined with the context vector of the attention mechanism, and then processed by a stack composed of 2*LSTM units. The output of the 2*LSTM unit is then connected in series with the context vector of the attention mechanism using a linear projection unit to achieve accurate prediction of each frame of the Mel spectrogram.

[0122] In the embodiment of the present application, the residual processing module may include a 5*convolutional Post-Net, which predicts the residual through the 5*convolutional Post-Net and adds it to the prediction result to further optimize the overall prediction effect. Thus, Waveglow is used as a vocoder to realize the conversion from the Mel spectrogram to speech data, and the augmented pronunciation of the unregistered word is obtained.

[0123] In the embodiment of the present application, the feature processing model can augment the pronunciation data of Mongolian unregistered words by combining visual text features. Thus, the prediction generation of pronunciation is assisted by extracting visual features of Mongolian text, such as character shape, font style and size. Among them, the visual text feature F can satisfy:

[0124] F=VisualFeatureExtractor(T);

[0125] Among them, T is the image corresponding to the Mongolian text.

[0126] The visual text feature F is input into the pre-trained TTS augmented model M for feature fusion to obtain the pronunciation data, where the pronunciation data satisfy:

[0127]

[0128] For the Mongolian unregistered word set Vunknown, the TTS augmented model combined with visual text features is used to generate pronunciation predictions, and the generated results are added to the OOV pronunciation dataset. The generated results meet the following requirements:

[0129]

[0130] in, represents the augmented pronunciation data set of unregistered words, Represents the initial unregistered word pronunciation data set.

[0131] The fine-grained prosody control unit is used to improve the prosody and emotional expression of the augmented pronunciation data. This unit controls the pitch, duration, and intensity of the synthesized pronunciation to make the generated pronunciation data more natural.

[0132] First, the prosodic features are extracted from the out-of-domain pronunciation data, where the prosodic feature F v satisfy:

[0133] F v =ExtractRhythmFeatures(v);

[0134] Wherein, v represents voice data.

[0135] Then, the fine-grained prosody control unit takes the prosody features and the emotion label as input to generate adjusted prosody parameters, where the adjusted prosody parameters satisfy:

[0136]

[0137] Among them, E represents the emotion label, P v Represents the original prosodic parameters.

[0138] Finally, the adjusted rhythmic parameters and the unregistered word text information set are used to perform pronunciation synthesis to generate pronunciation data with the required rhythmic expression and emotional expression. satisfy:

[0139]

[0140] Among them, T v Indicates the text information corresponding to the pronunciation v.

[0141] The pronunciation enhancement unit enhances the content, rhythm and emotional features in the generated pronunciation data through separation contrast learning, thereby suppressing the impact of overfitting of the augmented model to the noise data. The loss function L satisfies:

[0142]

[0143] Among them, S represents the similarity calculation function, F x and They represent the feature vectors of the original data x and the positive samples respectively, and τ is used to control the distribution.

[0144] Through the above loss function, we can get the enhanced feature representation. Finally, we use the enhanced features to generate pronunciation data. satisfy:

[0145]

[0146] in, Represents the enhanced feature representation.

[0147] The data augmentation method provided in the embodiment of the present application introduces visual text features in the encoding part, so that the speech generator can generate word pronunciation data according to the relationship between the text morphological features and the corresponding pronunciation, thereby realizing data augmentation of the pronunciation of unregistered Mongolian words and improving the pass rate of the augmented pronunciation.

[0148] Figure 5 A schematic diagram of the structure of the data augmentation device provided in this application, such as Figure 5 As shown, the data augmentation device 50 provided in this embodiment includes:

[0149] An acquisition module 501 is used to acquire a Mongolian text word sequence containing an unregistered word and visual text features of a text slice in the text word sequence;

[0150] A speech synthesis module 502 is used to perform speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain the spectrogram information of the text word sequence;

[0151] The obtaining module 503 is used to obtain the augmented pronunciation of the unregistered word according to the spectrogram information.

[0152] In a possible implementation, the acquisition module 501 may also be specifically used for:

[0153] Obtaining a Mongolian text word sequence containing an unregistered word;

[0154] According to the text word sequence, generate a visual text representation of the text word sequence;

[0155] Slicing the visual text representation of the text word sequence to obtain text slices of the visual text representation;

[0156] The text slices are encoded to obtain the visual text features of the text slices.

[0157] In a possible implementation, the acquisition module 501 may also be specifically used for:

[0158] According to a sliding window that meets size requirements and word formation relationships in a text word sequence, a visual text representation of the text word sequence is sliced ​​to obtain text slices of the visual text representation.

[0159] In a possible implementation, the acquisition module 501 may also be specifically used for:

[0160] The text slice is input into the convolution layer module in the feature processing model for convolution processing to obtain visual feature information; the convolution layer module includes a convolution layer, an activation function layer, a pooling layer and a fully connected layer arranged in sequence;

[0161] Inputting the visual feature information into a reshaping module in a feature processing model for reshaping to obtain reshaping feature information;

[0162] The reshaped feature information is input into the linear layer module to obtain visual text features.

[0163] In a possible implementation, the speech synthesis module 502 may also be specifically used for:

[0164] Performing feature fusion on the phoneme features corresponding to the text slice group in the text word sequence and the visual text features of the text slice in the text word sequence to obtain fused features;

[0165] Encoding the fused features to obtain feature information of the fused features;

[0166] Decode the feature information of the fused features to obtain the prediction information of the spectrum graph;

[0167] The predicted information of the spectrogram is subjected to residual optimization processing to obtain the spectrogram information of the text word sequence.

[0168] In a possible implementation, the speech synthesis module 502 may also be specifically used for:

[0169] The feature information of the fused features is input into the preprocessing network in the decoder for processing to obtain the feature vector after dimensionality reduction;

[0170] Input the feature information of the fused features into the attention mechanism module in the decoder to obtain the context vector;

[0171] Concatenate the feature vector after dimension reduction and the context vector to obtain a concatenated feature vector;

[0172] Inputting the concatenated feature vector into a stack module in a decoder for processing to obtain a feature representation representing the time step, wherein the stack module includes at least two unidirectional LSTM layers;

[0173] The feature representation representing the time step and the context vector are linearly transformed to obtain the prediction information of the spectrogram.

[0174] In a possible implementation, the speech synthesis module 502 may also be specifically used for:

[0175] Input the prediction information of the spectrum graph into the convolutional network for residual prediction processing to obtain the residual prediction result;

[0176] The residual prediction results and the prediction information of the spectrogram are fused to obtain the spectrogram information of the text word sequence.

[0177] In a possible implementation, the obtaining module 503 may also be specifically used for:

[0178] The spectrogram information is input into the vocoder to obtain the augmented pronunciation of the unregistered word.

[0179] The data augmentation device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0180] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0181] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0182] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0183] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0184] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0185] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0186] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0187] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0188] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0189] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0190] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0191] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0193] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0194] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0195] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data augmentation method, characterized in that: include: Acquire a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence; Performing speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence; The augmented pronunciation of the unregistered word is obtained according to the spectrogram information.

2. The method according to claim 1, characterized in that The obtaining of a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence includes: Obtaining a Mongolian text word sequence containing an unregistered word; Based on the text word sequence, generating a visual text representation of the text word sequence; Slicing the visual text representation of the text word sequence to obtain text slices of the visual text representation; The text slice is encoded to obtain visual text features of the text slice.

3. The method according to claim 2, characterized in that Slicing the visual text representation of the text word sequence to obtain text slices of the visual text representation includes: According to the sliding window that meets the size requirement and the word formation relationship in the text word sequence, the visual text representation of the text word sequence is sliced ​​to obtain text slices of the visual text representation.

4. The method according to claim 2, characterized in that: The text slice is encoded to obtain visual text features of the text slice, including: Inputting the text slice into a convolutional layer module in a feature processing model for convolution processing to obtain visual feature information; the convolutional layer module includes a convolutional layer, an activation function layer, a pooling layer and a fully connected layer arranged in sequence; Inputting the visual feature information into a reshaping module in the feature processing model for reshaping to obtain reshaping feature information; The reshaped feature information is input into a linear layer module to obtain visual text features.

5. The method according to claim 1, characterized in that Performing speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain spectrogram information of the text word sequence, including: Performing feature fusion on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain fused features; Performing encoding processing on the fused features to obtain feature information of the fused features; Decoding the feature information of the fused feature to obtain prediction information of the spectrum graph; The prediction information of the spectrogram is subjected to residual optimization processing to obtain the spectrogram information of the text word sequence.

6. The method according to claim 5, characterized in that Decoding the feature information of the fused feature to obtain prediction information of the spectrum graph includes: Inputting the feature information of the fused features into a preprocessing network in a decoder for processing to obtain a feature vector after dimensionality reduction; Inputting the feature information of the fused features into the attention mechanism module in the decoder to obtain a context vector; Concatenate the feature vector after dimension reduction and the context vector to obtain a concatenated feature vector; Inputting the concatenated feature vector into a stack module in a decoder for processing to obtain a feature representation representing a time step, wherein the stack module includes at least two unidirectional LSTM layers; The feature representation representing the time step and the context vector are linearly transformed to obtain prediction information of the spectrogram.

7. The method according to claim 5, characterized in that Performing residual optimization processing on the prediction information of the spectrogram to obtain the spectrogram information of the text word sequence includes: Inputting the prediction information of the spectrum graph into a convolutional network for residual prediction processing to obtain a residual prediction result; Feature fusion is performed on the residual prediction result and the prediction information of the spectrogram to obtain the spectrogram information of the text word sequence.

8. The method according to claim 1, characterized in that The step of obtaining the augmented pronunciation of the unregistered word according to the spectrogram information includes: The spectrogram information is input into a vocoder to obtain an augmented pronunciation of the unregistered word.

9. A data augmentation device, characterized in that: include: An acquisition module, used for acquiring a Mongolian text word sequence containing unregistered words and visual text features of text slices in the text word sequence; A speech synthesis module, used for performing speech synthesis processing on the phoneme features corresponding to the text slice groups in the text word sequence and the visual text features of the text slices in the text word sequence to obtain the spectrogram information of the text word sequence; The obtaining module is used to obtain the augmented pronunciation of the unregistered word according to the spectrum diagram information.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

Images