A method for constructing Vietnamese speech recognition corpus

Abstract

The present invention proposes a method for constructing Vietnamese speech recognition corpus, which belongs to the field of artificial intelligence technology. The present invention uses the speech representation and text representation obtained by the speech pre-training model and the text pre-training model, first performs temporal alignment through CTC, and then uses the hybrid attention combining local attention and global attention to align the speech and text twice, and the aligned speech and text can be directly used as Vietnamese speech recognition corpus. Experimental results show that this method effectively reduces the problem of semantic alignment misalignment in the Vietnamese speech-text alignment task, improves the alignment robustness of the model in noisy data, reduces the gap between different modes of speech and text, and improves the quality of Vietnamese speech recognition corpus.

Description

Technical field:

[0001] The invention relates to a Vietnamese speech recognition corpus construction method and belongs to the technical field of artificial intelligence. Background technology:

[0002] Vietnamese is a language with scarce resources. The public data set for Vietnamese speech recognition only has hundreds of hours. Compared with tens of thousands of hours for major languages ​​such as English and Chinese, the scarcity and sparsity of annotated data make the model unable to adapt well to data deviation caused by factors such as accent, age, environmental noise, speech speed, near and far fields. This leads to unsatisfactory recognition rate of the recognition model on Vietnamese. Vietnamese has a large amount of long speech text data such as audio books, film and television dramas that can be used as training corpus, but the manual annotation method is labor-intensive and costly. Therefore, it is of great research significance and value to study the automatic alignment algorithm of Vietnamese speech and text and to automatically annotate and align Vietnamese speech data. At the same time, Internet data (speech and text data) often carries a large amount of noise data (speech noise, text noise). Common problems such as missing words, swallowing words, wrong words, and inconsistency between text and speech are common, such as mismatch between speech and text. The speech-text alignment task needs to solve not only the problem of temporal alignment with text according to acoustic boundaries, but also semantic alignment when speech and text do not completely correspond. Summary of the invention:

[0003] The present invention proposes a method for constructing Vietnamese speech recognition corpus, which solves the problem of misalignment caused by noise data such as text addition or deletion in the Vietnamese speech-text alignment task. The method uses large pre-trained speech and text models as speech feature extractors and text feature extractors respectively, and then uses CTC-hybrid cross attention to align speech and text. The model is given the ability to understand text and speech, so that the model can learn semantic alignment information between different modalities; and on the basis of the above, a Vietnamese phoneme text encoder with Vietnamese tone features is integrated, so that the model can pay more attention to the characteristics of tone changes in Vietnamese speech while learning contextual semantic information.

[0004] The technical solution of the present invention is: a method for constructing Vietnamese speech recognition corpus, the method comprising a training process and a parsing process, and the specific steps are as follows:

[0005] The training process includes:

[0006] (1) Collect speech and text containing Vietnamese language with a length of less than 20 seconds as training and test corpora;

[0007] (2) The open source speech recognition dataset was preprocessed, including data cleaning, which included removing punctuation, numbers, English, and illegal characters, unifying Unicode encoding, and converting Vietnamese text into phonemes using the Vietnamese phoneme conversion tool Viphoneme.

[0008] (3) Preprocess the audio and remove the silent segments;

[0009] (4) constructing a phoneme-granular vocabulary based on step (2);

[0010] (5) Fine-tune the pre-trained speech encoder, where the speech encoder uses the pre-trained model of Wav2vec2, which is composed of a 7-layer convolutional neural network and a 12-layer transformer network. It can shorten the sequence length of speech and encode the speech into a vector with a hidden layer dimension of 768. The model needs to be pre-trained using unsupervised speech and then fine-tuned using labeled Vietnamese speech recognition data.

[0011] (6) The pre-trained text encoder is re-pre-trained, where the text encoder uses the pre-trained text encoder Bert. The model adopts smaller model parameters than the original Bert, and the number of hidden layers is 4.

[0012] (7) Initialize the parameters of the fine-tuned speech encoder and pre-trained text encoder to the speech encoder and text encoder of the alignment model.

[0013] (8) Use CTC-mixed cross attention to jointly train speech features and text features; use mixed cross attention to train speech representation X S and the text representation Y P Perform semantic alignment, where the obtained speech representation is in Represents the speech encoder, the text features obtained in For the text encoder, the parameters of the local attention of the cross attention are divided into chunks Global attention parameters Divide into blocks to obtain non-overlapping subsequences For example Where G is the number of groups, T is the sequence length, and C is the length of the subsequence. Similarly, the A matrix represents the similarity matrix between the speech sequence and the text sequence;

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] H = concatenate[Z X ,O g ] (9)

[0023] As shown in formula (1), the parameters in the local attention is represented by the phonetic representation Z X Affine transformation is obtained, where W Q ,W K , is the model linear layer parameter; as shown in formula (2) Through By group splitting and completing, for Similarity matrix; As shown in formula (3), the similarity matrix of local attention parameters is calculated through the relu nonlinear activation function; As shown in formula (4), the similarity matrix is ​​mapped to between 0 and 1 through the softmax function, and the cross attention mechanism model will learn an alignment matrix Where N is the length of the text sequence and T is the length of the speech sequence, which is used to align the speech sequence and the phoneme text sequence; as shown in formula (5), by paying attention to weighting, we get As shown in formula (6) and formula (7), the global attention is the same; as in formula (7), the local features are added to the global features and then affine transformation is performed to obtain O g ; As shown in formula (9), the speech representation Z X Concatenate the features obtained by formula (8) to obtain H, where H is used for CTC loss function training; Local query vector;

[0024] The correlation vector between the local queried information and other information;

[0025] By local query vector The correlation vector between the local queried information and other information The calculated local attention similarity matrix; b is the bias matrix;

[0026] Local information matrix;

[0027] The weighted local information matrix is ​​obtained by calculating the local attention similarity matrix and the local information matrix;

[0028] Global query vector;

[0029] The correlation vector between the global queried information and other information;

[0030] Global information matrix;

[0031] Weighted global information matrix;

[0032] W o : The learnable parameter matrix in the neural network model.

[0033] U g : Gating parameter matrix in neural network.

[0034] The parsing process includes:

[0035] (9) Send a piece of speech and text into the trained alignment model;

[0036] (10) The model uses CTC-mixed cross attention to perform secondary semantic alignment on the alignment results to obtain the alignment results. For the results of the first CTC alignment with confidence less than the threshold, hybrid cross attention needs to be used for secondary semantic alignment; There are two decoder outputs in the Vietnamese speech-text alignment model, namely the probability matrix of the Vietnamese speech-text Vietnamese phoneme vocabulary output by wav2vec through the linear layer Where C is the size of the vocabulary; the Vietnamese phoneme-to-phoneme Vietnamese phoneme-text similarity matrix obtained by hybrid cross attention The character probability k corresponding to each frame is obtained from the probability matrix F j,t =p(c j |t,F)=F c,t ,c∈[1,N],t∈[1,T], so the maximum joint probability of alignment is obtained by using the dynamic programming algorithm as shown in formula (10);

[0037]

[0038] Where p(blank|t) is the probability that the current time t is a blank character. Set the transition probability of the first character to zero, and then calculate the maximum transition probability when the next character is a blank character or the next character. Let the current last character probability be t = argmax t′ (k′t ,N-1), and then by backtracking the transition probability matrix k j,t Find a path with the highest probability corresponding to the character and the frame, and get the alignment relationship S t , as shown in formula (11);

[0039]

[0040] By minimizing the average probability, we can penalize the probability of wrong matching and obtain the alignment relationship S. att =[x1=y1,…,x i =y j ,x i +1 = y j ,…,x t =y n ]; N: the length of the current text, a positive integer; j: the subscript of the probability matrix F, 0≤j≤N and j is an integer; x i :i is a subscript, indicating the speech frame at a certain moment, 0≤i≤T, where T is the frame length of the speech, y j : j is a subscript, indicating the character corresponding to a certain moment, 0≤j≤N. N is the length of the text; x t :t is a subscript, indicating the speech frame corresponding to time t, 0≤t≤T. Where T is the frame length of the speech; y n :n is a subscript, indicating the nth character, 0≤n≤N. N is the length of the text.

[0041] The beneficial effects of the present invention are as follows: the present method proposes a method for constructing Vietnamese speech recognition corpus. Aiming at the problem of misalignment caused by noise data such as text addition or deletion in Vietnamese speech-text alignment tasks, the present invention combines Vietnamese acoustic tone features and proposes a dual decoder realignment method based on CTC-hybrid cross attention of a text encoder incorporating Vietnamese tone features, which effectively reduces the phenomenon of semantic alignment misalignment in alignment tasks and improves the alignment robustness of the model in noisy data. Experiments show that the alignment method based on the hybrid cross attention mechanism helps the model pay attention to speech-text alignment information locally and globally, and the use of contrastive learning enables the model to learn the semantic alignment information between speech-text modalities and reduce the gap between different speech-text modalities. Description of the drawings:

[0042] Figure 1 Schematic diagram of the alignment model training process of the Vietnamese speech recognition corpus construction method proposed in the present invention;

[0043] Figure 2 It is a schematic diagram of the decoding and parsing process of the alignment model of the Vietnamese speech recognition corpus construction method proposed in the present invention;

[0044] Figure 3 It is a structural diagram of a hybrid attention model of a Vietnamese speech recognition corpus construction method proposed in the present invention;

[0045] Figure 4 It is a schematic diagram of the accelerated experiment of cross attention and mixed attention training of the present invention.

[0046] Specific implementation method:

[0047] Embodiment 1, as Figure 1-Figure 4 As shown, a method for constructing Vietnamese speech recognition corpus, the specific steps of the method are as follows:

[0048] Step 1: Collect the corresponding speech and Vietnamese text as training and testing corpora; specifically, use the open source dataset VIVOS, which contains 11,660 parallel sentence pairs;

[0049] Table 1 shows the data set settings.

[0050]

[0051] Step 2: Preprocess the text of the dataset and clean the Vietnamese text data. Data cleaning includes removing punctuation, numbers, English and illegal characters, and unifying Unicode encoding. Use the open source Vietnamese text to phoneme tool Viphoneme to obtain Vietnamese phoneme sequences;

[0052] Step 3. For audio, all audio files are resampled into wav format files with a bit rate of 16kHz using the computer automatic text processing tool sox;

[0053] Step 4: Construct a vocabulary of phoneme granularity based on Step 2;

[0054] Step 5. Fine-tune the Vietnamese Wav2vec2 speech encoder on the Vietnamese supervised dataset and pre-train the Vietnamese text Bert encoder on the unsupervised data.

[0055] Step 6. Initialize the parameters of the fine-tuned speech encoder and pre-trained text encoder to the speech encoder and text encoder of the alignment model;

[0056] Step 7: Use CTC-hybrid cross attention to jointly train speech features and text features;

[0057] Using hybrid cross attention to speech representation X S and the text representation Y P Perform semantic alignment, where the obtained speech representation is in Represents the speech encoder, the text features obtained in For the text encoder, the parameters of the local attention of the cross attention are divided into chunks Global attention parameters Divide into blocks to obtain non-overlapping subsequences For example Where G is the number of groups, T is the sequence length, and C is the length of the subsequence. Similarly, the A matrix represents the similarity matrix between the speech sequence and the text sequence;

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] H = concatenate[Z X ,O g ] (9)

[0067] As shown in formula (1), the parameters in the local attention is represented by the phonetic representation Z X Affine transformation is obtained, where W Q ,W K , is the model linear layer parameter; as shown in formula (2) Through By group splitting and completing, for Similarity matrix; As shown in formula (3), the similarity matrix of local attention parameters is calculated through the relu nonlinear activation function; As shown in formula (4), the similarity matrix is ​​mapped to between 0 and 1 through the softmax function, and the cross attention mechanism model will learn an alignment matrix Where N is the length of the text sequence and T is the length of the speech sequence, which is used to align the speech sequence and the phoneme text sequence; as shown in formula (5), by paying attention to weighting, we get As shown in formula (6) and formula (7), the global attention is the same; as in formula (7), the local features are added to the global features and then affine transformation is performed to obtain O g ; As shown in formula (9), the speech representation Z X Concatenate the features obtained by formula (8) to obtain H, where H is used for CTC loss function training;

[0068] Local query vector;

[0069] The correlation vector between the local queried information and other information;

[0070] By local query vector The correlation vector between the local queried information and other information The calculated local attention similarity matrix; b is the bias matrix;

[0071] Local information matrix;

[0072] The weighted local information matrix is ​​obtained by calculating the local attention similarity matrix and the local information matrix;

[0073] Global query vector;

[0074] The correlation vector between the global queried information and other information;

[0075] Global information matrix;

[0076] Weighted global information matrix;

[0077] W o : The learnable parameter matrix in the neural network model.

[0078] U g : Gating parameter matrix in neural network.

[0079] Step 8. Send a piece of speech and text into the trained alignment model;

[0080] Step 9: The model performs secondary semantic alignment on the alignment results through CTC-mixed cross attention to obtain the alignment results. For the results of the first CTC alignment with confidence less than the threshold, hybrid cross attention needs to be used for secondary semantic alignment; There are two decoder outputs in the Vietnamese speech-text alignment model, which are the probability matrix of the Vietnamese speech-text Vietnamese phoneme vocabulary output by wav2vec through the linear layer Where C is the size of the vocabulary; the Vietnamese phoneme-to-phoneme Vietnamese phoneme-text similarity matrix obtained by hybrid cross attention The character probability k corresponding to each frame is obtained from the probability matrix F j,t =p(c j |t,F)=F c,t ,c∈[1,N],t∈[1,T], so the maximum joint probability of alignment is obtained by using the dynamic programming algorithm as shown in formula (10);

[0081]

[0082] Where p(blank|t) is the probability that the current time t is a blank character. Set the transition probability of the first character to zero, and then calculate the maximum transition probability when the next character is a blank character or the next character. Let the current last character probability be t = argmax t′ (k′t,N-1), and then backtrack the transition probability matrix k j,t Find a path with the highest probability corresponding to the character and the frame, and get the alignment relationship S t , as shown in formula (11);

[0083]

[0084] By minimizing the average probability, we can penalize the probability of wrong matching and obtain the alignment relationship S. att =[x1=y1,…,x i =y j ,x i +1 = y j ,…,x t =y n ]; N: the length of the current text, a positive integer;

[0085] j: is the subscript of the probability matrix F, 0≤j≤N and j is an integer; x i :i is a subscript, indicating the speech frame at a certain moment, 0≤i≤T, where T is the frame length of the speech, y j : j is a subscript, indicating the character corresponding to a certain moment, 0≤j≤N. N is the length of the text; x t :t is a subscript, indicating the speech frame corresponding to time t, 0≤t≤T. Where T is the frame length of the speech; y n :n is a subscript, indicating the nth character, 0≤n≤N. N is the length of the text.

[0086] With the help of CTC-hybrid attention alignment method, the audio and its corresponding text are aligned at the phoneme level to obtain the audio time segment corresponding to each word in the text. For the original audio corresponding to the text sequence, the audio and text are first forced to be aligned. This operation allows us to obtain the approximate alignment of the audio frames corresponding to each word in the text sequence. For example:

[0087] Text sequence:

[0088] Audio alignment: 1.080-1.880

[0089] 1.880-2.320

[0090] 2.320-2.7840

[0091] The present invention uses a pre-trained model for and extracts speech and text features. Table 2 shows the accuracy of speech-text alignment experiments using different algorithms.

[0092] Table 2 Results of Vietnamese speech-text alignment using different algorithms

[0093]

[0094] The experimental results are shown in Table 2. Compared with the traditional hidden Markov model alignment model MFA, Gentle, and CTC-hybrid cross attention model, the F1 values ​​on the noisy dataset are greatly improved, which fully reflects the advantages of neural network alignment. Compared with the baseline model of the present invention, the F1 value on the VIVOS clean dataset differs by about 0.5%, but the accuracy can be improved by 8% with the VIVOS noisy dataset, which means that in the noisy scene, the CTC-hybrid cross attention mechanism can effectively reduce the phenomenon of alignment misalignment and improve the quality of corpus construction.

[0095] The present invention mentions that the theoretical time complexity of the hybrid attention mechanism is smaller than that of the ordinary attention mechanism, which can accelerate the training speed and reduce the time cost for large-scale data training. In order to explore the acceleration of model training under the same number of training steps of linear cross-mixed attention and cross-attention, a speed pair experiment was designed. The aligned models of the cross-attention and linear hybrid experiments were trained on the VIVOS dataset for 2400 steps using a Tesla T4 graphics card. The loss of the two models and the average running time of every 100 steps were recorded. In order to keep the data level of the loss training time consistent, the loss value was displayed with a reduction of 100 times, as shown in the figure. Figure 4 As shown. Figure 4There are two groups of comparisons in the paper, namely the training speed comparison between linear mixed cross attention and cross attention, and the loss comparison between linear mixed cross attention and cross attention. The experiment shows that when the loss values ​​of linear mixed cross attention and cross attention are similar, the average training time per step of linear mixed cross attention is 9.9 seconds, and the average training time per step of cross attention is 22.57 seconds. That is to say, under similar performance, the training speed of linear mixed cross attention is about 2.27 times that of cross attention.

[0096] Table 3 Mixed attention ablation experiment

[0097] Experiment number Phoneme Word Embedding Cross-Attention Mixed Attention Contrastive loss CTC Decoding Accuracy Recall F1 Experiment 1 - - √ √ √ 0.64 0.65 0.64 Experiment 2 √ - √ √ - 0.60 0.60 0.60 Experiment 3 √ - - √ √ 0.58 0.56 0.57 Experiment 4 - √ - √ - 0.59 0.59 0.59 Experiment 5 - - √ √ - 0.58 0.59 0.58 Experiment 6 - √ - - - 0.55 0.55 0.55 Experiment 7 - - √ - - 0.56 0.56 0.56

[0098] The experimental results are shown in Table 3. Compared with Experiment 5 and Experiment 6, hybrid attention uses a linear attention mechanism, which effectively reduces the training time when the accuracy is similar. The use of linear hybrid attention in the alignment model is about 2.27 times faster than ordinary cross attention. The F1 values ​​of Experiment 4 and Experiment 6 are 7% and 3% higher than those of Experiment 5 and Experiment 7, respectively, indicating that adding contrast loss is beneficial for the model to reduce the differences between different modalities. In Experiment 2 and Experiment 5, Experiment 2 combines the characteristics of Vietnamese tones, and the F1 value of Vietnamese tone information is increased by 3% compared with that without tone features, proving that the integration of tone information into the Vietnamese phoneme encoder helps to improve the alignment accuracy. In Experiment 1 and Experiment 2, Experiment 1 uses CTC-Attention joint decoding, which improves the F1 value by 6%, proving that Attention secondary realignment helps to correct misaligned alignment when Vietnamese speech and text are not completely consistent.

Claims

1. A method for constructing Vietnamese speech recognition corpus, characterized in that: The method comprises two processes and 10 steps as follows: The training process includes: (1) Collecting speech and text corresponding to Vietnamese as training and testing corpora; (2) Preprocess the text of the dataset and convert the Vietnamese text into Vietnamese phonemes; (3) Preprocess the audio and remove the silent segments; (4) constructing a phoneme-granular vocabulary based on step (2); (5) Fine-tune the pre-trained speech encoder; (6) Re-pretraining the pre-trained text encoder; (7) Initializing the parameters of the fine-tuned speech encoder and the pre-trained text encoder to the speech encoder and text encoder of the alignment model; (8) Use CTC-hybrid cross attention to jointly train speech features and text features; The parsing process includes: (9) Send a piece of speech and text into the trained alignment model; (10) The model performs secondary semantic alignment on the alignment results through CTC-hybrid cross attention to obtain the alignment results; In (8), a hybrid cross attention is used to represent the speech representation X S and the text representation Y P Perform semantic alignment, where the obtained speech representation is in Represents the speech encoder, the text features obtained in For the text encoder, the parameters of the local attention of the cross attention are divided into chunks Global attention parameters Divide into blocks to obtain non-overlapping subsequences A matrix represents the similarity matrix between the speech sequence and the text sequence; H=concatenate[Z X ,0g] (9) As shown in formula (1), the parameters in the local attention is represented by the phonetic representation Z X Affine transformation is obtained, where W Q , W K , is the model linear layer parameter; as shown in formula (2) Through By group splitting and completing, for Similarity matrix; As shown in formula (3), the similarity matrix of local attention parameters is calculated through the relu nonlinear activation function; As shown in formula (4), the similarity matrix is ​​mapped to between 0 and 1 through the softmax function, and the cross attention mechanism model will learn an alignment matrix Where N is the length of the text sequence and T is the length of the speech sequence, which is used to align the speech sequence and the phoneme text sequence; as shown in formula (5), by paying attention to weighting, we get As shown in formula (6) and formula (7), the global attention is the same; as shown in formula (7), the local features are added to the global features and then affine transformation is performed to obtain O g ; As in formula (9), the speech representation Z X The features obtained by concatenating with formula (8) are obtained to obtain H, where H Used for CTC loss function training; Local query vector; The correlation vector between the local queried information and other information; By local query vector The correlation vector between the local queried information and other information The calculated local attention similarity matrix; b is the bias matrix; Local information matrix; The weighted local information matrix is ​​obtained by calculating the local attention similarity matrix and the local information matrix; Global query vector; The correlation vector between the global queried information and other information; Global information matrix; Weighted global information matrix; W o : The learnable parameter matrix in the neural network model; U g : Gating parameter matrix in neural network; In (10), for the result of the first CTC alignment confidence less than the threshold, it is necessary to use hybrid cross attention for secondary semantic alignment; there are two decoder outputs in the Vietnamese speech-text alignment model, namely, the probability matrix of the Vietnamese speech-text Vietnamese phoneme vocabulary output by wav2vec after the linear layer Where C is the size of the vocabulary; the Vietnamese phoneme-to-phoneme Vietnamese phoneme-text similarity matrix obtained by hybrid cross attention The character probability k corresponding to each frame is obtained from the probability matrix F j，t = p(c j |t, F) = F c , t, c∈[1, N], t∈[1, T], the maximum joint probability of alignment is obtained by using the dynamic programming algorithm as shown in formula (10); Where p(blank|t) is the probability that the current time t is a blank character. Set the transition probability of the first character to zero, and then calculate the maximum transition probability when the next character is a blank character or the next character. Let the current last character probability be t = argmax t′ (k′ t , N-1), and then by backtracking the transition probability matrix k j，t Find a path with the highest probability corresponding to the character and the frame, and get the alignment relationship S t , as shown in formula (11); By minimizing the average probability, we can penalize the probability of wrong matching and obtain the alignment relationship S. at t=[x1=y1,...,x i =y j , x i +1=y j , ..., x t =y n ]; N: the length of the current text, a positive integer; j: the subscript of the probability matrix F, 0≤j≤N and j is an integer; x i : i is a subscript, indicating the speech frame at a certain moment, 0≤i≤T, where T is the frame length of the speech, y j : j is a subscript, indicating the character corresponding to a certain moment, 0≤j≤N. N is the length of the text; xt: t is a subscript, indicating the speech frame corresponding to time t, 0≤t≤T. T is the frame length of the speech; y n : n is the subscript, indicating the nth character, 0≤n≤N. N is the length of the text.

2. The method for constructing Vietnamese speech recognition corpus according to claim 1, characterized in that: In (2), the open source speech recognition dataset is preprocessed, including data cleaning, which includes removing punctuation, numbers, English and illegal characters, unifying Unicode encoding, and converting Vietnamese text into phonemes using the Vietnamese phoneme conversion tool Viphoneme.

3. The method for constructing Vietnamese speech recognition corpus according to claim 1, characterized in that: In (5), the speech encoder uses the pre-trained model of Wav2vec2, which is composed of a 7-layer convolutional neural network and a 12-layer transformer network. It can shorten the sequence length of speech and encode the speech into a vector with a hidden layer dimension of 768. The model needs to be pre-trained using unsupervised speech and then fine-tuned using labeled Vietnamese speech recognition data.

4. The method for constructing Vietnamese speech recognition corpus according to claim 1, characterized in that: In (6), the text encoder uses a pre-trained text encoder Bert, which has smaller model parameters than the original Bert and has 4 hidden layers.

Images