Speech-to-text translation method from Vietnamese to English based on multi-feature fusion

Abstract

The present invention proposes a Vietnamese to English speech-to-text translation method based on multi-feature fusion, which belongs to the field of artificial intelligence. First, three speech features, namely Fbank features, Wav2vec2 features and Pitch features, are extracted for Vietnamese audio. Fbank features are used as the main and Pitch features are used as the auxiliary. The spectral representation of pitch perception is obtained by effective mixing through alternating coding, and the Wav2vec2 features are encoded to obtain self-supervisory representation. Based on the difference between spectral representation and self-supervisory representation, the two types of features are fused using a cross-attention mechanism to construct a Vietnamese-English speech translation device, which can effectively model the acoustic information and semantic information in Vietnamese speech and realize Vietnamese-English speech translation. Experimental results show that the constructed speech translation device achieves good translation effect, and the BLEU value of speech translation from Vietnamese to English reaches 39.56.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for translating speech from Vietnamese to English into text. Background Art

[0002] Speech-to-text translation aims to translate the speech of the source language into text in the target language. It is widely used in various fields such as speeches, meetings, overseas travel, and simultaneous interpretation. Building a speech translation dataset requires high requirements on annotators and costs. Currently, only a few language pairs have public corpora, such as English to German, French and a few European languages. Most language pairs lack speech translation annotated corpora, such as Vietnamese. There is no public speech translation dataset for English, and research work is relatively scarce. There is an urgent need to carry out relevant research.

[0003] End-to-end speech translation uses a model to directly map source language speech to target language text, avoiding the inherent defects of the cascade method, such as error accumulation and high latency, and has therefore attracted much attention from researchers. End-to-end speech translation models simultaneously perform cross-modal and cross-language mapping, and training data is relatively scarce, so there is still a large gap in translation performance compared to cascade models. In addition, speech data is subject to variability due to factors such as speaker emotions, volume, accent, and external noise, which limits the performance of end-to-end speech translation models. Therefore, exploring effective speech representations is crucial for speech translation tasks.

[0004] Feature extraction is an important step in speech translation. The quality of features directly affects the translation effect. In speech translation, artificially designed Fbank features or self-supervised Wav2vec2 features are often used as model input. The extraction process of Fbank features is based on a frame division and is transformed frame by frame. Under the settings of sampling rate of 16K, frame length of 25m5, and frame shift of 10, each frame of Fbank% features covers 400 sampling points, which can represent the local features in speech acoustic information. However, the Fbank feature extraction method is pre-set according to human voice and cannot be dynamically learned, which has certain limitations. The Wav2vec2 model extracts features by sequentially calculating layer by layer through stacking 7 layers of convolutional neural networks with different step lengths and convolution kernel sizes. Each frame of Wav2vec2 features covers 3200 audio sampling points. The stacked CNN increases the coverage of speech acoustic information, which is conducive to the representation and learning of semantic information in speech. However, since the representation is self-supervised pre-trained on large-scale unlabeled speech, the performance of the representation on the target task is highly dependent on the correlation between the training domain and the target domain. When a single Fbank or Wav2vec2 feature is used as the input of the model, it cannot meet the model's requirement of modeling both acoustic and semantic information in speech. Summary of the invention

[0005] The purpose of the present invention is to provide a Vietnamese to English speech to text translation method based on multi-feature fusion. The method extracts multiple speech features from the same speech data, and effectively fuses the features based on the differences between the features, so as to prompt the model to learn richer coding representations, alleviate the problem that a single speech feature in speech translation is insufficient in modeling both acoustic and semantic information, and improve the robustness of speech representation for complex speech data.

[0006] The technical solution of the present invention is a method for translating speech from Vietnamese to English into text based on multi-feature fusion. The specific steps of the method are as follows:

[0007] Step 1, feature extraction: extract artificial filter bank Fbank spectrum features and pitch Pitch features, as well as self-supervised features Wav2vec2 for Vietnamese audio sequences;

[0008] Step 2, construction of encoder for multi-feature fusion: Based on Step 1, different encoding methods are selected to encode different types of features; convolutional encoding is performed on self-supervised features, and alternating encoding is performed using Transformer block-based alternating encoding blocks for artificial spectral features and pitch features. The two types of encoded representations of self-supervised features, spectral features, and pitch features are fused using a fusion layer based on a cross-attention mechanism to obtain the final encoded output representation;

[0009] Step 3, construction of a Vietnamese-to-English speech translation device with multi-feature fusion: Based on Step 1 and Step 2, feature extraction is first performed, and after inputting into a multi-feature fusion encoder to obtain an output representation, the encoded output representation vector is input into a Transformer decoder to obtain the probability distribution of the vector sequence on the target vocabulary, and the MBGD (Mini-Batch Gradient Descent) algorithm is used to train the speech translation model and construct a Vietnamese-to-English speech translation device;

[0010] Step 4, multi-feature fusion Vietnamese to English speech translation: Based on Step 1, Step 2, and Step 3, the three features of Fbank, Wav2vec2, and Pitch are first extracted from the Vietnamese speech to be translated, and the three types of features are used as input vectors of the translation model, and the final translation result is obtained using the translation device.

[0011] As a further solution of the present invention, the specific steps of Step 1 are:

[0012] Step 1.1: Extract 80-dimensional Fbank features, 1-dimensional Pitch features and 512-dimensional Wav2vec2 features from the Vietnamese audio sequence with a sampling rate of 16000. The frame shift of the Fbank feature extraction is set to 10ms, the window size of each frame is 25ms, and an 80-dimensional filter group is used. The Pitch feature uses the fundamental frequency feature of the speech signal, and the search frequency range is set to 50Hz to 400Hz. The Wav2vec2 feature uses the Wav2vec2 model pre-trained on 100 hours of Vietnamese audio to extract 512-dimensional speech features.

[0013] As a further solution of the present invention, the specific steps of Step 2 are:

[0014] Step 2.1: For Fbank features and Pitch features, based on the Transformer architecture, Fbank coding blocks and Fbank-Pitch hybrid coding blocks are used for alternating coding. In an alternating cycle C, C-1 Fbank coding blocks and 1 Fbank-Pitch hybrid coding block are included. Each coding block encodes the Fbank features, and the Pitch features are encoded at intervals of multiple blocks to obtain a spectrum coding representation.

[0015] Step 2.2: Encode the Wav2vec2 features based on CNN to obtain self-supervised encoding representation;

[0016] Step 2.3: For the spectral representation and self-supervised representation output by Step 2.1 and Step 2.2, the cross-attention mechanism is used to align and fuse the two types of representations to achieve complementarity and enhancement of different types of representations.

[0017] As a further solution of the present invention, the specific steps of Step 3 are:

[0018] Step 3.1: Use a multi-feature fusion encoder and Transformer decoder to build a Vietnamese-English speech translation model, and use Vietnamese-English speech translation data for training; extract Fbank features, Wav2vec2 features, and Pitch features from Vietnamese audio as encoder input; use an alternating feature encoding layer to perform mixed encoding on the Fbank and Pitch spectral features to output a spectral representation, and enhance the model's ability to represent semantic information by adding Vietnamese pitch information; use the Wav2vec2 encoding layer to encode the Wav2vec2 features to output a self-supervised representation; input the spectral representation and self-supervised representation into the representation fusion layer, learn the alignment and fusion between representations through the cross-attention mechanism, achieve complementarity between different types of representations, and obtain the final encoder output representation. Finally, input the encoder output representation into the decoder to output the target language text word sequence.

[0019] As a further solution of the present invention, the specific steps of Step 4 are:

[0020] Step 4.1: When performing speech translation from Vietnamese to English, three types of Fbank, Pitch and Wav2vec2 speech features are first extracted from the input audio. The translation device uses the three features as encoder input, realizes multi-feature fusion through cross-attention mechanism and alternating encoding, and obtains the encoder-side output representation containing acoustic and semantic information. This representation and the target text word embedding are jointly input into the decoder for decoding, and the probability distribution of the target language text word sequence is output.

[0021] The beneficial effects of the present invention are as follows: the method of the present invention extracts multiple speech features from the same speech data, and based on the differences between the features, effectively fuses the features, prompts the model to learn richer coding representations, and constructs a Vietnamese-English speech translation device. The device can effectively model the acoustic information and semantic information in Vietnamese speech, realize Vietnamese-English speech translation, and the constructed speech translation device achieves good translation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the encoder of the multi-feature fusion speech translation model proposed in the present invention;

[0023] Figure 2 is the alternating feature encoding layer;

[0024] Figure 3 Cross-attention mechanism in representation fusion layer. DETAILED DESCRIPTION

[0025] Embodiment 1, as Figure 1-Figure 3As shown, a Vietnamese to English speech to text translation method based on multi-feature fusion, the specific steps of the method are as follows:

[0026] (1) Collection and organization of training data:

[0027] Collect English texts corresponding to Vietnamese speech and Vietnamese texts, collect Vietnamese audio and Vietnamese texts from the public VLSP2019 corpus, and obtain the English texts corresponding to Vietnamese texts by calling Google text translation service. Thus, the obtained data is expressed as a five-tuple SEU, namely:

[0028] SEU=(SPEECH_VI,TEXT_VI,TEXT_EN,DURATION)

[0029] Among them, SPEECH_VI is the Vietnamese voice, TEXT_VI is the language text, TEXT_EN is the English text, and DURATION is the Vietnamese audio length.

[0030] The collected data set is divided into training set, validation set and test set. The Vietnamese audio data contains about 416 hours in total, of which about 395.5 hours are divided into training set, which contains 300k sentences in total; about 13.1 hours are divided into development set, which contains 10k sentences in total; and 7.2 hours are divided into test set, which contains 5.5k sentences in total.

[0031] (2) Feature extraction and data preprocessing of Vietnamese audio data:

[0032] Audio feature extraction is an important basic link in building a speech translation model. The model needs to model both acoustic information and semantic information based on the input speech features. It is very challenging to use a single feature to model both types of information. Three features, Fbank, Pitch and Wav2vec2, are extracted from the audio sequence. Among them, Fbank and Pitch are artificial features, and Wav2vec2 is a feature extracted by the self-supervised method. Input speech signal sequence Extract Fbank feature sequences Extract Pitch feature sequence Extract Wav2vec2 feature sequence The feature extraction process is as follows:

[0033]

[0034] The Fbank feature uses the torchaudio package, sets the frame shift to 10ms, the frame window size to 25ms, and extracts the 80-dimensional Fbank feature sequence as follows: Among them is d f Fbank feature dimension, l f is the sequence length; the Pitch feature is extracted using the SWIPE algorithm in the pySPTK tool, the search frequency range is set to 50Hz to 400Hz, and the extracted Pitch feature sequence is in d p is the Pitch feature dimension, l p is the sequence length; the Wav2vec2 feature is pre-trained on 100 hours of Vietnamese audiobooks using the open source w2v-vi model 3, and the 512-dimensional vector output by the 7th layer CNN of the model is used for experiments. The feature sequence is in d w is the feature dimension of Wav2vec2, l w is the sequence length.

[0035] For the Fbank feature of speech, the LB strategy of SpecAugment is used on the training set to enhance the speech data to ensure better generalization and robustness. Among them, the Fbank feature and Pitch feature sequence of speech are normalized by mean and variance. For the Wav2vec2 feature of speech, the open source w2v.vi model is used to extract 512-dimensional speech features for experiments, hereinafter referred to as w2v.vi features. Since the computational complexity of the self-attention layer of the ordinary Transrmer model is the square of the input length, in order to more effectively calculate the input data, a convolutional neural network is used in the experiment to downsample the input sequence. By setting different numbers of layers, the sequence length of the input model is reasonably controlled, so that the sequence lengths of different features are basically consistent. All convolutional layers use the same configuration, with a step size of 2, a convolution kernel size of 5, and a hidden layer dimension of 1024 for multi-layer convolution. Audio with less than 5 frames and more than 3000 frames is filtered.

[0036] For the target language text, case-sensitivity is maintained while punctuation is preserved. Sentences are segmented using the UnigramSentencesPiece model with a vocabulary size of , using 256-dimensional word embeddings and superimposed position embeddings.

[0037] (3) Construction process of Vietnamese to English speech translation encoder with multi-feature fusion:

[0038] Different from the previous speech translation model that encodes a single feature, the present invention proposes to encode three speech features, Fbank, Wav2vec2 and Pitch, at the encoding end. The model encoder consists of three parts: Wav2vec2 feature encoding layer, Fbank-Pitch alternating feature encoding layer and representation fusion layer. Among them, the Fbank-Pitch alternating feature encoding layer is used to explicitly add the Pitch feature to further assist the encoder in modeling semantic information. Figure 1 shown.

[0039] Fbank-Pitch alternating feature encoding layer: Fbank feature sequence f and its position encoding pos f After addition, they are used together with the Pitch feature sequence p as the input of the module to obtain the hidden layer state sequence h1, hereinafter referred to as the spectrum representation, as shown in formula (2).

[0040] h1=AlternatedEncoder(D(f)+pos f ,D(p)) (2)

[0041] In Vietnamese, tone can be used to distinguish the meaning of words, and the tone information in speech is represented by pitch features (Pitch). Explicitly adding tone information to speech features can enhance the modeling ability of the model for semantic information. Different from the previous work of using a single Fbank feature as the input of the Transformer coding block, the present invention uses the Fbank feature as the main feature and the Pitch feature as the auxiliary feature according to the speech and tone characteristics of Vietnamese, and uses two coding blocks to perform feature coding in an alternating manner. The two coding blocks include a Transformer coding block based on self-attention with Fbank as input, and a Transformer coding block based on cross-attention with Fbank and Pitch as input, hereinafter referred to as F feature coding block (F-Block) and FP mixed coding block (FP-Block). The alternating coding method integrates Pitch information for more effective coding without increasing the number of coding blocks and model complexity.

[0042]

[0043] As attached Figure 2As shown in the figure, the alternating feature encoding layer contains a total of L encoding blocks. If the alternating period is set to C, it contains (L / C) alternating periods. As shown in the above formula (3), there are C-1 F-Blocks and 1 FP-Block in each alternating period, where i is the number of blocks in the current encoding block. F-Block focuses on encoding Fbank features, while FP-Block uses a cross-attention mechanism to simultaneously encode Pitch features and Fbank features. Both F-Block and FP-Block encode Fbank features, and encode Pitch features at intervals of multiple blocks. This design is consistent with the actual pronunciation, that is, the distinction between word meaning and sentence meaning is mainly based on the pronunciation of different phonemes, supplemented by different pitches.

[0044] Wav2vec2 feature encoding layer: For the Wav2vec2 feature sequence w, CNN is used as the encoder, and the hidden state sequence h2 is obtained through dimension conversion, which is referred to as self-supervised representation below. The encoding process is as follows:

[0045] h2=Wav2vec2Encoder(w) (4)

[0046] Representation fusion layer: The input is the spectral representation h1 and the self-supervised representation h2, and the cross-attention mechanism is used to fuse and align the representations, and the fused representation vector is output. Without increasing the representation length and feature dimension, the alignment between the two representations is automatically learned through the cross-attention mechanism to complement and enhance each other. The feature fusion process can be formulated as follows (5):

[0047]

[0048] Fbank features extract features based on the human ear's perception of acoustic signals and manually design structures, which has limitations in extracting features from complex audio; the self-supervised representation obtained based on the self-supervised pre-training method lacks adaptability to specific tasks and data. In order to better represent the audio, the spectrum representation with pitch information and the self-supervised representation are fused in the encoder using a cross-attention mechanism, so that the features complement each other and meet the requirements of the speech translation task to model both acoustic and semantic information.

[0049] The spectral representation h1 output by the Fbank-Pitch alternating feature encoding layer and the self-supervised representation h2 output by the Wav2Vec2 feature encoding layer are fused through the multi-head cross attention mechanism. The calculation process of the multi-head cross attention is shown in the attached figure. Figure 3 As shown, the spectrum representation h1 is used as q, and the self-supervised representation h2 is used as k and v. First, the vectors Q, K, and V are obtained by linear transformation of the following formula (6), where Wi Q , W i K , W i V All are randomly initialized parameter matrices;

[0050] Q=qW i Q ,K=kW i K ,V=vW i V (6)

[0051] Then, the single-head attention is calculated by the following formula (7) to obtain the vector sequence head i , where d m The hidden dimension of the model is equal to the dimension of vectors Q, K and V;

[0052]

[0053] Then, the vector sequences of each head are concatenated by the following equation (8) to output the vector representation c of the audio sequence, where h is the number of heads of multi-head attention, and W is i O is a randomly initialized parameter matrix;

[0054] c=MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O (8)

[0055] Finally, c and h2 are added through the residual network and normalized by layer to obtain the final encoder output representation As shown in the following formula (9).

[0056]

[0057] (4) Vietnamese to English speech translation with multi-feature fusion: Based on steps (1), (2), and (3), in the encoder-decoder framework, Fbank features, Wav2vec2 features, and Pitch features are used as encoder inputs. Multi-feature fusion is achieved through a cross-attention mechanism and alternating encoding to obtain an encoder-end output representation containing acoustic and semantic information. This representation is combined with the target text word embedding h y The decoder is input together for decoding, and the probability distribution p(y|x) of the target language text word sequence is output, as shown in the following formula (10):

[0058] p(y|x)=softmax(h y ) (10)

[0059] In order to prove the effect of the present invention, the above method is tested on the data set used in the experiment, and the details are as follows:

[0060] Step a1: Fbank features, Pitch features and Wav2vec2 features should be extracted for the selected dataset respectively.

[0061] Step a2: Following Wang's end-to-end speech-to-text model setting, first use Fbank features and w2v2-vi features on the encoder-decoder model for speech translation and speech recognition tasks, respectively, and use a 2-layer convolutional network to downsample both features.

[0062] Table 1 shows the results of feature comparison experiments.

[0063]

[0064] The experimental results are shown in Table 1 above. The experimental results on the test set are shown in Table 1. The word error rate of the Fbank feature in the ASR task is 0.15 lower than that of the w2v2-vi feature, and the BLEU value in the ST task is 0.98 higher. When resources are relatively sufficient, the performance of the Fbank feature using Vietnamese speech in the ASR task and ST task is slightly better than that of the w2v2-vi feature. The Fbank feature performs better in speech translation tasks and speech recognition. This is because the calculation process of the Fbank feature is to use a manually designed structure to reduce noise and perform time-frequency analysis on the speech signal, which can achieve better results. The w2v2-vi feature achieves relatively poor results. This is because the feature encoder performs self-supervised learning on large-scale unlabeled speech data during the pre-training stage. The extracted features have good generalization for speech data, but lack adaptability to the target task and data set.

[0065] Step a3: To select the optimal alternating period, 20% of the training set is selected, and the alternating feature encoder is used as the encoder for preliminary speech translation experiments. The alternating feature coding block consists of 12 coding blocks. To ensure that the two coding blocks are evenly distributed, the alternating period C is selected from 2, 3, 4, and 6, respectively, and the Pitch feature coding ratio decreases successively. The experimental results are shown in Table 2, where FP / F represents the total ratio of FP-Block to F-Block in the encoder. It can be seen from the table that when the alternating period is 3, the encoder obtains the best translation effect, and this setting is used in the following experiments.

[0066] Table 2 BLEU values ​​of different alternating cycles C on the test set

[0067]

[0068] Step a4: To select the optimal feature fusion method, the commonly used fusion methods and different Wav2vec2 features were compared. The features extracted from the two Wav2vec2 models w2v2-vi and XLSR-53 were used as fusion features. The experiments were conducted using three fusion methods: splicing fusion-L, splicing fusion-F, and attention fusion. Splicing fusion-L splices the spectral representation h1 and the self-supervised representation h2 in the length dimension. The feature dimensions of the two features are the same, i.e., d w =d z , the vector finally encoded by concatenating in the length dimension As the encoder output, it is shown as follows:

[0069]

[0070] Concatenation Fusion-F concatenates h1 and h2 in the feature dimension, and applies 2 layers and 1 layer of convolution downsampling to Fbank features and Wav2vec2 features respectively, so that the length dimensions of the two are close to the same order of magnitude. The length difference mainly lies in the padding during convolution. When concatenating in the feature dimension, the feature length with the largest length is selected as the length of the final output representation. After concatenation, the output vector is twice the original vector in the feature dimension, and then the linear layer is applied to transform it to the original representation dimension. The vector finally encoded in this fusion method is As shown in the following formula.

[0071]

[0072] Table 3. Experimental results of fusion method comparison

[0073]

[0074] The experimental results are shown in Table 3. The BLEU value of the proposed model on the test set is consistently higher than the two baseline models of speech translation in Table 4, which shows that compared with a single Fbank feature or w2v2-vi feature, the encoding representation obtained by fusing the two types of features can improve the performance of the speech translation model. From the perspective of fusion methods, the BLEU value based on the attention fusion method is higher than that of the splicing fusion-L method and the splicing fusion-F method. The alignment relationship between the two features is learned through the cross-attention mechanism. The sequence of the encoded output remains unchanged in the dimension of the sequence length and the hidden layer dimension, and no additional computational overhead is added during decoding. The experimental results show that the attention fusion method is the best fusion method. From the perspective of feature type, the w2v2-vi feature is slightly better than the XLSR-53 feature in the attention fusion method and the splicing fusion-L method. The possible reason for this is that the pre-training corpus of w2v2-vi is 100 hours of Vietnamese audiobooks, which has an advantage in extracting Vietnamese audio features, while the training data of XLSR-53 is 53k hours of multilingual audio, of which Vietnamese accounts for a small proportion, so it may interfere with the feature extraction of Vietnamese audio. The performance of the splicing fusion-F method is much worse than that of the other two methods. The reason may be that the linear layer is used to reduce the dimension of the feature dimension after splicing it, and using a single linear layer to reduce the dimension of two spliced ​​features is not the optimal dimensionality reduction method.

[0075] Step a5: To verify the effectiveness of the proposed method, the Fairseq S2T model, the ST model with the encoder pre-trained by ASR, and the model with MT and ST multi-task joint training are used as baseline models for experiments on the dataset, hereinafter referred to as the Fairseq ST baseline, ST+ASR PT baseline, and MTL ST baseline. For fair comparison, none of the models uses additional data for pre-training or training. Among them, the losses of MT and ST in the MTL ST baseline are both NLL losses, with a weight distribution of 4:6, and the model is based on the Tranformer architecture. As shown in 4.1, the translation effect of Fbank features is better than Wav2vec2 features when resources are sufficient, so the baseline models all use Fbank features as input features.

[0076] Table 4. Feature fusion experimental results

[0077]

[0078] The experimental results are shown in Table 4. Compared with the Fairseq ST baseline model, the ST+ASR PT baseline uses the encoder parameters pre-trained by ASR to initialize the encoder of the ST model, making full use of monolingual data to learn the acoustic information in speech, which improves 1.18 BLEU values. The MTL ST baseline performs joint training of text translation tasks and speech translation tasks by sharing decoder parameters. Since the training process does not use additional data and the loss distribution of the joint training process leads to non-optimal performance of a single task, it decreases by 3.64 BLEU values ​​compared with the Fairseq ST baseline. The proposed model improves 0.79 BLEU values ​​compared to the optimal ST+ASR PT baseline and 1.97 BLEU values ​​compared to the FairseqST baseline using Fbnak features. The Fbank-Pitch alternating feature encoding layer and representation fusion layer are used to fuse w2v2-vi features and Pitch features. The differences between different features enable them to complement each other to obtain richer encoding representations, so the translation quality is further improved. In addition, compared with the ST+ASR PT baseline and the MTL ST baseline, the proposed model does not require additional pre-training or joint training steps for some modules of the model, and has higher training efficiency.

[0079] Step a6: Compared with the three end-to-end baseline models listed in Table 4, the present invention proposes an alternating feature encoder and a representation fusion layer to fuse additional Pitch features and Wav2vec2 features. This section conducts an ablation experiment on the proposed model to evaluate the contribution of different modules and additional features in the proposed method to the model performance. Table 5 shows the ablation experiment results.

[0080] Table 5 Ablation experiment results

[0081]

[0082] As shown in Table 5, the proposed different feature encoding modules and the fusion of different features can bring positive gains to the model performance, and the optimal result can be achieved by using all of them. The alternating fusion module effectively fuses the Pitch feature and the Fbank feature according to the proportion of the components that actually distinguish the semantics, which can bring an improvement of 0.59 compared with the method of directly splicing the Fbank feature and the Pitch feature in the feature dimension. Further removing the Pitch feature and directly using the self-attention encoding representation of Fbank, the translation performance continues to decrease by 0.11 BLEU values. This means that the Pitch feature of Vietnamese may contain semantic information that can improve the effect of speech translation, proving the necessity of the Pitch feature for speech modeling of tonal languages. The representation fusion layer deeply fuses the self-supervised representation and the spectral representation using the cross-attention mechanism, so that the two features complement each other to enhance the representation of the output of the encoder. Removing this layer and directly splicing the two representations in the feature dimension will reduce the performance by 1.97. Further removing the Wav2vec2 features, that is, only using the alternating fusion module to mix the Fbank features and the Pitch features, the translation performance will continue to drop by 0.43, which shows that the Wav2vec2 features can supplement the Fbank features, and this supplement is beneficial to improving the performance of speech translation.

Claims

1. A Vietnamese to English speech-to-text translation method based on multi-feature fusion, characterized in that: The specific steps of the method are: Step 1, feature extraction: extract the artificial filter bank Fbank spectrum features and pitch Pitch features, as well as the self-supervised feature Wav2vec2 for the Vietnamese audio sequence; Step 2, construction of encoder for multi-feature fusion: Based on Step 1, different encoding methods are selected to encode different types of features; convolutional encoding is performed on self-supervised features, and alternating encoding is performed using Transformer block-based alternating encoding blocks for artificial spectral features and pitch features. The two types of encoded representations of self-supervised features, spectral features, and pitch features are fused using a fusion layer based on a cross-attention mechanism to obtain the final encoded output representation; Step 3, construction of a Vietnamese-to-English speech translation device with multi-feature fusion: Based on Step 1 and Step 2, feature extraction is first performed, and after inputting into a multi-feature fusion encoder to obtain an output representation, the encoded output representation vector is input into a Transformer decoder to obtain the probability distribution of the vector sequence on the target vocabulary, and the MBGD algorithm is used to train the speech translation model to construct a Vietnamese-to-English speech translation device; Step 4, multi-feature fusion Vietnamese to English speech translation: Based on Step 1, Step 2, and Step 3, the three features of Fbank, Wav2vec2, and Pitch are first extracted for the Vietnamese speech to be translated, and the three types of features are used as input vectors of the translation model, and the final translation result is obtained using the translation device.

2. The method for Vietnamese to English speech to text translation based on multi-feature fusion according to claim 1, characterized in that: The specific steps of Step 1 are: Step 1.1: Extract 80-dimensional Fbank features, 1-dimensional Pitch features and 512-dimensional Wav2vec2 features from the Vietnamese audio sequence with a sampling rate of 16000. The frame shift of the Fbank feature extraction is set to 10ms, the window size of each frame is 25ms, and an 80-dimensional filter group is used. The Pitch feature uses the fundamental frequency feature of the speech signal, and the search frequency range is set to 50Hz to 400Hz. The Wav2vec2 feature uses the Wav2vec2 model pre-trained on 100 hours of Vietnamese audio to extract 512-dimensional speech features.

3. The method for Vietnamese to English speech to text translation based on multi-feature fusion according to claim 1, characterized in that: The specific steps of Step 2 are: Step 2.1: For Fbank features and Pitch features, based on the Transformer architecture, Fbank coding blocks and Fbank-Pitch hybrid coding blocks are used for alternating coding. In an alternating cycle C, C-1 Fbank coding blocks and 1 Fbank-Pitch hybrid coding block are included. Each coding block encodes the Fbank features, and the Pitch features are encoded at intervals of multiple blocks to obtain the spectrum coding representation. Step 2.2: Encode the Wav2vec2 features based on CNN to obtain self-supervised encoding representation; Step 2.3: For the spectral representation and self-supervised representation output by Step 2.1 and Step 2.2, the cross-attention mechanism is used to align and fuse the two types of representations to achieve complementarity and enhancement of different types of representations.

4. The method for Vietnamese to English speech to text translation based on multi-feature fusion according to claim 1, characterized in that: The specific steps of Step 3 are: Step 3.1: Use a multi-feature fusion encoder and Transformer decoder to build a Vietnamese-English speech translation model, and use Vietnamese-English speech translation data for training; extract Fbank features, Wav2vec2 features, and Pitch features from Vietnamese audio as encoder input; use an alternating feature encoding layer to perform mixed encoding on the Fbank and Pitch spectral features to output a spectral representation, and enhance the model's ability to represent semantic information by adding Vietnamese pitch information; use the Wav2vec2 encoding layer to encode the Wav2vec2 features to output a self-supervised representation; input the spectral representation and self-supervised representation into the representation fusion layer, learn the alignment and fusion between representations through the cross-attention mechanism, achieve complementarity between different types of representations, and obtain the final encoder output representation. Finally, input the encoder output representation into the decoder to output the target language text word sequence.

5. The method for Vietnamese to English speech to text translation based on multi-feature fusion according to claim 1, characterized in that: The specific steps of Step 4 are: Step 4.1: When performing speech translation from Vietnamese to English, three types of Fbank, Pitch and Wav2vec2 speech features are first extracted from the input audio. The translation device uses the three features as encoder input, realizes multi-feature fusion through cross-attention mechanism and alternating encoding, and obtains the encoder-side output representation containing acoustic and semantic information. This representation and the target text word embedding are jointly input into the decoder for decoding, and the probability distribution of the target language text word sequence is output.

Images