A method and apparatus for modeling an end-to-end speech translation model based on dual attention
By employing a dual attention mechanism in the end-to-end speech translation model, combined with the encoding of acoustic and text encoders, the problem of inaccurate speech translation results in existing technologies is solved, achieving more accurate speech translation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAONIU FANYI
- Filing Date
- 2022-12-28
- Publication Date
- 2026-07-14
AI Technical Summary
Existing speech translation methods use attention mechanisms to compute only the output of one encoder during decoding, which makes it impossible to correct for errors in speech representation compared to text representation in cross-modal and cross-linguistic generation tasks, resulting in inaccurate generation results.
An end-to-end speech translation model based on dual attention is adopted. It encodes the text using primary and advanced acoustic encoders and text encoders respectively, and uses the dual attention mechanism to fuse acoustic and text information at the decoding end. Information is obtained through serial or parallel processing to achieve accurate translation.
It achieves translation that simultaneously considers acoustic and textual information, improves the accuracy of the generated results, and significantly enhances the performance of the speech translation model.
Smart Images
Figure CN116011472B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and particularly to an end-to-end speech translation model modeling method and device based on dual attention. Background Art
[0002] When conversing in different languages, speakers need speech translation assistance. However, existing speech translation methods have problems such as error propagation, resulting in significant translation deviations. For example, if a "ma" is missing at the end of a sentence during recognition, the translation model may translate an interrogative sentence into a declarative sentence; or paralinguistic information in the speech is lost, i.e., during the process of speech recognition into text, information such as tone, emotion, and pitch in the speech is lost, and this information is usually not expressed in written form. The meaning of the same sentence can be very different in different tones. In addition, for some texts with multiple possible word segmentations, the information in the original speech may be more helpful in capturing the correct segmentation method.
[0003] Recently, researchers have proposed end-to-end speech translation. Since the end-to-end model has no intermediate output information, it is difficult to perform targeted optimization for problems that occur during the translation process. Especially in actual scenarios, the audio may contain a lot of noise, and the spoken sentence structure is very irregular. How to handle this situation is a difficulty and pain point for end-to-end systems in practical applications. In addition, a key advantage of speech recognition and text translation is the rich data accumulation. End-to-end speech translation is a new direction, and most of the datasets have been labeled in recent years. The lack of data volume is the biggest obstacle to the development of speech translation. Moreover, early speech translation methods only learned acoustic encoding and did not consider the learning of source language text encoding. In subsequent work, a common approach is to stack two encoders and complete the conversion from acoustic information to text information in between. However, in this process, acoustic information may be lost. Therefore, how to better learn the representations in both acoustic and text dimensions has become an urgent problem to be solved for optimizing the speech translation model. Summary of the Invention
[0004] In view of the above analysis, the present invention aims to provide an end-to-end speech translation model modeling method and device based on dual attention; to solve the problem that in the existing speech translation method, during decoding, only the output of one encoder is calculated through the attention mechanism; for a cross-modal and cross-language generation task such as speech translation, the error of the text representation cannot be corrected by the speech representation containing the original input information, resulting in inaccurate generation results. [[ID=第十八]]
[0005] The object of the present invention is mainly achieved by the following technical solutions:
[0006] On the one hand, this invention provides a method for modeling an end-to-end speech translation model based on dual attention, comprising the following steps:
[0007] Obtain a speech dataset; the speech dataset includes source language speech data, source language annotated text corresponding to the source language speech data, and target language annotated text.
[0008] An initial speech translation model is constructed, comprising a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder. The primary acoustic encoder is used to extract and encode features from the source language speech data. The high-level acoustic encoder and the text encoder encode based on the output of the primary encoder, respectively, to obtain the latent vectors of the high-level acoustic encoder and the text encoder. The decoder is used to decode the latent vectors of the high-level acoustic encoder and the text encoder based on a dual attention mechanism to obtain the target language translation text corresponding to the source language speech data.
[0009] The initial speech translation model is trained using the speech dataset, and then iteratively updated using a loss function to obtain the final speech translation model.
[0010] Furthermore, the decoder is a multi-layer Transformer structure;
[0011] The decoder is used to decode the latent vectors of the advanced acoustic encoder and the text encoder based on a dual attention mechanism, including:
[0012] In each Transformer layer of the decoder, a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationship and identical structure are set;
[0013] The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the latent vectors of the high-level acoustic encoder and the latent vectors of the text encoder, respectively. They perform attention calculations through either serial or parallel processing to obtain a tensor representation that fuses acoustic and text information.
[0014] Furthermore, the first encoder-decoder attention module and the second encoder-decoder attention module adopt a serial processing method, including:
[0015] The latent vector of the text encoder is input into the first encoder-decoder attention module for attention calculation. The output of the first encoder-decoder attention module is subjected to residual and layer regularization and then input into the second encoder-decoder attention module.
[0016] The second encoder-decoder attention module receives the output of the first encoder-decoder attention module and the latent vector of the high-level acoustic encoder, performs attention calculations, and obtains a tensor representation that integrates acoustic and textual information.
[0017] Furthermore, the first encoder-decoder attention module and the second encoder-decoder attention module adopt a parallel processing method, including: inputting the latent vector of the text encoder and the latent vector of the high-level acoustic encoder into the first encoder-decoder attention module and the second encoder-decoder attention module respectively for attention calculation, adding the outputs of the two attention modules, performing residual and layer regularization, and obtaining a tensor representation that fuses acoustic and text information.
[0018] Furthermore, before performing attention calculation using the text encoder latent vector and the advanced acoustic encoder latent vector, the method further includes: randomly discarding the representation of each position in the text encoder latent vector and the advanced acoustic encoder latent vector with a preset probability, and inputting the remaining text encoder latent vector and advanced acoustic encoder latent vector after random discarding into the corresponding attention module for attention calculation.
[0019] Furthermore, training the initial speech translation model using the speech dataset includes: inputting the source language speech data into the primary acoustic encoder;
[0020] The loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively.
[0021] The loss for the target language annotated text is calculated for the outputs of the text encoder and decoder, respectively.
[0022] By iteratively minimizing the loss, a converged speech translation model is obtained.
[0023] Furthermore, a converter is also included between the primary acoustic encoder and the text encoder;
[0024] The advanced acoustic encoder and text encoder encode based on the output of the primary encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector, including:
[0025] The output representation of the primary acoustic encoder is input into the advanced acoustic encoder, and encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector.
[0026] The output representation of the primary acoustic encoder is transformed into a text modality using the converter and input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the text encoder latent vector.
[0027] Furthermore, the step of using the converter to perform text modal conversion on the output representation of the primary acoustic encoder includes:
[0028] The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder;
[0029] By using the CTC prediction distribution, the word embedding matrix of the source language is weighted to obtain the primary text modality representation;
[0030] The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.
[0031] Furthermore, the step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the magnitude of the CTC distribution to obtain a text modality representation of the corresponding text length.
[0032] On the other hand, a computer device is also provided, including at least one processor and at least one memory communicatively connected to said processor;
[0033] The memory stores instructions that can be executed by the processor to implement the aforementioned end-to-end speech translation modeling method based on dual attention.
[0034] The beneficial effects of this technical solution are:
[0035] This invention utilizes an acoustic encoder and a text encoder to explicitly encode the acoustic and text representations separately. At the same time, it captures information from both acoustic and text representations at the decoding end through a dual attention mechanism, thereby achieving simultaneous consideration of acoustic and text information and making the generated results more accurate.
[0036] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description
[0037] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0038] Figure 1 This is a flowchart of the end-to-end speech translation modeling method based on dual attention, according to an embodiment of the present invention.
[0039] Figure 2 This is a schematic diagram of the speech translation model structure according to an embodiment of the present invention; Detailed Implementation
[0040] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0041] This embodiment presents a modeling method for an end-to-end speech translation model based on dual attention, such as... Figure 1 As shown, it includes the following steps:
[0042] Step S1: Obtain the speech dataset.
[0043] Specifically, this embodiment uses the MuST-C English-German dataset, which is commonly used in the field of speech translation and has annotations. The dataset includes source language speech data, source language annotation text corresponding to the speech data, and target language annotation text.
[0044] Step S2: Construct the initial speech translation model.
[0045] Specifically, such as Figure 2 As shown, the initial speech translation model in this embodiment includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder;
[0046] Among them, the primary acoustic encoder is used to extract and encode features from the source language speech data;
[0047] Specifically, the primary acoustic encoder can employ a 12-layer Transformer structure to encode the feature sequences of the input source language speech data. This embodiment uses fewer layers to encode the input audio data, reducing computational resources while ensuring reasonable top-level CTC prediction.
[0048] After primary encoding using a primary acoustic encoder, the advanced acoustic encoder and the text encoder encode based on the output of the primary encoder, respectively, to obtain the latent vectors of the advanced acoustic encoder and the text encoder.
[0049] Preferably, both the advanced acoustic encoder and the text encoder are multi-layer Transformer structures; the advanced acoustic encoder is used to further learn on the representation learned by the primary acoustic encoder, and is encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector.
[0050] A converter is also included between the primary acoustic encoder and the text encoder;
[0051] The output representation of the primary acoustic encoder is transformed into a text modality using a converter and then input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the latent vector of the text encoder.
[0052] Specifically, the converter is used to perform text modal conversion on the output representation of the primary acoustic encoder, including:
[0053] The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder;
[0054] The source language word embedding matrix is weighted using the CTC prediction distribution to obtain a primary text modality representation. In this embodiment, in order to convert acoustic features into text features, the word embedding matrix is weighted using the CTC prediction distribution. In this way, a representation similar to the word embedding can be obtained.
[0055] Furthermore, the primary text modal representation is compressed using the CTC prediction distribution to obtain a text modal representation of the text length corresponding to the output of the primary acoustic encoder. Preferably, by iterating through the word with the highest CTC prediction probability at each position in the primary text modal representation, if the word with the highest CTC prediction probability is empty, the representation at that position is discarded; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, the CTC prediction probabilities of consecutive and identical words are averaged, summed, or weighted according to the magnitude of the CTC distribution to obtain a text modal representation of the corresponding text length. It should be noted that the representation obtained by weighting the aforementioned word embedding matrix belongs to the same representation space as the text representation. However, the length of the audio sequence is generally much larger than the length of the text representation, which is different from traditional text translation models. To alleviate this problem, by observing the CTC prediction distribution, for each position in the sequence, if the word with the highest prediction probability is empty, it means that the position does not contain effective information and can be discarded; otherwise, for consecutive positions predicted to be the same word, it means that these positions correspond to the same content, and they can be fused by averaging, summing, or weighting according to the size of the CTC distribution to obtain a text-level length representation.
[0056] After processing by the transformer to obtain the text modal representation, it is input into the text encoder, where it is encoded through multiple Transformer layers. The text encoder is responsible for further encoding on the word embedding-style input. This branch is similar to a text translation model and can be initialized using a pre-trained text translation encoder to achieve better performance. Furthermore, methods from existing text translation tasks can be applied to this encoder, alleviating the need for independently researching new methods in speech translation.
[0057] Furthermore, the decoder is used to decode the latent vectors of the high-level acoustic encoder and the text encoder based on the dual attention mechanism to obtain the target language translation text corresponding to the source language speech data;
[0058] Specifically, the decoder in this embodiment is a multi-layer Transformer structure; in each Transformer layer of the decoder, a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationship and identical structure are set.
[0059] The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the latent vectors of the high-level acoustic encoder and the latent vectors of the text encoder, respectively. They perform attention calculations through either serial or parallel processing to obtain a tensor representation that integrates acoustic and textual information.
[0060] Specifically, the two encoder-decoder attention modules can be used sequentially. First, the encoder-decoder attention is calculated on the text representation, and operations such as residual and layer regularization are performed. Then, another encoder-decoder attention module is used to obtain information from the acoustic representation. That is, the first and second encoder-decoder attention modules are processed serially, including:
[0061] The latent vector of the text encoder is input into the first encoder-decoder attention module for attention calculation. The output of the first encoder-decoder attention module is subjected to residual and layer regularization and then input into the second encoder-decoder attention module.
[0062] The second encoder-decoder attention module receives the output of the first encoder-decoder attention module and the latent vector of the high-level acoustic encoder, performs attention calculations, and obtains a tensor representation that integrates acoustic and textual information.
[0063] Alternatively, the two encoder-decoder attention modules can be used in parallel, that is, the encoder-decoder attention operations are calculated simultaneously on the text representation and the acoustic representation, then the two are added together, and residual and layer regularization operations are performed. Specifically, the first and second encoder-decoder attention modules are processed in parallel, including: inputting the latent vectors of the text encoder and the latent vectors of the high-level acoustic encoder into the first and second encoder-decoder attention modules respectively for attention calculation; adding the outputs of the two attention modules; performing residual and layer regularization to obtain a tensor representation that fuses acoustic and text information.
[0064] To improve the decoder's predictive ability, before using the latent vectors of the text encoder and the advanced acoustic encoder for attention calculation, the representations at each position in the latent vectors of the text encoder and the advanced acoustic encoder can be randomly discarded with a preset probability. The remaining latent vectors of the text encoder and the advanced acoustic encoder are then input into their respective attention modules for attention calculation. In this embodiment, the random discard probability is set to 0.5, meaning there is a 50% probability of randomly discarding one representation and a 50% probability of using both representations together for decoding. By randomly discarding representations when acquiring acoustic and text representations, the contribution of each representation to the decoding is enhanced.
[0065] It should be noted that the decoder in this embodiment acquires information through two encoder-decoder attention modules, and fuses the obtained representations into the network in a serial or parallel manner. In the standard Transformer model, the attention mechanism usually only calculates the output of one encoder. However, for cross-modal and cross-lingual generation tasks such as speech translation, the source language text corresponds to a more direct type of information, which is helpful for the generation of target language text. However, this representation may contain large errors. The representation encoded by the source language speech data contains the original input information, which can correct erroneous text representations. The acoustic and text information complement each other, which is beneficial to the accuracy of target text prediction.
[0066] Step S3: Train the initial speech translation model using the speech dataset, and obtain the speech translation model through iterative updates of the loss function.
[0067] Specifically, after acquiring the speech dataset, the first step is to extract frame-level feature sequences from the audio files using signal processing methods, focusing on the acoustic task. In this embodiment, pre-emphasis, framing, and windowing operations in signal processing are used to obtain frame-level feature sequences of the speech data in the dataset through Discrete Fourier Transform. The frame-level feature sequences can be 80-dimensional MFCC features or FBank features.
[0068] The frame-level feature sequence corresponding to the obtained source language speech data is input into the primary acoustic encoder.
[0069] The loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively.
[0070] Calculate the loss for the target language annotated text for the outputs of the text encoder and decoder respectively;
[0071] By iteratively minimizing the loss, a converged speech translation model is obtained.
[0072] The proposed method was validated on a speech translation task. Utilizing the commonly used MuST-C English-German dataset, the baseline method is based on a Transformer architecture, employing a 12-layer acoustic encoder and a 6-layer decoder, with a hidden layer dimension of 256. The base model achieved a BLEU score of 23.6. This embodiment, using a 12-layer primary acoustic encoder, a 6-layer advanced acoustic and text encoder, and a 6-layer decoder, achieved a BLEU score of 24.5. Theoretically, by combining the method of this embodiment with pre-training techniques, the difference between pre-training and fine-tuning is reduced, thus achieving better results.
[0073] In practical applications, the trained speech translation model is loaded, the speech to be recognized is received, the corresponding FBank features are extracted by signal processing tools, and the features are input into the speech translation model. The model then undergoes feature extraction and prediction through a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder to obtain the target language translation text corresponding to the speech to be recognized.
[0074] Another embodiment of the present invention also provides a computer device, including at least one processor and at least one memory communicatively connected to the processor;
[0075] The memory stores instructions that can be executed by the processor to implement the aforementioned end-to-end speech translation modeling method based on dual attention.
[0076] In summary, the embodiments of the present invention provide an end-to-end speech translation modeling method based on dual attention. By explicitly encoding acoustic and text representations separately through acoustic encoders and text encoders, and simultaneously capturing information from both acoustic and text representations at the decoding end through a dual attention mechanism, the method achieves simultaneous consideration of acoustic and text information, thereby making the generated results more accurate and significantly improving the performance of the speech translation model.
[0077] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0078] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for modeling an end-to-end speech translation model based on dual attention, characterized in that, Includes the following steps: Obtain a speech dataset; the speech dataset includes source language speech data, source language annotated text corresponding to the source language speech data, and target language annotated text. An initial speech translation model is constructed, comprising a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder. The primary acoustic encoder is used for feature extraction and encoding of the source language speech data. The high-level acoustic encoder and the text encoder encode based on the output of the primary acoustic encoder, respectively, to obtain the high-level acoustic encoder latent vector and the text encoder latent vector. The decoder is a multi-layer Transformer structure used to decode the high-level acoustic encoder latent vector and the text encoder latent vector based on a dual attention mechanism to obtain the target language translation text corresponding to the source language speech data. This includes setting a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationships and identical structures in each Transformer layer of the decoder. The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the high-level acoustic encoder latent vector and the text encoder latent vector, respectively, and perform attention calculation through either serial or parallel processing to obtain a tensor representation that integrates acoustic and text information. The initial speech translation model is trained using the speech dataset, and then iteratively updated using a loss function to obtain the final speech translation model.
2. The end-to-end speech translation modeling method based on dual attention according to claim 1, characterized in that, The first encoder-decoder attention module and the second encoder-decoder attention module adopt a serial processing method, including: The latent vector of the text encoder is input into the first encoder-decoder attention module for attention calculation. The output of the first encoder-decoder attention module is subjected to residual and layer regularization and then input into the second encoder-decoder attention module. The second encoder-decoder attention module receives the output of the first encoder-decoder attention module and the latent vector of the high-level acoustic encoder, performs attention calculations, and obtains a tensor representation that integrates acoustic and textual information.
3. The end-to-end speech translation modeling method based on dual attention according to claim 1, characterized in that, The first encoder-decoder attention module and the second encoder-decoder attention module adopt a parallel processing method, including: inputting the latent vector of the text encoder and the latent vector of the high-level acoustic encoder into the first encoder-decoder attention module and the second encoder-decoder attention module respectively for attention calculation, adding the outputs of the two attention modules, performing residual and layer regularization, and obtaining a tensor representation that fuses acoustic and text information.
4. The end-to-end speech translation modeling method based on dual attention according to claim 3, characterized in that, Before performing attention calculation using the text encoder latent vector and the advanced acoustic encoder latent vector, the method further includes: randomly discarding the representation of each position in the text encoder latent vector and the advanced acoustic encoder latent vector with a preset probability, and inputting the remaining text encoder latent vector and advanced acoustic encoder latent vector after random discarding into the corresponding attention module for attention calculation.
5. The end-to-end speech translation modeling method based on dual attention according to claim 1, characterized in that, Training the initial speech translation model using the speech dataset includes: inputting the source language speech data into the primary acoustic encoder; The loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively. The loss for the target language annotated text is calculated for the outputs of the text encoder and decoder, respectively. By iteratively minimizing the loss, a converged speech translation model is obtained.
6. The end-to-end speech translation modeling method based on dual attention according to claim 1, characterized in that, A converter is also included between the primary acoustic encoder and the text encoder; The advanced acoustic encoder and the text encoder encode based on the output of the primary acoustic encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector, including: The output representation of the primary acoustic encoder is input into the advanced acoustic encoder, and encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector. The output representation of the primary acoustic encoder is transformed into a text modality using the converter and input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the text encoder latent vector.
7. The end-to-end speech translation modeling method based on dual attention according to claim 6, characterized in that, The process of using the converter to perform text modal conversion on the output representation of the primary acoustic encoder includes: The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder; By using the CTC prediction distribution, the word embedding matrix of the source language is weighted to obtain the primary text modality representation; The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.
8. The end-to-end speech translation modeling method based on dual attention according to claim 7, characterized in that, The step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the size of the CTC distribution to obtain a text modality representation of the corresponding text length.
9. A computer device, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions that can be executed by the processor to implement the end-to-end speech translation modeling method based on dual attention as described in any one of claims 1-8.