A method and apparatus for speech translation based on cooperative attention

Abstract

The application relates to a speech translation method and device based on collaborative attention, and belongs to the technical field of natural language processing; the speech translation method cannot encode acoustic and text representation separately, cannot capture and integrate the information of the opposite party in the attention calculation process, the encoding is not sufficient, and the translation result is inaccurate in the prior art; the translation method comprises the following steps: obtaining source language speech data to be translated; based on the source language speech data, a frame-level feature sequence is extracted; the frame-level feature sequence is input into a pre-trained speech translation model to obtain translated text; the high-level acoustic encoder and the text encoder of the speech translation model both comprise a collaborative attention module, high-level acoustic encoder hidden vectors and text encoder hidden vectors which fuse acoustic and text information are obtained through collaborative attention calculation; the target language translation text corresponding to the source language speech data is obtained through decoding by a decoder.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a speech translation method and device based on collaborative attention. Background Technology

[0002] When engaging in cross-language listening, speaking, reading, and writing, especially with non-English languages, most people lack the relevant comprehension skills. Since both listening and speaking in cross-language contexts are based on phonetic expression, a new application area has emerged: speech translation.

[0003] Speech translation, simply put, is translation where the input is a speech modality. Early speech translation models were designed by first using a speech recognition model to transcribe the speech, and then using a text translation model to translate the transcribed text; this was called cascaded speech translation. While cascaded speech translation achieved good performance, this approach suffers from problems such as error propagation, low translation efficiency, and loss of paralinguistic information in the speech.

[0004] Early methods only learned acoustic encodings without considering the source language text encodings. In later work, a common approach was to stack two encoders, with the acoustic information being converted to text information in between. However, in this process, existing speech translation models cannot simultaneously take into account both acoustic and text encodings, which may lead to the loss of acoustic information and low translation accuracy. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a speech translation method and device based on collaborative attention; to solve the problem that existing speech translation methods cannot encode acoustic and text representations separately, and cannot capture and integrate each other's information separately during attention calculation, resulting in insufficient encoding and inaccurate translation results.

[0006] The objective of this invention is mainly achieved through the following technical solutions:

[0007] On one hand, the present invention provides a speech translation method based on collaborative attention, comprising the following steps:

[0008] Obtain the source language speech data to be translated;

[0009] Based on the source language speech data, extract frame-level feature sequences;

[0010] The frame-level feature sequence is input into a pre-trained speech translation model to obtain the target language translation text corresponding to the source language speech data. The speech translation model includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, a gating unit, and a decoder. Both the high-level acoustic encoder and the text encoder include a collaborative attention module, which is used to perform collaborative attention calculation based on the output of the primary acoustic encoder to obtain the high-level acoustic encoder latent vector and the text encoder latent vector that fuse acoustic and text information. The gating unit is used to perform weighted fusion of the high-level acoustic encoder latent vector and the text encoder latent vector, and the weighted fused representation is input into the decoder for decoding to obtain the target language translation text corresponding to the source language speech data to be translated.

[0011] Furthermore, both the advanced acoustic encoder and the text encoder are n-layer Transformer structures, and the collaborative attention modules are all located between the self-attention layer and the feedforward neural network layer of each Transformer layer.

[0012] Furthermore, the collaborative attention module of the advanced acoustic encoder performs collaborative attention calculation based on the output of the primary acoustic encoder to obtain the latent vector of the advanced acoustic encoder, including:

[0013] The advanced acoustic encoder receives the output of the primary acoustic encoder and calculates the acoustic representation of the source language speech data to be translated through the self-attention layer.

[0014] The collaborative attention module uses the acoustic representation as the query and the representation output by the text encoder as the key and value to perform attention calculation. The result is then input into the feedforward neural network layer to obtain the advanced acoustic encoder latent vector that integrates acoustic and text information.

[0015] Furthermore, a converter is also included between the primary acoustic encoder and the text encoder;

[0016] The collaborative attention module of the text encoder performs collaborative attention calculation based on the output of the primary acoustic encoder to obtain the text encoder latent vector, including:

[0017] The output representation of the primary acoustic encoder is converted into a text modal representation using the converter described above.

[0018] The text encoder receives the text modal representation and calculates the text representation corresponding to the source language speech data to be translated through the self-attention module;

[0019] The collaborative attention module uses the text representation as the query and the representation of the advanced acoustic encoder as the key and value to perform attention calculation. The result is then input into the feedforward neural network layer to obtain the text encoder latent vector with acoustic and textual information.

[0020] Furthermore, the gating unit is used to perform weighted fusion of the latent vectors of the advanced acoustic encoder and the latent vectors of the text encoder, including:

[0021] The latent vectors of the advanced acoustic encoder and the text encoder are linearly transformed to obtain the corresponding acoustic coding vector representation and text coding vector representation, respectively.

[0022] The acoustic encoded vector representation and the text encoded vector representation are added together, and a gating value is calculated using the Sigmoid function; the acoustic encoded vector representation and the text encoded vector representation are then weighted and fused based on the gating value.

[0023] Furthermore, the step of using the converter to transform the output representation of the primary acoustic encoder through text modality conversion includes:

[0024] The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder;

[0025] By using the CTC prediction distribution, the word embedding matrix of the source language is weighted to obtain the primary text modality representation;

[0026] The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.

[0027] Furthermore, the step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the magnitude of the CTC distribution to obtain a text modality representation of the corresponding text length.

[0028] Furthermore, based on the source language speech data, frame-level feature sequences are extracted, including: pre-emphasizing, framing, and windowing the source language speech data, and obtaining the frame-level feature sequences corresponding to the source language speech data through discrete Fourier transform.

[0029] Furthermore, training the initial speech translation model using the speech dataset includes: inputting the source language speech data into the primary acoustic encoder;

[0030] The loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively.

[0031] The loss for the target language annotated text is calculated for the outputs of the text encoder and decoder, respectively.

[0032] By iteratively minimizing the loss, a converged speech translation model is obtained.

[0033] On the other hand, a computer device is also provided, including at least one processor and at least one memory communicatively connected to said processor;

[0034] The memory stores instructions that can be executed by the processor to implement the aforementioned collaborative attention-based speech translation method.

[0035] The beneficial effects of this technical solution are:

[0036] This invention models the acoustic and textual representations of source language speech data using two encoders, respectively. It captures information from both through a collaborative attention mechanism and allows for full interaction between the acoustic and textual representations, thus making the encoding process more comprehensive. The method mimics the structure of text translation models, explicitly learning the textual representation, which can significantly improve the performance of speech translation models.

[0037] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0039] Figure 1 This is a speech translation method based on collaborative attention, as described in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the speech translation model structure according to an embodiment of the present invention; Detailed Implementation

[0041] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0042] This embodiment presents a speech translation method based on collaborative attention, such as... Figure 1 As shown, it includes the following steps:

[0043] Step S1: Obtain the source language speech data to be translated.

[0044] The speech translation method in this embodiment is applicable to speech translation in any language. This method is implemented using a pre-trained speech translation model. Before training the model, the source language of the speech to be translated and the target language of the target text must be determined. During training, corresponding source language speech data and target language text data are selected for training. In practical applications, the source language speech data to be translated is acquired through any speech acquisition method and input into the pre-trained speech translation model to achieve the translation of the source language speech data into target language text.

[0045] Step S2: Extract frame-level feature sequences based on the source language speech data.

[0046] Specifically, by performing pre-emphasis, framing, and windowing operations on the source language speech data, the corresponding frame-level feature sequence, such as 80-dimensional MFCC features or FBank features, can be obtained through Discrete Fourier Transform.

[0047] Step S3: Input the frame-level feature sequence into the pre-trained speech translation model to obtain the target language translation text corresponding to the source language speech data.

[0048] Specifically, such as Figure 2 As shown, the speech translation model in this embodiment includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, a gating unit, and a decoder. Both the high-level acoustic encoder and the text encoder include a collaborative attention module. The collaborative attention module is used to perform collaborative attention calculation based on the output of the primary acoustic encoder to obtain the latent vectors of the high-level acoustic encoder and the text encoder that fuse acoustic and text information. The gating unit is used to perform weighted fusion of the latent vectors of the high-level acoustic encoder and the text encoder, and input the weighted fused representation into the decoder for decoding to obtain the target language translation text corresponding to the source language speech data to be translated.

[0049] Specifically, the pre-trained speech translation model in this embodiment is a translation model trained using the MuST-C English-German dataset, which is commonly used in the field of speech translation and has annotations. The dataset includes source language speech data, source language annotated text corresponding to the source language speech data, and target language annotated text.

[0050] During training, firstly, for the acoustic task, frame-level feature sequences are extracted from the audio file using signal processing methods. Operations in signal processing such as pre-emphasis, framing, and windowing can be used to obtain frame-level feature sequences through Discrete Fourier Transform.

[0051] The extracted frame-level feature sequences are input into a primary acoustic encoder for encoding. The primary acoustic encoder can employ a 12-layer Transformer structure to encode the feature sequences of the input source language speech data. In this embodiment, fewer layers are used to encode the input audio features, reducing computational resources while ensuring reasonable top-level CTC prediction.

[0052] Both the advanced acoustic encoder and the text encoder are multi-layer Transformer structures, with the collaborative attention module positioned between the self-attention layer and the feedforward neural network layer in each Transformer layer. The advanced acoustic encoder receives the output of the primary acoustic encoder and calculates the acoustic representation of the source language speech data to be translated through the self-attention layer. The collaborative attention module uses the acoustic representation of the source language speech data as the query and the representation output from the same layer of the text encoder as the key and value, performs attention calculations, and inputs the result into the feedforward neural network layer to obtain the advanced acoustic encoder latent vector that integrates acoustic and textual information.

[0053] It should be noted that the advanced acoustic encoder in this embodiment takes the output of the primary acoustic encoder as input and encodes it through multiple Transformer layers. In addition to the original self-attention layer and feedforward neural network layer in each layer, a collaborative attention module is introduced between them. This module uses the acoustic representation as the query and the representation of the corresponding text encoder layer as the key and value, to capture information about the text modality. The advanced acoustic encoder is responsible for further learning based on the representation learned by the primary acoustic encoder, while simultaneously capturing the representation of the text encoder, thus incorporating text information into the acoustic representation information.

[0054] Furthermore, a converter is included between the primary acoustic encoder and the text encoder; the collaborative attention module of the text encoder performs collaborative attention calculation based on the output of the primary acoustic encoder to obtain the text encoder latent vector, including:

[0055] The output representation of the primary acoustic encoder is transformed into a text modality using a converter. Specifically, the converter performs text modality transformation on the output of the primary acoustic encoder using the following method: First, the CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder. Then, the source language word embedding matrix is ​​weighted using the CTC prediction distribution to obtain the primary text modality representation. In this step, in order to convert acoustic features into text features, the word embedding matrix is ​​weighted using the CTC prediction distribution. In this way, a representation similar to word embeddings can be obtained.

[0056] After obtaining the primary text modal representation, it is compressed using the CTC prediction distribution to obtain a text modal representation of the same length as the output of the primary acoustic encoder. Specifically, it first iterates through the word with the highest CTC prediction probability at each position in the primary text modal representation. If the word with the highest CTC prediction probability is empty, the representation at that position is discarded. If the word with the highest CTC prediction probability is a representation of consecutive and identical words, the CTC prediction probabilities of these consecutive and identical words are averaged, summed, or weighted according to the magnitude of the CTC distribution to obtain the corresponding text modal representation of the same length. In this step, the weighted representation obtained by the word embedding matrix and the text representation belong to the same representation space. However, unlike traditional text translation models, the length of the audio sequence is generally much larger than the length of the text representation. To alleviate this problem, by observing the CTC prediction distribution, for each position in the sequence, if the word with the highest prediction probability is empty, it means that the position does not contain effective information and can be discarded; otherwise, for consecutive positions predicted to be the same word, it means that these positions correspond to the same content, and they can be fused by averaging, summing, or weighting according to the size of the CTC distribution to obtain a text-level length representation.

[0057] After obtaining the text modality representation through the converter, it is input into the text encoder. The text encoder receives the text modality representation and calculates the text representation corresponding to the source language speech data to be translated through the self-attention module.

[0058] Specifically, the collaborative attention module of the text encoder uses the text representation corresponding to the source language speech data as the query and the output representation of the same layer of the high-level acoustic encoder as the key and value to perform attention calculation. The result is then input into the feedforward neural network layer to obtain the text encoder latent vector with acoustic and text information.

[0059] It should be noted that in this embodiment, the text encoder takes the weighted representation processed by the transformer as input and encodes it through multiple Transformer layers. In addition to the original self-attention and feedforward neural network layers in each layer, a collaborative attention module is introduced between them. This module uses the text representation as the query and the acoustic encoder's representation as the key and value to capture information about the acoustic modality. The text encoder further encodes the input in a word embedding style; this branch is similar to a text translation model and can be initialized using a pre-trained text translation encoder to achieve better performance. Furthermore, methods from existing text translation tasks can be applied to this encoder, alleviating the need for independently researching new methods for speech translation.

[0060] Furthermore, a gating unit is used to selectively fuse the outputs of the advanced acoustic encoder and the text encoder. Traditional Transformer decoders only accept the output of a single encoder. To adapt to this design, the representations of the outputs of the advanced acoustic encoder and the text encoder are linearly transformed separately, then the two are added together and a gating value between [0,1] is calculated using the Sigmoid function. Finally, the two representations are weighted and fused using this gating value.

[0061] Finally, the weighted and fused representation from the gated units is input into the decoder to generate the target language text corresponding to the speech to be translated.

[0062] The decoder of the speech translation model consists of multiple identical Transformer layers, each containing a self-attention module, an encoder-decoder attention module, and a feedforward neural network module. Since the text translation model already possesses a certain ability to predict the target language text within the encoder, the learning burden on the decoder is significantly reduced. The output of the text encoder is fed into the decoder of the end-to-end speech translation model to generate the final target language text.

[0063] During training, the loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively.

[0064] Calculate the loss for the target language annotated text for the outputs of the text encoder and decoder respectively;

[0065] By iteratively minimizing the loss, a converged speech translation model is obtained.

[0066] The proposed method was validated on a speech translation task. On the commonly used MuST-C English-German dataset for speech translation, the baseline method, based on a Transformer architecture, uses a 12-layer acoustic encoder and a 6-layer decoder, with a hidden layer dimension of 256. The base model achieved a BLEU score of 23.6. Our proposed method, employing a 12-layer primary acoustic encoder, a 6-layer advanced acoustic and text encoder, and a 6-layer decoder, achieved a BLEU score of 24.6. Theoretically, combining this method with pre-training techniques could achieve even better results.

[0067] In practical applications, the trained speech translation model is loaded, the speech to be recognized is received, the corresponding FBank features are extracted by signal processing tools, and the features are input into the speech translation model. The model then undergoes feature extraction and prediction through a speech encoder, adapter, text encoder, and decoder to obtain the target language translation text corresponding to the speech to be recognized.

[0068] Another embodiment of the present invention also provides a computer device, including at least one processor and at least one memory communicatively connected to the processor;

[0069] The memory stores instructions that can be executed by the processor to implement the aforementioned collaborative attention-based speech translation method.

[0070] In summary, the embodiments of the present invention provide a speech translation method based on collaborative attention, which explicitly encodes acoustic and text representations through acoustic encoders and text encoders, and simultaneously captures information from the other party through a collaborative attention mechanism during the encoding process to fuse acoustic and text representations, thereby making the encoding more comprehensive and significantly improving the performance of the translation model.

[0071] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0072] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech translation method based on collaborative attention, characterized in that, Includes the following steps: Obtain the source language speech data to be translated; Based on the source language speech data, extract frame-level feature sequences; The frame-level feature sequence is input into a pre-trained speech translation model to obtain the target language translation text corresponding to the source language speech data. The speech translation model includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, a gating unit, and a decoder. Both the high-level acoustic encoder and the text encoder include a collaborative attention module, which is used to perform collaborative attention calculation based on the output of the primary acoustic encoder to obtain the high-level acoustic encoder latent vector and the text encoder latent vector that fuse acoustic and text information. The gating unit is used to perform weighted fusion of the high-level acoustic encoder latent vector and the text encoder latent vector, and the weighted fused representation is input into the decoder for decoding to obtain the target language translation text corresponding to the source language speech data to be translated. Both the advanced acoustic encoder and the text encoder are multi-layer Transformer structures, and the collaborative attention modules are set between the self-attention layer and the feedforward neural network layer of each Transformer layer. The collaborative attention module of the advanced acoustic encoder performs collaborative attention calculation based on the output of the primary acoustic encoder to obtain the advanced acoustic encoder latent vector, including: the advanced acoustic encoder receives the output of the primary acoustic encoder and calculates the acoustic representation corresponding to the source language speech data to be translated through the self-attention layer; The collaborative attention module uses the acoustic representation as the query and the representation output by the text encoder as the key and value to perform attention calculation. The result is then input into the feedforward neural network layer to obtain the advanced acoustic encoder latent vector that integrates acoustic and text information.

2. The speech translation method based on collaborative attention according to claim 1, characterized in that, A converter is also included between the primary acoustic encoder and the text encoder; The collaborative attention module of the text encoder performs collaborative attention calculation based on the output of the primary acoustic encoder to obtain the text encoder latent vector, including: The output representation of the primary acoustic encoder is converted into a text modal representation using the converter described above. The text encoder receives the text modal representation and calculates the text representation corresponding to the source language speech data to be translated through the self-attention module; The collaborative attention module uses the text representation as the query and the representation of the advanced acoustic encoder as the key and value to perform attention calculation. The result is then input into the feedforward neural network layer to obtain the text encoder latent vector with acoustic and textual information.

3. The speech translation method based on collaborative attention according to claim 1, characterized in that, The gating unit is used to perform weighted fusion of the latent vectors of the advanced acoustic encoder and the latent vectors of the text encoder, including: The latent vectors of the advanced acoustic encoder and the text encoder are linearly transformed to obtain the corresponding acoustic coding vector representation and text coding vector representation, respectively. The acoustic encoded vector representation and the text encoded vector representation are added together, and a gating value is calculated using the Sigmoid function; the acoustic encoded vector representation and the text encoded vector representation are then weighted and fused based on the gating value.

4. The speech translation method based on collaborative attention according to claim 2, characterized in that, The process of using the converter to transform the output representation of the primary acoustic encoder into a text modality includes: The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder; By using the CTC prediction distribution, the word embedding matrix of the source language is weighted to obtain the primary text modality representation; The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.

5. The speech translation method based on collaborative attention according to claim 4, characterized in that, The step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the size of the CTC distribution to obtain a text modality representation of the corresponding text length.

6. The speech translation method based on collaborative attention according to claim 1, characterized in that, Based on the source language speech data, extract frame-level feature sequences, including: performing pre-emphasis, framing, and windowing operations on the source language speech data, and obtaining the frame-level feature sequences corresponding to the source language speech data through discrete Fourier transform.

7. The speech translation method based on collaborative attention according to claim 1, characterized in that, Training the speech translation model using a speech dataset includes: inputting source language speech data from the speech dataset into the primary acoustic encoder; The loss for the source language annotated text is calculated for the outputs of the primary acoustic encoder and the advanced acoustic encoder, respectively. The loss for the target language annotated text is calculated for the outputs of the text encoder and decoder, respectively. By iteratively minimizing the loss, a converged speech translation model is obtained.

8. A computer device, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions executable by the processor to implement the collaborative attention-based speech translation method according to any one of claims 1-7.

Images