A Modeling Method and Device for an Acoustic and Text Joint Encoding Speech Translation Model

Abstract

The present invention relates to a method and device for modeling a speech translation model with acoustic and text joint coding, belonging to the technical field of natural language processing; it solves the problem in the prior art that the speech translation method cannot be modeled by combining acoustic and text joint representations, resulting in insufficient coding and information loss; the modeling method of the present invention includes: obtaining a speech data set; constructing an initial speech translation model; the initial speech translation model includes an acoustic conversion module and a joint encoding and decoding module; the acoustic conversion module is used to extract features from the source language speech data and perform conversion processing to obtain an acoustic-text joint representation; the joint encoding and decoding module is used to perform joint coding on the acoustic-text joint representation and decode to obtain the target language translation text corresponding to the source language speech data; using the speech data set to train the initial speech translation model, and through iterative update of the loss function, obtaining the speech translation model.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a method and device for modeling a speech translation model with joint acoustic and text encoding. Background Art

[0002] In the early stage, most commonly used translation functions were text translations, such as web page translations, document translations, etc. However, among the four aspects of language ability, namely listening, speaking, reading, and writing, text translation can only rely on reading and writing. For the more direct and more difficult listening and speaking, which are both expressed based on speech. Therefore, a new application field has emerged, namely speech translation.

[0003] Recently, researchers have proposed end-to-end speech translation, that is, through a single model, to complete the generation of the target language text from the specified source language speech. However, this method has problems of complex task modeling and insufficient data accumulation. To alleviate the above two problems, one idea is to optimize the model structure so that it can be more suitable for the speech translation task and reduce the demand for data. The goal of speech translation is to learn cross-modal and cross-language conversion from source language audio to target language text. Early methods only learned acoustic encoding and did not consider the learning of source language text encoding. In subsequent work, a common idea is to stack two encoders and complete the conversion of acoustic information to text information in between. However, in this process, acoustic information may be lost, and the speech translation model cannot simultaneously take into account acoustic and text encoding and enable them to interact, resulting in low accuracy and poor effects of current speech translation methods. Summary of the Invention

[0004] In view of the above analysis, the present invention aims to provide a method and device for modeling a speech translation model with joint acoustic and text encoding; to solve the problem that the existing speech translation methods cannot model by combining acoustic and text representations, resulting in insufficient encoding and information loss.

[0005] The object of the present invention is mainly achieved through the following technical solutions:

[0006] On the one hand, the present invention provides a method for modeling a speech translation model with joint acoustic and text encoding, and this method includes the following steps:

[0007] Obtain a speech data set; the speech data set includes source language speech data, source language annotation text corresponding to the source language speech data, and target language annotation text;

[0008] Construct an initial speech translation model; the initial speech translation model includes an acoustic conversion module and a joint encoding and decoding module; the acoustic conversion module is used to extract features from the source language speech data and perform conversion processing to obtain an acoustic text joint representation; the joint encoding and decoding module is used to jointly encode the acoustic text joint representation and decode to obtain the target language translation text corresponding to the source language speech data;

[0009] Use the speech data set to train the initial speech translation model, and through iterative update of the loss function, obtain the speech translation model.

[0010] Furthermore, the acoustic conversion module includes a primary acoustic encoder and a converter;

[0011] The primary acoustic encoder includes multiple Transformer or Conformer layers, which are used to extract features from the input source language speech data and perform CTC prediction to obtain an acoustic encoder hidden vector h s ;

[0012] The converter is used to perform semantic embedding and concatenation processing on the acoustic encoder hidden vector h s to obtain an acoustic text joint representation.

[0013] Furthermore, the converter is used to perform semantic embedding and concatenation processing on the acoustic encoder hidden vector h s to obtain an acoustic text joint representation, including:

[0014] Perform semantic embedding on the acoustic encoder hidden vector h s to obtain a representation of the text modality

[0015]

[0016] Compress the length of the representation of the text modality through the CTC prediction distribution to obtain a compressed representation of the text modality h t ;

[0017] Concatenate the acoustic encoder hidden vector h s with the compressed representation of the text modality h t in the sequence dimension to obtain an acoustic text joint representation h st .

[0018] Furthermore, performing semantic embedding on the acoustic encoder hidden vector h s to obtain a text modality representation includes:

[0019] Calculate the CTC prediction distribution P(π|h on the predicted source language text using the acoustic encoder hidden vector s );

[0020] Minimize the CTC prediction loss through the reverse iteration method;

[0021] Weight the source language word embedding matrix W through the CTC prediction distribution P(π|h s ) after minimizing the loss to obtain the text modality representation

[0022] Furthermore, compress the length of the text modality representation through the CTC prediction distribution to obtain the compressed text modality representation h t , including:

[0023] Traverse the word with the highest CTC prediction probability corresponding to each position in the text modality representation . If the word with the highest CTC prediction probability is empty, discard the representation of that position; if the word with the highest CTC prediction probability is the representation of consecutive and identical words, average, sum, or weight according to the size of the CTC prediction distribution the CTC prediction probabilities of the consecutive and identical words to obtain the compressed text modality representation h t .

[0024] Furthermore, the joint encoding and decoding module includes a joint encoder and a decoder;

[0025] The joint encoder is based on a multi-layer Transformer or Conformer structure and is used to perform feature extraction and attention calculation on the acoustic text joint representation h st to obtain the joint encoding hidden vector;

[0026] The decoder is used to decode the joint encoding hidden vector to obtain the target language text word probability distribution.

[0027] Furthermore, the joint encoder is an acoustic text joint encoding model pre-trained based on the SLAM or mSLAM method.

[0028] Furthermore, training the initial speech translation model using the speech dataset and iteratively updating through the loss function to obtain the speech translation model includes:

[0029] Train the acoustic conversion module using the source language speech data and the source language annotated text, and iteratively update through the loss function to obtain the trained acoustic conversion module;

[0030] Train the joint encoding and decoding module using the acoustic text joint representation and the target language annotated text; through iterative updating of the loss function, obtain the trained joint encoding and decoding module;

[0031] Obtain the speech translation model based on the trained acoustic conversion module and joint encoding and decoding module.

[0032] Further, after training the joint encoding and decoding module using the acoustic text joint representation and the target language annotated text, it further includes: performing word embedding on the source language annotated text to obtain the word embedding representation of the source language annotated text; using the source language word embedding representation and the target language annotated text to perform enhanced training on the joint encoding and decoding module; based on the trained acoustic conversion module and the enhanced trained joint encoding and decoding module, obtain the speech translation model.

[0033] On the other hand, a computer device is also disclosed, including at least one processor and at least one memory communicatively connected to the processor;

[0034] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the foregoing method for modeling a speech translation model with joint acoustic and text encoding.

[0035] Advantages of the technical solution of the present invention:

[0036] Aiming at the problem that the existing speech translation methods only model the acoustic representation, the present invention constructs a representation that simultaneously includes both acoustic and text modalities, and uses a pre-trained acoustic-text joint encoder to uniformly model the combined acoustic and text representations, thereby making the encoding process more sufficient. The present invention utilizes a powerful acoustic-text joint pre-training model trained on a large amount of data, and has great potential in the end-to-end speech translation task. By jointly encoding the representations of both sound and text modalities through a pre-trained model, more sufficient interaction can be carried out, providing a richer encoding representation for the generation of the target language text, and significantly improving the performance of the speech translation model.

[0037] Other features and advantages of the present invention will be described in the following specification, and part of them will become obvious from the specification, or be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings are only for the purpose of showing specific embodiments, and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference numerals represent the same components.

[0039] Figure 1 Flow chart of the method for modeling an acoustic and text jointly encoded speech translation model according to an embodiment of the present invention.

[0040] Figure 2 Schematic diagram of the process of acoustic and text joint encoding according to an embodiment of the present invention.

[0041] Figure 3 Schematic diagram of compressing the length of the text representation according to an embodiment of the present invention. Detailed implementation manners

[0042] The preferred embodiments of the present invention will be specifically described below with reference to the accompanying drawings, in which the accompanying drawings form a part of this application and are used together with the embodiments of the present invention to explain the principles of the present invention, and are not used to limit the scope of the present invention.

[0043] The present invention discloses a method for modeling a speech translation model based on acoustic and text joint encoding. In the modeling process, the acoustic representation and text representation corresponding to the source language speech data are cascaded, and the cascaded representation containing both acoustic and text information is used for modeling. During the encoding process, a pre-trained acoustic-text joint encoder is used to fully encode the cascaded acoustic-text joint representation, which can effectively utilize the potential of the pre-trained model and enable full interaction of information in both acoustic and text modalities, so that the translation result of the speech translation model modeled by this method is more accurate.

[0044] A method for modeling an acoustic and text jointly encoded speech translation model in this embodiment, as Figure 1 shown, includes the following steps:

[0045] Step S1, obtain a speech data set.

[0046] Specifically, in this embodiment, the commonly used MuST-C English-German data set with annotations in the field of speech translation can be adopted; the data set includes source language speech data, source language annotation text corresponding to the speech data, and target language annotation text.

[0047] Step S2, construct an initial speech translation model.

[0048] Specifically, as Figure 2 shown, the initial speech translation model in this embodiment includes an acoustic conversion module and a joint encoding and decoding module;

[0049] Among them, the acoustic conversion module is used to extract features from the source language speech data and perform conversion processing to obtain an acoustic-text joint representation;

[0050] Preferably, the acoustic conversion module includes a primary acoustic encoder and a converter;

[0051] The primary acoustic encoder includes multiple layers of Transformer or Conformer layers for feature extraction and CTC prediction of the input source language speech data to obtain the acoustic encoder hidden vector h s Specifically, in this embodiment, a 12-layer Transformer structure is adopted. For the input audio feature s, it is encoded using 12 encoding layers. On the basis of ensuring reasonable CTC prediction at the top layer, the computational resources used are reduced to obtain the acoustic representation h s , as shown in the following formula:

[0052] h s = JAE(s).

[0053] The Transformer is used to perform semantic embedding and concatenation processing on the acoustic encoder hidden vector h s to obtain the acoustic-text joint representation;

[0054] Specifically, during the process of performing semantic embedding and concatenation processing on the acoustic encoder hidden vector h s , first, semantic embedding is performed on the acoustic encoder hidden vector h s to obtain the representation in the text modality Specifically, performing semantic embedding on the acoustic encoder hidden vector h s includes: calculating the CTC prediction distribution P(π|h s ) on the predicted source language text using the acoustic encoder hidden vector; minimizing the CTC prediction loss through the reverse iteration method during training; weighting the source language word embedding matrix W by the CTC prediction distribution P(π|h s ) after minimizing the loss to obtain the representation in the text modality as shown in the following formula:

[0055]

[0056] The weighted representation in the text modality is obtained. After that, the length of the representation in the text modality is compressed by the CTC prediction distribution to obtain the compressed representation h in the text modality; in this embodiment, as t shown, the length of the representation in the text modality is compressed by judging the word with the highest prediction probability corresponding to each position; first, traverse the representation in the text modality Figure 3 ​For each position in, the word with the highest CTC prediction probability is selected. If the word with the highest CTC prediction probability is empty, the representation of that position is discarded. If the word with the highest CTC prediction probability is a representation of consecutive and identical words, the CTC prediction probabilities of the consecutive and identical words are averaged, summed, or weighted according to the size of the CTC prediction distribution to obtain the compressed text modality representation h t ,

[0057] After obtaining the compressed text modality representation, the acoustic encoder hidden vector h s is concatenated with the compressed text modality representation h t in the sequence dimension to obtain the acoustic-text joint representation h st ; Specifically, through a special separator [SEP], the acoustic encoder hidden vector h s and the text modality representation h t are concatenated to form a longer sequence h st =(h s ; [SEP]; h t ) to obtain the acoustic-text joint representation that contains both acoustic and text modalities.

[0058] Furthermore, the joint encoding and decoding module is used to jointly encode the acoustic-text joint representation and decode it to obtain the target language translation text corresponding to the source language speech data;

[0059] Preferably, the joint encoding and decoding module includes a joint encoder and a decoder;

[0060] Among them, the joint encoder is an acoustic-text joint encoding model pre-trained based on a multi-layer Transformer or Conformer structure and based on the SLAM or mSLAM method, which is used to extract features and calculate attention for the acoustic-text joint representation h st to obtain the joint encoding hidden vector;

[0061] Compared with traditional models, the acoustic-text joint encoding model pre-trained based on the SLAM or mSLAM method used in this embodiment is a model trained uniformly on a large amount of data through labeled speech recognition, text translation data, and unlabeled pure speech and pure text data. This model has the ability to jointly encode speech and text. And compared with the traditional method where only the acoustic representation is input into the joint encoder for training, this embodiment uses the acoustic-text joint representation that contains both acoustic and text information for training, and uses the self-attention module to interact between the two modalities during the encoding process, so as to obtain a more sufficient encoding representation.

[0062] Specifically, since the acoustic-text joint encoder has the ability to model both acoustic and text modalities, text data can be further utilized during training. That is, the word embedding representation corresponding to the source language text is used as the input to the acoustic-text joint encoder, which is used to predict the target language text at the decoding end. Through multi-task learning, in the case of less speech translation data, the model's demand for data can be reduced.

[0063] Furthermore, the decoder is used to decode the jointly encoded hidden vector to obtain the target language translation text corresponding to the input source language text.

[0064] The decoder of the speech translation model consists of multiple identical Transformer layers, each layer containing a self-attention module, an encoder-decoder attention module, and a feed-forward neural network module. Since in the encoder, the text translation model already has a certain ability to predict the target language text, the learning pressure of the decoder prediction can be greatly reduced. The output of the text encoder is fed into the decoder of the end-to-end speech translation model to generate the final required target language text.

[0065] Step S3: Use the speech dataset to train the initial speech translation model, and through iterative update of the loss function, obtain the speech translation model.

[0066] Specifically, after obtaining the speech dataset, first for the acoustic task, frame-level feature sequences are extracted from the audio files through signal processing methods. In this embodiment, operations such as pre-emphasis, framing, and windowing in signal processing are used to obtain the frame-level feature sequences of the speech data in the dataset through discrete Fourier transform. The frame-level feature sequences can be 80-dimensional MFCC features or FBank features.

[0067] The extracted frame-level feature sequences are input into the speech translation model, and the acoustic conversion module is trained using the source language speech data and the source language annotated text, and is iteratively updated through the loss function to obtain the trained acoustic conversion module;

[0068] The joint encoding and decoding module is trained using the acoustic-text joint representation and the target language annotated text; through iterative update of the loss function, the trained joint encoding and decoding module is obtained.

[0069] The speech translation model is obtained based on the trained acoustic conversion module and joint encoding and decoding module.

[0070] After training the joint encoding and decoding module using the acoustic text joint representation and the target language annotated text, the joint encoder can be further enhanced using the text data to improve the performance of the model. First, word embeddings are performed on the source language annotated text to obtain the word embedding representation of the source language annotated text. The joint encoding and decoding module is enhanced using the source language word embedding representation and the target language annotated text. Based on the trained acoustic conversion module and the enhanced joint encoding and decoding module, a speech translation model is obtained.

[0071] In practical applications, the trained speech translation model is loaded, the speech to be recognized is received, the corresponding FBank features are extracted through signal processing tools, input into the speech translation model, and encoded and predicted through the acoustic conversion module and the joint encoding and decoding module to obtain the target language translation text corresponding to the speech to be recognized.

[0072] Another embodiment of the present invention further provides a computer device, including at least one processor and at least one memory communicatively connected to the processor;

[0073] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the foregoing method for modeling a speech translation model based on joint encoding of acoustics and text.

[0074] In summary, the method for modeling a speech translation model based on joint encoding of acoustics and text provided by the embodiments of the present invention jointly encodes the representations of two modalities through a pre-trained model, conducts more sufficient interaction, and provides a richer encoding representation for the generation of the target language text. It has great potential in the end-to-end speech translation task, can enrich the encoding representation, and achieve significant performance improvement.

[0075] Those skilled in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0076] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. An acoustic and text joint encoding method for modeling a speech translation model, characterized in that Including the following steps: Obtain a speech dataset; the speech dataset includes source language speech data, source language annotation text corresponding to the source language speech data, and target language annotation text; Construct an initial speech translation model; the initial speech translation model includes an acoustic conversion module and a joint encoding and decoding module; the acoustic conversion module is used to extract features from the source language speech data and perform conversion processing to obtain an acoustic text joint representation; the joint encoding and decoding module is used to jointly encode the acoustic text joint representation and decode to obtain the target language translation text corresponding to the source language speech data; the acoustic conversion module includes a primary acoustic encoder and a converter; the primary acoustic encoder includes multiple Transformer or Conformer layers, which are used to extract features from the input source language speech data and perform CTC prediction to obtain an acoustic encoder hidden vector h s ; The converter is used to perform semantic embedding and concatenation processing on the acoustic encoder hidden vector h s to obtain the acoustic-text joint representation, which specifically includes: performing semantic embedding on the acoustic encoder hidden vector h s to obtain the representation of the text modality compressing the length of the representation of the text modality through the CTC prediction distribution to obtain the compressed representation h of the text modality t ; concatenating the acoustic encoder hidden vector h s and the compressed representation h of the text modality t in the sequence dimension to obtain the acoustic-text joint representation h st ; Use the speech dataset to train the initial speech translation model, and through iterative update of the loss function, obtain the speech translation model.

2. The method for modeling an acoustic and text jointly encoded speech translation model according to claim 1, characterized in that The acoustic encoder hidden vector h s is semantically embedded to obtain a text modality representation including: Calculate the CTC prediction distribution P(π|h on the predicted source language text using the acoustic encoder hidden vector s ); Minimize the CTC prediction loss through the reverse iteration method; By weighting the source language word embedding matrix W with the CTC prediction distribution P(π|h after minimizing the loss s ), the text modality representation is obtained 3. The method for modeling an acoustic and text joint encoding speech translation model according to claim 1, wherein, Compress the length of the text modality representation through the CTC prediction distribution to obtain the compressed text modality representation h which includes: t ​ Traverse the text modality representation for the word with the highest CTC prediction probability corresponding to each position. If the word with the highest CTC prediction probability is empty, discard the position representation. If the word with the highest CTC prediction probability is a representation of consecutive and identical words, average, sum, or weight according to the size of the CTC prediction distribution for the consecutive and identical words to obtain the compressed text modality representation h t .

4. The method for modeling an acoustic and text joint encoding speech translation model according to claim 1, wherein The joint encoding and decoding module includes a joint encoder and a decoder; The joint encoder is based on a multi-layer Transformer or Conformer structure and is used for jointly representing the acoustic text h st to perform feature extraction and attention calculation to obtain a jointly encoded hidden vector; The decoder is used to decode the joint encoded hidden vector to obtain the target language text word probability distribution.

5. The method for modeling an acoustic and text jointly encoded speech translation model according to claim 4, wherein The joint encoder is an acoustic text joint encoding model pre-trained based on the SLAM or mSLAM method.

6. The method for modeling an acoustic and text joint encoding speech translation model according to claim 1, wherein The step of using the speech dataset to train the initial speech translation model and obtaining the speech translation model through iterative update of the loss function includes: Use the source language speech data and source language annotation text to train the acoustic conversion module, and perform iterative update through the loss function to obtain the trained acoustic conversion module; Use the acoustic text joint representation and target language annotation text to train the joint encoding and decoding module; through iterative update of the loss function, obtain the trained joint encoding and decoding module; Obtain the speech translation model based on the trained acoustic conversion module and joint encoding and decoding module.

7. The method for modeling an acoustic and text jointly encoded speech translation model according to claim 6, wherein After using the acoustic text joint representation and target language annotation text to train the joint encoding and decoding module, it further includes: performing word embedding on the source language annotation text to obtain the word embedding representation of the source language annotation text; using the source language word embedding representation and target language annotation text to perform enhanced training on the joint encoding and decoding module; based on the trained acoustic conversion module and the enhanced trained joint encoding and decoding module, obtain the speech translation model.

8. A computer device, characterized in that, Including at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the method for modeling an acoustic and text jointly encoded speech translation model according to any one of claims 1-7.

Images