A speech translation model modeling method and apparatus based on speech synthesis data

Abstract

The application relates to a speech translation model modeling method and device based on speech synthesis data, and belongs to the technical field of natural language processing; the method solves the problem that, in the prior art, a speech translation model is inaccurate in translation results due to insufficient training data and insufficient utilization; the modeling method comprises the following steps: acquiring a general speech synthesis data set, and training a general speech synthesis model; acquiring a speech translation data set of a target field; fine-tuning the general speech synthesis model in the speech translation data set to obtain a special speech synthesis model; inputting source language labeled text into the special speech synthesis model, generating a plurality of pieces of speech synthesis pseudo data according to a pre-set proportion, and obtaining a pseudo speech data set; constructing an initial speech translation model, training the initial speech translation model by using the speech translation data set of the target field and the pseudo speech data set, and obtaining a speech translation model through loss function iteration and updating.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and device for modeling a speech translation model based on speech synthesis data. Background Technology

[0002] With rapid economic development and increasingly extensive international exchanges, language barriers remain a major obstacle for many. This has led to the emergence of an important application area: speech translation.

[0003] Compared to speech recognition and text translation technologies, speech translation is an emerging field, with most datasets labeled in recent years. The most commonly used MUST-C dataset currently only contains a few hundred hours of audio data. In contrast, for mainstream languages, speech recognition typically involves thousands or even tens of thousands of hours of data, while text translation data reaches tens or even hundreds of millions of hours. This lack of data is the biggest obstacle to the development of speech translation. Therefore, how to achieve high-quality, high-precision speech translation using limited datasets has become an urgent problem to be solved. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a method and device for modeling a speech translation model based on speech synthesis data; and to solve the problem that the existing speech translation models have inaccurate translation results due to insufficient training data and inadequate utilization.

[0005] The objective of this invention is mainly achieved through the following technical solutions: On the one hand, the present invention provides a method for modeling a speech translation model based on speech synthesis data, comprising the following steps: Obtain a general speech synthesis dataset and train a general speech synthesis model using the general speech synthesis dataset; Obtain a speech translation dataset for the target domain; the speech translation dataset includes source language speech data, source language annotation text corresponding to the source language speech data, and target language annotation text; use the source language speech data and corresponding source language annotation text in the speech translation dataset to fine-tune the general speech synthesis model to obtain a dedicated speech synthesis model; The source language annotated text is input into the dedicated speech synthesis model, and multiple pseudo-speech data are generated according to a preset ratio to obtain a pseudo-speech dataset. An initial speech translation model is constructed, and the initial speech translation model is trained using the speech translation dataset and pseudo-speech dataset of the target domain. After iterative updates using a loss function, the speech translation model is obtained.

[0006] Furthermore, the step of inputting the source language-annotated text into the dedicated speech synthesis model to generate multiple pseudo-speech data according to a pre-set ratio, resulting in a pseudo-speech dataset, includes: Set the ratio of generated speech synthesis pseudo data to original speech data; Using the dedicated speech synthesis model, different styles of speech synthesis data are generated from the source language annotated text according to the specified ratio to obtain the first speech synthesis pseudo data; the different styles of speech synthesis data include different speech rates, pitches, and speaker speech synthesis pseudo data. Retrieve the source language annotated text and the target language annotated text, select words that appear less than n times, and use the corresponding source language annotated text to generate speech synthesis data to obtain the second speech synthesis pseudo data; Based on the first and second pseudo-speech synthesis data, a pseudo-speech dataset is obtained.

[0007] Furthermore, the initial speech translation model is trained using the speech dataset and pseudo-speech dataset of the target domain, including: Add the [TTS] tag to the beginning of sentences in the speech synthesis pseudo data, and add the [ST] tag to the beginning of sentences in the source language speech data; For the same source language annotated text, the source language speech data and multiple speech synthesis pseudo data are trained in the same batch; By iteratively updating the cross-entropy loss function and the consistency loss function, a converged speech translation model is obtained.

[0008] Furthermore, the convergent speech translation model is obtained through iterative updates using the cross-entropy loss function and the consistency loss function, including: The target prediction probability is obtained by averaging the prediction probabilities generated from the source language speech data corresponding to the same source language annotated text and multiple speech synthesis pseudo data; Calculate the KL loss of the predicted probability of the source language speech data and each speech synthesis pseudo-data and the predicted probability of the target, and weight it with cross-entropy; By iteratively minimizing the loss, a converged speech translation model is obtained.

[0009] Furthermore, the speech translation model includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder; The primary acoustic encoder is used to extract and encode features from source language speech data and speech synthesis pseudo data; The advanced acoustic encoder and the text encoder encode based on the output of the primary encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector. The decoder is used to decode the latent vectors of the advanced acoustic encoder and the latent vectors of the text encoder based on the dual attention mechanism to obtain the target language translated text corresponding to the source language speech data.

[0010] Furthermore, a converter is also included between the primary acoustic encoder and the text encoder; The advanced acoustic encoder and text encoder encode based on the output of the primary encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector, including: The output representation of the primary acoustic encoder is input into the advanced acoustic encoder, and encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector. The output representation of the primary acoustic encoder is transformed into a text modality using the converter and input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the text encoder latent vector.

[0011] Furthermore, the step of using the converter to perform text modal conversion on the output representation of the primary acoustic encoder includes: The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder; The source language word embedding matrix is ​​weighted using the CTC prediction distribution to obtain a primary text modality representation; The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.

[0012] Furthermore, the step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the magnitude of the CTC distribution to obtain a text modality representation of the corresponding text length. Furthermore, the decoder is a multi-layer Transformer structure; The decoder is used to decode the latent vectors of the advanced acoustic encoder and the text encoder based on a dual attention mechanism, including: In each Transformer layer of the decoder, a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationship and identical structure are set; The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the latent vectors of the high-level acoustic encoder and the latent vectors of the text encoder, respectively. They perform attention calculations through either serial or parallel processing to obtain a tensor representation that fuses acoustic and text information.

[0013] On the other hand, a computer device is also provided, including at least one processor and at least one memory communicatively connected to said processor; The memory stores instructions that can be executed by the processor to implement the aforementioned speech translation modeling method based on speech synthesis data.

[0014] The beneficial effects of this technical solution are: To address the scarcity of training data for speech translation in existing technologies, this invention provides a method for augmenting training data and training a speech translation model using pseudo-speech synthesis data. This method obtains a speech synthesis model within the target domain through training, generates pseudo-speech synthesis data using text translation data from the target domain, and incorporates real source language speech data for targeted training. This approach explicitly increases the scale of training data, reduces the need for labeled data, increases data diversity, improves model performance on different datasets, makes the training process more stable, and leads to faster and better model convergence.

[0015] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0017] Figure 1 This is a flowchart of a speech translation modeling method based on speech synthesis data according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the training of the speech synthesis and speech translation model according to an embodiment of the present invention; Detailed Implementation Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0019] This embodiment presents a speech translation modeling method based on speech synthesis data, such as... Figure 1 As shown, it includes the following steps: Step S1: Obtain a general speech synthesis dataset and train a general speech synthesis model using the general speech synthesis dataset.

[0020] Specifically, this embodiment trains a high-performance speech synthesis model using large-scale speech synthesis data. It collects as much general-purpose speech synthesis data as possible, such as LibriSpeech and Librits datasets. Specifically, this step focuses on maximizing the data size to increase the model's robustness. The acquired large-scale general-purpose speech synthesis data is then used to train the tacotron2 or fastspeech model, resulting in a powerful general-purpose speech synthesis model capable of synthesizing general-domain speech based on text data.

[0021] Step S2: Obtain the speech translation dataset for the target domain; the speech translation dataset includes source language speech data, source language annotation text corresponding to the source language speech data, and target language annotation text; fine-tune the general speech synthesis model using the source language speech data and the corresponding source language annotation text in the speech translation dataset to obtain a dedicated speech synthesis model.

[0022] Specifically, while general-purpose speech synthesis models trained using generic speech synthesis data can synthesize high-quality speech, speech data from different domains typically exhibits variations. For example, audio from different domains has different characteristics in terms of pitch and speech rate. The target domain can be a speech or a conversation. In the speech domain, sentences are longer and pronunciation is more standard; while in conversation, sentences are shorter and speech rates vary considerably. To make the speech generated by the speech synthesis model more consistent with the feature distribution of the speech dataset within the target domain, in practical applications, the general-purpose speech synthesis model can be further fine-tuned using the speech recognition portion of the speech translation data from the target domain (i.e., source language speech data and corresponding source language annotated text). This enhances the model's performance on target domain data and generates speech data that better reflects the characteristics of the target domain.

[0023] Step S3: Input the source language labeled text into the dedicated speech synthesis model, generate multiple pseudo-speech data according to a pre-set ratio, and obtain the pseudo-speech dataset.

[0024] Specifically, the first step is to set the ratio of generated pseudo-speech data to the original speech data. The purpose of this step is to control the ratio of pseudo-speech data to the original real data (i.e., the source language speech data in the speech translation dataset). If the proportion of pseudo-speech data is too large, the distribution of the generated pseudo-data will cover the distribution of the original real data, causing the trained model to deviate from the expected performance. If the amount of pseudo-speech data is too small, the training effect will not be significant. This embodiment achieves a better training effect by controlling the ratio of pseudo-speech data to original real data to 10:1.

[0025] Furthermore, a dedicated speech synthesis model is used to generate speech synthesis data of different styles from the source language labeled text on a proportional basis, resulting in the first speech synthesis pseudo-data. The speech synthesis data of different styles includes speech synthesis pseudo-data of different speech rates, pitches, and speakers. Specifically, when using the speech synthesis model to generate speech synthesis data, different styles of pseudo-data can be generated by adding perturbations, adjusting parameters, etc., in order to increase the diversity of the generated data and improve the generalization ability of the model.

[0026] Specifically, due to the limited training corpus data, some words may appear less frequently and therefore cannot be adequately trained. In this embodiment, low-frequency words in the training data are retrieved separately and pseudo-speech synthesis data is generated to improve the model's prediction ability for low-frequency words. Specifically, source language labeled text and target language labeled text are retrieved, and words that appear less than n times are selected. In this embodiment, n=10 is set, and the corresponding source language labeled text is used to generate speech synthesis data to obtain the second pseudo-speech synthesis data. A pseudo-speech dataset is obtained based on the first and second pseudo-speech synthesis data.

[0027] Step S4: Construct an initial speech translation model. Train the initial speech translation model using the speech translation dataset and pseudo-speech dataset of the target domain. Iterate and update the model using the loss function to obtain the speech translation model.

[0028] Specifically, the initial speech translation model in this embodiment includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder; Among them, the primary acoustic encoder is used to extract and encode features from source language speech data and speech synthesis pseudo data; The primary acoustic encoder can employ a 12-layer Transformer structure to encode the feature sequences of the input source language speech data and speech synthesis pseudo-data. This embodiment uses fewer layers to encode the input audio data, reducing computational resources while ensuring reasonable top-level CTC prediction.

[0029] After primary encoding using a primary acoustic encoder, the advanced acoustic encoder and the text encoder encode based on the output of the primary encoder, respectively, to obtain the latent vectors of the advanced acoustic encoder and the text encoder. Preferably, both the advanced acoustic encoder and the text encoder are multi-layer Transformer structures; the advanced acoustic encoder is used to further learn on the representation learned by the primary acoustic encoder, and is encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector.

[0030] A converter is also included between the primary acoustic encoder and the text encoder; The output representation of the primary acoustic encoder is transformed into a text modality using a converter and then input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the latent vector of the text encoder.

[0031] Specifically, the converter is used to perform text modal conversion on the output representation of the primary acoustic encoder, including: The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder; The source language word embedding matrix is ​​weighted using the CTC prediction distribution to obtain a primary text modality representation. In this embodiment, in order to convert acoustic features into text features, the word embedding matrix is ​​weighted using the CTC prediction distribution. In this way, a representation similar to the word embedding can be obtained.

[0032] Furthermore, the primary text modal representation is compressed using the CTC prediction distribution to obtain a text modal representation of the corresponding text length from the output of the primary acoustic encoder. Preferably, by iterating through the word with the highest CTC prediction probability at each position in the primary text modal representation, if the word with the highest CTC prediction probability is empty, the representation at that position is discarded; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, the CTC prediction probabilities of consecutive and identical words are averaged, summed, or weighted according to the magnitude of the CTC distribution to obtain a text modal representation of the corresponding text length. It should be noted that the aforementioned weighted representation obtained by the word embedding matrix belongs to the same representation space as the text representation; however, the length of the audio sequence is generally much larger than the length of the text representation, which is different from traditional text translation models. To alleviate this problem, by observing the CTC prediction distribution, for each position in the sequence, if the word with the highest prediction probability is empty, it means that the position does not contain effective information and can be discarded; otherwise, for consecutive positions predicted to be the same word, it means that these positions correspond to the same content, and they can be fused by averaging, summing, or weighting according to the size of the CTC distribution to obtain a text-level length representation.

[0033] After processing by the transformer to obtain the text modal representation, it is input into the text encoder, where it is encoded through multiple Transformer layers. The text encoder is responsible for further encoding on the word embedding-style input. This branch is similar to a text translation model and can be initialized using a pre-trained text translation encoder to achieve better performance. Furthermore, methods from existing text translation tasks can be applied to this encoder, alleviating the need for independently researching new methods in speech translation.

[0034] Furthermore, the decoder is used to decode the latent vectors of the high-level acoustic encoder and the text encoder based on the dual attention mechanism to obtain the target language translation text corresponding to the source language speech data; Specifically, the decoder in this embodiment is a multi-layer Transformer structure; in each Transformer layer of the decoder, a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationship and identical structure are set. The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the latent vectors of the high-level acoustic encoder and the latent vectors of the text encoder, respectively. They perform attention calculations through either serial or parallel processing to obtain a tensor representation that integrates acoustic and textual information.

[0035] Specifically, the two encoder-decoder attention modules can be used sequentially. First, the encoder-decoder attention is calculated on the text representation, and operations such as residual and layer regularization are performed. Then, another encoder-decoder attention module is used to obtain information from the acoustic representation. That is, the first and second encoder-decoder attention modules are processed serially, including: The latent vector of the text encoder is input into the first encoder-decoder attention module for attention calculation. The output of the first encoder-decoder attention module is subjected to residual and layer regularization and then input into the second encoder-decoder attention module. The second encoder-decoder attention module receives the output of the first encoder-decoder attention module and the latent vector of the high-level acoustic encoder, performs attention calculations, and obtains a tensor representation that integrates acoustic and textual information.

[0036] Alternatively, the two encoder-decoder attention modules can be used in parallel, that is, the encoder-decoder attention operations are calculated simultaneously on the text representation and the acoustic representation, then the two are added together, and residual and layer regularization operations are performed. Specifically, the first and second encoder-decoder attention modules are processed in parallel, including: inputting the latent vectors of the text encoder and the latent vectors of the high-level acoustic encoder into the first and second encoder-decoder attention modules respectively for attention calculation; adding the outputs of the two attention modules; performing residual and layer regularization to obtain a tensor representation that fuses acoustic and text information.

[0037] To improve the decoder's predictive ability, before using the latent vectors of the text encoder and the advanced acoustic encoder for attention calculation, the representations at each position in the latent vectors of the text encoder and the advanced acoustic encoder can be randomly discarded with a preset probability. The remaining latent vectors of the text encoder and the advanced acoustic encoder are then input into their respective attention modules for attention calculation. In this embodiment, the random discard probability is set to 0.5, meaning there is a 50% probability of randomly discarding one representation and a 50% probability of using both representations together for decoding. By randomly discarding representations when acquiring acoustic and text representations, the contribution of each representation to the decoding is enhanced.

[0038] It should be noted that the decoder in this embodiment acquires information through two encoder-decoder attention modules, and fuses the obtained representations into the network in a serial or parallel manner. In the standard Transformer model, the attention mechanism usually only calculates the output of one encoder. However, for cross-modal and cross-lingual generation tasks such as speech translation, the source language text corresponds to a more direct type of information, which is helpful for the generation of target language text. However, this representation may contain large errors. The representation encoded by the source language speech data contains the original input information, which can correct erroneous text representations. The acoustic and text information complement each other, which is beneficial to the accuracy of target text prediction.

[0039] Furthermore, after constructing the initial speech translation model, the initial speech translation model is trained using a speech translation dataset and a pseudo-speech dataset from the target domain.

[0040] Specifically, after acquiring the speech dataset, the first step is to extract frame-level feature sequences from the audio files using signal processing methods, focusing on the acoustic task. In this embodiment, pre-emphasis, framing, and windowing operations in signal processing are used to obtain frame-level feature sequences of the speech data in the dataset through Discrete Fourier Transform. The frame-level feature sequences can be 80-dimensional MFCC features or FBank features. A [TTS] tag is added to the beginning of sentences in the speech synthesis pseudo-data to identify that the sample is generated by the speech synthesis model; a [ST] tag is added to the beginning of sentences in the source language speech data to identify that the sample is the original real data, thus distinguishing the two types of data. Since speech synthesis data usually does not contain interference information, such as silence or background noise, more noise can be injected into the speech synthesis pseudo-data to improve the robustness of the model.

[0041] The source language speech data corresponding to the same source language annotated text and multiple speech synthesis pseudo data are trained in the same batch; a converged speech translation model is obtained by iteratively updating the model through cross-entropy loss function and consistency loss function.

[0042] It should be noted that although the styles or speakers of the multiple speech-synthesized pseudo-data points synthesized from the same source language annotated text may differ, they correspond to the same text and should theoretically predict the same distribution. Therefore, in calculating the loss, in addition to the default cross-entropy loss, this embodiment adds an extra consistency loss to constrain the predicted distribution of the speech-synthesized pseudo-data points, making it the same as the predicted distribution of the source language speech data, thus providing stronger guidance for model training. Specifically, the predicted probabilities obtained from the source language speech data and multiple speech-synthesized pseudo-data points corresponding to the same source language annotated text are averaged as the target predicted probability. Then, the KL loss of the predicted probability of each sample and the average predicted probability are calculated and weighted with cross-entropy. After back-integration, the loss is minimized to obtain a converged speech translation model. In this embodiment, the weights of both losses are 1 by default.

[0043] The proposed method was validated on a speech translation task. Using the commonly used MuST-C English-German dataset, a speech synthesis model was trained on the LibriTTS dataset and then fine-tuned using the MuST-C speech recognition dataset. Then, corresponding pseudo-speech synthesis data was generated for the source language-annotated text in the MuST-C dataset. Simultaneously, data was extracted from similar domain text translation datasets IWSLT and Opensubtitle for generation, resulting in a BLEU score improvement of 1 compared to the original model. Therefore, the proposed method is highly beneficial for end-to-end speech translation models, enabling larger datasets, significant performance improvements, and increased model generalization.

[0044] In practical applications, the trained speech translation model is loaded, the speech to be recognized is received, the corresponding FBank features are extracted using signal processing tools, [ST] tags are added to the beginning of the sentence, and the speech is input into the speech translation model. The model then performs feature extraction and prediction through a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder to obtain the target language translation text corresponding to the speech to be recognized.

[0045] Another embodiment of the present invention also provides a computer device, including at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions that can be executed by the processor to implement the aforementioned speech translation modeling method based on speech synthesis data.

[0046] In summary, the embodiments of the present invention provide a speech translation modeling method based on speech synthesis data, such as... Figure 2 As shown, a general-domain speech synthesis model is first trained using general speech data. Then, a specialized speech synthesis model is trained using speech data from the target domain. Pseudo-speech data for speech synthesis is generated using text translation data. During training, real source language speech data is incorporated for targeted training. This explicitly increases the scale and diversity of the training data, significantly improving the model's performance on different datasets. Furthermore, this invention explicitly encodes acoustic and text representations separately using acoustic and text encoders. Simultaneously, a dual attention mechanism is used to capture information from both acoustic and text representations at the decoding end, achieving a balance between acoustic and textual information. This results in more accurate generated results and significantly improves the performance of the speech translation model.

[0047] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0048] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for modeling a speech translation model based on speech synthesis data, characterized in that, Includes the following steps: Obtain a general speech synthesis dataset and train a general speech synthesis model using the general speech synthesis dataset; Obtain a speech translation dataset for the target domain; the speech translation dataset includes source language speech data, source language annotated text corresponding to the source language speech data, and target language annotated text. By using the source language speech data and corresponding source language annotated text in the speech translation dataset, the general speech synthesis model is fine-tuned to obtain a dedicated speech synthesis model; The source language annotated text is input into the dedicated speech synthesis model, and multiple pseudo-speech data are generated according to a preset ratio to obtain a pseudo-speech dataset. This includes: setting the ratio of generated pseudo-speech data to original speech data; using the dedicated speech synthesis model to generate speech synthesis data of different styles from the source language annotated text according to the preset ratio to obtain first pseudo-speech data; the different styles of speech synthesis data include pseudo-speech data with different speech rates, pitches, and speakers; for low-frequency words in the training data, separate retrieval and generation of pseudo-speech data are performed to improve the model's prediction ability for low-frequency words, including: retrieving the source language annotated text and target language annotated text, selecting words that appear less than n times, and generating speech synthesis data using the corresponding source language annotated text to obtain second pseudo-speech data; and obtaining a pseudo-speech dataset based on the first and second pseudo-speech data. An initial speech translation model is constructed, and the initial speech translation model is trained using the speech translation dataset and pseudo-speech dataset of the target domain. After iterative updates using a loss function, the speech translation model is obtained.

2. The speech translation modeling method based on speech synthesis data according to claim 1, characterized in that, The initial speech translation model is trained using the speech dataset and pseudo-speech dataset of the target domain, including: Add the [TTS] tag to the beginning of sentences in the speech synthesis pseudo data, and add the [ST] tag to the beginning of sentences in the source language speech data; For the same source language annotated text, the source language speech data and multiple speech synthesis pseudo data are trained in the same batch; By iteratively updating the cross-entropy loss function and the consistency loss function, a converged speech translation model is obtained.

3. The speech translation modeling method based on speech synthesis data according to claim 2, characterized in that, The convergent speech translation model is obtained through iterative updates using the cross-entropy loss function and the consistency loss function, including: The target prediction probability is obtained by averaging the prediction probabilities generated from the source language speech data corresponding to the same source language annotated text and multiple speech synthesis pseudo data; Calculate the KL loss of the predicted probability of the source language speech data and each speech synthesis pseudo-data and the predicted probability of the target, and weight it with cross-entropy; By iteratively minimizing the loss, a converged speech translation model is obtained.

4. The speech translation modeling method based on speech synthesis data according to claim 1, characterized in that, The speech translation model includes a primary acoustic encoder, a high-level acoustic encoder, a text encoder, and a decoder; The primary acoustic encoder is used to extract and encode features from source language speech data and speech synthesis pseudo data; The advanced acoustic encoder and the text encoder encode based on the output of the primary acoustic encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector. The decoder is used to decode the latent vectors of the advanced acoustic encoder and the latent vectors of the text encoder based on the dual attention mechanism to obtain the target language translated text corresponding to the source language speech data.

5. The speech translation modeling method based on speech synthesis data according to claim 4, characterized in that, A converter is also included between the primary acoustic encoder and the text encoder; The advanced acoustic encoder and the text encoder encode based on the output of the primary acoustic encoder, respectively, to obtain the advanced acoustic encoder latent vector and the text encoder latent vector, including: The output representation of the primary acoustic encoder is input into the advanced acoustic encoder, and encoded through multiple feature extraction layers in the advanced acoustic encoder to obtain the advanced acoustic encoder latent vector. The output representation of the primary acoustic encoder is transformed into a text modality using the converter and input into the text encoder. It is then encoded through multiple feature extraction layers to obtain the text encoder latent vector.

6. The speech translation modeling method based on speech synthesis data according to claim 5, characterized in that, The process of using the converter to perform text modal conversion on the output representation of the primary acoustic encoder includes: The CTC prediction distribution on the predicted source language text is calculated using the output of the primary acoustic encoder; The source language word embedding matrix is ​​weighted using the CTC prediction distribution to obtain a primary text modality representation; The primary text modality representation is compressed using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder.

7. The speech translation modeling method based on speech synthesis data according to claim 6, characterized in that, The step of compressing the representation of the primary text modality using the CTC prediction distribution to obtain a text modality representation of the text length corresponding to the output of the primary acoustic encoder includes: traversing the word with the highest CTC prediction probability at each position in the primary text modality representation; if the word with the highest CTC prediction probability is empty, then discarding the representation at that position; if the word with the highest CTC prediction probability is a representation of consecutive and identical words, then averaging, summing, or weighting the CTC prediction probabilities of consecutive and identical words according to the size of the CTC distribution to obtain a text modality representation of the corresponding text length.

8. The speech translation modeling method based on speech synthesis data according to claim 4, characterized in that, The decoder is a multi-layer Transformer structure; The decoder is used to decode the latent vectors of the advanced acoustic encoder and the text encoder based on a dual attention mechanism, including: In each Transformer layer of the decoder, a first encoder-decoder attention module and a second encoder-decoder attention module with dual relationship and identical structure are set; The first encoder-decoder attention module and the second encoder-decoder attention module obtain information through the latent vectors of the high-level acoustic encoder and the latent vectors of the text encoder, respectively. They perform attention calculations through either serial or parallel processing to obtain a tensor representation that fuses acoustic and text information.

9. A computer device, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor; The memory stores instructions that can be executed by the processor to implement the speech translation modeling method based on speech synthesis data as described in any one of claims 1-8.

Images