Speech translation method and device

Abstract

The invention provides a speech translation method and device, and the method comprises the steps: translating source language speech data based on a speech translation model, and obtaining a target language text; a training target of the speech translation model comprises minimizing a difference between a first target language prediction text generated based on source language sample speech data and a translation label corresponding to the source language sample speech data, and minimizing a difference between speech features of the source language sample speech data and text features of a first source language sample text. And minimizing the difference between a second target language prediction text generated based on the pseudo source language speech features and a translation label corresponding to the second source language sample text. According to the method and the device, the problem of scarcity of annotated voice data is solved by efficiently utilizing relatively rich text data in a low-resource language scene, so that the performance of a voice translation model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a speech translation method and device. BACKGROUND

[0002] Speech translation technology is limited by the severe lack of speech training data for low-resource languages, making it difficult to meet the extensive application needs of education, tourism, media and other industries in these regions.

[0003] In the prior art, to solve the problem of data sparsity, one method is to simulate speech features through text, which usually uses an acoustic model to obtain the phoneme sequence and pronunciation duration of the text to generate approximate speech features. However, this method requires training an acoustic model for each language separately, which is cumbersome and inefficient, and it is difficult to effectively align speech and text features to a unified space, resulting in poor utilization of text data and limited improvement in translation effect. SUMMARY

[0004] The present application provides a speech translation method and device to solve the defects in the prior art.

[0005] The present application provides a speech translation method, comprising the following steps: determining source language speech data; translating the source language speech data based on a speech translation model to obtain target language text; The speech translation model is trained based on first training data and second training data, the first training data includes source language sample speech data and first source language sample text paired with the sample speech data, and the second training data includes second source language sample text. The training target of the speech translation model includes minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on the pseudo-source language speech features and the corresponding translation label of the second source language sample text, the pseudo-source language speech features are generated according to the second source language sample text, and are used to simulate the text features of the real speech features under the source language.

[0006] According to the speech translation method provided by the present application, the speech translation model is trained based on the following steps: extracting the speech features of the source language sample speech data, the text features of the first source language sample text, and the pseudo-source language speech features; determine a difference between the speech feature of the source language sample voice data and the text feature of the first source language sample text according to a similarity between the speech feature of the source language sample voice data and the text feature of the first source language sample text; in a case where the difference between the speech feature of the source language sample voice data and the text feature of the first source language sample text is less than a threshold value, determine a training loss value based on a difference between the first target language predicted text and a corresponding translation label of the source language sample voice data, the difference between the speech feature of the source language sample voice data and the text feature of the first source language sample text, a difference between the second target language predicted text and a corresponding translation label of the second source language sample text, and a difference between a third target language predicted text generated based on the first source language sample text and a corresponding translation label of the source language sample voice data; update the parameters of the speech translation model based on the training loss value.

[0007] According to the speech translation method provided by the application, the difference between the speech feature of the source language sample voice data and the text feature of the first source language sample text is determined according to the similarity between the speech feature of the source language sample voice data and the text feature of the first source language sample text, which comprises: perform feature alignment on the speech feature of the source language sample voice data and the text feature of the first source language sample text to obtain aligned speech feature and aligned text feature; determine a first loss value according to the similarity between the aligned speech feature and the aligned text feature; determine a second loss value according to the similarity between the speech feature of the source language sample voice data and the text feature of the first source language sample text; determine the difference between the speech feature of the source language sample voice data and the text feature of the first source language sample text according to the first loss value and / or the second loss value.

[0008] According to the speech translation method provided by the application, the first loss value is determined according to the similarity between the aligned speech feature and the aligned text feature, which comprises: obtain first reconstructed speech feature and first reconstructed text feature, wherein the first reconstructed speech feature is obtained by performing self-attention weighting on the aligned speech feature, and the first reconstructed text feature is obtained by performing self-attention weighting on the aligned text feature; Obtain a second reconstructed speech feature and a second reconstructed text feature. The second reconstructed speech feature is obtained by performing cross-attention weighting on the aligned speech feature as the query feature. The second reconstructed text feature is obtained by performing cross-attention weighting on the aligned text feature as the query feature. Calculate the first mean square error between the first reconstructed speech feature and the second reconstructed speech feature, and the second mean square error between the first reconstructed text feature and the second reconstructed text feature; The first mean square error and the second mean square error are added together to obtain the first loss value.

[0009] According to a speech translation method provided by the present invention, determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text based on a first loss value and a second loss value includes: Based on the sum of the first loss value and the second loss value, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined.

[0010] According to a speech translation method provided by the present invention, after determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, the method further includes: If the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is greater than or equal to the threshold, the parameters of the speech translation model are frozen, and the parameters of the text processing module are updated based on the difference between the speech features of the source language sample speech data and the text features of the first source language sample text. The text processing module is used to extract the transliterated letter sequence from the input text, upsample the letter sequence according to the duration of the speech corresponding to the input text, and extract text features from the upsampled character sequence.

[0011] According to a speech translation method provided by the present invention, the pseudo-source language speech features are generated based on the following steps: Extract the transliterated letter sequence from the second source language sample text; The letter sequence is upsampled based on the duration of the speech corresponding to the second source language sample text; Extract the pseudo-source language speech features from the upsampled character sequence.

[0012] According to a speech translation method provided by the present invention, the step of extracting the pseudo-source language speech features from the upsampled character sequence includes: Extract initial pseudo-speech features from the upsampled character sequence; Using the speech translation model, the initial pseudo-speech features are aligned with the real speech features to obtain the pseudo-source language speech features.

[0013] According to a speech translation method provided by the present invention, the second target language predicted text is generated based on the following steps: Using the aforementioned speech translation model, the pseudo-source language speech features are masked, and the masked pseudo-source language speech features are decoded to obtain the predicted text of the second target language.

[0014] The present invention also provides a voice translation device, comprising the following modules: The determining unit is used to determine the source language speech data; The translation unit is used to translate the source language speech data based on a speech translation model to obtain target language text; The speech translation model is trained based on first training data and second training data. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text. The training objectives of the speech translation model include minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on pseudo-source language speech features and the corresponding translation label of the second source language sample text. The pseudo-source language speech features are text features generated based on the second source language sample text to simulate the real speech features under the source language.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech translation methods.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech translation method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech translation method as described above.

[0018] The speech translation method and apparatus provided by this invention construct a joint training framework containing three training objectives. By using paired first training data and plain text second training data to jointly train the speech translation model, it achieves efficient use of relatively abundant text data to compensate for the scarcity of labeled speech data in low-resource language scenarios, thereby improving the performance of the speech translation model. It effectively solves the problem that traditional methods are difficult to train usable speech translation models due to the lack of large-scale paired speech data, and avoids the complex acoustic modeling process required in traditional methods, thus improving the efficiency of model training and translation effect. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the speech translation method provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the distribution of speech and text features provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the distribution of aligned speech and text features provided by the present invention.

[0023] Figure 4 This is a flowchart illustrating the speech translation model training method provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the speech translation device provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] Machine speech translation technology for low-resource languages ​​has significant social and commercial value, capable of meeting the needs of various industries such as education, tourism, and media. However, current mainstream end-to-end speech translation systems heavily rely on large-scale speech-text pairing data for training. For low-resource languages, the labeled speech data is typically only a few dozen hours or even less. This scarcity of data severely limits the performance and application of existing speech translation models, making it exceptionally difficult to train a usable model.

[0028] To address the aforementioned data scarcity issue, a common technical solution is to convert text into approximate speech features through acoustic modeling. This method typically uses limited paired data to train an acoustic model that can extract the phoneme sequence corresponding to each character or word in the text and estimate its pronunciation duration, thereby generating an intermediate feature representation that simulates the acoustic characteristics of real speech, which drives the training of the translation model.

[0029] However, the above approach has significant drawbacks. First, performing independent acoustic modeling for each low-resource language is a tedious and time-consuming task, resulting in an overly complex and inefficient text data preprocessing workflow. Second, due to the vast differences in phonemes and pronunciation rules among different languages, this method struggles to effectively align features from different modalities (speech and text) into a unified feature space, leading to poor utilization of text data and ultimately limited improvement in translation quality.

[0030] To address this, the present invention provides a speech translation method that aims to generate pseudo-source language speech features from source language text and to jointly train the method with the goal of minimizing the differences between real speech translation, speech-text feature differences, and pseudo-speech translation differences. This method achieves efficient utilization of rich text data in low-resource languages, simplifies the data processing flow, and effectively improves the performance of speech translation models.

[0031] in, Figure 1 This is a flowchart illustrating the speech translation method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110 and 120.

[0032] Step 110: Determine the source language speech data.

[0033] In this embodiment, source language speech data can be understood as the raw audio signal input by the user that needs to be translated into the target language. The source language can refer to a language with relatively scarce data resources, or any language lacking large-scale, high-quality annotated speech data. It can also be a specific domain or dialect of a high-resource language (such as Chinese or English). In this case, although the overall data for that language is abundant, paired speech data for specific application scenarios (such as professional fields like medicine or law) remains scarce. The source language speech data can be a speech stream recorded in real-time by the user through audio acquisition devices such as microphones, or a pre-recorded audio file stored on the device. This embodiment does not specifically limit this.

[0034] Step 120: Based on the speech translation model, translate the source language speech data to obtain the target language text; The speech translation model is trained based on first training data and second training data. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text. The training objectives of the speech translation model include minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on pseudo-source language speech features and the corresponding translation label of the second source language sample text. The pseudo-source language speech features are text features generated based on the second source language sample text to simulate the real speech features under the source language.

[0035] In this embodiment, the training process of the speech translation model does not rely solely on traditional paired data, but is obtained through collaborative training based on the first training data and the second training data.

[0036] The first training data can be understood as a limited but high-quality pairing data, which includes source language sample speech data and first source language sample text paired with the sample speech data. The pairing here can be understood as the spoken content contained in the speech data and the text content recorded in the text being semantically consistent.

[0037] The second training data can be understood as a large amount of text but little corresponding speech data, or speech data that is difficult to obtain, or pure text data that does not contain any corresponding speech data. The second training data includes sample text of the second source language.

[0038] Among them, the second source language sample text can be obtained by transliterating the characters or words in the original text into the alphabet representation of the source language according to language-specific rules or open-source tools. If the source language is English, the characters or words in the original text can be transliterated into English alphabet representation to obtain the second source language sample text. Among them, considering that the text corresponding to the source language may be relatively small in quantity, therefore, in this embodiment, text data in other languages (such as high-resource languages with rich data resources) can also be introduced and transliterated into the alphabet representation of the source language to further expand the scale and diversity of the second training data, thereby enhancing the model's ability to learn the general phonetic-form mapping relationship.

[0039] For example, if the source language is English, the Chinese text "欢迎你来" can be obtained, transliterated into "huanyingnilai", and the transliterated text can be used as the second source language sample text.

[0040] After obtaining the first training data and the second training data, the initial model is trained based on the first training data and the second training data to obtain a speech translation model. Among them, the speech translation model includes three training objectives, which are specifically as follows: The first training objective is to minimize the difference between the first target language prediction text generated based on the source language sample speech data and the translation label corresponding to the source language sample speech data. This objective is used to train the end-to-end translation ability of the model. Among them, the first target language prediction text refers to the translation result text of the corresponding target language generated by the model after inputting the source language sample speech data into the initial model. The translation label corresponding to the source language sample speech data refers to the correct target language translation text that is manually pre-annotated and used as the standard answer for this sample speech data in the first training data.

[0041] Specifically, the source language sample speech data in the first training data is input into the initial model to generate the first target language prediction text. Then, the difference between the first target language prediction text and the pre-annotated translation label is calculated. Among them, the cross-entropy loss function can be used to calculate the difference between the two, or other text similarity measurement methods (such as the distance based on the bag-of-words model or word embedding) can be used to calculate the difference between the two. This embodiment does not make specific limitations on this. The smaller the difference between the two, the closer the translation result of the model is to the standard answer, and the higher the translation accuracy. Minimizing the difference between the two can adjust the parameters inside the model through the backpropagation algorithm, so that the model can learn the translation ability to directly map from the source language speech to the target language text.

[0042] The second training objective is to minimize the difference between the speech features of the source language sample speech data and the text features of the first source language sample text. The core of this objective is to achieve cross-modal feature alignment. Speech features can be understood as high-dimensional vector representations of the acoustic and semantic information of the speech extracted from the source language sample speech data by the model's internal speech encoder. Text features can be understood as high-dimensional vector representations of the semantic information of the corresponding first source language sample text extracted from the model's internal text encoder.

[0043] Specifically, the source language sample speech data from the first training data is input into a speech encoder to obtain speech features, and the paired first source language sample text is input into a text encoder to obtain text features. Then, the difference between these two types of features from different modalities but with the same content is calculated. For example, the mean squared error loss function can be used to calculate the distance between feature vectors, or the cosine similarity loss can be used to calculate the directional consistency of feature vectors; this embodiment does not specifically limit this. The smaller the difference between the two, the more it indicates that the model has mapped the speech and text with the same content to neighboring locations in the latent feature space. Minimizing this difference can encourage the model to learn a unified, modality-independent shared representation space, thereby enabling the model to learn the ability to recognize the same semantic content in different modalities.

[0044] The third training objective is to minimize the difference between the predicted target language text generated based on pseudo-source language speech features and the corresponding translation labels of the second source language sample text. This objective aims to simulate a speech translation task using the second training data (i.e., plain text). Here, pseudo-source language speech features refer to text features generated in the feature space to simulate real speech features based on the second source language sample text through a text processing workflow. The predicted target language text refers to the translated text of the corresponding target language generated by the speech translation model after inputting the pseudo-source language speech features. The corresponding translation labels of the second source language sample text refer to the standard target language translation text prepared in advance for this plain text in the second training data.

[0045] Specifically, arbitrary second source language sample text (such as a Russian news text without paired speech) is extracted from the second training data and converted into pseudo-source language speech features by the text processing module. Subsequently, the pseudo-source language speech features are input into a speech translation model (such as shared encoding) to generate predicted text in the second target language. Then, the difference between the predicted text in the second target language and its corresponding translation label is calculated. This can be done using methods such as cross-entropy loss commonly used in masked language models, or other text difference measurement methods; this embodiment does not specifically limit this. The smaller the difference, the more accurate the translation result based on the simulated pseudo-source language speech features. Minimizing this difference allows the model to optimize its translation capabilities using only text data without real speech, thus enabling the model to learn to utilize more readily available text resources to enhance its performance in low-resource speech translation scenarios.

[0046] Therefore, the first training objective provides the model with a basic translation capability from real speech to target text. The second training objective constructs a unified feature space shared by speech and text, establishing a crucial bridge for simulating speech input using text data. The third training objective utilizes this shared feature space, enabling the model to learn from large-scale, easily accessible plain text data and enhance its translation capabilities, achieving effective knowledge transfer. Combining these three objectives in a clearly defined and complementary manner allows the speech translation model to not only learn direct translation knowledge from limited paired data but also to learn broader and deeper translation patterns and linguistic knowledge from massive amounts of plain text data. This results in overall translation performance and generalization ability far exceeding what can be achieved using only paired data for training.

[0047] The speech translation method provided in this embodiment constructs a joint training framework containing three training objectives. It uses paired first training data and plain text second training data to jointly train the speech translation model. This enables efficient use of relatively abundant text data to compensate for the scarcity of labeled speech data in low-resource language scenarios, thereby improving the performance of the speech translation model. It effectively solves the problem that traditional methods are difficult to train usable speech translation models due to the lack of large-scale paired speech data, and avoids the complex acoustic modeling process required in traditional methods, thus improving the efficiency of model training and translation effect.

[0048] Based on the above embodiments, the speech translation model is trained using the following steps: Step 210: Extract the speech features of the source language sample speech data, the text features of the first source language sample text, and the speech features of the pseudo-source language.

[0049] Specifically, speech features refer to high-dimensional vector representations extracted from raw audio signals that reflect the acoustic and semantic information of the speech content. They are used to characterize the intrinsic properties of speech signals. Speech features can be extracted through a speech encoder within the model. For example, by inputting a sample of source language speech data into the speech encoder, its corresponding speech features can be obtained.

[0050] Text features are high-dimensional vector representations extracted from source language text sequences that reflect the semantic information of the text content. They are used to characterize the inherent meaning of the text sequence. Text features can be extracted through a text encoder within the model. For example, inputting a sample text from the first source language into the text encoder will yield its corresponding text features.

[0051] Pseudo-source language speech features refer to text features generated by specific processing of plain text data (i.e., second source language sample text), which are used in the feature space to simulate real speech features. They are used to provide speech-like input signals for speech translation models in the absence of real speech. As an optional embodiment, the second source language sample text can first be transliterated into a letter sequence, then the letter sequence can be upsampled according to the predicted speech duration, and finally, the pseudo-source language speech features can be extracted from the upsampled sequence using a text encoder.

[0052] Step 220: Determine the differences between the speech features of the source language sample speech data and the text features of the first source language sample text based on the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text.

[0053] Specifically, the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text refers to the degree of proximity of the features of these two different modalities in the same high-dimensional latent space. It is used to characterize the quality of cross-modal feature alignment. The greater the similarity between the two, the more effective the unified representation space learned by the model is, that is, speech and text with the same content are mapped to very close positions in the feature space.

[0054] As an optional embodiment, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text can be determined by calculating the mean square error or cosine distance between the speech features and the text features.

[0055] Step 230: If the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is less than a threshold, determine the training loss value based on the differences between the first target language predicted text and the corresponding translation labels of the source language sample speech data, the differences between the speech features of the source language sample speech data and the text features of the first source language sample text, the differences between the second target language predicted text and the corresponding translation labels of the second source language sample text, and the differences between the third target language predicted text generated based on the first source language sample text and the corresponding translation labels of the source language sample speech data.

[0056] Specifically, if the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is less than a threshold, it indicates that the text processing module has the ability to generate high-quality speech-like features, and the feature spaces of speech and text have been initially aligned. At this point, a comprehensive joint optimization of the core part of the speech translation model can be initiated to efficiently improve the overall translation performance of the model.

[0057] The training loss value is determined based on the differences between the predicted text of the first target language and the corresponding translation labels of the source language sample speech data, the differences between the speech features of the source language sample speech data and the text features of the first source language sample text, the differences between the predicted text of the second target language and the corresponding translation labels of the second source language sample text, and the differences between the predicted text of the third target language generated based on the first source language sample text and the corresponding translation labels of the source language sample speech data.

[0058] Specifically, the difference between the predicted text of the first target language and the corresponding translation labels of the source language sample speech data can be measured using cross-entropy loss; the difference between the speech features of the source language sample speech data and the text features of the first source language sample text can be measured using mean squared error; the difference between the predicted text of the second target language and the corresponding translation labels of the second source language sample text can be measured using cross-entropy loss in the masked language model; and the difference between the predicted text of the third target language and the corresponding translation labels of the source language sample speech data can also be measured using cross-entropy loss. Here, the predicted text of the third target language refers to the predicted text obtained after processing the first source language sample text in the first training data through a language translation model.

[0059] As an optional embodiment, loss value 1 can be determined based on the difference between the predicted text of the first target language and the corresponding translation labels of the source language sample speech data; loss value 2 can be determined based on the difference between the speech features of the source language sample speech data and the text features of the first source language sample text; loss value 3 can be determined based on the difference between the predicted text of the second target language and the corresponding translation labels of the second source language sample text; and loss value 4 can be determined based on the difference between the predicted text of the third target language and the corresponding translation labels of the source language sample speech data. The sum of loss value 1, loss value 2, loss value 3 and loss value 4 can be used as the training loss value, or the four loss values ​​can be weighted and added together to obtain the training loss value. This embodiment does not specifically limit this.

[0060] Step 240: Update the parameters of the speech translation model based on the training loss value.

[0061] Specifically, since the training loss value integrates the overall error of the model in multiple tasks such as real speech translation, cross-modal feature alignment, and simulated translation using plain text, the parameters of the speech translation model can be updated based on this training loss value. This enables the model to learn in the direction of simultaneously optimizing multiple objectives, thereby comprehensively improving the translation accuracy and robustness of the model in low-resource language scenarios.

[0062] Furthermore, this embodiment introduces a conditional judgment mechanism based on cross-modal feature differences in the training process (i.e., judging whether the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is less than a threshold), thereby enabling an intelligent phased training strategy of alignment first and optimization later. This avoids invalid or unstable model parameter updates when the feature space is not aligned, ensuring the efficiency of the training process and the superior performance of the final model.

[0063] Based on any of the above embodiments, determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, according to the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text, includes: Align the speech features of the source language sample speech data with the text features of the first source language sample text to obtain aligned speech features and aligned text features; The first loss value is determined based on the similarity between the aligned speech features and the aligned text features; The second loss value is determined based on the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text. Based on the first loss value and / or the second loss value, determine the difference between the speech features of the source language sample speech data and the text features of the first source language sample text.

[0064] Specifically, considering that the initial speech features and text features belong to different modalities, there is a natural gap in their data distribution and feature representation. Direct comparison or replacement would cause the model to fail to correctly understand the information. Therefore, this embodiment performs feature alignment on the speech features and text features after acquisition, thereby laying the foundation for subsequent model training using text data to simulate speech data. Feature alignment can be understood as a process of mapping features from different modalities to a unified, modality-independent shared latent space.

[0065] As an optional implementation, the speech features of the source language sample speech data and the text features of the first source language sample text can be input into a shared encoder. The shared encoder processes and transforms these two types of features through its internal network layers, and its output is the aligned speech features and aligned text features. After feature alignment, the resulting aligned features become more similar in feature distribution, even difficult to distinguish, thereby enabling the model to understand and fuse information from speech and text at a deeper level.

[0066] Figure 2 This is a schematic diagram of the distribution of speech and text features provided by the present invention. Figure 3 This is a visualization diagram of the aligned speech-text feature distribution provided by the present invention, such as... Figure 2 and Figure 3 As shown, red represents speech features, and blue represents text features paired with speech features. Figure 2 In step 3, the unaligned speech features and text features are in a classified state, while the aligned speech features and text features are in a mixed state.

[0067] Furthermore, the similarity between aligned speech features and aligned text features refers to the degree of closeness of the two types of features in a unified latent space after deep fusion by the shared encoder, and it is used to characterize the alignment quality in the deep alignment stage. A high similarity between the two indicates that the shared encoder has successfully mapped inputs from different modalities to a highly unified representation space. The first loss value can be determined by calculating the mean squared error between the aligned speech features and the aligned text features.

[0068] The similarity between the speech features of the source language sample speech data and the text features of the first source language sample text refers to the degree of closeness between the features generated by their respective initial encoders (i.e., the speech encoder and the text encoder) before entering the shared encoder. It is used to characterize the alignment quality in the initial encoding stage. The greater the similarity between the two, the more likely the initial encoder has been able to generate features with a certain degree of similarity. The second loss value can also be determined by calculating the mean squared error between the speech features and the text features.

[0069] It should be noted that the first loss value supervises the output of the shared encoder at the depth alignment level, ensuring the uniformity of the final features. The second loss value supervises the outputs of the speech encoder and text encoder at the initial encoding level, providing a more direct and auxiliary supervision signal. The difference between the speech features of the source language sample speech data and the text features of the first source language sample text can be determined using either the first loss value to focus on optimizing the deep fusion effect, or the second loss value to focus on optimizing the extraction of initial features. Considering that single loss supervision may not be robust enough—for example, supervising only depth alignment (first loss value) may lead to sparse gradients in the initial encoder, while supervising only the initial encoding (second loss value) may not guarantee sufficient fusion of the final features—this embodiment preferably determines the difference between the speech features of the source language sample speech data and the text features of the first source language sample text based on the first and second loss values. For example, the difference can be determined based on the sum of the first and second loss values, or it can be determined by weighting and adding the first and second loss values; this embodiment does not specifically limit this approach.

[0070] Since the aforementioned hierarchical supervision mechanism can be applied to different depths of the model simultaneously, it can more stably and comprehensively guide the entire model to learn feature alignment from initial encoding to deep fusion based on the first and second loss values, thereby obtaining higher quality and more robust cross-modal unified representations, which significantly improves the effect of subsequent training using plain text data.

[0071] Based on any of the above embodiments, determining a first loss value according to the similarity between aligned speech features and aligned text features includes: Obtain the first reconstructed speech features and the first reconstructed text features. The first reconstructed speech features are obtained by self-attention weighting of the aligned speech features, and the first reconstructed text features are obtained by self-attention weighting of the aligned text features. The second reconstructed speech feature and the second reconstructed text feature are obtained. The second reconstructed speech feature is obtained by cross-attention weighting of the aligned speech feature with the aligned text feature as the query feature. The second reconstructed text feature is obtained by cross-attention weighting of the aligned text feature with the aligned speech feature as the query feature. Calculate the first mean square error between the first reconstructed speech feature and the second reconstructed speech feature, and the second mean square error between the first reconstructed text feature and the second reconstructed text feature; The first mean square error and the second mean square error are added together to obtain the first loss value.

[0072] Specifically, the first reconstructed speech feature is obtained by self-attention weighting of the aligned speech feature, and thus the first reconstructed speech feature contains contextual information about the relationships between the parts within the aligned speech feature. Similarly, the first reconstructed text feature is obtained by self-attention weighting of the aligned text feature, and thus the first reconstructed text feature contains contextual information about the relationships between the words or letters within the aligned text feature.

[0073] The second reconstructed speech feature is obtained by using the aligned text feature as the query feature and applying cross-attention weighting to the aligned speech feature, thus incorporating contextual guidance from the text modality. Similarly, the second reconstructed text feature is obtained by using the aligned speech feature as the query feature and applying cross-attention weighting to the aligned text feature, thus incorporating contextual guidance from the speech modality.

[0074] Considering that simply calculating the direct distance (such as mean squared error) between aligned speech features and aligned text features may not be sufficient to measure their consistency in deep semantic interaction, as such a static comparison cannot verify whether they exhibit similar behavior when interacting with other information, this embodiment introduces a reconstruction and comparison mechanism based on self-attention and cross-attention when calculating the first loss value. This allows for a more profound measurement of the alignment degree of the two modal features from the perspective of dynamic interaction, i.e., verifying whether they are functionally equivalent.

[0075] Specifically, the first mean square error between the first reconstructed speech features and the second reconstructed speech features, and the second mean square error between the first reconstructed text features and the second reconstructed text features are calculated, and the first mean square error and the second mean square error are added together to obtain the first loss value.

[0076] Since the first loss value measures the consistency of feature reconstruction results under two different contexts (self-modality and peer-modality), when this loss value decreases, it indicates that the aligned speech features and text features are not only similar in static representation, but also show a high degree of consistency in dynamic interaction and information fusion, proving higher quality feature alignment.

[0077] Among them, the first loss value It can be determined based on the following formula: in, Representing speech features, Representing text features, This represents the first reconstructed speech feature. This represents the first reconstructed text feature. This represents the second reconstructed speech feature. This represents the second reconstructed text feature.

[0078] Based on any of the above embodiments, determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, according to the first loss value and the second loss value, includes: Based on the sum of the first loss value and the second loss value, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined.

[0079] Specifically, the first loss value supervises the output of the shared encoder at the depth alignment level, while the second loss value supervises the outputs of the speech encoder and text encoder at the initial encoding level. Adding the two together constructs a comprehensive supervision signal covering the entire path from the initial encoding to the depth alignment pipeline, thereby enabling end-to-end optimization of the entire feature alignment pipeline.

[0080] By minimizing the difference obtained from the above summation, the differences in both the initial encoding stage and the depth alignment stage are reduced simultaneously, avoiding potential imbalances between different levels of the model. For example, if only the first loss value is used for optimization, the initial encoder may be undertrained, resulting in sparse gradient signals. The model would then need to expend more effort in the subsequent depth alignment stage, affecting overall training efficiency and stability. If only the second loss value is used for optimization, it may only guarantee the similarity of the initial features in the shallow representation, but it cannot ensure that these features can be effectively fused by the subsequent shared encoder, potentially leading to differences in the deep semantics of the final aligned features.

[0081] Based on any of the above embodiments, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined, and then the method further includes: If the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is greater than or equal to a threshold, the parameters of the speech translation model are frozen, and the parameters of the text processing module are updated based on the difference between the speech features of the source language sample speech data and the text features of the first source language sample text. The text processing module is used to extract transliterated letter sequences from the input text, upsample the letter sequences according to the duration of the speech corresponding to the input text, and extract text features from the upsampled character sequences.

[0082] Specifically, if the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is greater than or equal to a threshold, it indicates that the current training is still in its early stages, the text processing module's ability to generate text features is insufficient, and its output text features are far removed from the true speech features in the latent space, resulting in poor quality of cross-modal feature alignment. In this case, resources should be concentrated on prioritizing the training and optimization of the text processing module to enable it to generate high-quality speech-like features.

[0083] To avoid backpropagating erroneous gradient information to the entire translation model when feature representations are severely inconsistent, which could lead to unstable model training or even performance degradation, the parameters of the speech translation model are frozen (i.e., the parameters of the speech translation model are not updated). Instead, the parameters of the text processing module are updated based on the differences between the speech features of the source language sample speech data and the text features of the first source language sample text. This ensures that the text processing module can provide high-quality input before proceeding to the next stage of joint optimization.

[0084] The text processing module extracts transliterated letter sequences from the input text. Based on the duration of the corresponding speech in the input text, it upsamples the letter sequences and extracts text features from the upsampled character sequences. The transliterated letter sequences refer to the conversion of different forms of writing in the source language (such as Cyrillic letters, Chinese characters, Arabic letters, etc.) into a phoneme or pinyin representation based on a universal alphabet (such as the English alphabet). The speech duration refers to the length of time each letter or phoneme in the text actually occupies during pronunciation, predicted by the duration model.

[0085] Since speech is a continuous signal with a temporal length, while text is a discrete sequence of symbols, there is a natural difference in the sequence length between the two. Therefore, after extracting the letter sequence, it is necessary to upsample the letter sequence to simulate the time dimension of the real speech signal in terms of structure, so that the generated text feature sequence matches the speech feature sequence in length.

[0086] For example, the source language text "North Carolina" is transliterated into the letter sequence "beikaluolaina" after text conversion. Then, a duration model predicts the duration of each letter (e.g., 'b', 'e', ​​'i', etc.). Based on this duration, the letter sequence is upsampled, potentially resulting in a longer sequence like "bbeeeiikkaaalluuuoollaaiinnaa". Finally, this upsampled sequence is input into a text encoder to extract the final text features, which can be considered as pseudo-source language speech features.

[0087] Based on any of the above embodiments, the pseudo-source language speech features are generated based on the following steps: Extract the transliterated letter sequence from the second source language sample text; Upsample the letter sequence according to the speech duration corresponding to the second source language sample text; Extract the pseudo-source language speech features from the upsampled character sequence.

[0088] Specifically, transliteration can be to convert the words in the text into a spelling form based on letters (such as English letters), and this process can be achieved through preset language-specific rules or by using open-source tools (such as uroman). For example, the Chinese "North Carolina" can be converted to "beikaluolaina".

[0089] In addition, considering..., in this embodiment, the letter sequence is upsampled according to the speech duration corresponding to the second source language sample text to bridge the gap between text and speech structurally, so that the generated pseudo-source language speech features can effectively simulate real speech features in terms of sequence length. Among them, the speech duration refers to the time length or number of frames that each phoneme unit (such as a letter) in the text may occupy during actual pronunciation, which can be predicted by a pre-trained duration model and can estimate the time length or number of frames that each letter in the above letter sequence may occupy during actual pronunciation.

[0090] Upsampling is to repeat each letter in the letter sequence according to the predicted speech duration. For example, if it is predicted that the letter 'b' lasts for 2 frames and 'e' lasts for 3 frames, then "be" will be upsampled to "bbeee". Through this process, the letter sequence can be extended into a long sequence that is comparable to the real speech feature sequence in length, thus structurally simulating the speech timeliness.

[0091] Finally, extract the pseudo-source language speech features from the upsampled character sequence. Optionally, the upsampled character sequence can be input into the text encoder in the model to output the final pseudo-source language speech features. Due to the fact that the generation process of this feature integrates the phoneme information in the transliteration process and the timing information in the upsampling process, it can effectively simulate real speech features in the feature space.

[0092] This embodiment converts discrete text data into a high-dimensional representation that is both structurally and content-wise近似于真实语音特征的高维表示,从而使得语音翻译模型能够在仅有纯文本数据的情况下进行有效的模拟语音翻译训练,极大地扩展了可用于训练的数据量,有效解决了低资源语种场景下标注语音数据严重匮乏的问题.

[0093] It should be noted that there is an unclear expression "近似于真实语音特征的高维表示" in the original text. It may need to be further clarified for a more accurate translation. The above translation is based on the existing text as much as possible.Based on any of the above embodiments, extracting pseudo-source language speech features from the upsampled character sequence includes: Extract initial pseudo-speech features from the upsampled character sequence; By using a speech translation model, the initial pseudo-speech features are aligned with the real speech features to obtain pseudo-source language speech features.

[0094] Specifically, the initial pseudo-speech features can be understood as a vector representation obtained by a preliminary encoding of the character sequence after transliteration and upsampling. Considering that although the features have simulated speech in terms of phonemes and temporal structure, the data distribution in the high-dimensional feature space may still differ from the real speech features, this embodiment further utilizes a speech translation model to align the initial pseudo-speech features with the real speech features to obtain pseudo-source language speech features.

[0095] Because the shared encoder has already learned during the training phase to map features from different modalities (real speech and text) to a unified latent space, the initial pseudo-speech features can be input into the shared encoder, which processes and transforms them, and its output is the final, depth-aligned pseudo-source language speech features.

[0096] Based on any of the above embodiments, the second target language predicted text is generated based on the following steps: Using a speech translation model, the speech features of the pseudo-source language are masked, and the masked pseudo-source language speech features are decoded to obtain the predicted text of the second target language.

[0097] Specifically, a mask can be understood as randomly replacing a portion of the feature vectors in the pseudo-source language speech features with a special "[MASK]" marker or a random vector.

[0098] Considering that if the model always sees complete, lossless pseudo-source language speech features, its task of predicting the corresponding translated text may become relatively simple, easily leading to overfitting and insufficient generalization ability. To increase the difficulty of the training task and force the model to learn deeper and more robust contextual dependencies, this embodiment introduces a masking mechanism to transform the original translation task into a more challenging cloze test task.

[0099] Specifically, this embodiment uses a speech translation model to mask the pseudo-source language speech features and decode the masked pseudo-source language speech features. It then generates the final target language text sequence word by word or character by autoregression. This sequence is the second target language predicted text.

[0100] Based on any of the above embodiments Figure 4This is a flowchart illustrating the speech translation model training method provided by the present invention, as shown below. Figure 4 As shown, the speech translation model includes a speech encoder, a shared encoder, an adapter, and a large language model (LLM). During the training process, the speech translation model also needs to be trained in conjunction with a text processing module, which includes text transfer, a duration model, and a text encoder.

[0101] like Figure 4 As shown, first training data and second training data are obtained. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text.

[0102] Next, the source language sample speech data is input into the Speech Encoder, which encodes the source language sample speech data to obtain the speech features H of the source language sample speech data. s Simultaneously, the first source language sample text is input into Text Transfer, which upsamples the letter sequence of the first source language sample text based on the speech duration predicted by the Duration Model. The upsampled letter sequence is then input into TextEncoder to obtain the text features H of the first source language sample text. t Based on the speech features H of the source language sample speech data s Textual features H of the first source language sample text t The difference between them is used to calculate the second loss value L. AM3 .

[0103] Similarly, the second source language sample text is input into Text Transfer, which upsamples the letter sequence of the second source language sample text based on the speech duration predicted by the Duration Model, and then inputs the upsampled letter sequence into Text Encoder to obtain the pseudo-source language speech features.

[0104] Subsequently, the speech features H of the source language sample speech data were analyzed. s Text features H of the first source language sample text t The input is fed into the Shared Encoder, which performs feature alignment between the two to obtain the aligned speech feature H. s1and aligned text feature H t1 Based on the aligned speech feature H s1 and aligned text feature H t1 The difference between them is used to calculate the first loss value L. ’ AM3 .

[0105] Align speech features H s1 The inputs are fed into the adapter, which adjusts them into vectors that are easier for LLM to process. The processed speech features are then input into the LLM to obtain the first target language predicted text. The loss value L is determined based on the difference between the first target language predicted text and the corresponding translation labels of the source language sample speech data. LM .

[0106] Aligned text features H t1 The text is masked, and the masked features are sequentially input into the Adaptor and LLM to obtain the predicted text for the third target language. The loss value L is determined based on the difference between the predicted text for the third target language and the corresponding translation labels of the source language sample speech data. MLM .

[0107] Similarly, the pseudo-source language speech features are input into the Shared Encoder for feature alignment. After masking the aligned features, the masked features are sequentially input into the Adaptor and LLM to obtain the second target language predicted text. The loss value L is determined based on the difference between the second target language predicted text and the corresponding translation labels of the second source language sample text. ’ MLM .

[0108] If the first loss value L ’ AM3 With the second loss value L AM3 If the sum is greater than or equal to the threshold, then freeze the parameters of Speech Encoder, Shared Encoder, Adaptor, and LLM, and update the parameters of Duration Model and Text Encoder.

[0109] If the first loss value L ’ AM3 With the second loss value L AM3 If the sum is less than the threshold, then according to L AM3 L ’ AM3 L LM L MLM L ’ MLMCalculate the training loss value and update the parameters of Speech Encoder, Shared Encoder, Adaptor, and LLM based on the training loss value.

[0110] The speech translation device provided by the present invention will be described below. The speech translation device described below can be referred to in correspondence with the speech translation method described above.

[0111] Based on any of the above embodiments Figure 5 This is a schematic diagram of the speech translation device provided by the present invention, as shown below. Figure 5 As shown, the device includes: Determining unit 510 is used to determine the source language speech data; Translation unit 520 is used to translate source language speech data into target language text based on a speech translation model; The speech translation model is trained based on first training data and second training data. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text. The training objectives of the speech translation model include minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on pseudo-source language speech features and the corresponding translation label of the second source language sample text. The pseudo-source language speech features are text features generated based on the second source language sample text to simulate the real speech features under the source language.

[0112] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a speech translation method.

[0113] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech translation methods provided by the above methods.

[0115] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the speech translation methods provided by the methods described above.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech translation method, characterized in that, include: Identify the source language speech data; Based on a speech translation model, the source language speech data is translated to obtain target language text. The speech translation model is trained based on first training data and second training data. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text. The training objectives of the speech translation model include minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on pseudo-source language speech features and the corresponding translation label of the second source language sample text. The pseudo-source language speech features are text features generated based on the second source language sample text to simulate the real speech features under the source language.

2. The speech translation method according to claim 1, characterized in that, The speech translation model is trained based on the following steps: Extract the speech features of the source language sample speech data, the text features of the first source language sample text, and the speech features of the pseudo source language; Based on the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined. If the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is less than a threshold, the training loss value is determined based on the difference between the first target language predicted text and the corresponding translation label of the source language sample speech data, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, the difference between the second target language predicted text and the corresponding translation label of the second source language sample text, and the difference between the third target language predicted text generated based on the first source language sample text and the corresponding translation label of the source language sample speech data. The parameters of the speech translation model are updated based on the training loss value.

3. The speech translation method according to claim 2, characterized in that, The step of determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text based on the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text includes: The speech features of the source language sample speech data are aligned with the text features of the first source language sample text to obtain aligned speech features and aligned text features. A first loss value is determined based on the similarity between the aligned speech features and the aligned text features; The second loss value is determined based on the similarity between the speech features of the source language sample speech data and the text features of the first source language sample text. Based on the first loss value and / or the second loss value, determine the difference between the speech features of the source language sample speech data and the text features of the first source language sample text.

4. The speech translation method according to claim 3, characterized in that, The step of determining the first loss value based on the similarity between the aligned speech features and the aligned text features includes: Obtain a first reconstructed speech feature and a first reconstructed text feature, wherein the first reconstructed speech feature is obtained by self-attention weighting of the aligned speech feature, and the first reconstructed text feature is obtained by self-attention weighting of the aligned text feature; Obtain a second reconstructed speech feature and a second reconstructed text feature. The second reconstructed speech feature is obtained by performing cross-attention weighting on the aligned speech feature as the query feature. The second reconstructed text feature is obtained by performing cross-attention weighting on the aligned text feature as the query feature. Calculate the first mean square error between the first reconstructed speech feature and the second reconstructed speech feature, and the second mean square error between the first reconstructed text feature and the second reconstructed text feature; The first mean square error and the second mean square error are added together to obtain the first loss value.

5. The speech translation method according to claim 3, characterized in that, Based on the first loss value and the second loss value, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined, including: Based on the sum of the first loss value and the second loss value, the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is determined.

6. The speech translation method according to claim 2, characterized in that, The process of determining the difference between the speech features of the source language sample speech data and the text features of the first source language sample text further includes: If the difference between the speech features of the source language sample speech data and the text features of the first source language sample text is greater than or equal to the threshold, the parameters of the speech translation model are frozen, and the parameters of the text processing module are updated based on the difference between the speech features of the source language sample speech data and the text features of the first source language sample text. The text processing module is used to extract the transliterated letter sequence from the input text, upsample the letter sequence according to the duration of the speech corresponding to the input text, and extract text features from the upsampled character sequence.

7. The speech translation method according to any one of claims 1 to 6, characterized in that, The pseudo-source language speech features are generated based on the following steps: Extract the transliterated letter sequence from the second source language sample text; The letter sequence is upsampled based on the duration of the speech corresponding to the second source language sample text; Extract the pseudo-source language speech features from the upsampled character sequence.

8. The speech translation method according to claim 7, characterized in that, The extraction of the pseudo-source language speech features from the upsampled character sequence includes: Extract initial pseudo-speech features from the upsampled character sequence; Using the speech translation model, the initial pseudo-speech features are aligned with the real speech features to obtain the pseudo-source language speech features.

9. The speech translation method according to any one of claims 1 to 6, characterized in that, The second target language predicted text is generated based on the following steps: Using the aforementioned speech translation model, the pseudo-source language speech features are masked, and the masked pseudo-source language speech features are decoded to obtain the predicted text of the second target language.

10. A voice translation device, characterized in that, include: The determining unit is used to determine the source language speech data; The translation unit is used to translate the source language speech data based on a speech translation model to obtain target language text; The speech translation model is trained based on first training data and second training data. The first training data includes source language sample speech data and first source language sample text paired with the sample speech data. The second training data includes second source language sample text. The training objectives of the speech translation model include minimizing the difference between the first target language predicted text generated based on the source language sample speech data and the corresponding translation label of the source language sample speech data, minimizing the difference between the speech features of the source language sample speech data and the text features of the first source language sample text, and minimizing the difference between the second target language predicted text generated based on pseudo-source language speech features and the corresponding translation label of the second source language sample text. The pseudo-source language speech features are text features generated based on the second source language sample text to simulate the real speech features under the source language.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech translation method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech translation method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the speech translation method as described in any one of claims 1 to 9.

Images