Simultaneous translation from source language to target language

Abstract

The embodiment of the invention provides a speech translation method and device, equipment and a storage medium. The method comprises the following steps: in response to a first audio clip in a source language, converting the first audio clip into first audio embedding; in response to determining that the at least one audio clip is obtained before the first audio clip and / or at least one translation clip in a target language corresponding to the at least one audio clip, constructing a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio clip, and at least one translation embedding corresponding to the at least one translation clip; and using the trained machine learning model to generate a first translation segment in the target language corresponding to the first audio segment based on the first model input.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for speech translation. Background Art

[0002] Spoken translation of conversations, statements, questions, etc. involves the translation of utterances spoken in a source language into utterances spoken in a target language. Simultaneous speech translation (SiST) is identified as one of the most challenging tasks in the translation domain. Machine-assisted automatic interpretation has received a lot of attention in the natural language processing (NLP) community. Traditional simultaneous translation methods typically employ a cascade system involving a streaming automatic speech recognition (ASR) model, a punctuation model, and a machine translation (MT) model. However, such cascade systems often suffer from error propagation and delays from the ASR module. Despite these advances in both academic SiST models and commercial SiST engines, the translation quality is still far from satisfactory. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for speech translation is provided. The method includes: in response to obtaining a first audio segment in a source language, converting the first audio segment into a first audio embedding; in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment, constructing a first model input based on a first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment; and using a trained machine learning model, based on the first model input, generating a first translation segment in the target language corresponding to the first audio segment.

[0004] In a second aspect of the present disclosure, an electronic device is provided. The device includes: at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device performs actions including: in response to obtaining a first audio segment in a source language, converting the first audio segment into a first audio embedding; in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment, based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment, construct a first model input; and using the trained machine learning model, based on the first model input, generate a first translation segment in the target language corresponding to the first audio segment.

[0005] In a third aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0006] In a fourth aspect of the present disclosure, a device for speech translation is provided. The device includes: an embedding conversion module configured to convert the first audio segment into a first audio embedding in response to obtaining a first audio segment in a source language; an input construction module configured to construct a first model input based on a first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment; and a translation generation module configured to generate a first translation segment in the target language corresponding to the first audio segment using a trained machine learning model based on the first model input.

[0007] It should be understood that the contents described in the summary of the present invention are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A schematic diagram showing an architecture of model-based speech translation according to some embodiments of the present disclosure;

[0011] Figures 3A-3C An example of a machine learning model according to some embodiments of the present disclosure is shown;

[0012] Figure 3D-Figure 3I Some examples of comparison of translation results between related products and some embodiments of the present disclosure are shown;

[0013] Figure 4 A flowchart showing a process for speech translation according to some embodiments of the present disclosure is shown;

[0014] Figure 5A schematic structural block diagram of an apparatus for speech translation according to some embodiments of the present disclosure is shown; and

[0015] Figure 6 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the association between various data. For example, the above-mentioned correlation relationship can be obtained based on various technical solutions currently known and / or to be developed in the future.

[0018] It should be understood that before applying the technical solutions disclosed in various embodiments of the present disclosure, the type, scope of use and usage scenarios of personal information involved in the technical solutions described herein should be informed to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0019] It should be understood that before applying the technical solutions disclosed in various implementation methods of the present disclosure, the type, scope of use and usage scenarios of personal information involved in the technical solutions described herein should be informed to the user in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly inform the user that the requested operation will obtain and use the user's personal information. Therefore, based on the prompt message, the user can decide whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution described in this article.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending the prompt information to the user may include, for example, a pop-up window, and the prompt information may be presented in the pop-up window in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not limit the implementation of the present disclosure. Other methods that meet relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0023] As used herein, the term "model" can learn the correlation between corresponding inputs and outputs from training data so that corresponding outputs can be generated for given inputs after training. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, a "model" may also be referred to as a "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0024] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing an input and providing a corresponding output, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.

[0025] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the reasoning stage). In the training stage, a large amount of training data can be used to train a given model, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent reasoning that meets the expected goals from the training data. Through training, the model can be considered to be able to learn the association from input to output (also called input to output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input and determine the corresponding output based on the parameter values ​​obtained through training.

[0026] Figure 1 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 may include a speech translation system 110 that applies a machine learning model 112 to implement speech translation.

[0027] The speech translation system 110 receives audio 101 captured in a physical environment, such as in a meeting or from any other data source. The audio 101 contains content in some language, referred to as the source language. The speech translation system 110 translates the audio 101 in the source language using a machine learning model 112 to obtain a translation result 102 in a target language, which may be different from the source language. The translation result 102 may then be presented to (one or more) users. The source language and the target language may be any language, such as a natural language.

[0028] The machine learning model 112 can be a model suitable for the audio modality of processing data. The translation result 102 can be any appropriate modality, for example, it can be a translated text sequence or a translated audio. The translated text sequence and the translated audio correspond to each other in the target language. For example, the machine learning model 112 can output a translated text sequence, and then the translated text sequence is converted to a translated audio via text to speech (TTS) technology to play back to (one or more) users. The machine learning model 112 can output a translated audio, and the translated audio is then converted into a translated text sequence to be presented to (one or more) users via audio speech recognition (ASR) technology.

[0029] In the use case of simultaneous speech translation, the speech translation system 110 is configured to provide translation results for audio segments in an audio stream, rather than waiting for all audio to be captured.

[0030] The machine learning model 112 can be, for example, any neural network that can perform feature extraction and feature aggregation, including but not limited to a fully convolutional network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc., and the embodiments of the present disclosure are not limited in this respect. In some embodiments, the machine learning model 112 can be stored locally in the speech translation system 110. The speech translation system 110 can directly use the local machine learning model 112 to implement feature extraction and feature aggregation when it is necessary to perform tasks associated with feature extraction and feature aggregation. In some embodiments, the machine learning model 112 can also be a model stored in the cloud. The speech translation system 110 can send the input 101 to the machine learning model 112 in the cloud, and obtain the translation result 102 from the machine learning model 112 in the cloud.

[0031] The speech translation system 110 may include or be implemented in any type of device capable of computing, including a terminal device or a server device. The terminal device may be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination thereof, including accessories and peripheral devices of these devices or any combination thereof. The server device may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc.

[0032] It should be understood that the structure and functionality of environment 100 are described for illustrative purposes only and does not imply any limitation on the scope of the present disclosure.

[0033] Traditional simultaneous translation methods usually adopt cascade systems, involving streaming automatic speech recognition (ASR) models, punctuation models and machine translation (MT) models. However, this cascade system often suffers from error propagation and delays from the ASR module. Despite these advances in both academic SiST models and commercial SiST engines, the translation quality is still far from satisfactory. However, from a user-centric perspective, these systems cannot deliver enough effective information to the audience, which seriously affects the effectiveness of communication. In contrast, professional human interpreters usually deliver more necessary information. This difference means further improvements for machine-assisted SiST systems to reach human capabilities.

[0034] Motivated by the great success of large-scale language models in machine translation and speech translation, we proposed to apply language models to the SiST task. The challenge of incorporating language models into the SiST task is the read-write strategy, where the language model needs to provide a partial translation for the input speech.

[0035] According to an example embodiment of the present disclosure, an improved scheme for speech translation is provided. In the scheme, a model-based end-to-end method is proposed to achieve simultaneous interpretation by iteratively performing multiple actions for audio segments. For a first audio segment in a source language, it is converted into a first audio embedding. Based on the first audio feature embedding, at least one audio embedding corresponding to at least one previous audio segment (if available), and at least one translation embedding corresponding to at least one translation segment, a model input is constructed. The model input is provided to a trained machine learning model to generate a translation segment in a target language corresponding to the first audio segment. The translation segment and / or the first audio segment can be stored and provided for use in the translation of subsequent audio segments.

[0036] In this way, the accuracy of the translation of an audio segment may be improved by taking into account previous audio segments and previous translated segments to provide a translation with semantic completeness.

[0037] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0038] Figure 2 FIG. 2 is a schematic diagram of an architecture 200 for speech translation according to some embodiments of the present disclosure. The architecture 200 may be implemented in Figure 1 The various units and / or encoders in the architecture 200 may be or utilize the speech translation system 110 in the speech translation system 110. Figure 1 For the sake of discussion, reference will be made to Figure 1 The architecture 200 is described with reference to the environment 100 in FIG.

[0039] like Figure 2 As shown, at step 201, the speech translation system 110 receives an audio clip 212 in an environment 210. This step is marked as an <input> operation to obtain input to the machine learning model 112.

[0040] In some embodiments, the machine learning model 112 can be constructed based on an encoder-conditioned language model architecture. The audio clip 212 is converted into an audio embedding (also called an audio representation) by the audio encoder for processing by the conditional language model.

[0041] Figure 3A An example architecture of the machine learning model 112 is shown, including an audio encoder 310 and a conditional language model 330. The conditional language model 330 can be based on a large language model. The audio encoder 310 converts the audio clip into an audio embedding to be input to the conditional language model 330.

[0042] In some embodiments, the audio encoder 310 may include a large-scale speech conformer pre-trained on a large amount of speech data and an audio adapter connected to the conditional language model 330. The audio adapter may downsample the audio representation, and the resulting representation is linearly projected to match the dimension of the embedding layer of the conditional language model 330. In some examples, the projected audio representation may reduce computational latency for SiST.

[0043] In some embodiments, the language model is a medium-sized decoder-only transformer to balance performance and computational efficiency. The conditional language model 330 can be pre-trained on large amounts of text data and fine-tuned using instructions. The conditional language model 330 can directly take continuous embeddings from both the audio encoder and the text embedder as input. The conditional language model 330 can autoregressively generate a translation response for a provided audio clip (and optionally, provide a transcription of the audio clip).

[0044] Assume that Figure 3A In the example of , an audio segment 212 is obtained for translation. If no previous audio segment is obtained or stored for the current audio segment 212 to be processed, the model input to the conditional language model 330 may include the audio embedding of the audio segment 212 and instructions 324, which contain prompt information to indicate the translation from the source language to the target language. Figure 3AAs shown, instruction 324 may instruct streaming translation into "English" (which is the target language). Conditional language model 330 may then perform translation of audio segment 212 based on such instructions. In some examples, instruction 324 may be configured to also instruct conditional language model 330 to perform transcription on the audio segment. For example, instruction 324 may instruct conditional language model 330 to transcribe into Chinese and then translate into English. In this case, in addition to providing the translation result of the input audio in English, conditional language model 330 may also provide the Chinese transcription text of the input audio segment in Chinese.

[0045] In some embodiments, the machine learning model 112 is configured to perform streamlined generation. The machine learning model 112 can generate translation results by detecting whether the currently generated text sequence has semantic completeness. In spoken language, people may have modal particles, stuttering, repetitions, etc., and sometimes background speech may be inserted into the user who is speaking. In some cases, in order to reduce the complexity in capture and upload decisions, audio clips with relatively fixed time periods are input for simultaneous translation. In these cases, the audio clip can start at any time point in the audio stream and end at any time point in the audio stream. This makes it inaccurate to translate the ASR results of the audio clip in the source language directly into the target language.

[0046] Therefore, by considering semantic completeness in speech translation, rather than translating all the content contained in the audio segment, some content in the audio segment may not be translated into the target language because they may damage the semantic completeness of the translated text sequence.

[0047] In some embodiments, because the machine learning model 112 can output the translation results (e.g., a translated text sequence) in a streamlined manner, it can sequentially detect the semantic completeness of one or more translated text units in the target language that is translated from the audio segment. If the machine learning model 112 determines that a first portion of the current audio segment with semantic completeness is translated, and a second portion after the first portion may compromise the semantic completeness, the machine learning model 112 may only generate a translated text sequence corresponding to the first portion. For example, if Figure 3B As shown, portion 302 in audio segment 212 may not contribute to semantic completion, and thus conditional language model 330 may not output translated text sequence 226 to include a translation of portion 302 .

[0048] In some embodiments, the machine learning model 112 can detect the semantic completeness of the translated text sequence in the target language from which the audio segment is translated based on contextual information (e.g., one or more previous audio segments and / or translation results corresponding to one or more previous audio segments that may be stored in the memory 240).

[0049] The challenge for machine learning models is to understand and translate words and uncommon phrases that language models cannot learn from training data. To address this challenge, in some embodiments, Figure 2 As shown, one or more of the following modules are included to enhance the machine learning model 112: a knowledge base 230 storing words and pairwise translations, and a memory 240 storing the context of speech. The knowledge base 230 and the memory 240 may be embedded in the speech translation system 110, or may be external modules.

[0050] Memory 240 stores previous transcriptions (optional) and translations during an interactive session for an audio stream in environment 210. At turn r, it first reads the previous audio For example, one or more audio segments obtained before the current audio segment 212, where t r-1 is the prediction cutoff timestamp of round r-1, and T r is the end time of the audio stream at turn r. In some embodiments, at step 202, the speech translation system 110 can retrieve relevant information k from the knowledge base 230. r (marked as <retrieval>). At step 203, the speech translation system 110 also loads the historical context information y from the memory of the last round 1：r-1 (marked as <loading information>(<LOAD_MEM> )). Then, the model input constructed for the audio segment 212 can be expressed as k r ,y 1:r-1 In some examples, such as Figure 3A As shown, the model input to the conditional language model 330 of the machine learning model 112 may also include contextual information 326.

[0051] In some embodiments, if predicted cutoff timestamp(s) for one or more previous audio segments are output by the machine learning model 112 and stored in the memory 240, such cutoff timestamps may also be included in the model input.

[0052] The speech translation system 110 may provide a model input to the machine learning model 112 to generate a translation result in the target language for the audio segment 212 in the source language, such as a translated text sequence. In some embodiments, the machine learning model 112 may determine a cutoff timestamp in the audio segment 212 at round r, which identifies a time in the audio segment 212 before the complete semantics are detected. The machine learning model 112 may then generate a translation result corresponding to the audio portion of the audio segment 212 before the cutoff timestamp.

[0053] At round r, the output of the machine learning model 112 may be represented as follows: Among them, t r is the predicted deadline timestamp, indicating the end time of the current translation round r. Then y r is forwarded to update the memory 240. In some embodiments, when instructed to output a transcription, the machine learning model 112 may optionally participate in a chain of thoughts (CoT) to first generate a transcription and then generate a translation of the audio segment at turn r. For the next turn r+1, the audio stream begins at the predicted cutoff timestamp.

[0054] In some embodiments, the model output y at each round r At least the translation segment (e.g., the translation text sequence) in the target language may be included. The translation segment (e.g., the translation text sequence) may then be presented to the user. For example, in Figure 2 In step 204, the translated text sequence 226 (labeled as <output>) in English corresponding to the audio segment 212 may be presented to the user in the environment 210. In some embodiments, the model output y at each round r It may also include a transcription of the audio clip in the source language. In this case, the transcription may be presented to the user. For example, Figure 2 In the example, Chinese transcribed text sequence 224 corresponding to audio segment 212 may be presented to a user in environment 210 .

[0055] In architecture 200, a new driver's read and write strategy is applied. The strategy can effortlessly balance translation quality and latency without complex human pre-designed requirements. Unlike most systems, which frequently rewrite output during the translation process for better quality, the strategy proposed in this article ensures that all outputs are deterministic while maintaining high quality.

[0056] In real-world scenarios, accurate speech transcription or translation of professional and domain-specific words is challenging. Even human interpreters require prior domain knowledge to understand those words, including people's names, locations, jargon, or special words in the field. For example, when hearing machine learning theory, interpreters who are not familiar with machine learning theory may not recognize the word "Rademacher complex". Therefore, in various scenarios, human interpreters are often familiar with the corresponding domain knowledge in advance. Motivated by the preparation trajectory of human interpreters, in some embodiments, it is proposed to integrate external databases to give machine learning models the necessary domain-specific knowledge. Each item in the database contains keywords and corresponding values ​​in text modality. Keywords that can appear in speech are used as input for retrievers. The value of the item can be itself, a paired translation in the target language, or even an interpretation of the keyword.

[0057] In theory, all items in the external database can be added to the prompt words to provide information for translation. Considering that the knowledge base may contain a large number of terms, it not only increases the reasoning time, but also may reduce the model performance due to noise intervention. Simply using all items to prompt the machine learning model not only increases the reasoning time, but also may damage the performance of the machine learning model 112 due to noise intervention. Therefore, in some embodiments, a multimodal retrieval enhanced generation (MM-RAG) process may be incorporated into the architecture 200. The speech translation system 110 can apply a multimodal retriever to extract knowledge from the knowledge base 230 based on the audio clip input. The multimodal retriever first retrieves relevant words from the database based on the input speech. Figure 2 As shown, a small number of filtered items are incorporated into the prompts of the machine learning model 112 for learning in context.

[0058] When constructing a model input for the audio segment 212, the speech translation system 110 can determine at least one word from the knowledge base 230 (also referred to as a knowledge base) that matches the audio segment 212. In some embodiments, the speech translation system 110 can apply a multimodal retriever to determine a match between a word and an audio segment.

[0059] Using the knowledge retrieved from the memory 240 and the previous context, the machine learning model 112 has the learning ability in context to better utilize the provided context information. To achieve this, a series of learning data in context can be collected to train the model. Compared with previous methods for interventions, such as shallow fusion and traditional replacement-based methods, which generate fixed translations for a given translation pair, the merging of knowledge bases can achieve better results and generate more coherent text. For example, in some Internet companies, the Chinese character "大盘" means "overall performance" in English, while in most cases, it should be translated as "stock market" in English. In some embodiments of the present disclosure, different contexts can be given by the speech translation system to select the correct translation. Additionally, in some embodiments, monolingual texts from both the source language and the target language can be used to assist in translation.

[0060] In some embodiments, the knowledge base 230 may include word pairs in the source language and their corresponding word pairs in the target language, or vice versa. For example, as Figure 2 shown, the CN-EN pair for "Ising model" and the EN-CN pair for "伊辛" can be included in the knowledge base 230. In some embodiments, the knowledge base 230 may additionally or alternatively include words in one of the source language or the target language. For example, in Figure 2 , the knowledge base 230 may include Chinese and / or English expressions for "ferromagnetism". It should be understood that the knowledge base 230 may include many other word pairs in two or more different languages, or words in one language. The scope of the present disclosure is not limited in this regard.

[0061] In some embodiments, the multimodal retriever may employ audio and text encoders to independently encode the audio segments and text keys of the words in the knowledge base 230. To enhance the alignment between the audio embedding and the text embedding, the multimodal retriever may include an embedding fusion layer, which includes a multi-head attention module followed by a pooling layer. The resulting pooled representation is then fed into a linear projection layer to produce a final score indicating the probability of the presence of the text key in the audio segment. The word with the highest score is forwarded to the machine learning model 112 to enhance the translation quality.

[0062] For example, for the audio segment 212, the speech translation system 110 may employ the multimodal retriever to retrieve the CN-EN pair for "Ising model" and / or the EN-CN pair for "伊辛" at step 202, as these two pairs are determined to match the audio segment 212. Then, the model input to the machine learning model 112 for the translation of the audio segment 212 may also include the retrieved words, for example, the text embedding including the retrieved words. In some examples, as Figure 3A and Figure 3C As shown, the model input to the conditional language 330 of the machine learning model 112 may also include context information 326. In some embodiments, in the translation result for the current audio segment, the matching words in the target language may be included in the translated text sequence or translated audio corresponding to the input audio segment. Figure 2 In the example, the English translated text sequence 226 may include the matching words “Ising model”, “Ising”.

[0063] In some embodiments, the multimodal retriever can be independently trained using a substantial data set of speech recognition data. During training, sentences are randomly selected from the speech transcription to serve as positive samples, indicating their occurrence in the speech. In some embodiments, negative sentences are selected from different sentences, indicating that the speech does not mention these sentences. A label of 1 can be assigned to the positive sample, and a label of 10 can be assigned to the negative sample, aiming to minimize the binary cross entropy (BCE) loss. This method helps to improve the model's ability to distinguish between relevant and irrelevant information, enhancing its overall performance and accuracy. In some embodiments, positive samples can be labeled as 1, and negative samples can be labeled as 0, minimizing the binary cross entropy (BCE) loss. In order to evaluate the effectiveness of the multimodal retriever, an internal retrieval development set can be constructed. Each sample in the development set includes a short audio block and a word mentioned in the audio. Note that the words here can be defined as special keywords, such as names, locations, abbreviations, and words for specific domains.

[0064] According to an embodiment of the present disclosure, the speech translation system 110 may iteratively translate the corresponding audio segments obtained from the environment 210. For example, Figure 3C, after the audio segment 212 is translated, a subsequent audio segment is further obtained, which can be processed in a similar manner to the audio segment 212. The subsequent audio segment can be considered to be connected to the previous audio segment 212 as an audio segment 314, because the previous audio segment 212 is obtained as a context to facilitate translation. The connected audio segment can be converted by the audio encoder 310 to obtain the corresponding audio embedding. The model input constructed for the audio segment can also include audio embedding, embedding corresponding to context information 326, instructions 324, and knowledge 322 retrieved from the knowledge base 250 (if available). Context information (e.g., (one or more) translation segments corresponding to the previous (one or more) audio segments) can also be converted into the embedding space as a translation embedding for processing by the conditional language model. Based on the model input, the conditional language model 330 can output a translation segment, for example, a translation text sequence for a new audio segment. In some embodiments, by considering the semantic completeness of the translation result, the conditional language model 330 can output a partial translation that is not included in the translated text sequence, because the partial translation may damage the semantic completeness of the previous translation result, but is beneficial to the semantic completeness of the current audio segment. Similarly, a portion of the new audio segment that damages the semantic completeness of the current audio segment (e.g., Figure 3C The portion 304 in the translation text sequence may not be directly translated into the target language in the translation text sequence.

[0065] In architecture 200, memory 240 stores the previous round y 1：r-1 It has at least two functions. First, it works with the input speech to determine which part of the speech has been translated and which part has not yet been translated, thereby helping the machine learning model 112 make reading and writing decisions, and output the translation of the unfinished part. Second, understanding human speech often requires context. For example, when a speaker talks about a "barrel bridge", it often refers to a bridge built on a river supported by barrels. However, in the context of "watch", it refers to the mechanical structure in a watch. Polysemy in different contexts may lead to very different translation results. Therefore, the machine learning model 112 may be able to retrieve the context of a long speech for translating some keywords, and make appropriate translations in different contexts.

[0066] In each round, the translation results of the previous round are loaded as context information to facilitate the machine learning model 112 to detect the semantic completeness of the translated text sequence generated for the audio segment to be translated in the current round. Figure 3B A schematic diagram for such an iterative translation is shown in FIG. Figure 3BAs shown, at the turn r of the translation for the audio segment 212, the information marked as <loading information> (<LOAD_MEM> ) can obtain the relevant translation 350 of the machine learning model 112 from the memory 210 as a prompt word (part of the model input) at round r-1. After the machine learning model 112 generates the translation output 352 at round r, it is marked as <update information> (<UPDATE_MEM> ) can store the translation output 352 in the memory 240 and obtain y 1：r That is, the translation outputs generated from consecutive rounds may be aggregated and stored in memory 240. In some embodiments, depending on an upper limit on the size configured for memory 240, the translation outputs stored in memory 240, or a portion thereof, may be flushed.

[0067] For example, if the upper limit of the translation result corresponding to a total of 30 seconds of audio is configured, the information stored in the memory 240 may be deleted following a first-in-first-out (FIFO) principle. In other examples, all information stored in the memory 240 before the first 30 seconds may be deleted to wait for further context information in the next 30 seconds. It should be understood that there may be various other ways to control the time window of the context information stored in the memory.

[0068] In order to enable the machine learning model 112 to provide partial translations with semantic completeness for the input partial audio, during the model training phase, professional human interpreters are imitated to learn their strategies for segmenting complete sentences into several semantic fragments by syntactic boundaries (pauses, commas, conjunctions, etc.) and contextual meanings. In order to enable the machine learning model 112 to learn such strategies, a data-driven strategy learning process is applied. The training data set may include partial training samples annotated by human interpreters, which include read and write timings for segmentation. Based on the training data, the machine learning model 112 can obtain a robust read and write strategy for SiST.

[0069] Unlike the predetermined reading and writing probabilities and heuristic waiting strategies detailed in previous studies, interpreters engage in a dynamic process of listening (reading) and translating (writing). They listen intently to the speaker's speech and segment lengthy sentences into semantic chunks, which represent the smallest language unit that can independently convey a complete thought. Once a chunk that encapsulates enough information is identified, they proceed to translate that segment into the target language, providing an accurate and contextually appropriate translation.

[0070] Simulating the strategies of human interpreters, in some embodiments of the present disclosure, the read and write strategies are not explicitly defined for the machine learning model 112. The machine learning model 112 can determine the strategy by waiting for a complete semantic segment. Specifically, given a partial speech, the machine learning model 112 can generate a translation for a complete segment of the input speech (with complete semantics). The machine learning model 112 can be trained with segmented speech data to learn this capability.

[0071] In the training phase, given the source audio x 1：M , which can be split into a series of n segments y 1：n Then we can get n training samples, expressed as in and j represents the jth segment of the audio and the corresponding translation. For training, the goal is to 1：t In the case of , output all complete segmented translations and cutoffs (optional). The training objective can be defined as minimizing the predicted output translation and cutoff of the segments in the training sample, which can be expressed as follows: in, Indicates the uniform distribution of speech over time. Using equation (2) training, the machine learning model 112 is able to generate a cutoff timestamp for the input speech. Additionally, the objective function causes the machine learning model 112 to wait for an appropriate time to start translation, because the machine learning model 112 will not output anything when it determines that the current audio segment does not contain complete semantics.

[0072] In some embodiments, for model-based solutions, the scarcity of training data continues to hinder performance on the SiST task. In some embodiments, the data scarcity of the SiST task is addressed with a three-stage training methodology: pre-training, continuous training, and fine-tuning. In some embodiments, in the pre-training stage, the language model and audio encoder are independently pre-trained on a large-scale dataset. Then, the machine learning model 112 can be continuously trained using billions of tokens of mediocre quality synthetic speech translation data, aiming at its speech and text modalities. In some embodiments, multiple training tasks can be included to enhance the learning capabilities of the machine learning model 112 in context to better utilize context-related information from the retriever and previous translations. In the final stage, the machine learning model 112 is fine-tuned using a small amount of manually annotated data to improve robustness and translation quality.

[0073] Specifically, for streaming and higher quality translation, three types of tasks can be applied to train the machine learning model 112: automatic speech recognition (ASR), speech translation (ST), and text translation (MT). In order to align the pre-trained language model (e.g., LLM) and the modality of the audio encoder, the machine learning model 112 is continuously trained on various tasks with a large amount of paired data. It can further enhance the learning ability in the context of the method of this article by merging the translation into memory and knowledge from an external database. Therefore, the ST task can be extended to different configurations as shown in Table 1. ST translation can be streaming or offline, direct or COT, with or without context, which leads to 8 different tasks.

[0074] In some embodiments, the main challenge in developing an end-to-end SiST model is the data scarcity of simultaneous ST. To this end, a synthetic data construction pipeline is proposed. Using a strong language model, two types of speech translation data (ASR training data and MT training data) can be synthesized for continuous training: offline ST data and context-aware segmented streaming ST data. In some embodiments, in the first training phase of multi-task continuous training, a first training data set can be used to train the machine learning model 112, the first training data set including a first number of samples, the samples including sample audio or sample audio fragments in the source language, and corresponding translation results in the target language. In some embodiments, the first training data set may include offline ST data and / or context-aware segmented streaming ST data.

[0075] In some embodiments, ASR data can be used to construct offline ST data. Given the true value transcription of speech, the trained language model can be used to translate the source language into the target language. In order to ensure the readability and conciseness of the target language, the trained language model is prompted to perform inverse text normalization (ITN), filler word smoothing, etc. In some embodiments, streaming ST data consists of fine-grained audio-text alignment and translation pairs for segmented semantic blocks. Compared with offline ST data, the collection of streaming ST data is even more challenging. It is found that some human interpreters often segment long speech into several semantic blocks, each of which can be translated independently to ensure effective and smooth translation. Inspired by such findings, the trained language model is used to construct streaming ST data by imitating the chunking process. Long speech data is used to construct streaming ST data because the additional history can provide better context-related information. First, the prompt word information to the language model is to instruct the model to decompose the ASR transcription into multiple independent semantic blocks, which are then translated into the target language. Subsequently, the semantic blocks are aligned with the corresponding audio blocks to obtain streaming ST data. Such training data enables the machine learning model 112 to process incomplete speech input and generate coherent semantic partial translations.

[0076] In some embodiments, the first training stage can be performed based on at least one of the following: a first training task, configured to cause the machine learning model to translate a first sample audio in a source language into a first translation result in a target language; a second training task, configured to cause the machine learning model to translate a transcribed text corresponding to a second sample audio in the source language into a second translation result in the target language; a third training task, configured to cause the machine learning model to translate a sample audio segment in the source language into a third translation segment with complete semantics in the target language; a fourth training task, configured to cause the machine learning model to translate a third sample audio with complete semantics into a fourth translation result in the target language. Each of the above training tasks can be performed with or without historical translations as context information. As shown in Table 1 below, multiple training tasks can be defined. Table 1 Configuration explain Direct The speech is translated directly into the target language. COT The speech is first transcribed into the source language and then translated into the target language. Streaming Given a partial speech, translate the segment with complete semantics into the target language. Offline Full speech, translate the entire content into the target language. w / o context No historical translations and outside knowledge were provided. w / Context Provide historical translations or external knowledge as context.

[0077] Even though the machine learning model 112 may have good translation quality on the SiST task after the previous multi-task continuous training phase, in some embodiments, it is proposed to further improve performance by fine-tuning on streaming ST data with manual annotations for different tasks. In some embodiments, for the second training phase for multi-task supervised fine-tuning, a second training data set may be used to train the machine learning model 112, the second training data set comprising a second number of samples, the samples comprising sample audio or sample audio segments in the source language, and corresponding translation results in the target language. In some embodiments, the translation quality level of the second training data set may be higher than the translation quality level of the first training data set. Such high-quality data enables the model to better align with the segmentation methodology of professional human interpreters. In addition, this process enhances the robustness of the model to speech influencies (such as stuttering), ensuring smoother communication in real-world scenarios.

[0078] The source of manually annotated streaming ST data is derived from real-world scenarios containing various speech characteristics, such as disfluencies, stuttering, code mixing, and professional terminology. Such features ensure the robustness of the machine learning model 112 under different conditions. In some embodiments, professional human interpreters can be requested to provide high-quality annotations for simultaneous segmentation and interpretation of speech data. Additionally, terms and jargon are identified and translated within the context, further enhancing the context-awareness of the machine learning model 112.

[0079] In some examples, for Figures 3D to 3F Some examples in the table and FIG. 3G to FIG. 3I The study demonstrates the ability of the machine learning model 112 in translating complex speech using the cases of CN-EN and EN-CN in some example tables. One of the most performed cascade systems, Product X, was selected for comparison with the translation results according to an example embodiment of the present disclosure (labeled as CLASI translation). Product X adopts the cascade method for SiST. It is one of the best SiST systems in the market. The correct transcription (Golden Transcription) indicates the true translation in the test, and "CLASI ASR" indicates the ASR result of the test audio clip according to an example embodiment of the present disclosure, and "ASR of Product X" indicates the ASR result of the test audio clip from Product X.

[0080] The detailed description is described in Table . For the CN-EN direction, cases on robustness to recognition errors, reasoning ability for translation, and buzzwords are presented in Figure 3D-3F For the EN-CN direction, examples of natural, expressive, and accurate terminology translations are presented in Figure 3G-3I middle.

[0081] Figure 4 4 is a flowchart of a process 400 for speech translation according to some embodiments of the present disclosure. The process 400 may be implemented, for example, at the speech translation system 110. For ease of discussion, reference will be made to Figure 1 The process 400 is described with reference to the environment 100 of FIG.

[0082] At block 410 , in response to obtaining a first audio segment in a source language, the speech translation system 110 converts the first audio segment into a first audio embedding.

[0083] At box 420, in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in the target language corresponding to the at least one audio segment, the speech translation system 110 constructs a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment.

[0084] At box 430, the speech translation system 110 uses the trained machine learning model to generate a first translation segment in the target language corresponding to the first audio segment based on the first model input.

[0085] In some embodiments, generating the first translation segment includes: determining a cutoff timestamp in the first audio segment, the cutoff timestamp identifying a time in the first audio segment before a complete semantic is detected; and generating the first translation segment corresponding to an audio portion of the first audio segment before the cutoff timestamp.

[0086] In some embodiments, constructing the first model input further comprises: constructing the first model input based on at least one cutoff timestamp in the at least one audio segment, respectively.

[0087] In some embodiments, constructing the first model input further comprises: determining at least one word matching the first audio segment from a knowledge base; and constructing the first model input further based on the at least one word. In some embodiments, generating the first translation segment comprises: generating the first translation segment to include the at least one word in the target language.

[0088] In some embodiments, constructing the first model input also based on the at least one word includes: constructing the first model input also based on: at least one pair consisting of at least one word in the source language and a translation of the at least one word in the target language, at least one pair consisting of at least one word in the target language and a translation of the at least one word in the source language, the at least one word in the source language, or the at least one word in the target language.

[0089] In some embodiments, the first model input further includes prompt word information, wherein the prompt word information is used to indicate translation from the source language to the target language.

[0090] In some embodiments, the first audio segment is obtained in an interactive session, and constructing the first model input further includes: constructing the first model input based on historical context information in the interactive session.

[0091] In some embodiments, process 400 further includes: in response to obtaining a second audio segment following the first audio segment in the source language, converting the second audio segment into a second audio embedding; constructing a second model input based on the second audio feature embedding and at least the first audio embedding and a first translation embedding corresponding to the first translation segment; and providing the second model input to the trained machine learning model to obtain a second model output, the second model output indicating a second translation segment in the target language corresponding to the second audio segment.

[0092] In some embodiments, the training process of the machine learning model includes: a first training stage, configured to use a first training data set to train the machine learning model, the first training data set including a first number of samples, the samples including sample audio or sample audio fragments in the source language, and the corresponding translation results in the target language; and a second training stage, configured to use a second training data set to train the machine learning model, the second training data set including a second number of samples, the samples including sample audio or sample audio fragments in the source language, and the corresponding translation results in the target language, the translation quality level of the second training data set being higher than the translation quality level of the first training data set.

[0093] In some embodiments, the first training stage is performed based on at least one of the following: a first training task, configured to enable the machine learning model to translate a first sample audio in the source language into a first translation result in the target language; a second training task, configured to enable the machine learning model to translate a transcribed text corresponding to a second sample audio in the source language into a second translation result in the target language; a third training task, configured to enable the machine learning model to translate a sample audio segment in the source language into a third translation segment with complete semantics in the target language; and a fourth training task, configured to enable the machine learning model to translate a third sample audio with complete semantics into a fourth translation result in the target language.

[0094] Figure 5 : A schematic structural block diagram of an apparatus 500 for speech translation according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the speech translation system 110. Each unit / component in the apparatus 500 may be implemented by hardware, software, firmware or any combination thereof.

[0095] As shown in the figure, the device 500 includes: an embedding conversion module, configured to convert a first audio segment in a source language into a first audio embedding in response to obtaining the first audio segment; an input construction module, configured to construct a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment; and a translation generation module, configured to use a trained machine learning model to generate a first translation segment in the target language corresponding to the first audio segment based on the first model input.

[0096] In some embodiments, the translation generation module 530 is further configured to: determine a cutoff timestamp in the first audio segment, the cutoff timestamp identifying a time in the first audio segment before a complete semantic is detected; and generate the first translation segment corresponding to an audio portion of the first audio segment before the cutoff timestamp.

[0097] In some embodiments, the input construction module 520 is further configured to: construct the first model input based on at least one cutoff timestamp in the at least one audio segment, respectively.

[0098] In some embodiments, the input construction module 520 is further configured to: determine at least one word matching the first audio segment from a knowledge base; and construct the first model input based on the at least one word. In some embodiments, the translation generation module 530 is further configured to: generate the first translation segment to include the at least one word in the target language.

[0099] In some embodiments, the input construction module 520 is further configured to: construct the first model input based on: at least one pair consisting of at least one word in the source language and a translation of the at least one word in the target language, at least one pair consisting of at least one word in the target language and a translation of the at least one word in the source language, the at least one word in the source language, or the at least one word in the target language.

[0100] In some embodiments, the first model input further includes prompt word information, wherein the prompt word information is used to indicate translation from the source language to the target language.

[0101] In some embodiments, the first audio segment is obtained in an interactive session, and constructing the first model input further includes: constructing the first model input based on historical context information in the interactive session.

[0102] In some embodiments, the device 500 also includes: a second embedding conversion module, configured to convert the second audio segment into a second audio embedding in response to obtaining a second audio segment following the first audio segment in the source language; a second input construction module, configured to construct a second model input based on the second audio feature embedding and at least the first audio embedding and a first translation embedding corresponding to the first translation segment; and an output acquisition module, configured to provide the second model input to the trained machine learning model to obtain a second model output, wherein the second model output indicates a second translation segment in the target language corresponding to the second audio segment.

[0103] In some embodiments, the training process of the machine learning model includes: a first training stage, configured to use a first training data set to train the machine learning model, the first training data set including a first number of samples, the samples including sample audio or sample audio fragments in the source language, and the corresponding translation results in the target language; and a second training stage, configured to use a second training data set to train the machine learning model, the second training data set including a second number of samples, the samples including sample audio or sample audio fragments in the source language, and the corresponding translation results in the target language, the translation quality level of the second training data set being higher than the translation quality level of the first training data set.

[0104] In some embodiments, the first training stage is performed based on at least one of the following: a first training task, configured to enable the machine learning model to translate a first sample audio in the source language into a first translation result in the target language; a second training task, configured to enable the machine learning model to translate a transcribed text corresponding to a second sample audio in the source language into a second translation result in the target language; a third training task, configured to enable the machine learning model to translate a sample audio segment in the source language into a third translation segment with complete semantics in the target language; and a fourth training task, configured to enable the machine learning model to translate a third sample audio with complete semantics into a fourth translation result in the target language.

[0105] Figure 6 6 is a block diagram of an electronic device 600 that can implement multiple implementations of the present disclosure. It should be understood that Figure 6 The illustrated electronic device 600 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. Figure 6 The electronic device 600 shown in FIG. 1 may be used to implement Figure 1 The speech translation system 110 or Figure 5 The device 500 in.

[0106] like Figure 6 As shown, the electronic device 600 is in the form of a general electronic device. The components of the electronic device 600 may include, but are not limited to, one or more processors or processing units 610, a memory 620, a storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. The processing unit 610 may be an actual or virtual processor and may perform various processes based on a program stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 600.

[0107] The electronic device 600 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. The storage device 630 may be any removable or non-removable media, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be used to store information and / or data (such as training data for training) and may be accessed within the electronic device 600.

[0108] The electronic device 600 may also include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 620 may include a computer program product 625 having one or more program elements configured to perform various methods or actions of various implementations of the present disclosure.

[0109] The communication unit 640 communicates with another electronic device via a communication medium. In addition, the functions of the components in the electronic device 600 can be implemented by a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0110] The input device 650 may be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 660 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed. The external device (such as a storage device, a display device, etc.) communicates with one or more devices that enable a user to interact with the electronic device 600, or communicates with any device (e.g., a network card, a modem, etc.) that enables the electronic device 600 to communicate with one or more other electronic devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0111] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer executable instructions or computer programs are stored, and the computer executable instructions are executed by a processor to implement the above method. According to an exemplary implementation of the present disclosure, a computer program product is also provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer executable instructions, which are executed by a processor to implement the above method. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method as described above is implemented.

[0112] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of each frame in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to the processing unit of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and this machine generation device is to realize the function / action specified in one or more frames in the flow chart and / or the block diagram when these instructions are executed by a computer or other programmable data processing device. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions enable a computer, a programmable data processing device and / or other equipment to work in an ad hoc manner. Therefore, the computer-readable medium comprising instructions includes a product, and this product includes instructions for realizing the various aspects of the function / action specified in one or more frames in the flow chart and / or the block diagram.

[0114] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps can be executed on the computer, other programmable data processing apparatus, or other device to generate a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0115] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product implemented according to the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a unit, a program segment or an instruction, which contains one or more executable instructions for realizing a specified logical function. In some alternative embodiments, the function marked in the frame can also occur in an order different from the order marked in the accompanying drawings. For example, two continuous blocks can actually be executed in parallel, and sometimes can also be executed in reverse order, depending on the function involved. It should also be noted that each frame in the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be realized by a special hardware-based system or by a combination of special hardware and computer instructions that performs a specified function or action.

[0116] Each embodiment of the present disclosure has been described above. The above description is exemplary rather than exhaustive and is not limited to the disclosed implementations. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used in this disclosure is intended to best explain the principles, practical applications, or technical improvements in the market for each implementation, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for speech translation, comprising: In response to obtaining a first audio segment in a source language, converting the first audio segment into a first audio embedding; In response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment, constructing a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment; as well as Using the trained machine learning model, a first translation segment in the target language corresponding to the first audio segment is generated based on the first model input.

2. The method of claim 1, wherein generating the first translation segment comprises: determining a cutoff timestamp in the first audio segment, the cutoff timestamp identifying a time in the first audio segment before a complete semantic is detected; as well as The first translated segment is generated corresponding to the audio portion of the first audio segment before the cutoff timestamp.

3. The method of claim 1, wherein constructing the first model input further comprises: The first model input is also constructed based on at least one cutoff timestamp in the at least one audio segment, respectively.

4. The method of claim 1, wherein constructing the first model input further comprises: determining at least one word matching the first audio clip from a knowledge base; constructing the first model input also based on the at least one term; as well as Wherein generating the first translation segment comprises: The first translation segment is generated to include the at least one term in the target language.

5. The method of claim 4, wherein constructing the first model input further based on the at least one term comprises: The first model input is also constructed based on: at least one pair of at least one word in the source language and a translation of the at least one word in the target language, at least one pair of at least one word in the target language and a translation of the at least one word in the source language, the at least one word in the source language, or The at least one word in the target language. 6 . The method according to claim 1 , wherein the first model input further comprises prompt word information, the prompt word information being used to indicate translation from the source language to the target language.

7. The method of claim 1, wherein the first audio clip is obtained in an interactive session, and constructing the first model input further comprises: The first model input is also constructed based on historical context information in the interaction session.

8. The method according to claim 1, further comprising: In response to obtaining a second audio segment subsequent to the first audio segment in the source language, converting the second audio segment into a second audio embedding; constructing a second model input based on the second audio feature embedding and at least the first audio embedding and a first translation embedding corresponding to the first translation segment; as well as The second model input is provided to the trained machine learning model to obtain a second model output indicating a second translated segment in the target language corresponding to the second audio segment.

9. The method according to claim 1, wherein the training process of the machine learning model comprises: A first training phase is configured to train the machine learning model using a first training data set, wherein the first training data set includes a first number of samples, wherein the samples include sample audio or sample audio segments in the source language and corresponding translation results in the target language; as well as The second training stage is configured to train the machine learning model using a second training data set, wherein the second training data set includes a second number of samples, the samples include sample audio or sample audio fragments in the source language, and corresponding translation results in the target language, and the translation quality level of the second training data set is higher than the translation quality level of the first training data set.

10. The method of claim 9, wherein the first training phase is performed based on at least one of: A first training task is configured to enable the machine learning model to translate a first sample audio in the source language into a first translation result in the target language; A second training task is configured to cause the machine learning model to translate a transcription text corresponding to a second sample audio in the source language into a second translation result in the target language; A third training task is configured to enable the machine learning model to translate the sample audio segment in the source language into a third translation segment with complete semantics in the target language; The fourth training task is configured to enable the machine learning model to translate the third sample audio with complete semantics into a fourth translation result in the target language.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

12. A speech translation device, comprising: an embedding conversion module configured to, in response to obtaining a first audio segment in a source language, convert the first audio segment into a first audio embedding; an input construction module configured to, in response to determining that at least one audio segment is obtained before the first audio segment and / or at least one translation segment in a target language corresponding to the at least one audio segment, construct a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio segment, and at least one translation embedding corresponding to the at least one translation segment; A translation generation module is configured to use the trained machine learning model to generate a first translation segment in the target language corresponding to the first audio segment based on the first model input.

13. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images