Stable real-time translation of audio streams

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, facilitates generating a stable, real-time text translation in a target language of an input audio data stream recorded in a source language. An audio stream recorded in a first language is obtained. A partial transcription of the audio may be generated at each time interval among a plurality of consecutive time intervals. Each partial transcription may be translated into a second language different from the first language. Each translated partial transcription may be input to a model that determines whether a portion of the input translated partial transcription is stable. Based on the input translated partial transcription, the model identifies a portion of the translated partial transcription that is predicted to be stable. The stable portion of the translated partial transcription is provided for display on a user device.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is an international application and claims the benefit of U.S. Application No. 62 / 970,892, filed February 6, 2020. The disclosures of the above applications are incorporated herein by reference in their entireties.

[0002] This specification relates generally to generating a robust real-time text translation in a target language of an input audio data stream recorded in a source language. [Background technology]

[0003] A real-time translation system aims to provide a real-time text translation in a target language of an audio stream recorded in a source language. In other words, a real-time translation system aims to generate a text translation of an audio stream as (or shortly after) such a system receives the audio stream. Typically, conventional real-time translation systems generate a text translation of a received audio stream immediately after the first portion of the stream is received. As additional portions of the audio stream are received, these systems generate an updated text translation of the audio from the beginning of the stream to the end of the additional portion. This process continues until the entire audio stream has been translated. FIG. 1 shows an example of a chronological listing of a real-time Spanish translation of an audio stream of a sentence ("Set makers Feud over seat with with big orders at stake") generated by a conventional real-time translation system. Note that the example sentence shown in FIG. 1 includes an error introduced by an automatic machine translation (the original sentence spoken was "Jet makers feud over seat width with big orders at stake").

[0004] Conventional real-time translation systems allow visualization of the text translation of an audio stream as the stream is being received, but the text translation often changes repeatedly (referred to herein as "translation flicker" or simply "flicker") as additional portions of the audio stream are received. This is illustrated in FIG. 1, where the first translated word in the first few translations changes multiple times before finalizing the word "Establecer." Such translation flicker may occur because word orders (e.g., the order of verbs and adjectives relative to nouns) may vary from language to language. For example, such translation flicker may occur because existing translation algorithms generally make decisions between roughly equivalent words, phrases, and word orders, and those decisions may change as the audio stream continues to be received. Summary of the Invention [Means for solving the problem]

[0005] In general, one innovative aspect of the subject matter described herein includes operations of obtaining an audio data stream; determining a first language in which audio in the audio data stream is recorded; generating a partial transcription of the audio at each time interval among a plurality of consecutive time intervals, each partial transcription being (1) a transcription of the audio from a start of the audio to a particular time interval and (2) a first language; for each partial transcription, translating the partial transcription into a second language different from the first language; and inputting the translated partial transcription into a model, wherein the model determines whether a portion of the input translated partial transcription is stable and determines whether the portion of the input translated partial transcription is stable over a time interval relative to the input translated partial transcription. The method may include the operations of: determining that a portion of the input translated portion transcription is stable when the portion of the input translated portion transcription is not expected to change in another translated portion transcription of the recorded audio by a time interval later than the time interval; identifying the portion of the translated portion transcription as stable based on the input translated portion transcription by the model; and, in response to identifying the portion of the translated portion transcription as stable, providing the portion of the translated portion transcription using the identification for display on a user device. Other embodiments of this aspect include corresponding systems, devices, apparatus, and computer programs configured to perform the actions of the method. The computer program (e.g., instructions) may be encoded on a computer storage device.

[0006] These and other embodiments can each optionally include one or more of the following features.

[0007] In some implementations, the model may be trained based on translations of transcriptions generated from one or more training audio data files recorded in a first language.

[0008] In some implementations, the transcriptions generated from the one or more training audio data files may include, for each training audio data file, multiple partial transcriptions of the audio data file, where each partial transcription of the training audio data file is generated at a respective one of multiple consecutive time intervals, where each partial transcription (1) is a transcription of the audio from a start of the audio to a particular time interval and (2) is in a first language.

[0009] In some implementations, the model may be trained based on a stability value that identifies a portion of a respective translated partial transcription of a respective one of the one or more audio data files that is stable.

[0010] In some implementations, each stability value may be generated using the position of words within the respective translated partial transcriptions and the other translated partial transcriptions for each one of the one or more audio data files.

[0011] In some implementations, a method may include training a model to identify sequences of words in partial transcriptions that are predicted to be stable, the training step including obtaining training audio data files; identifying, from among the training audio data files, a plurality of training audio data files having audio recorded in a first language; generating, for each training audio data file in the plurality of training audio data files, a partial transcription of the audio in the training audio data file at a plurality of successive time intervals, each partial transcription (1) being a transcription of the audio from a start of the audio to a particular time interval and (2) being in the first language; and generating the partial transcriptions ordered from oldest to newest with respect to the start of the audio. generating a chronological list of translated partial transcriptions corresponding to the chronological list of partial transcriptions, comprising, for each partial transcription in the chronological list of partial transcriptions, translating the partial transcription into the target language; for each translated partial transcription, starting with a first translated partial transcription in the chronological list of translated partial transcriptions, generating a stability value using word positions in the translated partial transcription and other translated partial transcriptions to identify portions of the translated partial transcription that are stable; and training the model using the translated partial transcriptions as inputs to the model and the stability values ​​as outputs of the translation stability model.

[0012] In some implementations, generating the stability value may include calculating a correctness probability for each word in the translated partial transcription, where the correctness probability measures the probability that a word at a particular position in the translated partial transcription will appear at a particular position in one or more other translated partial transcriptions that appear after the translated partial transcription in a chronological list of translated partial transcriptions, and generating a stability value for the translated partial transcription using the correctness probability for each word in the translated partial transcription.

[0013] In some implementations, calculating a probability of correctness for each word in the translated partial transcription can include, for each word in the translated partial transcription, determining a specific position of the word in the translated partial transcription; calculating a first value, the first value being a number of translated partial transcriptions that are listed after the translated partial transcription in a chronological list of translated partial transcriptions; calculating a second value, the second value specifying the number of times the word appears at a specific position in a translated partial transcription that appears after the translated partial transcription in the chronological list of translated partial transcriptions; and dividing the second value by the first value to obtain a probability of correctness for the word.

[0014] In some implementations, generating the stability value may include calculating a change frequency metric for each word in the translated partial transcription, where the change frequency metric measures the number of times a word at a particular position in the translated partial transcription changes at a particular position in the translated partial transcription that is listed after the translated partial transcription in a chronological list of translated partial transcriptions, and generating a stability value for the translated partial transcription using the change frequency metric for each word in the translated partial transcription.

[0015] In some implementations, generating a stability value may include calculating an equal-to-last metric for each word in the partial transcription, where the equal-to-last metric identifies whether a position of the word in the partial transcription is the same as a position of a word in the partial transcription that appears last on a chronological list of the translated partial transcription, and generating a stability value for the translated partial transcription using the equal-to-last metric for each word in the translated partial transcription.

[0016] Another innovative aspect of the subject matter described herein includes operations of obtaining an audio data stream, determining a first language in which audio in the audio data stream is recorded, generating partial transcriptions of the audio at a plurality of successive time intervals, each partial transcription being (1) a transcription of the audio from a start of the audio to a particular time interval and (2) in the first language, and for each successive specific partial transcription, translating the specific partial transcription into a target language, and for each word of the translated specific partial transcription, determining a position of the word in the translated specific partial transcription prior to the translated specific partial transcription. The method may include the steps of: calculating a stability value for a word by determining whether a word position in a threshold number of translated partial transcriptions in a set of translated partial transcriptions that includes a plurality of words ...

[0017] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: The innovations described herein reduce translation fluctuations typically encountered by conventional real-time translation systems when translating an audio stream in real-time. To achieve such reduced translation fluctuations, the techniques described herein display only portions of the real-time translation that are determined to be stable (referring to a translation that is not expected / predicted to change as additional portions of the audio stream are received and translated, as described further herein).

[0018] Moreover, the techniques described herein have faster processing times compared to conventional real-time translation systems that add a certain delay before generating a translation. In particular, such conventional systems wait a longer period of time to generate the translated text than the present system. Thus, while these delay-based conventional systems achieve higher accuracy and / or lower translation fluctuation than other conventional systems, the accuracy comes at the expense of a longer delay, which may not be feasible in settings requiring faster real-time translation output. In contrast, the stability-based techniques described herein minimize the delays that would normally be present in delay-based conventional systems, but do so while providing similar or even better stability / reduced translation fluctuation than such conventional systems.

[0019] In one implementation described further herein, a model (e.g., a supervised machine learning model) is used to predict the portion of the translated portion of the input audio stream that is stable, and this portion is used to determine the portion of the translation to display. Rather than training the model with already translated text as may be done in conventional systems, the model training technique described herein uses actual audio files / streams that are transcribed and then translated. Using actual audio to curate the training data ensures that the model is robust (more so than conventional systems) since it is trained using multiple different audio scenarios of humans (or other entities) speaking or making sounds. Furthermore, any errors in the translation that may have been introduced during translation (e.g., in producing an inaccurate translation) and / or during transcription of the audio (e.g., in transcribing one or more words inaccurately) are propagated during model training. For example, transcription of an audio file may use context information to continually update the transcription as more audio data is received. By using audio to curate the training data, the model can in effect integrate uncertainties in the transcription of the audio signal into the model. As a result, the model is trained using potentially erroneous training data, which not only mimics how translation is performed in practice, but also ensures that the model is trained to adapt to such systematic errors in determining the stability of the translation.

[0020] Moreover, the model-based techniques described herein do not require any pre-labeled data for training, nor do they require significant resources to obtain or generate such labels. Indeed, the techniques described herein utilize a set of simple rules that require limited computational processing in generating training labels for the acquired training audio data.

[0021] In another implementation described further herein, an algorithm is used to predict the stable portion of a particular translation based on whether one or more previous translations from a previous time in the audio stream also included that portion of the translation in the same location as the particular translation. This technique is more resource-efficient than model-based techniques, as it requires fewer computing resources (e.g., processor and memory requirements) than are required to train, build, and store a model.

[0022] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an exemplary output of a real-time translation produced by a conventional real-time translation system. [Diagram 2] FIG. 1 is a block diagram of an exemplary environment in which an input audio stream in a source language is text translated in real time into a target language. [Diagram 3] 1 is a flow diagram of an exemplary process for generating a real-time text translation of an input audio stream recorded in a source language into a target language. [Figure 4] 3 is an exemplary list of translated partial transcriptions and corresponding stability values ​​for the translated partial transcriptions generated by the system of FIG. 2. [Diagram 5] FIG. 5 is a block diagram of an exemplary environment for training the translational stability model described with reference to FIGS. 2, 3, and 4. [Figure 6] FIG. 6 is a flow diagram of an exemplary process for training a translational stability model described with reference to FIGS. 2, 3, 4, and 5. [Figure 7]FIG. 1 is a block diagram of an example environment in which an input audio stream in a source language is text translated in real time into a target language. [Figure 8] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Like reference numbers and designations in the various drawings indicate like elements.

[0025] This specification relates generally to generating a robust real-time text translation in a target language of an input audio data stream recorded in a source language.

[0026] As summarized below and described in more detail throughout this specification, the real-time translator receives an audio stream and generates a stable, real-time text translation of the audio stream in a target language. As used herein, a stable, real-time text translation is a translation of a first portion of the audio stream (i.e., from the start of the audio stream until a particular time interval) that is not expected / predicted to change when a real-time translation is generated for a later portion of the audio stream (i.e., a portion of the audio stream from the start of the stream until a time interval that occurs after the particular time interval).

[0027] Upon receiving an audio stream (or a portion of this stream), the real-time translator determines a first language (also referred to as a source language) in which the audio stream is recorded. The real-time translator generates partial transcriptions of the audio stream at multiple consecutive time intervals during the streaming of the audio. As used herein, a partial transcription is (1) a transcription of the audio from the beginning of the audio to a particular time interval and (2) in the first language. Then, for each partial transcription, the real-time translator generates a text translation of the partial transcription in a second language (also referred to herein as a target language) different from the first language.

[0028] For each translated partial transcription, the real-time translator determines whether a portion of the translated partial transcription (which may include part or all of the translated partial transcription) is stable, i.e., whether the translated partial transcription is expected / predicted not to change in another translated partial transcription of audio recorded by a time interval later than the time interval for the input translated partial transcription.

[0029] The real-time translator can make this stability determination in several different ways. In some implementations, the real-time translator can use a model (e.g., a supervised or unsupervised machine learning model) trained to generate a stability value that identifies stable portions of the input translated partial transcription. Such a model is trained using a set of translated partial transcriptions (generated from actual audio data files) and corresponding stability values ​​for the translated partial transcriptions. In other implementations, the real-time translator calculates the stability value of a particular translated partial transcription by determining whether the position of each word in the particular translated partial transcription matches the position of that word in a threshold number of other translated partial transcriptions that precede the particular translated partial transcription. Each word in the transcription / translation can be generalized to a token in other languages.

[0030] Using the generated stability value to identify a portion of the translated partial transcript that is identified as stable, the real-time translator provides only that portion of the translated partial transcript for display.

[0031] The above-described features of the real-time translator, as well as additional features, are further described below with reference to FIGS.

[0032] FIG. 2 is a block diagram of an example environment 200 in which an input audio stream in a source language is text translated in real time into a target language.

[0033] The real-time translator 204 (shown in FIG. 2) generates a real-time translation of the input audio stream 202. The input audio stream 202 may be received from multiple sources. For example, the audio stream 202 may be a voice recording or an audio file stored on a user device. As another example, the audio stream 202 may be obtained from a server or cloud service over a network.

[0034] In some implementations, a user of a user device can send a request to the real-time translator 204 to generate a real-time translation of the audio stream 202. Such a request can include the audio stream 202 or, alternatively, can identify a location or source of the audio stream 202 that can be used by the real-time translator 204 to obtain the audio stream 202.

[0035] The real-time translator 204 is a data processing device that includes multiple components: (1) a language detector 206; (2) a transcriber 208; (3) a translator 210; (4) a translation stability model 212; and (5) a display engine 214. Each of these components is also a data processing device as further described herein. Although these components are shown as being part of the real-time translator 204, in alternative implementations, one or more of these components may be implemented separately. Furthermore, although they are shown as separate components, one or more of these components may be combined into one component. For example, the transcriber 208 and the language detector 206 may be implemented as a single component. The structure and operation of each of these components is described with reference to FIG. 3.

[0036] In some implementations, the real-time translator 204 may be implemented as a single software package that resides locally on the user device. The user device typically includes a user application, such as a web browser, to facilitate sending and receiving data over the network 108, although native applications executed by the client device 102 may also facilitate sending and receiving content over the network 108. Alternatively, the real-time translator may be deployed in the cloud or on a server accessible by the user device over a network (such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof).

[0037] 3 is a flow diagram of an exemplary process 300 for generating a real-time text translation of an input audio stream recorded in a source language into a target language. Operations of the process 300 are exemplarily described below with reference to the real-time translator 204 (and its components) of FIG. 2. The operations of the process 300 may be performed by any suitable device or system, for example, any suitable data processing apparatus. The operations of the process 300 may also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing apparatuses to perform the operations of the process 300.

[0038] The real-time translator 204 obtains (at 302) an audio data stream. As described with reference to FIG. 2, the input audio stream 202 may be received from a number of sources. For example, the audio stream 202 may be a voice recording or an audio file stored on a user device. As another example, the audio stream 202 may be obtained from a server or cloud service over a network.

[0039] The language detector 206 determines (at 304) a first language in which the audio in the audio data stream is recorded. In some implementations, the language detector 206 parses metadata included in the audio stream 202 to search for a tag identifying the first language in which the audio is recorded. Alternatively, such a language identifier may be included in a request by a user device to translate the input audio stream. If such a language identification tag is included in the audio stream 202 (or otherwise included in the request), the language detector 206 obtains data associated with this tag that identifies the first language in which the audio is recorded. If the audio stream or file does not include such a language identification tag, the language detector 206 analyzes a certain portion of the input audio stream 202 (e.g., a portion corresponding to a certain time interval of audio (e.g., 1-3 seconds)) to determine the source language in which the audio corresponding to the certain portion was recorded. In some implementations, to accomplish this, the language detector 206 can compare portions of the input audio stream or file to a repository of known signatures associated with different target languages. In some implementations, the language detector 206 may include a machine learning model, such as a residual neural network (RNN) (or another suitable statistical model), that identifies the language in which a certain snippet of audio is recorded and is trained using a set of audio snippets and a corresponding set of labels that identify the language in which each audio snippet is recorded.

[0040] The transcriber 208 generates (at 306) partial transcriptions of the audio at successive time intervals during the audio stream. In some implementations, the transcriber 208 is implemented using an automated speech recognition (ASR) algorithm or another suitable audio-to-text transcription service. Unlike conventional ASR algorithms that wait until the entire audio stream is received before generating a full transcription of the audio stream, the algorithm used to implement the transcriber 208 generates partial transcriptions of the audio at successive time intervals (which may be separated by a fixed time, e.g., every 1-2 seconds, or a variable time) of the audio stream or playback. As used herein, a partial transcription of the audio is (1) a transcription of the audio from the start of the audio to a particular time interval in the audio, and (2) in a first language, which is the language in which the audio is recorded. In other words, at each successive time interval, the transcriber 208 generates a partial transcription of the audio from the start of the audio to the end of that particular successive time interval.

[0041] As shown in FIG. 2, the transcriber 208 generates partial transcriptions (TR1 208-1 through TRN 208-N) of the audio in the audio stream 202 at each time interval (T1 through TN) shown on the timeline. Thus, the transcriber 208 generates a partial transcription TR1 208-1 for time T1, which is the transcription of the audio from the start of the audio (time T0) to time T1. Similarly, the transcriber 208 generates a partial transcription TR2 208-2 for time T2, which is the transcription of the audio from the start of the audio (time T0) to time T2. As another example, the transcriber 208 generates a partial transcription TRN 208-N for time TN, which is the transcription of the audio from the start of the audio (time T0) to time TN.

[0042] The following operations (operations 308, 310, 312, and 314) are performed iteratively for each partial transcription. In some implementations, these operations are performed starting with the first partial transcription (TR1) and continuing with each subsequent successive partial transcription (TR2, TR3...TRN).

[0043] The translator 210 translates (at 308) the partial transcriptions into a second language different from the first language. The translator 210 may be implemented using a conventional machine translation service that translates text in a source language (i.e., the first language) into a target language (i.e., the second language). In some implementations, the translator 210 accepts as input each partial transcription (generated by the transcriber 208) that is in the first / source language. For each such partial transcription, the translator 210 generates a text translation of the partial transcription in the target language. As shown in FIG. 2, the translator 210 uses the partial transcriptions (TR1 208-1 through TRN 208-N) to generate corresponding translated partial transcriptions (L1 210-1 through LN 210-N).

[0044] The real-time translator 204 inputs (at 310) the translated portion transcription (generated in operation 308) into the translation stability model 212. The translation stability model 212 accepts the translated portion transcription as input and outputs a stability value that identifies portions of the input translated portion transcription that are stable, i.e., portions of the input translated portion transcription that are not expected / predicted to change in a subsequent translated portion (or full) transcription (i.e., another translated transcription for a time interval later than the time interval for the input translated portion transcription). In some implementations, the stability value output by the translation stability model 212 indicates the number of words in the translated portion transcription that are stable. For example, for a translated portion transcription that includes 10 words, a stability value of 5 indicates that the first 5 words of the translated portion transcription are stable.

[0045] In some implementations, the translational stability model 212 may be implemented as a machine learning model, such as an RNN (e.g., a supervised or unsupervised model). Alternatively, the translational stability model 212 may be implemented as a rule-based engine or another suitable statistical model. When the translational stability model 212 is implemented as a supervised machine learning model (e.g., a recurrent neural network), the translational stability model 212 may be trained using a set of translated partial transcriptions and a corresponding set of stability values ​​for the translated partial transcriptions. Additional details regarding training the translational stability model 212 are described with reference to FIGS. 5 and 6.

[0046] In response to inputting the translated partial transcriptions into the translational stability model 212 in operation 308, the translational stability model 212 identifies (at 312) stability values ​​that identify portions of the translated partial transcriptions that are expected / predicted to be stable (and the real-time translator 204 obtains the stability values ​​from the model 212). This is further illustrated in Figure 2. For example, Figure 2 shows that the translational stability model 212 outputs a stability value of 0 for the first two translated partial transcriptions (L1 210-1 and L2 210-2) and a stability value of 4 for the third and fourth translated partial transcriptions (L3 210-3 and L4 210-4).

[0047] In response to identifying a portion of the translated partial transcription that is expected / predicted to be stable, the real-time translator 204 provides (at 314) the portion of the translated partial transcription for display. In some implementations, the translation stability model 212 outputs a stability value and provides this value as well as the corresponding translated partial transcription to the display engine 214. The display engine 214 uses the translated partial transcription and the corresponding stability value to provide for display only the portion of the translated partial transcription that the stability value identifies as stable. For example, if (1) the translated partial transcription includes 10 words and (2) the stability value is 5, indicating that the first five words of the translated partial transcription are expected to be stable, then the display engine 214 provides only the first five words of the translated partial transcription for display. For example, as shown in Figure 2, the fourth translated partial transcription L4 has a stability value of 4, and therefore the display engine 214 will provide only the first four words of this translated partial transcription. This is further illustrated with reference to Figure 4.

[0048] Operations 308-314 continue until the real-time translator 204 has transcribed and translated the entire audio stream 202 and provided an expected stable portion of the translation for display. In this manner, the real-time translator 204 uses the stability value in generating a real-time text translation (in the second / target language) of the input audio stream recorded in the first / source language.

[0049] FIG. 4 shows an exemplary list of translated partial transcriptions and corresponding stability values ​​for the translated partial transcriptions generated by the real-time translator of FIG.

[0050] Reference numeral 404 indicates translated partial transcriptions, and reference numeral 402 indicates stability values ​​for each of the translated partial transcriptions. The translated partial transcriptions are in Spanish, but the partial transcriptions and their translations may be in other languages ​​(in other words, the techniques described herein apply to translations between multiple pairs of languages). Black highlighting with white text represents text determined to be unstable by the translation stability model 212 and therefore is not provided for display (or is hidden). On the other hand, black text without highlighting represents text determined to be stable by the translation stability model 212 and therefore is provided for display.

[0051] In the following paragraphs, the first five translated partial transcriptions and their corresponding stability values ​​(shown in FIG. 4) are used as examples to explain what is provided for display by the display engine 214.

[0052] 4 ("soy," "yo no," and "no soy vidente") have stability values ​​of 0, indicating that none of the words in these translated partial transcriptions are expected to be stable. Thus, for these translated partial transcriptions, the display engine 214 does not provide any of the words of the translated partial transcriptions for display.

[0053] The fourth translated partial transcription ("no soy vidente divisores de cajones con dicho") has a stability value of 3, indicating that the first three words of this translated partial transcription are expected to be stable. Thus, the display engine 214 provides the first three words of this translated partial transcription ("no soy vidente") for display.

[0054] The fifth translated partial transcription ("no soy vidente divisors de cajones con dicho es") has a stability value of 8, indicating that the first eight words in this translated partial transcription are expected to be stable. Thus, the display engine 214 provides the first eight words of this translated partial transcription ("no soy vidente divisors de cajones con dicho es") for display.

[0055] FIG. 5 is a block diagram of an example environment 500 for training the translational stability model 212 described with reference to FIGS.

[0056] The environment 500 is used to train the translation stability model 212, which may be implemented, for example, as a supervised machine learning model, to generate a stability value for an input translated partial transcription obtained from an audio stream. All components of the real-time translator 204 shown in Figure 5 (as well as their associated structures and operations performed by these components) are the same as the identically named components of the real-time translator 204 described with reference to Figures 2-4, except for the stability metric generator 504.

[0057] In some implementations, any audio utterance may be used in curating the training audio data used in training the translation stability model. Curating a diverse set of training audio data ensures that the model is trained using multiple different audio scenarios of humans (or other entities) speaking or making sounds.

[0058] Furthermore, actual (i.e., verified for correctness) translated partial transcriptions are generally not used in training; rather, the translational stability model is preferably trained using translated partial transcriptions generated by the translator 210, which are generated from partial transcriptions generated by the transcriber 208. The generated translated partial transcriptions may have errors, which may have been introduced by the translator 210 (e.g., generating an inaccurate translation) and / or the transcriber 208 (e.g., transcribing one or more words inaccurately). Thus, training the translational stability model 212 on such potentially erroneous training data ensures that the model is trained to adapt to such system errors in determining stability values.

[0059] As described in more detail below, the translation stability model 212 is trained using a set of labeled training data including a set of translated partial transcriptions in the target / second language and a corresponding set of labels that identify a stability value for each translated partial transcription. The translation stability model is trained to reduce the loss (specified by a loss function (e.g., L1, L2 loss)) of the model 212. Generally, because stability values ​​for the translated partial transcriptions corresponding to the training audio data files do not yet exist, the stability metric generator 504 is used to generate stability values ​​for these translated partial transcriptions (as described further below with reference to FIG. 6), and these stability values ​​are then used as labels during training.

[0060] In some implementations, the stability metric generator 504 is a rule-based engine (i.e., a type of data processing device) that includes a set of rules used in generating stability values ​​for the translated partial transcriptions obtained from the training audio data files. Additional details regarding the stability metric generator 504 and the training of the translation stability model 212 are described below with reference to FIG.

[0061] 6 is a flow diagram of an exemplary process 600 for training the translational stability model 212 described with reference to FIGS. 2-5. Operations of the process 600 are exemplarily described below with reference to the real-time translator 204 of FIG. 5. The operations of the process 600 may be performed by any suitable device or system, for example, any suitable data processing apparatus. The operations of the process 600 may also be implemented as instructions stored on a non-transitory computer-readable medium. Execution of the instructions causes one or more data processing apparatuses to perform the operations of the process 600.

[0062] The real-time translator 204 obtains (at 602) a training audio data file. As described above, the real-time translator 204 can obtain the training audio data file from any source that stores recorded audio. Examples of such sources are described with reference to FIG.

[0063] The real-time translator 204 identifies (at 604) from among the training audio data files a plurality of training audio data files having audio recorded in a first language. The language detector 206 determines the language in which each of the training audio data files is recorded (in the same manner as described with reference to FIG. 2 and above) and identifies only those of the plurality of training audio data files that the language detector determines to be recorded in the first language (the source language).

[0064] The following operations (operations 606-614) are performed for each training audio data in the multiple training audio data files.

[0065] The transcriber 208 generates (at 606) partial transcriptions of audio in the training audio data file at a number of consecutive time intervals. This operation is performed in substantially the same manner as described with reference to the corresponding operation (operation 306) of FIG. 3. In some implementations, the transcriber 208 also assigns each partial transcription an audio order identifier that may be used in determining the relative order of the partial transcriptions. In some implementations, the audio order identifier identifies the total time of audio corresponding to the partial transcription. For example, assume that a partial transcription is generated for every 2 seconds of audio in the audio data file. In this example, a first partial transcription for the first 2 seconds of audio may be assigned an audio order identifier of 2, a second partial transcription for the next 2 seconds of audio may be assigned an audio order identifier of 4 (i.e., representing a total time of 4 seconds that has elapsed since the start of the audio), a third partial transcription for the next 2 seconds of audio may be assigned an audio order identifier of 6 (i.e., representing a total time of 6 seconds that has elapsed since the start of the audio), and so on.

[0066] The transcriber 208 generates (at 608) a chronological list of partial transcriptions. In some implementations, the transcriber 208 orders the partial transcriptions (generated in operation 606) from oldest to newest with respect to the start of the audio. In some implementations, the transcriber 208 uses the audio order identifiers in generating the chronological list of partial transcriptions. In such implementations, the transcriber 208 generates a list of audio order identifiers and their corresponding partial transcriptions, and sorts this list in ascending order of the audio order identifiers (assuming that the audio order identifier for a partial transcription is greater than the audio order identifier for the preceding partial transcription). Thus, the resulting list of partial transcriptions is enumerated in chronological order with respect to the start of the audio.

[0067] The translator 210 generates (at 610) a chronological list of translated partial transcriptions that correspond to the chronological list of partial transcriptions. In some implementations, for each partial transcription in the chronological list of partial transcriptions, the translator translates the partial transcription into the target language to obtain a translated partial transcription. This operation is performed in substantially the same manner as operation 308 described with reference to FIG. 3.

[0068] The following operations (operations 612-614) are performed for each translated partial transcription, starting with the first translated partial transcription in the chronological list of translated partial transcriptions, and then continuing with each successive translated partial transcription in that chronological list.

[0069] The stability metric generator 504 generates (at 612) a stability value for the translated partial transcription that identifies portions of the translated partial transcription that are stable. The stability value may be calculated in different ways, each of which may involve using the positions of words in the translated partial transcription and / or other translated partial transcriptions in a chronological list of translated partial transcriptions. Several different techniques for calculating the stability value are described below.

[0070] One technique for calculating the stability value is based on calculating a correctness probability for each word in the translated partial transcription. The correctness probability measures the probability that a word at a particular position in the translated partial transcription will appear at that same position in one or more subsequent translated partial transcriptions (i.e., one or more translated partial transcriptions that appear after the translated partial transcription in the chronological list of translated partial transcriptions).

[0071] To calculate this correctness probability for a word in the translated partial transcription, the stability metric generator 504 first determines the position (also called the first position or the specific position) of the word relative to the start (or first word) of the translated partial transcription. In some implementations, the stability metric generator 504 counts the number of words from the first word to the specific word in the translated partial transcription. The stability metric generator 504 then calculates a first value and a second value. The stability metric calculates the first value by determining the number of translated partial transcriptions that are listed after the translated partial transcription in the chronological list of translated partial transcriptions. The stability metric generator 504 calculates the second value by determining the number of times the specific word appears in the first position in a subsequent translated partial transcription, i.e., a translated partial transcription that appears after the translated partial transcription in the chronological list of translated partial transcriptions. The stability metric calculates a probability of correctness for each word by dividing the second value by the first value.

[0072] The stability metric generator 504 uses the resulting correctness probability values ​​for each word in the translated partial transcription to calculate a stability value for the translated partial transcription. In some implementations, the stability metric generator 504 uses the correctness probability values ​​for each word in the translated partial transcription to generate an array. The locations / positions of such an array correspond to the positions where the words appear in the translated partial transcription, and the data stored at these array positions correspond to the correctness probability values ​​for those words. For example, array position 1 may correspond to a first word in the translated partial transcription and include a correctness probability for the first word, array position 2 may correspond to a second word in the translated partial transcription and include a correctness probability for the second word, and so on.

[0073] In some implementations, if the correctness probability for a word satisfies a threshold (e.g., meets or exceeds the threshold), the word may be identified in the array as stable (e.g., by assigning the word a label of 1). On the other hand, if the correctness probability for a word does not satisfy the threshold, the word may be identified in the array as unstable (e.g., by replacing the correctness probability with 0).

[0074] The stability metric generator 504 then determines a stability value for the translated partial transcription by counting the number of stable words in the translated partial transcription until the first unstable word is identified. In some implementations, this involves starting from the first position in the array and counting the number of "1's" (or another value representing a stable word) until the first "0" (or another value representing an unstable word) is encountered.

[0075] Another technique for calculating a stability value is based on calculating a change frequency metric, which measures the number of times a word at a particular position in a translated partial transcription does not appear at that particular position in a translated partial transcription that is listed after the translated partial transcription in a chronological list of translated partial transcriptions.

[0076] To calculate this change frequency value for a word in a translated partial transcription, the stability metric generator 504 first determines the position (also called the first position or specific position) of the word relative to the start (or first word) of the translated partial transcription (as described above with respect to the correctness probability metric). The stability metric generator 504 then calculates a first value and a second value. The stability metric calculates the first value by determining the number of translated partial transcriptions that are listed after the translated partial transcription in the chronological list of translated partial transcriptions. The stability metric generator 504 calculates the second value by determining the number of times that the word at the specific position changes in subsequent translated partial transcriptions, i.e., translated partial transcriptions that appear after the translated partial transcription in the chronological list of translated partial transcriptions. The stability metric calculates the change frequency for each word by dividing the second value by the first value.

[0077] The stability metric generator 504 uses the resulting change frequency values ​​for each word in the translated partial transcription to calculate a stability value for the translated partial transcription. In some implementations, the stability metric generator 504 uses the change frequency values ​​for words in the translated partial transcription to generate an array. The locations / positions of such an array correspond to the positions where words appear in the translated partial transcription, and the data stored at these array positions correspond to the change frequency values ​​for those words. For example, array position 1 may correspond to a first word in the translated partial transcription and include a change frequency for the first word, array position 2 may correspond to a second word in the translated partial transcription and include a change frequency for the second word, and so on. In some implementations, if the change frequency value for a word satisfies a threshold (e.g., is at or below the threshold), the word may be identified in the array as stable (e.g., by assigning a label of 1 to the word). On the other hand, if the frequency of change value for a word does not satisfy the threshold, the word may be identified in the array as unstable (eg, by assigning the word a label of 0).

[0078] The stability metric generator 504 then determines a stability value for the translated partial transcription by counting the number of stable words in the translated partial transcription until the first unstable word is identified. In some implementations, this involves starting from the first position in the array and counting the number of "1's" (or another value representing a stable word) until the first "0" (or another value representing an unstable word) is encountered.

[0079] Another technique for calculating the stability value is based on an equal-to-last metric, which identifies whether the position of a word in a translated partial transcription is the same as the position of a word in the last translated partial transcription in a chronological list of translated partial transcriptions.

[0080] To calculate this equal-to-last metric for a word in a translated partial transcription, the stability metric generator 504 first determines the position (also called the first position or specific position) of the word relative to the start (or first word) of the translated partial transcription (as described above with respect to the correctness probability metric). The stability metric generator 504 then calculates a first value (also called the equal-to-last value for a word) by determining whether the particular word appears in the first / specific position in the "last" translated partial transcription, i.e., the translated partial transcription that appears at the end of the chronological list of translated partial transcriptions. The first value for a word indicates whether the word is stable or not. In some implementations, if the stability metric generator 504 determines that the particular word appears in the first position in the "last" translated partial transcription, it assigns a "1" (or another value indicating a stable word) to the first value, otherwise it assigns a "0" (or another value indicating an unstable word) to the first value.

[0081] The stability metric generator 504 uses the equal-to-last values ​​for each word in the translated partial transcription to calculate a stability value for the translated partial transcription. In some implementations, the stability metric generator 504 uses the equal-to-last values ​​for the words in the translated partial transcription to generate an array. The locations / positions of such an array correspond to the positions where the words appear in the translated partial transcription, and the data stored at these array positions correspond to the equal-to-last values ​​for those words. For example, array position 1 may correspond to a first word in the translated partial transcription and include an equal-to-last value for the first word, array position 2 may correspond to a second word in the translated partial transcription and include an equal-to-last value for the second word, and so on.

[0082] The stability metric generator 504 then determines a stability value for the translated partial transcription by counting the number of stable words in the translated partial transcription until the first unstable word is identified. As mentioned above, in some implementations, an equal-to-last value of 1 indicates a stable word, while an equal-to-last value of 0 indicates an unstable word. In such implementations, the stability value for the translated partial transcription is determined by starting from the first position in the array and counting the number of "1's" until the first "0" (or another value representing an unstable word) is encountered.

[0083] In some implementations, only one of the above-identified techniques for calculating a stability value for a translated partial transcription may be used by the stability metric generator 504. In other implementations, two or more of these techniques may be used in calculating the stability value, in which case the stability metric generator 504 calculates a stability value for the translated partial transcription using each of these techniques and then selects one of the stability values ​​(e.g., the lowest or highest stability value) as the final stability value for use in training the translation stability model 212.

[0084] Returning to process 600, the real-time translator 204 trains (at 614) the translational stability model 212 using the translated partial transcriptions and the stability values ​​determined in operation 612. In some implementations, training continues until all translated partial transcriptions of each of the training audio data files are complete. Alternatively, training continues until the loss of the translational stability model 212 meets a particular threshold (e.g., is at or below a particular threshold).

[0085] Once trained using process 600, the translation stability model 212 may be used in an inference mode to identify stability values ​​for the input translated partial transcriptions, and these stability values ​​may be used in determining expected / predicted stable translated words to display during real-time translation of a received audio stream (as described with reference to Figures 2-4).

[0086] FIG. 7 is a block diagram of an example environment 700 in which an input audio stream in a source language is text translated in real time into a target language.

[0087] Much like the exemplary environment described with reference to Figures 2-4, the exemplary environment 700 provides a system for generating a stable real-time text translation of an input audio stream using a calculated stability value. In fact, all components of the real-time translator 204 shown in Figure 2 (as well as their associated structure and operations performed by these components as described with reference to Figure 3) are the same as the identically named components of the real-time translator 204 shown in Figure 7, except for the translation stability model 212. However, instead of the translation stability model 212 shown in Figure 2, the real-time translator 204 of Figure 7 includes a translator comparator 702. Both the translation stability model 212 and the translator comparator 702 generate stability values ​​for the translated partial transcriptions, but the translator comparator 702 generates stability values ​​for the translated partial transcriptions in a different manner (as will be further described below).

[0088] To calculate a stability value for a translated partial transcription, the translator comparator 702 first calculates stability values ​​of words in the translated partial transcription. The translator comparator 702 does so by using words that appear in one or more preceding translated partial transcriptions (i.e., one or more translated partial transcriptions that correspond to portions of the audio stream that end in a previous time interval). In some implementations, the translator comparator 702 identifies the position of the word in the translated partial transcription. Thus, for each word, the translator comparator 702 calculates a first position value that represents the position of the word in the translated partial transcription (similar to the description of determining the position of a word given above with reference to FIG. 6).

[0089] Then, for each word in the translated partial transcription, the translator comparator 702 determines a stability value for the word by determining whether the word appears in a first position in a threshold number of preceding translated partial transcriptions (which may be any predetermined number greater than or equal to 1). In other words, the translator comparator 702 determines whether the position of the word in the translated partial transcription matches the position of the word in a threshold number of translated partial transcriptions in the set of translated partial transcriptions preceding the particular translated partial transcription. If the translator comparator 702 determines that the word appears in a first position in the threshold number of preceding translated partial transcriptions, it assigns the word a "1" (or another value indicating a stable word), otherwise it assigns the word a "0" (or another value indicating an unstable word).

[0090] The translator comparator 702 uses the stability values ​​for each word in the translated partial transcription to calculate a stability value for the translated partial transcription. In some implementations, the translator comparator 702 uses the stability values ​​for the words in the translated partial transcription to generate an array. The locations / positions of such an array correspond to the positions where the words appear in the translated partial transcription, and the data stored at these array positions identifies the stability values ​​for those words. For example, array position 1 may correspond to a first word in the translated partial transcription and includes a stability value for the first word, array position 2 may correspond to a second word in the translated partial transcription and includes a stability value for the second word, and so on.

[0091] The translator comparator 702 then determines a stability value for the translated partial transcription by counting the number of stable words in the translated partial transcription until the first unstable word is identified, starting from the first position in the array and counting the number of "1's" (or another appropriate value representing a stable word) until the first "0" (or another value representing an unstable word) is encountered.

[0092] Using the calculated stability value for the translated partial transcription, the real-time translator 204 provides a portion of the translated partial transcription for display. In some implementations, the translator comparator 702 outputs the stability value and provides this value as well as the corresponding translated partial transcription to the display engine 214. The display engine 214 uses the translated partial transcription and the corresponding stability value to provide for display only the portion of the translated partial transcription that the stability value identifies as expected / predicted to be stable. For example, if (1) the translated partial transcription includes 10 words and (2) the stability value is 5, indicating that the first five words of the translated partial transcription are stable, the display engine 214 provides only the first five words of the translated partial transcription for display.

[0093] Thus, the technique for calculating the stability value described with reference to FIG. 7 differs from the translational stability model 212 building approach described with reference to FIGS. 2-6. In general, the translational stability model 212 builds a stability prediction by comparing the current sequence with future partial results, with the aim of deviating as little as possible from the future results. In contrast, the algorithm described with reference to FIG. 7 uses past translated partial transcriptions, with the aim of deviating as little as possible from the past. Furthermore, since the algorithm described with reference to FIG. 7 does not require any model training and development, it is more computationally resource efficient than the model-based techniques described with reference to FIGS. 2-6.

[0094] 8 is a block diagram of an exemplary computer system 800 that may be used to perform the operations described above. The system 800 includes a processor 810, a memory 820, a storage device 830, and an input / output device 840. Each of the components 810, 820, 830, and 840 may be interconnected, for example, using a system bus 850. The processor 810 is capable of processing instructions for execution within the system 800. In some implementations, the processor 810 is a single-threaded processor. In another implementation, the processor 810 is a multi-threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 or on the storage device 830.

[0095] The memory 820 stores information within the system 400. In one implementation, the memory 820 is a computer-readable medium. In some implementations, the memory 820 is a volatile memory unit. In another implementation, the memory 820 is a non-volatile memory unit.

[0096] The storage device 830 is capable of providing mass storage for the system 800. In some implementations, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0097] The input / output device(s) 840 provide input / output operations for the system 400. In some implementations, the input / output device(s) 840 may include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., an RS-232 port, and / or a wireless interface device, e.g., an 802.11 card. In another implementation, the input / output device(s) may include a driver device configured to receive input data and send output data to other input / output devices, e.g., keyboards, printers, and display devices 860. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0098] Although an exemplary processing system has been described in FIG. 8, implementations of the subject matter and functional operations described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in combinations of one or more of them.

[0099] The subject matter and embodiments of the operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage medium(s) for execution by or to control the operation of a data processing device. Alternatively, or in addition, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or may be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, although a computer storage medium is not a propagating signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagating signal. A computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0100] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0101] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, programmable processors, computers, systems on chips, or a combination of the above. The apparatus may include special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, e.g., code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of these. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing infrastructures, and grid computing infrastructures.

[0102] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted, declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system, but need not. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0103] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0104] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors. Generally, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0105] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user may provide input to the computer. Other types of devices may also be used to provide for interaction with the user, e.g., feedback provided to the user may be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from that web browser.

[0106] An embodiment of the subject matter described herein may be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface or web browser through which a user may interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0107] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to the client devices (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., results of user interaction) may be received at the server from the client device.

[0108] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as working in some combinations and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0109] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all of the shown operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0110] Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0111] 102 client devices 108 Network 200 Environment 202 Input Audio Stream, Audio Stream 204 Real-time translator 206 Language Detector 208 Transcriber 208-1~208-N Partial transcription TR1~TRN 208-1 Partial Transcription TR1 208-2 Partial Transcription TR2 208-N Partial Transcription TRN 210 Translator 210-1~210-N Translated partial transcription L1~LN 210-1 Translated partial transcript L1 210-2 Translated partial transcription L2 210-3 Translated partial transcription L3 210-4 Translated partial transcription L4 210-N Translated partial transcription LN 212 Translation Stability Model, Model 214 Display Engine 300 processes 400 Systems 402 Translated Partial Transcription 404 Stability Value 500 environment 504 Stability Metric Generator 600 processes 700 Environment 702 Translator Comparator 800 Computer Systems, Systems 810 Component, processor 820 Components, Memory 830 Component, storage device 840 Components, Input / Output Devices 850 System Bus 860 Input / Output Devices T0, T1, T2, TN time T1~TN Time interval

Claims

1. 1. A computer-implemented method comprising: obtaining an audio data stream; determining a first language in which the audio in the audio data stream is recorded; generating a partial transcription of the audio for each time interval among a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio to a particular time interval and (2) being in the first language; For each partial transcription, translating the partial transcription into a second language different from the first language; inputting the translated partial transcription into a machine learning model, the machine learning model is configured to predict whether a portion of an input translated partial transcript is stable; the machine learning model is trained to minimize a loss function and is trained using a set of translations of transcriptions generated from one or more training audio data files recorded in the first language and a corresponding set of stability values ​​each indicating whether each set of translations is stable; said set of stability values ​​being generated using a set of rules; using the machine learning model to generate a prediction indicating that a portion of the translated partial transcription is stable; a portion of the input translated partial transcription is stable when the portion of the input translated partial transcription is not expected to change in another translated partial transcription of recorded audio by a time interval later than the time interval for the input partial transcription; in response to generating the prediction indicating the portion of the translated partial transcription as stable, providing the portion of the translated partial transcription for display on a user device using the prediction; 23. A computer-implemented method comprising:

2. the transcriptions generated from one or more training audio data files include, for each training audio data file, a plurality of partial transcriptions of the training audio data file; each partial transcription of the training audio data file is generated at a respective one of a plurality of successive time intervals; 2. The computer-implemented method of claim 1, wherein each partial transcription is (1) a transcription of the audio from a beginning of the audio to a particular time interval and (2) in the first language.

3. A computer-implemented method as described in claim 2, wherein a stability value for a training audio data file from the one or more training audio data files is generated using word positions within the respective translated partial transcriptions and other translated partial transcriptions for the training audio data file.

4. The method further includes training the machine learning model to identify sequences of words in the partial transcription that are expected to be stable, the training step comprising: obtaining a training audio data file; identifying, from among said training audio data files, a number of training audio data files having audio recorded in said first language; For each training audio data file in the plurality of training audio data files, generating partial transcriptions of the audio in the training audio data file at a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio up to a particular time interval and (2) being in the first language; generating a chronological list of partial transcriptions ordered from oldest to newest with respect to the start of the audio; generating a chronological list of translated partial transcriptions corresponding to the chronological list of partial transcriptions, the chronological list including, for each partial transcription in the chronological list of partial transcriptions, translating the partial transcription into the second language; for each translated partial transcription, starting with a first translated partial transcription in said chronological list of translated partial transcriptions; generating a stability value using word positions within the translated partial transcription and other translated partial transcriptions to identify portions of the translated partial transcription that are stable; training the machine learning model using the translated partial transcriptions as inputs to the machine learning model and the stability values ​​as outputs of a translation stability model; 2. The computer-implemented method of claim 1, comprising:

5. The step of generating the stability value further comprises: calculating a correctness probability for each word in the translated partial transcription, the correctness probability measuring the probability that a word at a particular position in the translated partial transcription appears at the particular position in one or more other translated partial transcriptions that appear after the translated partial transcription in a chronological list of translated partial transcriptions; generating the stability value for the translated partial transcription using the correctness probability for each word in the translated partial transcription; 5. The computer-implemented method of claim 4, comprising:

6. Calculating the probability of correctness for each word in the translated partial transcription, For each word in the translated partial transcription, determining a specific location of the word within the translated partial transcription; calculating a first value, the first value being the number of translated partial transcriptions listed after the translated partial transcription in the chronological list of translated partial transcriptions; calculating a second value, the second value specifying the number of times the word appears at the particular position within a translated partial transcription that appears after the translated partial transcription in the chronological list of translated partial transcriptions; dividing the second value by the first value to obtain the validity probability of the word; 6. The computer-implemented method of claim 5, comprising:

7. The step of generating the stability value further comprises: calculating a frequency of change metric for each word in the translated partial transcription, the frequency of change metric measuring the number of times a word at a particular position in the translated partial transcription changes at that particular position in a translated partial transcription that is listed after the translated partial transcription in a chronological list of translated partial transcriptions; generating the stability value for the translated partial transcription using the change frequency metric for each word in the translated partial transcription; 5. The computer-implemented method of claim 4, comprising:

8. The step of generating the stability value further comprises: calculating an equal-to-last metric for each word in the partial transcription, the equal-to-last metric identifying whether a position of a word in the partial transcription is the same as a position of the word in a partial transcription that appears last on the chronological list of translated partial transcriptions; generating the stability value for the translated partial transcription using the equal-to-last metric for each word in the translated partial transcription; 5. The computer-implemented method of claim 4, comprising:

9. 1. A computer-implemented method comprising: obtaining an audio data stream; determining a first language in which the audio in the audio data stream is recorded; generating partial transcriptions of the audio for a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio to a particular time interval and (2) being in the first language; For each specific consecutive partial transcription, translating the particular partial transcription into a second language; calculating a stability value for each word of the particular translated partial transcription by determining whether a position of the word in the particular translated partial transcription matches a position of the word in a threshold number of translated partial transcriptions in a set of translated partial transcriptions that precede the particular translated partial transcription; using a machine learning model to generate a stability value for the particular translated partial transcription using the calculated stability values ​​for the words in the particular translated partial transcription, the stability value for the translated specific partial transcription identifies a portion of the translated specific partial transcription that is stable; the machine learning model is trained to minimize a loss function and is trained using a set of translations of transcriptions generated from one or more training audio data files recorded in the first language and a corresponding set of stability values ​​each indicating whether each set of translations is stable; said set of stability values ​​being generated using a set of rules; providing the portion of the translated particular partial transcription for display on a user device in response to generating the stability value; 23. A computer-implemented method comprising:

10. A computer-implemented method as described in claim 9, wherein a stability value for a training audio data file from the one or more training audio data files is generated using the positions of words within the respective translated partial transcriptions and other translated partial transcriptions for the training audio data file.

11. 1. A system comprising: one or more memory devices for storing instructions; one or more data processing devices configured to interact with said one or more memory devices and to perform operations upon execution of said instructions; and wherein the operation comprises: receiving a content request from a client device; Obtaining an audio data stream; determining a first language in which audio in the audio data stream is recorded; generating a partial transcription of the audio for each time interval among a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio to a particular time interval and (2) being in the first language; For each partial transcription, translating the partial transcription into a second language different from the first language; inputting the translated partial transcription into a model, the model determines whether a portion of the input translated partial transcript is stable; inputting a portion of an input translated partial transcription being stable when the portion of the input translated partial transcription is not expected to change in another translated partial transcription of recorded audio by a time interval later than the time interval for the input partial transcription; identifying, by the model, a portion of the translated partial transcription as stable based on the input translated partial transcription; in response to identifying the portion of the translated partial transcription as stable, providing the portion of the translated partial transcription for display on a user device using the identification. Including, the model is trained based on translations of transcriptions generated from one or more training audio data files recorded in the first language; the transcriptions generated from one or more training audio data files include, for each training audio data file, a plurality of partial transcriptions of the training audio data file; each partial transcription of the training audio data file is generated at a respective one of a plurality of successive time intervals; The system, wherein each partial transcription is (1) a transcription of the audio from a beginning of the audio to a particular time interval, and (2) is in the first language.

12. The one or more data processing devices are configured to perform operations further including training the model to identify sequences of words within the partial transcription that are expected to be stable, the training comprising: obtaining a training audio data file; identifying, from among the training audio data files, a plurality of training audio data files having audio recorded in the first language; For each training audio data file in the plurality of training audio data files, generating partial transcriptions of the audio in the training audio data file at a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio to a particular time interval and (2) being in the first language; generating a chronological list of partial transcriptions ordered from oldest to newest relative to the start of the audio; generating a chronological list of translated partial transcriptions corresponding to the chronological list of partial transcriptions, the chronological list including, for each partial transcription in the chronological list of partial transcriptions, translating the partial transcription into the second language; for each translated partial transcription, starting with a first translated partial transcription in said chronological list of translated partial transcriptions; generating a stability value using word positions within the translated partial transcription and other translated partial transcriptions to identify portions of the translated partial transcription that are stable; training the model using the translated partial transcriptions as inputs to the model and the stability values ​​as outputs of a translation stability model; The system of claim 11 , comprising:

13. generating said stability value calculating a correctness probability for each word in the translated partial transcription, the correctness probability measuring the probability that a word at a particular position in the translated partial transcription appears at the particular position in one or more other translated partial transcriptions that appear after the translated partial transcription in a chronological list of translated partial transcriptions; generating the stability value for the translated partial transcription using the correctness probability for each word in the translated partial transcription; The system of claim 12, comprising:

14. generating said stability value calculating a frequency of change metric for each word in the translated partial transcription, the frequency of change metric measuring the number of times a word at a particular position in the translated partial transcription changes at that particular position in a translated partial transcription that is listed after the translated partial transcription in a chronological listing of translated partial transcriptions; generating the stability value for the translated partial transcription using the change frequency metric for each word in the translated partial transcription; The system of claim 12, comprising:

15. generating said stability value calculating an equal-to-last metric for each word in the partial transcription, the equal-to-last metric identifying whether a position of a word in the partial transcription is the same as a position of the word in a partial transcription that appears last on the chronological list of translated partial transcriptions; generating the stability value for the translated partial transcription using the equal-to-last metric for each word in the translated partial transcription; The system of claim 12, comprising:

16. A non-transitory computer readable storage medium having instructions stored thereon, the instructions, when executed by one or more data processing devices, causing the one or more data processing devices to: Obtaining an audio data stream; determining a first language in which audio in the audio data stream is recorded; generating a partial transcription of the audio for each time interval among a plurality of consecutive time intervals, each partial transcription (1) being a transcription of the audio from a beginning of the audio to a particular time interval and (2) being in the first language; For each partial transcription, translating the partial transcription into a second language different from the first language; inputting the translated partial transcription into a model, the model determines whether a portion of the input translated partial transcript is stable; inputting a portion of an input translated partial transcription being stable when the portion of the input translated partial transcription is not expected to change in another translated partial transcription of recorded audio by a time interval later than the time interval for the input partial transcription; identifying, by the model, a portion of the translated partial transcription as stable based on the input translated partial transcription; in response to identifying the portion of the translated partial transcription as stable, providing the portion of the translated partial transcription for display on a user device using the identification. Executing an operation including the model is trained based on translations of transcriptions generated from one or more training audio data files recorded in the first language; the transcriptions generated from one or more training audio data files include, for each training audio data file, a plurality of partial transcriptions of the training audio data file; each partial transcription of the training audio data file is generated at a respective one of a plurality of successive time intervals; A non-transitory computer-readable storage medium, wherein each partial transcription is (1) a transcription of the audio from a beginning of the audio to a particular time interval, and (2) is in the first language.

Images