Audio translation method and device for simultaneous interpretation, equipment and storage medium

Abstract

This application provides an audio translation method, apparatus, device, and storage medium for simultaneous interpreting. The method acquires source language audio, encodes it to obtain a source language token sequence, reverses the source language token sequence to obtain a reversed token sequence, concatenates the reversed token sequence to the beginning of the output sequence of a trained simultaneous interpreting model to obtain a concatenated token sequence, inputs the concatenated token sequence into the simultaneous interpreting model to generate a target language token sequence, and finally decodes the target language token sequence to obtain target language audio with the same semantics as the source language audio. Embodiments of this application can achieve streaming full-duplex simultaneous interpreting.

Description

Technical Field

[0001] This application relates to the field of audio signal processing technology, and more specifically, to an audio translation method, apparatus, device, and storage medium for simultaneous interpretation. Background Technology

[0002] With the development of intelligent voice technology, voice translation systems have made significant progress. However, in simultaneous interpreting, the source language audio and the target language translation audio are not strictly sequential, but rather alternate over time. That is, simultaneous interpreting requires both continuously receiving new voice input and generating the corresponding translation output before the input is complete. Therefore, how to generate accurate and fluent target language translation audio in real time while ensuring semantic consistency, even when the source language audio input is incomplete, is a key problem that simultaneous interpreting technology urgently needs to solve. Summary of the Invention

[0003] In view of this, this application provides an audio translation method, apparatus, device, and storage medium for simultaneous interpretation, in order to at least solve the problems existing in the related art.

[0004] Specifically, this application is implemented through the following technical solution: This application provides an audio translation method for simultaneous interpreting, including: Obtain the source language audio and encode the source language audio to obtain a source language token sequence; The source language token sequence is reversed to obtain a reversed token sequence; The reversed token sequence is concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain the concatenated token sequence. The concatenated token sequence is input into the simultaneous interpretation model to generate a target language token sequence. The target language token sequence is decoded to obtain target language audio with the same semantics as the source language audio.

[0005] This application also provides an audio translation device for simultaneous interpreting, comprising: An audio acquisition module is used to acquire source language audio and encode the source language audio to obtain a source language token sequence; The reverse order processing module is used to reverse the source language token sequence to obtain a reversed token sequence; The token concatenation module is used to concatenate the reversed token sequence to the beginning of the output sequence of the trained simultaneous interpretation model to obtain the concatenated token sequence. The sequence translation module is used to input the spliced ​​token sequence into the simultaneous interpretation model to generate a token sequence in the target language; The token decoding module is used to decode the target language token sequence to obtain target language audio with the same semantics as the source language audio.

[0006] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the audio translation methods for simultaneous interpretation described in the foregoing embodiments.

[0007] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the audio translation methods for simultaneous interpretation described in the foregoing embodiments.

[0008] This application also provides a computer program product, including a computer program that, when run by a processor, performs the steps of any of the possible audio translation methods for simultaneous interpreting described above.

[0009] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this embodiment, the source language token sequence is reversed and then concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain a concatenated token sequence. This concatenated token sequence is then input into the simultaneous interpretation model, enabling the model to explicitly distinguish between the reversed token sequence and the output sequence under causal constraints during inference. That is, for the reversed token sequence, a left-to-right order is used, allowing the model to utilize all the audio that has been heard. For the output sequence, a causal order is used, meaning that when generating a target language token, the model can only rely on the tokens that have already been generated and cannot see future tokens. This avoids the model's ability to predict the future, thus preventing logical errors in the translation process. Furthermore, the model can simultaneously handle both the input of the source language audio and the output of the target language audio, achieving streaming full-duplex simultaneous interpretation.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating an audio translation method for simultaneous interpreting, as shown in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating the training process of a simultaneous interpretation model according to an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the structure of an audio translation device for simultaneous interpretation, as shown in an exemplary embodiment of this application. Figure 4 This is a schematic diagram illustrating the structure of another audio translation device for simultaneous interpretation, as shown in an exemplary embodiment of this application. Figure 5 This is a hardware structure diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0014] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0015] With the continuous development of intelligent voice technology, voice translation systems have made significant progress in offline translation scenarios. However, in simultaneous interpreting scenarios, the system needs to output translation results in real time while voice input continues to arrive, i.e., perform streaming full-duplex voice translation tasks. Compared to offline voice translation, simultaneous interpreting tasks are more technically challenging, mainly in the following aspects: First, the input and output are highly intertwined in the time dimension. In simultaneous interpreting, the source language speech input and the target language translation output are not strictly sequential, but rather alternate continuously over time. The model needs to continuously receive new speech input while generating the corresponding translation output before the input process is complete, which places higher demands on the causal consistency of sequence modeling.

[0016] Secondly, the dual constraints of low latency and translation accuracy constitute a technical bottleneck. Simultaneous interpretation systems need to achieve the most accurate translation possible while ensuring low latency. The model cannot wait for the complete input before translating, and it must also make reasonable use of existing contextual information for reasoning, which makes the traditional "encode first, then decode" processing mode difficult to apply directly.

[0017] Furthermore, there are significant differences between training and inference scenarios. Existing speech translation models are typically trained on complete speech-text pairs, while the inference stage of simultaneous interpreting deals with incomplete and continuously growing input sequences. How to effectively adapt the model to streaming full-duplex inference during the training phase while maintaining its causal structure has been a long-standing technical challenge in this field.

[0018] In summary, streaming full-duplex speech translation tasks pose new challenges to existing speech translation models in terms of sequence organization, causal modeling mechanisms, and training method adaptation.

[0019] For simultaneous interpreting tasks, the relevant technologies can be mainly summarized into the following three types of solutions.

[0020] (1) Large language model method based on interleaved sequences. This type of method requires interleaving the input speech segments and the output translation segments into a sequence that satisfies causal constraints. It is difficult to directly use existing non-streaming speech translation data for training, the data construction is complex, and the training cost is high.

[0021] (2) Encoder-decoder based dual-channel model. This type of method models the input and output by placing them in different channels, but usually requires redesigning or customizing the model structure, making it difficult to fully inherit the general capabilities of existing base large language models in language understanding and generation.

[0022] (3) Methods based on streaming sequence models such as recurrent neural networks. Although these methods naturally support streaming processing, they are limited in terms of long sequence modeling and language generation capabilities, and the naturalness and expressive quality of the translation results are significantly lower than those based on large language models.

[0023] However, the above-mentioned technical approaches have obvious shortcomings in model structure, training methods, and practical applications, making it difficult to simultaneously meet the requirements of streaming, translation quality, and engineering feasibility. Therefore, how to generate accurate and fluent target language translation audio in real time when the source language audio input is incomplete, while ensuring semantic consistency, is a key problem that simultaneous interpretation technology urgently needs to solve.

[0024] Based on the above research, this disclosure provides an audio translation method for simultaneous interpreting. The method first acquires source language audio and encodes it to obtain a source language token sequence; then, it reverses the source language token sequence to obtain a reversed token sequence; the reversed token sequence is concatenated to the beginning of the output sequence of a trained simultaneous interpreting model to obtain a concatenated token sequence; then, the concatenated token sequence is input into the simultaneous interpreting model to generate a target language token sequence; finally, the target language token sequence is decoded to obtain target language audio with the same semantics as the source language audio.

[0025] In this embodiment, the source language token sequence is reversed and then concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain a concatenated token sequence. This concatenated token sequence is then input into the simultaneous interpretation model, enabling the model to explicitly distinguish between the reversed token sequence and the output sequence under causal constraints during inference. That is, for the reversed token sequence, a left-to-right order is used, allowing the model to utilize all the audio that has been heard. For the output sequence, a causal order is used, meaning that when generating a target language token, the model can only rely on the tokens that have already been generated and cannot see future tokens. This avoids the model's ability to predict the future, thus preventing logical errors in the translation process. Furthermore, the model can simultaneously handle both the input of the source language audio and the output of the target language audio, achieving streaming full-duplex simultaneous interpretation.

[0026] To facilitate understanding of this embodiment, a detailed description of the audio translation method for simultaneous interpreting disclosed in this disclosure is provided first. The execution entity of the audio translation method for simultaneous interpreting provided in this disclosure is generally a computer device. This computer device can be a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the computer device can also be a terminal device, which can be a mobile device, terminal, handheld device, computing device, vehicle-mounted device, etc.

[0027] In other embodiments, the method can also be applied to an implementation environment consisting of computer equipment and servers, or an implementation environment consisting of terminal equipment and servers. Furthermore, this audio translation method for simultaneous interpretation can also be implemented by a processor calling computer-readable instructions stored in memory.

[0028] Before introducing audio translation methods for simultaneous interpreting, we will first introduce the relevant technical terms.

[0029] Simultaneous interpreting: refers to speech translation technology that generates and outputs target speech in real time while the source speech is continuously input. It requires the system to complete understanding, reasoning and generation before the input is finished, and is widely used in scenarios such as international conferences, cross-language communication and real-time voice interaction.

[0030] Streaming speech translation refers to a method of processing and translating continuously input speech streams online. Its characteristic is that the translation results are output step by step during the process of receiving speech, rather than waiting for the complete input to end before generating them all at once.

[0031] Full-duplex speech translation: refers to a system that can continuously output target speech while receiving source speech input, realizing parallel interaction between input and output in the time dimension, which is different from the traditional half-duplex translation mode of "input first, output later".

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] Please see the appendix Figure 1 The above is a flowchart illustrating an audio translation method for simultaneous interpreting, as shown in an exemplary embodiment of this application. Figure 1 As shown, the audio translation method for simultaneous interpretation in this embodiment may include the following steps S101~S105: S101: Obtain the source language audio and encode the source language audio to obtain the source language token sequence.

[0034] The source language audio can refer to the original speech signal to be translated. The source language audio can include characters, words, etc., without limitation.

[0035] Here, the source language can be any language, such as Chinese, English, German, French, etc., without any limitation.

[0036] In simultaneous interpretation scenarios, the source language audio can refer to a continuously arriving streaming audio stream. Of course, in other implementations, the source language audio can also be non-streaming audio, such as a complete audio segment, which is not limited here.

[0037] In this embodiment, the source language audio can be encoded using an audio encoder to obtain a source language token sequence, which includes multiple tokens.

[0038] A token is a digital identifier in the computer field. In natural language processing, a token, as the smallest unit of text processing, refers to a character or sequence of characters and is the basic unit of input and output for large language models.

[0039] It is understandable that the source language token sequence can refer to the discrete symbol sequence obtained after encoding, where each token represents a short segment of semantic or acoustic features of the audio, arranged in ascending order according to the original speech time sequence.

[0040] This step serves as the starting point for streaming audio processing. By continuously acquiring the audio stream (e.g., acquiring a new audio segment every 200 milliseconds), the audio encoder converts it into a token sequence in the source language in real time. The key is "streaming encoding," which eliminates the need to wait for the entire sentence to end.

[0041] For example, if the source language audio is "Welcome to the conference" (total duration 3.2 seconds), and the "Welcome" audio is received at t=0.4s, the audio encoder encodes it and outputs the corresponding token. ;t=1.0s: Add the "tothe" fragment, and update the token sequence to ;t=2.0s: Add "conference" audio, token sequence updated to The source language token sequence is in ascending order: , corresponding to the audio timeline.

[0042] It should be noted that the number of words contained in the audio and the corresponding number of tokens obtained by encoding in the examples in this application are only illustrative. In general, the number of tokens obtained by encoding audio per second by the audio encoder is between 25 and 50, and there is no fixed correspondence between the numbers.

[0043] S102: Reverse the source language token sequence to obtain a reversed token sequence.

[0044] Here, reverse order processing refers to the operation of reversing the source language token sequence along the time dimension.

[0045] Following the example above, the source language token sequence is: After reversing the order, the reversed token sequence is: .

[0046] In this embodiment of the application, the reverse order processing can be achieved by dynamically maintaining a buffer. Specifically, the source language token sequence can be stored in the buffer in chronological order, and when storing each token, the token is inserted into the beginning position of the sequence in the buffer to obtain the reversed token sequence.

[0047] Here, the temporal order refers to the order in which the tokens arrive. That is, as mentioned earlier, the source language audio is streaming; therefore, when generating the source language token sequence, the tokens are also added streamingly. For example, tokens... First to arrive, token Arrived later.

[0048] It is understandable that the above process of storing each token in reverse order is completed without the need for a process of storing in forward order first and then reversing.

[0049] Following the example above, at time t1, the source language token Upon arrival, insert it into the beginning position of the sequence in the buffer; at this point, the sequence in the buffer is... At time t2, the source language token Upon arrival, insert it into the beginning position of the sequence in the buffer; at this point, the sequence in the buffer is... At time t3, the source language token Upon arrival, insert it into the beginning position of the sequence in the buffer; at this point, the sequence in the buffer is... .

[0050] Based on the above, it can be seen that the reverse processing flow simultaneously completes the processing of the input stream in chronological order (processing as soon as a new token arrives), realizes the sequence reversal (inserting the starting position), and generates a usable reverse sequence (the buffer state is the result), without the intermediate process of "first storing in ascending order and then reversing".

[0051] S103: The reversed token sequence is concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain the concatenated token sequence.

[0052] Here, the simultaneous interpretation model can be obtained by training a multimodal large model.

[0053] The output sequence of the simultaneous interpreting model can be empty or it can include the currently generated target language token sequence. For example, the case of an empty output sequence can refer to the beginning stage of simultaneous interpreting, when the simultaneous interpreting model has not yet processed it. As simultaneous interpreting continues, the simultaneous interpreting model continuously updates the output sequence. Therefore, its output sequence can also include the currently generated target language token sequence.

[0054] Continuing from the previous example, for the case where the output sequence is empty: if the reversed token sequence is... If the output sequence is empty, then the concatenated token sequence is: .

[0055] For the case where the output sequence is not empty: if the reversed token sequence is The output sequence is Then the concatenated token sequence is: .

[0056] S104: Input the spliced ​​token sequence into the simultaneous interpretation model to generate a target language token sequence.

[0057] Furthermore, after inputting the concatenated token sequence into the simultaneous interpretation model, it can be translated to generate a token sequence in the target language.

[0058] Here, the target language is different from the source language.

[0059] Specifically, for step S104, the spliced ​​token sequence can be input into the simultaneous interpretation model, and a one-way causal attention mechanism can be used to sequentially translate the spliced ​​token sequence in an autoregressive manner to obtain the target language token sequence.

[0060] Among them, the one-way causal attention mechanism refers to the process of generating sequences. At that time, only the sequence can be accessed. All tokens within the range.

[0061] The autoregressive approach refers to inputting a spliced ​​token sequence into a simultaneous interpretation model, where the model uses the spliced ​​token sequence itself as a training label (or regression target). This allows the output of the simultaneous interpretation model to regress to the training label, thereby improving the model's prediction accuracy.

[0062] Following the example above, the token sequence is concatenated as follows: In this case, the output sequence of the simultaneous interpretation model in the concatenated token sequence is empty. At this time, the concatenated token sequence is input into the simultaneous interpretation model to obtain the first target language token. (Joy), and the output sequence of the simultaneous interpretation model includes (Happy); Furthermore, the concatenated token sequence is updated to The data is then input into a simultaneous interpreting model to predict the next target language token. (Welcome), and the output sequence of the simultaneous interpretation model includes Similarly, the concatenated token sequence is updated to... The data is then input into a simultaneous interpreting model to predict the next target language token. (Refer to) In this way, the splicing token sequence is dynamically updated and continuously input into the simultaneous interpretation model to complete the translation.

[0063] Thus, the unidirectional causal attention mechanism combined with the autoregressive processing method can satisfy the streaming causal constraints. Each time a new batch of source language tokens (such as 200ms of speech) is received, the reverse buffer is updated, thereby generating 1 to N target language tokens. The input token sequence and the output token sequence continuously overlap on the physical time axis, realizing true "translation while listening".

[0064] S105: Decode the target language token sequence to obtain target language audio with the same semantics as the source language audio.

[0065] By using an audio decoder to decode the target language token sequence, the target language audio with the same semantics as the source language audio can be obtained, and the source language audio and the target language audio have an intersection in the time dimension, thus realizing simultaneous interpretation.

[0066] In this embodiment, the source language token sequence is reversed and then concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain a concatenated token sequence. This concatenated token sequence is then input into the simultaneous interpretation model, enabling the model to explicitly distinguish between the reversed token sequence and the output sequence under causal constraints during inference. That is, for the reversed token sequence, a left-to-right order is used, allowing the model to utilize all the audio that has been heard. For the output sequence, a causal order is used, meaning that when generating a target language token, the model can only rely on the tokens that have already been generated and cannot see future tokens. This avoids the model's ability to predict the future, thus preventing logical errors in the translation process. Furthermore, the model can simultaneously handle both the input of the source language audio and the output of the target language audio, achieving streaming full-duplex simultaneous interpretation.

[0067] Please see Figure 2 This is a flowchart illustrating the training process of a simultaneous interpretation model provided in an embodiment of this application. Figure 2 As shown, the training process includes S201~S204: S201: Obtain source language audio samples and target language audio samples that have the same semantics as the source language audio samples.

[0068] Here, the source language audio sample refers to a complete speech segment (non-streaming).

[0069] The semantic similarity can include the same audio information content. In other implementations, it can also include the same emotional tone, the same pragmatic intent, etc., which are not limited here.

[0070] For example, the source language audio sample could be "Driver should focus on driving", and the target language audio sample could be "Driver should focus on driving".

[0071] The source language audio sample can be any language (such as Chinese, English, Spanish, Japanese, etc.), and there are no restrictions here. The target language audio sample is a different language from the source language audio sample.

[0072] In this embodiment, the source language audio samples and the target language audio samples are non-streaming audio.

[0073] S202: Encode the source language audio sample to obtain a source language token sequence sample, and encode the target language audio sample to obtain a target language token sequence sample.

[0074] Here, the source language audio sample can be encoded using an audio encoder to obtain a source language token sequence sample, and the target language audio sample can be encoded using an audio encoder to obtain a target language token sequence sample.

[0075] Following the example above, the source language audio sample could be "Driver should focus on driving". The source language token sequence sample is... The target language audio sample could be "Drivers need to concentrate on driving," and the target language token sequence sample could be... .

[0076] It should be noted that, due to different languages, the number of tokens in the source language token sequence sample and the target language token sequence sample may be different. Of course, in some implementations, the number of tokens in the source language token sequence sample and the target language token sequence sample may also be the same for other source language audio samples and corresponding tagline audio samples.

[0077] S203: Reverse the source language token sequence sample to obtain a reversed token sequence sample, and concatenate the reversed token sequence sample with the target language token sequence sample to obtain a concatenated sequence sample.

[0078] Here, the reverse order processing is similar to step S102 above, and will not be repeated here.

[0079] Following the example above, a reversed token sequence sample could be... The target language token sequence sample is Then the spliced ​​sequence sample is .

[0080] S204: Input the spliced ​​sequence sample into the simultaneous interpretation model to be trained, and perform supervised training on the large model to be trained to obtain the simultaneous interpretation model.

[0081] Here, by inputting the spliced ​​sequence samples into the simultaneous interpretation model to be trained, the simultaneous interpretation model can learn the ability to "autoregressively generate target language token sequences based on reverse input context".

[0082] Specifically, the reversed token sequence samples in the spliced ​​sequence samples are used as training samples, and the target language token sequence samples in the spliced ​​sequence samples are used as the sample labels of the training samples to perform supervised training on the large model to be trained.

[0083] It is understandable that the reversed token sequence samples are used as training samples, while the simultaneous interpretation model outputs the predicted target language token sequence. Then, based on the predicted target language token sequence and the target language token sequence samples, the model loss is determined, and the model parameters of the simultaneous interpretation model to be trained are adjusted according to the model loss to complete supervised training and obtain the simultaneous interpretation model.

[0084] In this embodiment, the bidirectional sequence arrangement described above enables the simultaneous interpretation model to be trained directly using existing standard non-streaming speech translation data, without the need for complex interleaving or additional streaming annotation of the training data, thereby significantly reducing the training difficulty and engineering complexity.

[0085] Furthermore, by improving the sequence arrangement method, support for full-duplex tasks is achieved. This is simple to implement, easy to integrate with existing speech translation systems, and has good engineering feasibility and application promotion value.

[0086] Corresponding to the aforementioned embodiments of audio translation methods for simultaneous interpreting, this application also provides embodiments of audio translation apparatus for simultaneous interpreting.

[0087] Please see Figure 3 This is a schematic diagram illustrating the structure of an audio translation device for simultaneous interpretation, as shown in an exemplary embodiment of this application. Figure 3 As shown, the audio translation device 300 for simultaneous interpretation includes: The audio acquisition module 310 is used to acquire source language audio and encode the source language audio to obtain a source language token sequence; The reverse order processing module 320 is used to reverse the source language token sequence to obtain a reversed token sequence; The token concatenation module 330 is used to concatenate the reversed token sequence to the beginning of the output sequence of the trained simultaneous interpretation model to obtain a concatenated token sequence. Sequence translation module 340 is used to input the spliced ​​token sequence into the simultaneous interpretation model to generate a target language token sequence; The token decoding module 350 is used to decode the target language token sequence to obtain target language audio with the same semantics as the source language audio.

[0088] In some implementations, the source language audio and the target language audio overlap in the time dimension.

[0089] In some implementations, the source language token sequence includes multiple tokens ordered chronologically; the reverse order processing module 320 is specifically used for: The source language token sequence is stored in a buffer in chronological order, and when storing each token, the token is inserted into the beginning position of the sequence in the buffer to obtain the reversed token sequence.

[0090] In some embodiments, the sequence translation module 340 is specifically used for: The concatenated token sequence is input into the simultaneous interpretation model, and a one-way causal attention mechanism is used to sequentially translate the concatenated token sequence in an autoregressive manner to obtain the target language token sequence.

[0091] Please see Figure 4 This is a schematic diagram illustrating the structure of another audio translation device for simultaneous interpretation, as shown in an exemplary embodiment of this application. Figure 4 As shown, the audio translation device 300 for simultaneous interpretation also includes a model training module 360, which is used for: Obtain source language audio samples and target language audio samples that have the same semantics as the source language audio samples; The source language audio sample is encoded to obtain a source language token sequence sample, and the target language audio sample is encoded to obtain a target language token sequence sample. The source language token sequence sample is reversed to obtain a reversed token sequence sample, and the reversed token sequence sample is concatenated with the target language token sequence sample to obtain a concatenated sequence sample. The spliced ​​sequence samples are input into the simultaneous interpretation model to be trained, and the large model to be trained is subjected to supervised training to obtain the simultaneous interpretation model.

[0092] In some implementations, the model training module 360 ​​is specifically used for: The spliced ​​sequence samples are input into the simultaneous interpretation model to be trained. The reversed token sequence samples in the spliced ​​sequence samples are used as training samples, and the target language token sequence samples in the spliced ​​sequence samples are used as sample labels for the training samples. The large model to be trained is then subjected to supervised training.

[0093] In some implementations, the source language audio sample and the target language audio sample are non-streaming audio.

[0094] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0095] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0096] Corresponding to the above-described audio translation method for simultaneous interpreting, this disclosure also provides a computer device, such as... Figure 5 The diagram shown is a structural schematic of a computer device provided in an embodiment of this disclosure. Figure 5 As shown, the computer device 500 includes a processor 510, an internal bus 520, memory 530, a network interface 540, and non-volatile memory 550, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 510 reads the corresponding computer program from the non-volatile memory 550 into the memory 530 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0097] The memory 530, also known as internal memory, is used to temporarily store the computational data in the processor 510, as well as the data exchanged with non-volatile memory 550 such as hard disk. The processor 510 exchanges data with non-volatile memory 550 through the memory 530.

[0098] In this embodiment, memory 530 is specifically used to store application code that executes the solution of this application, and its execution is controlled by processor 510. That is, when the computer device is running, processor 510 communicates with network interface 540, memory 530 and non-volatile memory 550 through internal bus 520, so that processor 510 executes the application code stored in memory 530 and non-volatile memory 550, thereby executing the audio translation method for simultaneous interpretation described in the above method embodiment.

[0099] Processor 510 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware microservices. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0100] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 500. In other embodiments of this application, the computer device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0101] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the audio translation method for simultaneous interpretation described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0102] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the audio translation method for simultaneous interpretation in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0103] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0104] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0105] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0106] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. Basic computer microservices include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0107] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0108] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0109] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and microservices in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program microservices and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0110] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0111] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An audio translation method for simultaneous interpreting, characterized in that, include: Obtain the source language audio and encode the source language audio to obtain a source language token sequence; The source language token sequence is reversed to obtain a reversed token sequence; The reversed token sequence is concatenated to the beginning of the output sequence of the trained simultaneous interpretation model to obtain the concatenated token sequence. The concatenated token sequence is input into the simultaneous interpretation model to generate a target language token sequence. The target language token sequence is decoded to obtain target language audio with the same semantics as the source language audio.

2. The method according to claim 1, characterized in that, The source language audio and the target language audio overlap in the time dimension.

3. The method according to claim 1, characterized in that, The source language token sequence includes multiple tokens ordered chronologically. The process of reversing the source language token sequence to obtain a reversed token sequence includes: The source language token sequence is stored in a buffer in chronological order, and when storing each token, the token is inserted into the beginning position of the sequence in the buffer to obtain the reversed token sequence.

4. The method according to claim 1, characterized in that, The step of inputting the concatenated token sequence into the simultaneous interpretation model to generate a target language token sequence includes: The concatenated token sequence is input into the simultaneous interpretation model, and a one-way causal attention mechanism is used to sequentially translate the concatenated token sequence in an autoregressive manner to obtain the target language token sequence.

5. The method according to any one of claims 1-4, characterized in that, The simultaneous interpreting model is trained through the following steps: Obtain source language audio samples and target language audio samples that have the same semantics as the source language audio samples; The source language audio sample is encoded to obtain a source language token sequence sample, and the target language audio sample is encoded to obtain a target language token sequence sample. The source language token sequence sample is reversed to obtain a reversed token sequence sample, and the reversed token sequence sample is concatenated with the target language token sequence sample to obtain a concatenated sequence sample. The spliced ​​sequence samples are input into the simultaneous interpretation model to be trained, and the large model to be trained is subjected to supervised training to obtain the simultaneous interpretation model.

6. The method according to claim 5, characterized in that, The step of inputting the spliced ​​sequence sample into the simultaneous interpretation model to be trained, and performing supervised training on the large model to be trained, includes: The spliced ​​sequence samples are input into the simultaneous interpretation model to be trained. The reversed token sequence samples in the spliced ​​sequence samples are used as training samples, and the target language token sequence samples in the spliced ​​sequence samples are used as sample labels for the training samples. The large model to be trained is then subjected to supervised training.

7. The method according to claim 5, characterized in that, The source language audio samples and the target language audio samples are non-streaming audio.

8. An audio translation device for simultaneous interpreting, characterized in that, include: An audio acquisition module is used to acquire source language audio and encode the source language audio to obtain a source language token sequence; The reverse order processing module is used to reverse the source language token sequence to obtain a reversed token sequence; The token concatenation module is used to concatenate the reversed token sequence to the beginning of the output sequence of the trained simultaneous interpretation model to obtain the concatenated token sequence. The sequence translation module is used to input the spliced ​​token sequence into the simultaneous interpretation model to generate a token sequence in the target language; The token decoding module is used to decode the target language token sequence to obtain target language audio with the same semantics as the source language audio.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the audio translation method for simultaneous interpretation as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the audio translation method for simultaneous interpretation as described in any one of claims 1-7.

Images