Method, device, electronic device and storage medium for speech translation

Abstract

The application discloses a speech translation method and device, electronic equipment and storage medium, wherein the method comprises: acquiring continuous audio segments of a source language audio stream; for each audio segment in the continuous audio segments, performing feature extraction on each audio segment through a large language model (LLM) to obtain a first feature representation of each audio segment; based on the first feature representation of each audio segment, generating a source language text corresponding to each audio segment through a decoder, and displaying the source language text; the decoder is a decoder independent of the LLM; obtaining a second feature representation of the source language audio stream based on the first feature representation of each audio segment; and based on the second feature representation of the source language audio stream, generating a target language text corresponding to the source language audio stream through the LLM. The application embodiment can improve the real-time performance of ASR recognition result output while taking into account the device power consumption.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, electronic device and storage medium for speech translation. Background Technology

[0002] With increasingly frequent communication between people from different language backgrounds, the demand for voice translation is constantly growing. As an important branch of AI in the field of natural language processing, voice translation technology plays a crucial role in scenarios such as meetings, phone calls, speeches, and video subtitle translation. The core of voice translation lies in translating the source language into target language text and then converting it back into speech output. Automatic Speech Recognition (ASR) is one of the key steps in voice translation. With the normalization of voice translation functions on edge devices, how to reduce the output latency of ASR recognition results while also considering the power consumption of the devices is a problem that needs to be solved. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for speech translation, which can output streaming ASR recognition results through a decoder independent of a large language model (LLM). In speech translation schemes using LLM, this helps reduce device power consumption and ensures the real-time display of ASR recognition results on the screen.

[0004] In a first aspect, embodiments of this application provide a method for voice translation, applied to an electronic device or a chip within an electronic device. Taking the application of this method to an electronic device as an example, the method includes:

[0005] Get consecutive audio segments from the source language audio stream;

[0006] For each audio segment in a continuous audio segment, features are extracted from each audio segment using a large language model (LLM) to obtain the first feature representation of each audio segment;

[0007] Based on the first feature representation of each audio segment, the source language text corresponding to each audio segment is generated by the decoder and displayed; the decoder is an LLM-independent decoder.

[0008] The second feature representation of the source language audio stream is obtained based on the first feature representation of each audio segment;

[0009] Based on the second feature representation of the source language audio stream, the target language text corresponding to the source language audio stream is generated by LLM.

[0010] As can be seen from the embodiments of this application, the electronic device performs streaming segmentation on the source language audio stream to obtain continuous audio segments. Each audio segment is feature extracted by the LLM, and the extracted feature representations are input into a decoder independent of the LLM for ASR recognition. The ASR recognition results of each audio segment are displayed on the screen in real time, which helps to reduce the latency of ASR recognition output and ensure the timeliness of ASR recognition. In addition, the LLM obtains the second feature representation of the source language audio stream through the first feature representation of each audio segment and translates it into the corresponding target language text. The decoding of ASR recognition is completed by an independent decoder, so that the LLM only needs to focus on the translation task and does not need to perform ASR recognition decoding, which reduces the complexity of LLM inference and helps to reduce the power consumption of the electronic device.

[0011] In one possible implementation, a large language model (LLM) is used to extract features from each audio segment, resulting in a first feature representation for each audio segment, including:

[0012] For any audio segment in a continuous audio sequence, extract the audio features of that audio segment and convert the audio features into text features;

[0013] Obtain the task prompt and first context information; the prompt is used to prompt the LLM to translate the source language audio stream into the corresponding target language text in combination with the context information; if any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder; if any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of the historical audio segments of any audio segment in the source language audio stream.

[0014] The text features, Prompt, and first context information are concatenated to obtain the first concatenated feature.

[0015] By using LLM to extract features from the first splicing features, the first feature representation of any audio segment can be obtained.

[0016] In this implementation, the electronic device, based on the LLM's Prompt, combines the historical ASR recognition results (i.e., the first contextual information) of each audio segment with the ASR recognition results of each segment when performing ASR recognition on each audio segment. The historical ASR recognition results help the LLM understand the relationships between words and their context, forming a more accurate understanding of the audio segment. The Prompt provides the framework for the current task and can serve as an effective contextual guide. By concatenating the text features corresponding to the audio segment with the Prompt and historical ASR recognition results, it is beneficial to improve the accuracy of the LLM's understanding of the audio segment, thereby extracting feature representations that are more conducive to the decoder generating ASR recognition results.

[0017] In one possible implementation, based on the second feature representation of the source language audio stream, the target language text corresponding to the source language audio stream is generated via LLM, including:

[0018] Obtain the second context information; if the source language audio stream is the first source language audio stream in the current conversation, the second context information is the initial state of the LLM; if the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation.

[0019] The second feature representation is concatenated with the second contextual information to obtain the second concatenated feature.

[0020] Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through LLM.

[0021] In this implementation, the translation results of the historical audio stream can help LLM understand the connection between the preceding and following audio and the translation context. The second feature representation obtained based on the first feature representation integrates the Prompt and historical ASR recognition results. The second concatenated feature obtained by splicing the second feature representation with the second context information brings more effective information to LLM, providing better context understanding and translation knowledge. This is conducive to improving the translation accuracy and quality of the source language audio stream and generating target language text that is more in line with the expression of the other end.

[0022] In one possible implementation, the electronic device is a first terminal device, and the method further includes, before acquiring consecutive audio segments of the source language audio stream:

[0023] In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

[0024] In this implementation, the first terminal device has a real-time call voice translation function. Users can enable the voice translation function on the first terminal device through the voice translation function switch interface. This allows for the timely output of ASR recognition results for audio segments during calls and the output of voice translation results after the audio stream ends. This enhances user trust in the voice translation system and improves the user's interactive experience. When there is no need for voice translation, users can also disable the voice translation function through the voice translation function switch interface to reduce the power consumption of the first terminal device.

[0025] In one possible implementation, the first terminal device also displays a second user interface in response to the first operation. The second user interface includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device.

[0026] The method also includes:

[0027] In response to a second operation on the local language settings bar on the second user interface, set the local source language;

[0028] In response to a third action on the peer's language settings bar on the second user interface, set the target language for the peer.

[0029] In this implementation, the first terminal device displays a second user interface in response to a first operation on the first user interface. The second user interface is a setting interface for the local and remote languages ​​of the voice translation function. Based on a second operation by the user in the local language setting section of the second user interface, the source language of the local device can be set. Based on a third operation by the user in the remote language setting section of the second user interface, the target language of the remote device can be set. This setting facilitates the first terminal device in subsequently performing voice translation, displaying the ASR recognition result as source language text and translating the input audio stream into target language text.

[0030] In one possible implementation, the method further includes:

[0031] When a call with the second terminal device is answered, a first prompt tone is output; the first prompt tone is used to notify the user that the voice translation function of the second terminal device has been enabled.

[0032] In this implementation, since the voice translation function may cause a certain delay in the call between the first terminal device and the second terminal device, when the second terminal device has the voice translation function enabled, the first terminal device will output a first prompt tone when the call is answered to inform the user on the first terminal device that there is a delay in the call and that they need to wait patiently, thereby improving the call experience.

[0033] In one possible implementation, after generating the target language text corresponding to the source language audio stream via LLM, the method further includes:

[0034] Convert target language text into target language audio;

[0035] Transmit the target language audio to the second terminal device;

[0036] If the duration of the target language audio transmitted is greater than or equal to a preset duration and no response is received from the second terminal device, the method further includes:

[0037] Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

[0038] In this implementation, if the duration of the target language audio emitted by the first terminal device exceeds the preset duration and the first terminal device does not receive a response from the second terminal device during the call, the first terminal device will also output a second prompt tone to inform the user that the other end is performing a voice translation operation and that the user needs to wait for the voice translation result from the other end.

[0039] Secondly, embodiments of this application provide a speech translation apparatus, applied to an electronic device or a chip within an electronic device; the apparatus includes an acquisition unit and a processing unit; wherein:

[0040] The acquisition unit is used to acquire continuous audio segments from the source language audio stream;

[0041] The processing unit is used to extract features from each audio segment in a continuous audio segment using a large language model (LLM) to obtain a first feature representation for each audio segment; based on the first feature representation of each audio segment, it generates the source language text corresponding to each audio segment through a decoder and displays the source language text; the decoder is an LLM-independent decoder; based on the first feature representation of each audio segment, it obtains a second feature representation of the source language audio stream; based on the second feature representation of the source language audio stream, it generates the target language text corresponding to the source language audio stream through an LLM.

[0042] In one possible implementation, the processing unit is specifically used to extract features from each audio segment using a large language model (LLM) to obtain a first feature representation of each audio segment.

[0043] For any audio segment in a continuous audio sequence, extract the audio features of that audio segment and convert the audio features into text features;

[0044] Obtain the task prompt and first context information; the prompt is used to prompt the LLM to translate the source language audio stream into the corresponding target language text in combination with the context information; if any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder; if any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of the historical audio segments of any audio segment in the source language audio stream.

[0045] The text features, Prompt, and first context information are concatenated to obtain the first concatenated feature.

[0046] By using LLM to extract features from the first splicing features, the first feature representation of any audio segment can be obtained.

[0047] In one possible implementation, regarding the generation of target language text corresponding to the source language audio stream via LLM based on the second feature representation of the source language audio stream, the processing unit is specifically used for:

[0048] Obtain the second context information; if the source language audio stream is the first source language audio stream in the current conversation, the second context information is the initial state of the LLM; if the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation.

[0049] The second feature representation is concatenated with the second contextual information to obtain the second concatenated feature.

[0050] Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through LLM.

[0051] In one possible implementation, the device is applied to a first terminal device; the processing unit is further configured to:

[0052] In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

[0053] In one possible implementation, the processing unit, in response to the first operation, also causes the first terminal device to display a second user interface, which includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device.

[0054] The processing unit is also used for:

[0055] In response to a second operation on the local language settings bar on the second user interface, set the local source language;

[0056] In response to a third action on the peer's language settings bar on the second user interface, set the target language for the peer.

[0057] In one possible implementation, the acquisition unit is also used for:

[0058] When a call with the second terminal device is answered, a first prompt tone is output; the first prompt tone is used to notify the user that the voice translation function of the second terminal device has been enabled.

[0059] In one possible implementation, the processing unit is also used for:

[0060] Convert target language text into target language audio;

[0061] Transmit the target language audio to the second terminal device;

[0062] If the duration of the target language audio sent is greater than or equal to a preset duration and no response is received from the second terminal device, the acquisition unit is further configured to:

[0063] Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

[0064] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0065] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs; the processor is coupled to the memory, the one or more programs are stored in the memory, and are configured to cooperate with the communication interface to implement the method in any of the embodiments of the first aspect described above when executed by the processor.

[0066] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing an electronic device on which the chip is installed to perform the method as described in any of the embodiments of the first aspect above.

[0067] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program for execution by an electronic device, wherein when executed, the computer program causes the electronic device to perform the method as described in any of the embodiments of the first aspect above.

[0068] Sixthly, embodiments of this application provide a computer program product that, when run by an electronic device, causes the electronic device to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0070] Figure 1 A schematic diagram of a cascaded language translation system provided in an embodiment of this application;

[0071] Figure 2 A schematic diagram of an adapter-based multimodal LLM speech translation system provided in this application embodiment;

[0072] Figure 3 A schematic diagram of a system architecture provided for an embodiment of this application;

[0073] Figure 4 A flowchart illustrating a speech translation method provided in an embodiment of this application;

[0074] Figure 5 A schematic diagram of the architecture of a speech translation model provided in this application embodiment;

[0075] Figure 6 A flowchart illustrating another speech translation method provided in an embodiment of this application;

[0076] Figure 7 A schematic diagram illustrating a user interface interaction provided in an embodiment of this application;

[0077] Figure 8 A schematic diagram illustrating another user interface interaction provided in an embodiment of this application;

[0078] Figure 9 A schematic diagram illustrating another user interface interaction provided in an embodiment of this application;

[0079] Figure 10 A schematic diagram of the structure of a speech translation device provided in an embodiment of this application;

[0080] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0082] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0083] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0084] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0085] (1) Automatic Speech Recognition (ASR): A technology that converts speech into text;

[0086] (2) Machine Translation (MT): Converting source text into target language text;

[0087] (3) Text-to-Speech (TTS): Achieving the conversion of text to speech;

[0088] (4) Large Language Model (LLM): This refers to a deep learning model with a large number of parameters that is pre-trained using a large amount of text data. These models can understand and generate natural language text and are widely used in natural language processing tasks such as text generation, translation, and question answering.

[0089] (5) Mel-Frequency Cepstral Coefficients (MFCC);

[0090] (6) Neural Machine Translation (NMT): A method for language translation that uses deep learning techniques, especially neural networks. It learns the mapping relationships between languages ​​by training on a large bilingual corpus, thereby achieving automatic translation;

[0091] (7) Connectionist Temporal Classification (CTC): An algorithm for sequence learning, particularly in speech recognition and handwriting recognition. It allows models to be trained without pre-aligning input and output sequences.

[0092] Cascaded speech translation systems are one of the more widely adopted speech translation technologies in recent years. They break down the speech translation task into multiple sub-tasks and achieve translation from the source language to the target language by cascading multiple modules. For example... Figure 1 As shown, a cascaded language translation system typically includes several main modules: ASR, FusionAdaptor, MT, and TTS.

[0093] 1) The ASR module is the first step in the system's processing, and its main task is to convert the user's speech input into text. This process involves multiple steps, including preprocessing of the speech signal, feature extraction, and decoding. With the development of deep learning technology, neural network-based ASR systems have made significant progress in accuracy and robustness. These systems can handle various accents, noise, and speaking styles, providing more accurate text output.

[0094] 2) The Fusion Adaptation module ensures accuracy and fluency in translation results through semantic unit segmentation and retranslation strategies. This module first segments the continuous text stream into semantically independent units, which can be words, phrases, or sentences, depending on the context and language structure. Precise semantic unit segmentation allows the system to better understand the inherent logic and context of the language, providing clear and orderly input for subsequent translation steps. Subsequently, the Fusion Adaptation module employs a retranslation strategy to correct and refine the initial translation results. In real-time speech translation, imperfections in speech recognition and machine translation can lead to translation errors or incoherence. The retranslation strategy introduces feedback mechanisms and iterative optimization to retranslate certain words or phrases, or reconstruct the entire sentence using a language model, achieving better translation quality. Furthermore, the retranslation strategy can continuously learn and adapt by incorporating user feedback and historical data, improving the system's adaptability and robustness. The introduction of the Fusion Adaptation module enables the real-time speech translation system to maintain efficiency and accuracy in dynamically changing communication environments, providing global users with a more convenient and efficient cross-language communication experience.

[0095] 3) The machine translation module is responsible for translating the source language text output by the ASR module into the target language text. Traditional machine translation systems are based on rule-based or statistical models, while current machine translation systems mainly employ neural networks, especially sequence-to-sequence (Seq2Seq) models. These models, through an encoder-decoder architecture, can capture deep semantic and contextual information of the language, thus providing more fluent and accurate translation results. Furthermore, the introduction of attention mechanisms further improves translation quality, enabling the model to focus on key parts of the input sequence.

[0096] 4) The text-to-speech module converts the target language text output by the machine translation module into speech output. A TTS system typically includes steps such as text analysis, speech synthesis, and post-processing. The text analysis stage is responsible for converting the text into an intermediate representation for speech synthesis, such as a phoneme sequence. The speech synthesis stage generates the corresponding speech signal, which can be achieved by concatenating pre-recorded speech segments (concatenation synthesis) or using parametric methods (such as a deep learning-based vocoder). The post-processing step aims to optimize the naturalness and intelligibility of the speech output.

[0097] In cascaded speech translation systems, the output of one module is usually the input of the next. However, this cascading approach has the following drawbacks: 1. Error propagation and poor translation quality. Because the system consists of multiple cascaded modules, errors in any one module can be amplified or mispropagated by subsequent modules. For example, an error in the speech recognition module may cause the machine translation module to receive inaccurate text input, resulting in an incorrect translation. This error propagation effect is particularly pronounced in long sentences or complex contexts, severely impacting translation quality and causing a significant deviation between the final translation output and the actual meaning of the source language. 2. Information loss. In the process from speech to text and then to the translated text, each layer of conversion may lose some original speech information. For example, the speech recognition module may not be able to fully capture the speaker's intonation and emotion, while the machine translation module may lose some subtle meanings of the source language due to differences in language structure. This information loss is particularly prominent when dealing with polysemous words, idioms, or culturally specific expressions, resulting in translations that lack the richness and accuracy of the source language. 3. Accumulated latency leading to excessive delays. Because the system needs to sequentially process multiple modules such as speech recognition, fusion adaptation, machine translation, and text-to-speech, the processing time of each module accumulates, leading to an overall extended response time. In real-time dialogue, this latency can disrupt the natural flow of communication, making it difficult for both parties to provide immediate feedback. This latency issue can be exacerbated, especially under high concurrency or poor network conditions, impacting user experience and system usability. 4. Difficulty in utilizing contextual information. In a cascaded architecture, each module typically operates independently, lacking a comprehensive understanding of the entire dialogue context. The speech recognition module focuses on converting the current speech segment into text, while the machine translation module translates based on this isolated text segment; neither fully utilizes contextual information to aid translation. This lack of contextual information can lead to translations that are out of context or semantically incoherent, especially when dealing with pronouns, tenses, and context-dependent expressions, affecting both accuracy and naturalness.

[0098] The adapter-based multimodal LLM speech translation solution is a novel translation system that integrates advanced ASR and LLM-based NMT technologies. Utilizing the adapter module as a bridge, it optimizes the interaction between different modal data, thereby achieving high-quality speech-to-text translation. For example... Figure 2As shown, this system processes the entire speech stream. An ASR encoder extracts audio features, an adapter fuses these features with other modal information to obtain corresponding feature representations, and an LLM (Local Language Translator) translates the source language audio into target language text based on the task prompt and these feature representations. This approach requires the complete speech signal to be input into the system before outputting the ASR result, resulting in a long delay and preventing real-time on-screen display. Furthermore, the LLM needs to perform ASR decoding and translation simultaneously, leading to high power consumption.

[0099] To overcome the defects and shortcomings of the prior art, embodiments of this application provide a speech translation method, which can be applied to... Figure 3 The system architecture is shown below. Figure 3 As shown in the embodiments of this application, the system architecture includes a transmitting user equipment and a receiving user equipment, which communicate through a base station. The transmitting user equipment deploys a real-time speech translation algorithm based on LLM (Local Language Management). This algorithm segments the input source language audio stream using an ASR encoder, extracts audio features from each segment, and then converts these audio features into text features using a modality fusion unit. The text features of each audio segment are concatenated with a Prompt and historical ASR recognition context information, and then fed into the LLM for classification to generate corresponding posterior probabilities. The posterior probabilities of each audio segment are then sent to a CTC (Concurrent Translation Control) decoder for decoding to obtain the ASR recognition result for each audio segment. This recognition result is displayed in real-time on the user interface of the transmitting user equipment, and the output latency of the ASR recognition result is low. Based on the accumulated posterior probabilities corresponding to each audio segment, the LLM obtains the posterior probability of the entire audio stream. Combined with historical translation context information, it generates the target language text corresponding to the source language audio stream and can convert the target language text into target language audio, which is then transmitted to the receiving user equipment via the base station. When the receiving user equipment needs to reply to the sending user equipment, the original receiving user equipment becomes the sending user equipment, and vice versa. It can be seen that the LLM only needs to perform the translation task, while the ASR recognition task is handled by the CTC decoder. Considering factors such as model structure, computational complexity, and inference mode, compared to having both ASR recognition and translation handled by the LLM, this approach helps reduce the power consumption of the sending user equipment.

[0100] For example, Figure 3 The transmitting and receiving user devices in the system architecture shown can be smartphones, tablets, personal computers (PCs), wearable devices, conferencing equipment, in-vehicle equipment, etc.

[0101] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0102] Please see Figure 4 , Figure 4 This is a flowchart illustrating a speech translation method provided in an embodiment of this application. This method can be applied to electronic devices. For example, the electronic device can be a server, cloud server, terminal device, wearable device, conferencing equipment, vehicle-mounted device, etc. Figure 3 The transmitting user equipment in the system architecture shown. For example... Figure 4 As shown, the method includes steps 401-405:

[0103] 401: Get consecutive audio segments from the source language audio stream.

[0104] In this embodiment of the application, the source language audio stream can be a prepared audio stream (such as a user's recording or an audio stream in an existing audio database), or it can be an audio stream input by the user in real time on the electronic device side (such as an audio stream input by the user in Chinese in scenarios such as meetings or calls).

[0105] For example, for an input source language audio stream, an electronic device can segment it using a preset time window, such as extracting every 25 milliseconds of audio stream as an audio segment. A complete audio stream will be extracted into multiple consecutive audio segments. Adjacent audio segments within these consecutive segments may or may not overlap. In some implementations, audio segments may also be referred to as audio frames, audio slices, streaming audio segments, audio chunks, etc.

[0106] 402: For each audio segment in a continuous audio segment, feature extraction is performed on each audio segment using a large language model (LLM) to obtain the first feature representation of each audio segment.

[0107] In this embodiment, for any extracted audio segment, the electronic device can extract its audio features and convert these audio features into text features. For example, the audio features can be Mel-Frequency Cepstral Coefficients (MFCC), Chroma features, regular cepstral coefficients, spectrograms, etc., and this application does not limit the specific features thereto. For example, the text features can be a token sequence corresponding to the audio segment, or an embedding vector of the token sequence corresponding to the audio segment.

[0108] For example, the speech translation method provided in this application embodiment can be implemented through a speech translation model, such as... Figure 5As shown, the speech translation model includes a streaming ASR encoder, a modal fusion unit, an LLM (Local Language Translation) module, and a decoder. The electronic device extracts audio segments from the user's source language audio stream using a streaming ASR encoder, and then extracts audio features from these segments. The audio features extracted by the streaming ASR encoder are fed into the modal fusion unit, which converts them into text features. Specifically, the modal fusion unit captures the temporal dependencies of the audio features and combines them with contextual and environmental features to generate text features corresponding to the audio features. The electronic device uses these text features, a task prompt, and historical context information as input to the LLM. The prompt guides the LLM to combine the context information with the source language audio stream to translate it into the corresponding target language text.

[0109] For example, LLM needs to perform two-stage tasks: the first stage is to assist the decoder in recognizing audio segments, and the second stage is to translate speech.

[0110] In the first stage, the historical context information includes first context information. If any audio segment is the first audio segment in a consecutive audio segment, the first context information is the decoder's initial state. If any audio segment is not the first audio segment in a consecutive audio segment, the first context information is the decoder's recognition result of the historical audio segment of any audio segment in the source language audio stream. For example, assuming the source language audio stream is extracted into four audio segments, for the first audio segment, since it has not yet generated real historical context information, the decoder's initial state is used as the historical context information. For the second audio segment, since the decoder has already generated the recognition result of the first audio segment (i.e., the ASR recognition result), its first context information is the ASR recognition result of the first audio segment. For the third audio segment, the first context information is the ASR recognition results of the first two audio segments. For the fourth audio segment, the first context information is the ASR recognition results of the first three audio segments. Of course, electronic devices do not necessarily need to use the recognition results of all historical audio segments as the first context information of the current audio segment. For example, they can use the recognition result of the previous audio segment as the first context information of the current audio segment. The initial state of the decoder can be a specific initial value, which can be set or learned by the decoder during training.

[0111] In extracting the first feature representation for any audio segment, the electronic device concatenates text features, a prompt, and first contextual information to obtain a first concatenated feature. Then, it uses LLM to extract features from the first concatenated feature to obtain the first feature representation for any audio segment. For example, the first feature representation can be the classification probability (e.g., posterior probability) corresponding to the input text features, or it can be the sequence labeling result (e.g., phoneme labeling, character labeling, sub-word labeling) corresponding to the input text features.

[0112] In this implementation, the electronic device, based on the LLM's Prompt, combines the historical ASR recognition results (i.e., the first contextual information) of each audio segment with the ASR recognition results of each segment when performing ASR recognition on each audio segment. The historical ASR recognition results help the LLM understand the relationships between words and their context, forming a more accurate understanding of the audio segment. The Prompt provides the framework for the current task and can serve as an effective contextual guide. By concatenating the text features corresponding to the audio segment with the Prompt and historical ASR recognition results, it is beneficial to improve the accuracy of the LLM's understanding of the audio segment, thereby extracting feature representations that are more conducive to the decoder generating ASR recognition results.

[0113] 403: Based on the first feature representation of each audio segment, the source language text corresponding to each audio segment is generated by the decoder and the source language text is displayed; the decoder is an LLM-independent decoder.

[0114] In this embodiment, the electronic device inputs the first feature representation into the decoder for decoding, and the decoder outputs the ASR recognition result of the audio segment, that is, generates the source language text corresponding to the audio segment. For example, in a meeting scenario, a user inputs (Chinese) voice "I come from region A" through a meeting unit. Assuming that "I come from" is extracted as an audio segment, the meeting unit displays the ASR recognition result "I come from" on the meeting unit's screen in real time. Or, in a video call scenario, a user inputs "I come from region A" through a smartphone, and the smartphone displays the ASR recognition result "I come from" on the phone screen in real time. In a tour guide scenario, a user inputs (Chinese) voice "Please follow me closely" through a microphone. Assuming that "follow closely" is extracted as an audio segment, the terminal device connected to the microphone displays the ASR recognition result "follow closely" on the terminal device's screen in real time.

[0115] For example, the decoder can differ depending on the form of the first feature representation. For instance, if the first feature representation is a posterior probability, the decoder can be a CTC decoder, which can perform decoding using algorithms such as beam search or greedy search. If the first feature representation is a sequence labeling result, the decoder can be a sequence labeling model decoder, which can perform decoding using algorithms such as conditional random fields. The decoding task for ASR recognition is no longer completed by an LLM (Local Level Model), but by an independent decoder, which can relatively reduce the power consumption of electronic devices in speech translation.

[0116] 404: The second feature representation of the source language audio stream is obtained based on the first feature representation of each audio segment.

[0117] In the second stage, based on the first feature representation obtained for each audio segment, the electronic device accumulates the first feature representation of each audio segment through LLM to obtain the second feature representation of the source language audio stream, effectively utilizing the information extracted in the ASR recognition stage. For example, when the first feature representation is the posterior probability, the second feature representation is the posterior probability corresponding to the entire audio sequence; when the first feature representation is the sequence labeling result, the second feature representation is the sequence labeling result corresponding to the entire audio sequence.

[0118] 405: Based on the second feature representation of the source language audio stream, generate the target language text corresponding to the source language audio stream through LLM.

[0119] In this embodiment, the electronic device can acquire second contextual information after obtaining the second feature representation of the entire audio sequence. When the source language audio stream is the first source language audio stream in the current dialogue, the second contextual information is the initial state of the LLM; when the source language audio stream is not the first source language audio stream in the current dialogue, the second contextual information is the translation result of the LLM of the historical source language audio streams in the current dialogue. For example, similar to the initial state of the decoder, the initial state of the LLM can be a specific initial value, which can be set or learned by the LLM during training. The translation result of the historical source language audio streams can be the translation result of all source language audio streams preceding the current source language audio stream, or it can be the translation result of some historical source language audio streams, such as the translation result of the preceding sentence or the preceding two sentences of the current source language audio stream. This application does not limit this. The electronic device concatenates the second feature representation and the second contextual information using the LLM to obtain the second concatenated feature, and then generates the target language text corresponding to the source language audio stream using methods such as MT and NMT, thereby completing the translation of the entire audio stream.

[0120] In this implementation, the translation results of the historical audio stream can help LLM understand the connection between the preceding and following audio and the translation context. The second feature representation obtained based on the first feature representation integrates the Prompt and historical ASR recognition results. The second concatenated feature obtained by splicing the second feature representation with the second context information brings more effective information to LLM, providing better context understanding and translation knowledge. This is conducive to improving the translation accuracy and quality of the source language audio stream and generating target language text that is more in line with the expression of the other end.

[0121] As can be seen from the embodiments of this application, the electronic device performs streaming segmentation on the source language audio stream to obtain continuous audio segments. Each audio segment is feature extracted by the LLM, and the extracted feature representations are input into a decoder independent of the LLM for ASR recognition. The ASR recognition results of each audio segment are displayed on the screen in real time, which helps to reduce the latency of ASR recognition output and ensure the timeliness of ASR recognition. In addition, the LLM obtains the second feature representation of the source language audio stream through the first feature representation of each audio segment and translates it into the corresponding target language text. The decoding of ASR recognition is completed by an independent decoder, so that the LLM only needs to focus on the translation task and does not need to perform ASR recognition decoding, which reduces the complexity of LLM inference and helps to reduce the power consumption of the electronic device.

[0122] Please see Figure 6 , Figure 6 This is a flowchart illustrating another speech translation method provided in an embodiment of this application. This method can be applied to electronic devices. For example, the electronic device can be a first terminal device, such as a smartphone, tablet, wearable device, or in-vehicle infotainment system. Figure 6 As shown, the method includes steps 601-606:

[0123] 601: The first terminal device responds to a first operation on the first user interface of the first terminal device and enables the voice translation function of the first terminal device; the first user interface is the voice translation function switch interface of the first terminal device;

[0124] 602: The first terminal device acquires continuous audio segments from the source language audio stream;

[0125] 603: For each audio segment in a continuous audio segment, the first terminal device extracts features from each audio segment using a large language model (LLM) to obtain the first feature representation of each audio segment;

[0126] 604: The first terminal device generates the source language text corresponding to each audio segment through a decoder based on the first feature representation of each audio segment, and displays the source language text; the decoder is an LLM-independent decoder;

[0127] 605: The first terminal device obtains the second feature representation of the source language audio stream based on the first feature representation of each audio segment;

[0128] 606: The first terminal device generates the target language text corresponding to the source language audio stream through LLM based on the second feature representation of the source language audio stream.

[0129] The specific implementation methods of steps 602-606 can be found in the following references. Figure 4 The corresponding descriptions of steps 401-405 in the middle section can achieve the same or similar beneficial effects.

[0130] In this embodiment of the application, the first terminal device has a voice translation function, such as... Figure 7 As shown, Figure 7 The first terminal device is designated as the first user interface of the first terminal device. The first terminal device can respond to a first operation on the first user interface to enable the voice translation function, thereby executing the voice translation method provided in the embodiments of this application.

[0131] For example, the first user interface has a language translation function switch button. This button (with a prompt indicating whether to turn it on) is off by default. When a user on the first terminal device triggers this button on the first user interface, the language translation function is activated on the first terminal device. For example, a user can activate the voice translation function through the first user interface when the first terminal device powers on. For instance, the first user interface can pop up automatically upon power-on, and the user can activate the voice translation function by triggering a button on the first user interface. For example, a user can also access the first user interface through the settings function of the first terminal device when a call is needed, and then activate the voice translation function by triggering a button on the first user interface.

[0132] For example, in a call scenario, the first terminal device can determine whether to pop up the first user interface based on the location of the other party's number. For instance, when the other party's number is called, if the location of the other party's number is not the same as the location of the first terminal device's user's source language, the first terminal device can pop up the first user interface, and the first terminal device's user can perform a first operation on the first user interface to enable the voice translation function of the first terminal device.

[0133] For example, in a call scenario, the first terminal device can also determine whether to pop up the first user interface based on the first audio stream from the other end. For instance, if the user on the first terminal device side answers the call and the other end inputs the first audio stream in the target language, and the first terminal device determines based on the language recognition model that the input from the other end user is not the source language of the user on this end, then the first terminal device can pop up the first user interface, and the user on the first terminal device side can perform the first operation on the first user interface to enable the voice translation function of the first terminal device.

[0134] In this implementation, the first terminal device has a real-time call voice translation function. Users can enable the voice translation function on the first terminal device through the voice translation function switch interface. This allows for the timely output of ASR recognition results for audio segments during calls and the output of voice translation results after the audio stream ends. This enhances user trust in the voice translation system and improves the user's interactive experience. When there is no need for voice translation, users can also disable the voice translation function through the voice translation function switch interface to reduce the power consumption of the first terminal device.

[0135] In the embodiments of this application, such as Figure 8 As shown, in response to a first operation on the first user interface, the first terminal device also displays a second user interface, which includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device. For example, in response to a second operation on the second user interface regarding the local language setting bar, the first terminal device sets its source language; in response to a third operation on the second user interface regarding the remote language setting bar, it sets the target language for the remote end. For instance, if the user selects Chinese in the local language setting bar via a dropdown window, the first terminal device sets Chinese as the source language. Based on this setting, when performing voice translation, the ASR recognition result of the audio segment is displayed in Chinese. If the user selects English in the remote language setting bar via a dropdown window, the first terminal device sets English as the target language. Based on this setting, when performing voice translation, the input audio stream is translated into English text, and the English text can be converted into English audio and transmitted to the second terminal device.

[0136] In this implementation, the first terminal device displays a second user interface in response to a first operation on the first user interface. The second user interface is a setting interface for the local and remote languages ​​of the voice translation function. Based on a second operation by the user in the local language setting section of the second user interface, the source language of the local device can be set. Based on a third operation by the user in the remote language setting section of the second user interface, the target language of the remote device can be set. This setting facilitates the first terminal device in subsequently performing voice translation, displaying the ASR recognition result as source language text and translating the input audio stream into target language text.

[0137] For example, in a call scenario, such as Figure 9 As shown, the source language text in step 604 is displayed on a third user interface. The third user interface also includes accept and reject buttons for the source language text. When the accept button is triggered on the third user interface, the first terminal device confirms that the source language text is correctly recognized. When the reject button is triggered on the third user interface, the first terminal device confirms that the source language text is incorrectly recognized. It then re-decodes the first feature representation using the decoder and outputs updated source language text. For example, if the duration of displaying the third user interface exceeds a preset duration, the first terminal device assumes the source language text is correctly recognized. For example, to meet the latency requirements for ASR recognition result output, the first terminal device only performs a single decoding operation for the same audio segment. If the reject button is still triggered on the updated source language text for the same audio segment displayed on the third user interface, the first terminal device determines that the decoder's recognition error rate is high and sends feedback information to the speech translation model service provider. This feedback information indicates that the decoder needs to be updated or retrained. Alternatively, the first terminal device displays a corresponding prompt on the third user interface to remind the user that the speech translation model needs to be redeployed. Meanwhile, while the call is still ongoing, the first terminal device can temporarily perform the decoding task of ASR recognition for the current audio segment through LLM to ensure the accuracy of ASR recognition and the timeliness of ASR recognition output.

[0138] In this embodiment, taking a call between a first terminal device and a second terminal device as an example, when the call between the first and second terminal devices is answered (either by a user on the first terminal device or a user on the second terminal device), if the second terminal device has enabled the voice translation function, the first terminal device will also output a first prompt tone. This first prompt tone is used to inform the user on the first terminal device that the second terminal device has enabled the voice translation function, and that the call may be delayed. For example, the first prompt tone could be: "The other party has enabled the voice translation function; the call is delayed, please wait patiently." For the second terminal device, if the first terminal device has enabled the voice translation function, the second terminal device will output the same prompt tone.

[0139] In this implementation, since the voice translation function may cause a certain delay in the call between the first terminal device and the second terminal device, when the second terminal device has the voice translation function enabled, the first terminal device will output a first prompt tone when the call is answered to inform the user on the first terminal device that there is a delay in the call and that they need to wait patiently, thereby improving the call experience.

[0140] In this embodiment, after translating the source language audio stream into the corresponding target language text, the first terminal device can convert the target language text into target language audio and transmit the target language audio to the second terminal device. The call system records the time when the first terminal device sends the target language audio. If the duration of the sent target language audio is greater than or equal to a preset duration and no response is received from the second terminal device, the first terminal device outputs a second prompt tone. The second prompt tone is used to inform the user that the second terminal device is performing a voice translation operation, so as to guide the user on the first terminal device to wait patiently. For example, the second prompt tone can be a specific ticking sound set in the call system.

[0141] In this implementation, if the duration of the target language audio emitted by the first terminal device exceeds the preset duration and the first terminal device does not receive a response from the second terminal device during the call, the first terminal device will also output a second prompt tone to inform the user that the other end is performing a voice translation operation and that the user needs to wait for the voice translation result from the other end.

[0142] Please see Figure 10 , Figure 10 This is a schematic diagram of a speech translation device provided in an embodiment of this application. This device can be applied to electronic devices or chips within electronic devices. Figure 10 As shown, the device may include an acquisition unit 1001 and a processing unit 1002; wherein:

[0143] Acquisition unit 1001 is used to acquire continuous audio segments from the source language audio stream;

[0144] The processing unit 1002 is configured to: extract features from each audio segment in a continuous audio segment using a large language model (LLM) to obtain a first feature representation for each audio segment; generate source language text corresponding to each audio segment using a decoder based on the first feature representation of each audio segment, and display the source language text; the decoder is an LLM-independent decoder; obtain a second feature representation of the source language audio stream based on the first feature representation of each audio segment; and generate target language text corresponding to the source language audio stream using an LLM based on the second feature representation of the source language audio stream.

[0145] It can be seen that, Figure 10In the illustrated device, the source language audio stream is streamed and segmented to obtain continuous audio segments. Each audio segment undergoes feature extraction by an LLM (Liquid Logic Model). The extracted feature representations are input into a decoder independent of the LLM for ASR (Automatic Recognition Scripting). The ASR recognition results for each audio segment are displayed on the screen in real time, which helps reduce the latency of ASR recognition output and ensures the timeliness of ASR recognition. Furthermore, the LLM obtains the second feature representation of the source language audio stream from the first feature representation of each audio segment and translates it into the corresponding target language text. The decoding for ASR recognition is performed by a separate decoder, allowing the LLM to focus solely on the translation task without needing to perform ASR recognition decoding. This reduces the complexity of LLM inference and helps reduce the power consumption of the electronic device.

[0146] In one possible implementation, in terms of extracting features from each audio segment using a large language model (LLM) to obtain a first feature representation of each audio segment, the processing unit 1002 is specifically used for:

[0147] For any audio segment in a continuous audio sequence, extract the audio features of that audio segment and convert the audio features into text features;

[0148] Obtain the task prompt and first context information; the prompt is used to prompt the LLM to translate the source language audio stream into the corresponding target language text in combination with the context information; if any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder; if any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of the historical audio segments of any audio segment in the source language audio stream.

[0149] The text features, Prompt, and first context information are concatenated to obtain the first concatenated feature.

[0150] By using LLM to extract features from the first splicing features, the first feature representation of any audio segment can be obtained.

[0151] In one possible implementation, regarding the generation of target language text corresponding to the source language audio stream via LLM based on the second feature representation of the source language audio stream, the processing unit 1002 is specifically used for:

[0152] Obtain the second context information; if the source language audio stream is the first source language audio stream in the current conversation, the second context information is the initial state of the LLM; if the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation.

[0153] The second feature representation is concatenated with the second contextual information to obtain the second concatenated feature.

[0154] Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through LLM.

[0155] In one possible implementation, the device is applied to a first terminal device; the processing unit 1002 is further configured to:

[0156] In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

[0157] In one possible implementation, the processing unit 1002, in response to the first operation, also causes the first terminal device to display a second user interface, the second user interface including a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device;

[0158] Processing unit 1002 is also used for:

[0159] In response to a second operation on the local language settings bar on the second user interface, set the local source language;

[0160] In response to a third action on the peer's language settings bar on the second user interface, set the target language for the peer.

[0161] In one possible implementation, the acquiring unit 1001 is further used for:

[0162] When a call with the second terminal device is answered, a first prompt tone is output; the first prompt tone is used to notify the user that the voice translation function of the second terminal device has been enabled.

[0163] In one possible implementation, the processing unit 1002 is further configured to:

[0164] Convert target language text into target language audio;

[0165] Transmit the target language audio to the second terminal device;

[0166] If the duration of the target language audio sent is greater than or equal to a preset duration and no response is received from the second terminal device, the acquisition unit 1001 is further configured to:

[0167] Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

[0168] It should be noted that, Figure 10 The implementation of each unit described can also be referenced accordingly. Figures 4 to 9The corresponding description of the illustrated embodiments. Furthermore, Figure 10 The beneficial effects of the described device can be referred to Figures 4 to 9 The corresponding descriptions of the embodiments shown will not be repeated here.

[0169] Based on the description of the method and device embodiments above, this application also provides an electronic device. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes at least a processor 1101, a memory 1102, and a communication interface 1103, which are interconnected via a bus 1104. This electronic device can be used to execute relevant steps of a speech translation method. The processor 1101 in this electronic device is used to read the computer program code stored in the memory 1102 and execute it. Figures 4 to 9 The method of any of the embodiments shown.

[0170] The memory 1102 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used to store related computer programs and data.

[0171] Processor 1101 can be one or more CPUs. When processor 1101 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0172] For example, the processor 1101 in the electronic device can be used to read one or more programs stored in the memory 1102 described above, and perform the following operations:

[0173] Get consecutive audio segments from the source language audio stream;

[0174] For each audio segment in a continuous audio segment, features are extracted from each audio segment using a large language model (LLM) to obtain the first feature representation of each audio segment;

[0175] Based on the first feature representation of each audio segment, the source language text corresponding to each audio segment is generated by the decoder and displayed; the decoder is an LLM-independent decoder.

[0176] The second feature representation of the source language audio stream is obtained based on the first feature representation of each audio segment;

[0177] Based on the second feature representation of the source language audio stream, the target language text corresponding to the source language audio stream is generated by LLM.

[0178] In one possible implementation, processor 1101 performs feature extraction on each audio segment using a large language model (LLM) to obtain a first feature representation for each audio segment, including:

[0179] For any audio segment in a continuous audio sequence, extract the audio features of that audio segment and convert the audio features into text features;

[0180] Obtain the task prompt and first context information; the prompt is used to prompt the LLM to translate the source language audio stream into the corresponding target language text in combination with the context information; if any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder; if any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of the historical audio segments of any audio segment in the source language audio stream.

[0181] The text features, Prompt, and first context information are concatenated to obtain the first concatenated feature.

[0182] By using LLM to extract features from the first splicing features, the first feature representation of any audio segment can be obtained.

[0183] In one possible implementation, processor 1101 executes a second feature representation based on the source language audio stream, and generates target language text corresponding to the source language audio stream via LLM, including:

[0184] Obtain the second context information; if the source language audio stream is the first source language audio stream in the current conversation, the second context information is the initial state of the LLM; if the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation.

[0185] The second feature representation is concatenated with the second contextual information to obtain the second concatenated feature.

[0186] Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through LLM.

[0187] In one possible implementation, the electronic device is a first terminal device, and before acquiring consecutive audio segments of the source language audio stream, the processor 1101 is also used to perform:

[0188] In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

[0189] In one possible implementation, the first terminal device, in response to the first operation, also displays a second user interface, which includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device; the processor 1101 is further configured to execute:

[0190] In response to a second operation on the local language settings bar on the second user interface, set the local source language;

[0191] In response to a third action on the peer's language settings bar on the second user interface, set the target language for the peer.

[0192] In one possible implementation, the processor 1101 is also configured to: output a first prompt tone when a call with the second terminal device is answered; the first prompt tone is used to prompt the user that the second terminal device has enabled the voice translation function.

[0193] In one possible implementation, after generating the target language text corresponding to the source language audio stream via LLM, the processor 1101 is further configured to: convert the target language text into target language audio; and transmit the target language audio to a second terminal device.

[0194] If the duration of the target language audio sent is greater than or equal to a preset duration and no response is received from the second terminal device, the processor 1101 is also configured to perform:

[0195] Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

[0196] It should be noted that the implementation of each operation can also be referenced accordingly. Figures 4 to 9 A corresponding description of the method of any of the embodiments shown.

[0197] It should be noted that, although Figure 11The illustrated electronic device only shows the processor 1101, memory 1102, communication interface 1103, and bus 1104. However, in specific implementations, those skilled in the art should understand that the electronic device may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 11 All the devices shown.

[0198] This application also provides a chip, including: a processor, configured to retrieve and run a computer program from memory, causing a device with the chip installed to perform the above-described actions. Figures 4 to 9 The method described in any of the embodiments. The chip may be a chip in an electronic device.

[0199] This application also provides a computer-readable storage medium (memory) storing a computer program, which, when run, performs the functions described above. Figures 4 to 9 The method described in any of the embodiments is described above. It is understood that the computer-readable storage medium herein may include both built-in storage media within the device and extended storage media supported by the device. The computer-readable storage medium provides storage space containing the operating system of the device. Furthermore, one or more computer programs suitable for loading and execution by the processor of the device are also stored in this storage space. It should be noted that the computer-readable storage medium herein may be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it may also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0200] This application also provides a computer program product, which includes computer program code. When the computer program code is run by an electronic device,... Figures 4 to 9 The method flow described in any one of the embodiments can be implemented.

[0201] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0202] It should be understood that the processor mentioned in the embodiments of this application can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0203] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), EPROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DRRAM).

[0204] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0205] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0206] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0210] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0211] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0212] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0213] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for speech translation, characterized in that, The method includes: Get consecutive audio segments from the source language audio stream; For each audio segment in the continuous audio segments, features are extracted from each audio segment using a large language model (LLM) to obtain the first feature representation of each audio segment. Based on the first feature representation of each audio segment, the source language text corresponding to each audio segment is generated by the decoder and the source language text is displayed; the decoder is a decoder independent of the LLM. The second feature representation of the source language audio stream is obtained based on the first feature representation of each audio segment; Based on the second feature representation of the source language audio stream, the target language text corresponding to the source language audio stream is generated through the LLM.

2. The method according to claim 1, characterized in that, The step of extracting features from each audio segment using a large language model (LLM) to obtain a first feature representation for each audio segment includes: For any audio segment in the continuous audio segments, extract the audio features of the audio segment and convert the audio features into text features; The system acquires a task prompt and first context information. The prompt is used to instruct the LLM to translate the source language audio stream into the corresponding target language text, taking into account the context information. If any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder. If any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of historical audio segments of any audio segment in the source language audio stream. The text features, the Prompt, and the first context information are concatenated to obtain the first concatenated feature; The first feature representation of any audio segment is obtained by extracting features from the first splicing feature using the LLM.

3. The method according to claim 1 or 2, characterized in that, The generation of target language text corresponding to the source language audio stream through the LLM, based on the second feature representation of the source language audio stream, includes: Obtain second context information; if the source language audio stream is the first source language audio stream in the current conversation, the second context information is the initial state of the LLM; if the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation. The second feature representation is concatenated with the second context information to obtain the second concatenated feature; Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through the LLM.

4. The method according to any one of claims 1-3, characterized in that, The method is applied to a first terminal device; before acquiring continuous audio segments of the source language audio stream, the method further includes: In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

5. The method according to claim 4, characterized in that, In response to the first operation, the first terminal device also displays a second user interface, which includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device. The method further includes: In response to a second operation on the second user interface targeting the local language settings bar, the local source language is set; In response to a third operation on the second user interface targeting the peer's language settings bar, the target language of the peer is set.

6. The method according to claim 4 or 5, characterized in that, The method further includes: When a call with the second terminal device is answered, a first prompt tone is output; the first prompt tone is used to notify the user that the second terminal device has enabled the voice translation function.

7. The method according to claim 6, characterized in that, After generating the target language text corresponding to the source language audio stream using the LLM, the method further includes: Convert the target language text into target language audio; Transmit the target language audio to the second terminal device; If the duration of the target language audio transmitted is greater than or equal to a preset duration and no response is received from the second terminal device, the method further includes: Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

8. A speech translation device, characterized in that, The device includes an acquisition unit and a processing unit; wherein: The acquisition unit is used to acquire continuous audio segments of the source language audio stream; The processing unit is configured to: extract features from each audio segment in the continuous audio segments using a Large Language Model (LLM) to obtain a first feature representation of each audio segment; generate source language text corresponding to each audio segment using a decoder based on the first feature representation of each audio segment, and display the source language text; the decoder is an LLM-independent decoder; obtain a second feature representation of the source language audio stream based on the first feature representation of each audio segment; and generate target language text corresponding to the source language audio stream using the LLM based on the second feature representation of the source language audio stream.

9. The apparatus according to claim 8, characterized in that, In extracting features from each audio segment using a large language model (LLM) to obtain a first feature representation of each audio segment, the processing unit is specifically used for: For any audio segment in the continuous audio segments, extract the audio features of the audio segment and convert the audio features into text features; The system acquires a task prompt and first context information. The prompt is used to instruct the LLM to translate the source language audio stream into the corresponding target language text, taking into account the context information. If any audio segment is the first audio segment in a series of audio segments, the first context information is the initial state of the decoder. If any audio segment is not the first audio segment in a series of audio segments, the first context information is the decoder's recognition result of historical audio segments of any audio segment in the source language audio stream. The text features, the Prompt, and the first context information are concatenated to obtain the first concatenated feature; The first feature representation of any audio segment is obtained by extracting features from the first splicing feature using the LLM.

10. The apparatus according to claim 8 or 9, characterized in that, In generating target language text corresponding to the source language audio stream using the LLM based on the second feature representation of the source language audio stream, the processing unit is specifically used for: Obtain second context information; if the source language audio stream is the first source language audio stream of the user in the current conversation, the second context information is the initial state of the LLM; If the source language audio stream is not the first source language audio stream in the current conversation, the second context information is the translation result of the LLM of the historical source language audio streams in the current conversation; The second feature representation is concatenated with the second context information to obtain the second concatenated feature; Based on the second splicing feature, the target language text corresponding to the source language audio stream is generated through the LLM.

11. The apparatus according to any one of claims 8-10, characterized in that, The device is applied to a first terminal device; the processing unit is further configured to: In response to a first operation on the first user interface of the first terminal device, the voice translation function of the first terminal device is enabled; the first user interface is the on / off interface for the voice translation function of the first terminal device.

12. The apparatus according to claim 11, characterized in that, In response to the first operation, the processing unit also causes the first terminal device to display a second user interface, which includes a local language setting bar and a remote language setting bar for the voice translation function of the first terminal device. The processing unit is also used for: In response to a second operation on the second user interface targeting the local language settings bar, the local source language is set; In response to a third operation on the second user interface targeting the peer's language settings bar, the target language of the peer is set.

13. The apparatus according to claim 11 or 12, characterized in that, The acquisition unit is also used for: When a call with the second terminal device is answered, a first prompt tone is output; the first prompt tone is used to notify the user that the second terminal device has enabled the voice translation function.

14. The apparatus according to claim 13, characterized in that, The processing unit is also used for: Convert the target language text into target language audio; Transmit the target language audio to the second terminal device; If the duration of the target language audio transmitted is greater than or equal to a preset duration and no response is received from the second terminal device, the acquisition unit is further configured to: Output a second prompt tone; the second prompt tone is used to notify the user that the second terminal device is performing a voice translation operation.

15. An electronic device, characterized in that, It includes a processor, a memory, a communication interface, and one or more programs; the processor is coupled to the memory, the one or more programs are stored in the memory, and are configured to cooperate with the communication interface to implement the method as described in any one of claims 1-7 when executed by the processor.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for execution by an electronic device, which, when executed, causes the electronic device to perform the method of any one of claims 1-7.

17. A computer program product, characterized in that, When the computer program product is run by an electronic device, the electronic device performs the method as described in any one of claims 1-7.

Images