Translation method, video conference method, and electronic device

Abstract

This application provides a translation method, a video conferencing method, and an electronic device for the field of simultaneous interpretation. The method includes: acquiring the current word to be translated from the original language text; generating an embedding vector corresponding to the current word using a first object; transmitting the embedding vector to an encoder; acquiring a first encoder key-value cache corresponding to the embedding vector generated by the encoder; processing the first encoder key-value cache using a second object to generate a second encoder key-value cache; processing the second encoder key-value cache, the previous word corresponding to the current word, and the historical decoder key-value cache using a decoder to generate a target decoder key-value cache and a prediction matrix; and processing the prediction matrix using the first object to obtain the translation result of the current word. This application can reduce the memory footprint of the translation model, thereby embedding the translation model into the conferencing terminal without the need for server-side translation, effectively reducing computational costs, mitigating the impact of network latency, and improving the stability of video conferencing.

Description

Technical Field

[0001] This application relates to the field of simultaneous interpretation technology, and more specifically, to a translation method, a video conferencing method, and an electronic device. Background Technology

[0002] With the development of internet technology, users are no longer satisfied with just text-based communication and social interaction; their demand for real-time voice and video is growing stronger. Therefore, information exchange through audio and video conferencing is becoming increasingly common. Users can join different audio and video conferences according to their needs to exchange information within those conferences. In practice, it has been found that participants in audio and video conferences come from different countries or regions, and because participants from different countries or regions are familiar with different languages, current technologies require participants to consult large amounts of translation materials in real time to understand the content output by other participants, resulting in low efficiency and convenience. To solve this problem, a common approach is to set up a high-computing-power server and connect all conference terminals to this server to perform translation operations. This method has high translation costs and requires the simultaneous transmission of original audio and translation results, increasing bandwidth consumption and making data transmission susceptible to network latency, thus reducing the effectiveness of the meeting. Summary of the Invention

[0003] This application addresses the shortcomings of existing methods by proposing a translation method, a video conferencing method, and an electronic device that can solve the problems of high translation costs and susceptibility to network latency, which reduce the effectiveness of meetings.

[0004] In a first aspect, embodiments of this application provide a translation method for a conference terminal. The conference terminal includes a translation module equipped with a translation model. The translation model includes an encoder, a decoder, and a first object and a second object. The first and second objects are obtained by splitting data from the encoder and decoder. The method includes:

[0005] Determine that the decoder is not running for the first time, obtain the current word to be translated in the original language text, generate the embedding vector corresponding to the current word using the first object, transmit the embedding vector to the encoder, and obtain the first encoder kv cache generated by the encoder corresponding to the embedding vector;

[0006] The second object is used to process the first encoder kv cache to generate a second encoder kv cache, the second encoder kv cache including context information;

[0007] The decoder processes the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache to generate the target decoder kv cache and the prediction matrix. The first object is used to process the prediction matrix to obtain the translation result of the current word. The historical decoder kv cache is stored in the decoder and is the target decoder kv cache corresponding to the previous word.

[0008] In one possible implementation, the generation of the first object and the second object includes:

[0009] The translation model is deployed to the conference terminal using a preset deep learning model deployment tool. During the deployment process, the data of the encoder and the decoder are split to generate a first object and a second object, wherein the first object and the second object contain different data.

[0010] In one possible implementation, the data of the encoder and the decoder are split to generate a first object, including:

[0011] Extract the embedding portion from the encoder and the decoder, as well as the language model header of the decoder, and save the embedding portion and the language model header as the first object, which is a model file.

[0012] In one possible implementation, the generation of the second object includes:

[0013] Extract the portion of the decoder used to generate the encoder kv cache, and save the portion used to generate the encoder kv cache as a second object, which is a model file.

[0014] In one possible implementation, the method includes:

[0015] If it is determined that the decoder is running for the first time, an initial decoder kv cache is generated using the second object, and the initial decoder kv cache is sent to the decoder so that the decoder can perform decoding operations using the initial decoder kv cache.

[0016] In one possible implementation, the step of generating the target decoder kv cache and prediction matrix by processing the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache through the decoder includes:

[0017] The second encoder kv cache, the embedding vector of the previous word, and the history decoder cache are input into the decoder, wherein the embedding vector of the previous word is generated by the first object;

[0018] Obtain the target decoder kv buffer and prediction matrix output by the decoder.

[0019] In one possible implementation, processing the prediction matrix using the first object to obtain the translation result of the current word includes:

[0020] The prediction matrix is ​​processed using the first object to obtain the normalized probability distribution of the translation result, and the translation result of the current word is determined based on the normalized probability distribution.

[0021] In one possible implementation, the method includes:

[0022] Detect whether the word generated by the decoder is a preset word;

[0023] If so, then the translation process is considered complete, and the invocation of the first object is stopped.

[0024] Secondly, embodiments of this application provide a video conferencing method for a conferencing terminal equipped with a translation module, wherein the translation module performs text translation using the method described above. The video conferencing method includes:

[0025] In response to meeting instructions, obtain simultaneous interpretation information, which includes the original language and the target language;

[0026] Receive the audio stream to be translated and obtain the original language text of the audio stream;

[0027] The original language text is translated using the translation module to obtain the target language text corresponding to the original language text.

[0028] Audio is played based on the target language text.

[0029] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described above.

[0030] The beneficial technical effects of the technical solutions provided in this application include:

[0031] The translation method provided in this application is used in a conference terminal. The conference terminal includes a translation module equipped with a translation model. The translation model includes an encoder, a decoder, and a first object and a second object. The first and second objects are obtained by splitting the data in the encoder and decoder. When the decoder is not running for the first time, the current word to be translated in the original language text is obtained. The first object is used to generate an embedding vector corresponding to the current word, and the embedding vector is transmitted to the encoder to obtain the first encoder kv cache corresponding to the embedding vector generated by the encoder. The second object is used to process the first encoder kv cache to generate a second encoder kv cache, which includes context information. The decoder processes the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache to generate a target decoder kv cache and a prediction matrix. The first object is used to process the prediction matrix to obtain the translation result of the current word. The historical decoder kv cache is stored in the decoder and is the target decoder kv cache corresponding to the previous word. This application embodiment reduces the translation model by splitting it into a first object and a second object, thereby reducing the memory occupation of the translation model and embedding the translation model into the conference terminal. Translation does not require the use of a server, effectively reducing computing power costs, reducing the impact of network latency, and improving the stability and meeting effect of video conferencing.

[0032] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0033] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0034] Figure 1 A flowchart illustrating the translation method provided in this application embodiment;

[0035] Figure 2 A flowchart of a video conferencing method provided in an embodiment of this application;

[0036] Figure 3 A flowchart illustrating the operation of each module in the video conferencing method provided in the embodiments of this application;

[0037] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0039] Those skilled in the art will understand that, unless specifically stated otherwise, the terms "described" and "the" as used herein may also include plural forms. It should be further understood that the term "comprising" as used in this application's specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by this art. It should be understood that when we say an element is "connected" or "coupled" to another element, the element may be directly connected or coupled to the other element, or it may mean that the element and the other element are connected through an intermediate element. Furthermore, "connected" or "coupled" as used herein may include wireless connection or wireless coupling. The term "and / or" as used herein refers to at least one of the items defined by the term; for example, "A and / or B" may be implemented as "A," or as "B," or as "A and B."

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] This application provides a translation method that can be used in a conference terminal, such as a mobile phone, laptop, tablet, or other terminal capable of loading a translation model.

[0042] Optionally, the conference terminal includes a translation module equipped with a translation model, through which text translation is achieved.

[0043] Optionally, the conference terminal may also include a speech-to-text module and a text-to-speech module. A translation module is connected to both the speech-to-text module and the text-to-speech module. The speech-to-text module converts the audio stream into text. The translation module receives the text, translates it into a specified language, and transmits the translated text to the text-to-speech module. The text-to-speech module uses the received text to play audio.

[0044] Optionally, the translation model includes an encoder, a decoder, and a first object and a second object, which are obtained by splitting the data from the encoder and decoder. The first object and the second object perform different functions when translating text.

[0045] Optionally, the generation of the first object and the second object includes: deploying the translation model to the conference terminal using a preset deep learning model deployment tool, and splitting the data of the encoder and the decoder during the deployment process to generate the first object and the second object. The first object and the second object contain different data. By generating the first object and the second object when deploying the translation model, the memory required for the deployment of the translation model is reduced, thereby allowing the translation model to be deployed to a terminal with a small memory space.

[0046] In one embodiment, the translation model can use the Transformer architecture, and the deployment tool can be Optimum (a toolkit for optimizing and deploying deep learning models, suitable for deploying large pre-trained models to resource-constrained environments). Before deployment, the size of the translation model can be 2.5GB, which includes an encoder and decoder, and the execution time of 100 tokens is 8s. Using Optimum, the encoder and decoder are processed to generate a first object and a second object. After the processed encoder, decoder, first object, and second object are deployed to the conference terminal, the memory consumption of the translation model is reduced to 1.0GB, and the execution time of 100 tokens is also reduced to 2s, which can effectively reduce memory consumption and improve inference efficiency.

[0047] Optionally, the embedding parts in the encoder and decoder and the language model head of the decoder use the same matrix. To reduce redundant loading, the encoder and decoder data are split to generate a first object, including: extracting the embedding parts from the encoder and decoder and the language model head of the decoder, and saving the embedding parts and the language model head (lm-head) as the first object. The first object is a model file. Saving it as a separate model file avoids redundant loading when loading the encoder and decoder, reducing memory usage. The embedding parts convert words, characters, or sentences in the text into embedding vectors, which capture the semantic relationships between words. The lm-head (language model head) is the last layer of the Transformer model, responsible for converting the hidden states output by the decoder into a probability distribution in the vocabulary. Simply put, the role of the lm-head is to predict the probability of the translation result corresponding to each word, thereby generating sentences that conform to grammar and semantics.

[0048] In one embodiment, the portion of the decoder and encoder used for embedding is extracted from the decoder's lm-head and saved separately as a model file, which can be named embed_and_lm_head.onnx.

[0049] Optionally, the generation of the second object includes: extracting the portion of the decoder used to generate the encoder key-value cache, and saving the portion used to generate the encoder key-value cache as the second object, which is a model file. The deployed decoder is a copy of the decoder that can store historical decoder key-value caches.

[0050] In one embodiment, the deployment tool can be Optimum. When using the key-value pairs (KV) cache of the translation model exported by Optimum, two copies of the decoder need to be loaded: one for generating a "past" version (in Transformer architecture models, "past" refers to historical information generated during previous decoding processes, including key-value pairs. This historical information is used to accelerate subsequent decoding processes because it contains already processed contextual information, allowing the model to more efficiently utilize previous computational results when generating new words). In traditional Transformer architecture translation models, the decoder needs to consider both the current input and the historical information from previous times (i.e., "past" information) when generating the output at each time step. To achieve this, two copies of the decoder are typically loaded: one for generating the KV cache with historical information (the "past" version), and another for processing the current input (the "non-past" version). This approach increases memory usage. This application avoids this redundant loading through optimization, allowing only the "past" version of the decoder to be loaded. This means the decoder utilizes the previously generated key-value cache during each inference, eliminating the need to load the "non-past" version. This reduces memory usage and improves inference efficiency. Furthermore, the portion of the decoder used to generate the encoder key-value cache is extracted and saved as `cache_initializer.onnx`. During inference, only `cache_initializer.onnx` and the "past" version of the decoder are loaded.

[0051] Optionally, such as Figure 1 As shown, the translation method of this application includes:

[0052] S101: Determine that the decoder is not running for the first time, obtain the current word to be translated in the original language text, generate the embedding vector corresponding to the current word using the first object, transmit the embedding vector to the encoder, and obtain the first encoder kv buffer generated by the encoder corresponding to the embedding vector.

[0053] Optionally, the translation method of this application is used for video conferencing. When used for video conferencing, the encoder and decoder are initialized at each video conference. Furthermore, when the decoder is first run (i.e., initialized), an initial decoder kv buffer is generated using a second object, and the initial decoder kv buffer is sent to the decoder so that the decoder can perform decoding operations using the initial decoder kv buffer.

[0054] In one embodiment, the second object is cache_initializer.onnx. During decoder initialization, cache_initializer.onnx generates an initial decoder key-value buffer of length 0. The decoder stores this initial decoder key-value buffer and uses it for subsequent decoding operations.

[0055] Optionally, during the translation operation, a word segmenter is used to split the original language text into multiple translatable words. Based on the word order in the text and information about currently translated words, the word to be translated (i.e., the current word) is determined and transmitted to the first object. Specifically, the word segmenter can be a SentencePiece, and the current word can be represented as input_ids. The first object embeds the input_ids (i.e., converts the input_ids into an embedding vector, which can be used for mathematical operations). In this application, the embedding operation is performed by the first object (i.e., embed_and_lm_head.onnx). The encoder receives the embedding vector transmitted by the first object, processes the embedding vector, and obtains the first encoder kv buffer.

[0056] Optionally, during a video conference, when the first object initially embeds the input_ids, a special character can be embedded in the embedding vector to indicate the start of decoding. The decoder determines the start of decoding upon receiving this special character.

[0057] S102: Use the second object to process the first encoder kv cache to generate the second encoder kv cache.

[0058] Optionally, the second encoder kv cache includes contextual information. The second object receives the first encoder kv cache transmitted by the encoder, and generates a second encoder kv cache based on the first encoder kv cache for use by the decoder, thereby enabling the decoder to utilize this contextual information more efficiently when generating the translation result of the current word.

[0059] S103: The target decoder kv cache and prediction matrix are generated by processing the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache through the decoder. The prediction matrix is ​​then processed using the first object to obtain the translation result of the current word.

[0060] Optionally, the historical decoder key-value (kv) cache is stored in the decoder and is the target decoder kv cache corresponding to the previous word. The decoder performs decoding operations iteratively based on the historical decoder kv cache, thereby realizing the translation of the original language text.

[0061] Optionally, the target decoder kv cache and prediction matrix are generated by processing the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache through the decoder. This includes: inputting the second encoder kv cache, the embedding vector of the previous word, and the historical decoder cache into the decoder, where the embedding vector of the previous word is generated by the first object; and obtaining the target decoder kv cache and prediction matrix output by the decoder. Specifically, when decoding the current word, the first object generates the embedding vector of the previous word and transmits this embedding vector to the decoder to assist in decoding the current word. The prediction matrix can be a logits matrix.

[0062] In one embodiment, during each decoding iteration, the decoder is repeatedly invoked. The decoder receives the second encoder kv buffer transmitted from the second object, and inputs the word embeddings generated in the previous round (at the start of each decoding round, the decoder uses the embedding vector of the word generated in the previous round as input. This word is first converted into an embedding vector by the first object. This embedding vector represents the numerical representation of the word and captures its semantic information), the historical decoder kv buffer (the target decoder kv buffer corresponding to the previous word), and the second encoder kv buffer into the decoder. The decoder outputs a new decoder kv buffer (this new decoder kv buffer refers to intermediate state information stored to accelerate subsequent decoding steps) and a prediction matrix. The translation result corresponding to the current word is obtained using this prediction matrix.

[0063] Optionally, processing the prediction matrix using the first object to obtain the translation result of the current word includes: obtaining a normalized probability distribution of the translation result using the first object processing prediction matrix, and determining the translation result of the current word based on the normalized probability distribution. The normalized probability distribution includes the matching probability of each word in the word item corresponding to the current word with the current word, and the word with the highest matching probability is determined as the translation result of the current word.

[0064] Optionally, after obtaining the translation result of the current word, if there is still untranslated content in the original language text (i.e., there is a next word that needs to be translated), the embedding vector of the translation result is obtained using the first object. This embedding vector is then input into the decoder to assist in translating the next word. The iterative translation of the word segmentation results of the original language text is achieved in this way.

[0065] Optionally, the translation method of this application further includes: detecting whether the word generated by the decoder is a preset word; if so, determining that the translation process has ended and stopping the invocation of the first object.

[0066] In one embodiment, the preset word can be an end marker word, such as eos. The decoder outputs this end marker word after confirming the sentence is completely generated, indicating the end of the translation process. The process then stops calling the first object for further processing.

[0067] The translation method provided in this application is used in a conference terminal. The conference terminal includes a translation module equipped with a translation model. The translation model includes an encoder, a decoder, and a first object and a second object. The first and second objects are obtained by splitting the data in the encoder and decoder. When the decoder is not running for the first time, the current word to be translated in the original language text is obtained. The first object is used to generate an embedding vector corresponding to the current word, and the embedding vector is transmitted to the encoder to obtain the first encoder kv cache corresponding to the embedding vector generated by the encoder. The second object is used to process the first encoder kv cache to generate a second encoder kv cache, which includes context information. The decoder processes the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache to generate a target decoder kv cache and a prediction matrix. The first object is used to process the prediction matrix to obtain the translation result of the current word. The historical decoder kv cache is stored in the decoder and is the target decoder kv cache corresponding to the previous word. This application embodiment reduces the translation model by splitting the first and second objects, thereby reducing the memory occupation of the translation model and embedding the translation model into the conference terminal. Translation does not require the use of a server, effectively reducing computing power costs, reducing the impact of network latency, and improving the stability and meeting effect of video conferencing.

[0068] Based on the same inventive concept, embodiments of this application provide a video conferencing method, such as... Figure 2 , Figure 3 As shown, this method is used in a conference terminal equipped with a translation module, which performs text translation using the method described in the above embodiments.

[0069] Optionally, video conferencing methods include:

[0070] S201: Responding to meeting instructions, obtain simultaneous interpretation information.

[0071] Optionally, the simultaneous interpretation information includes the original language and the target language. Users of the conference terminal input conference instructions, join the video conference according to these instructions, and obtain the original and target languages ​​transmitted simultaneously.

[0072] S202: Receive the audio stream to be translated and obtain the original language text of the audio stream.

[0073] Optionally, the audio stream is an audio stream sent to the user by other participants in the video conference through their corresponding conference terminals. Upon receiving the audio stream, the user's conference terminal can cache the audio stream in the video conference-related audio stream queue.

[0074] Optionally, the conference terminal may include a speech-to-text module that extracts segments of the audio stream one by one from the audio stream queue and converts the audio stream into original language text.

[0075] S203: Use the translation module to translate the original language text to obtain the target language text corresponding to the original language text.

[0076] S204: Play audio based on the target language text.

[0077] Optionally, the conference terminal also includes a text-to-speech module, which can convert the target language text into PCM audio, play the PCM audio, and display the original language text and the target language text as bilingual subtitles on the screen of the conference terminal while playing the PCM audio.

[0078] Optionally, for easier review later, the original language text, target language text, and username can be written into the meeting minutes file.

[0079] Based on the same inventive concept, embodiments of this application provide an electronic device, such as... Figure 4 As shown, Figure 4 The illustrated electronic device 2000 includes a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are communicatively connected, for example, via a bus 2002.

[0080] Processor 2001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a FPGA (Field-Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0081] Bus 2002 may include a pathway for transmitting information between the aforementioned components. Bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 2002 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus.

[0082] The memory 2003 may be ROM (Read-Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0083] Optionally, the electronic device 2000 may also include a communication unit 2004. The communication unit 2004 can be used for receiving and transmitting signals. The communication unit 2004 allows the electronic device 2000 to communicate wirelessly or wiredly with other devices to exchange data. It should be noted that in practical applications, the communication unit 2004 is not limited to one.

[0084] Optionally, the electronic device 2000 may further include an input unit 2005. The input unit 2005 can be used to receive input numbers, characters, images, and / or sound information, or to generate key signal inputs related to user settings and function control of the electronic device 2000. The input unit 2005 may include, but is not limited to, one or more of the following: a touchscreen, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, a camera, a microphone, etc.

[0085] Optionally, the electronic device 2000 may also include an output unit 2006. The output unit 2006 can be used to output or display information processed by the processor 2001. The output unit 2006 may include, but is not limited to, one or more of a display device, a speaker, a vibration device, etc.

[0086] Although the electronic device 2000 with various devices is shown in the figure, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0087] Optionally, the memory 2003 is used to store a computer program for executing the scheme of this application, and its execution is controlled by the processor 2001. The processor 2001 is used to execute the computer program stored in the memory 2003 to implement the steps of any method provided in the embodiments of this application.

[0088] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by an electronic device / processor, implements the steps of any method provided in this application or the steps of various optional implementations of the method provided in this application.

[0089] Those skilled in the art will understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in related technologies that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0090] In the description of this application, the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate directions or positional relationships based on the exemplary directions or positional relationships shown in the accompanying drawings. They are used to facilitate the description or simplification of the embodiments of this application and are not intended to indicate or imply that the device or component referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0091] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0092] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0093] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0094] The above description is only a partial implementation of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A translation method, characterized in that, For use in a conference terminal, the conference terminal includes a translation module equipped with a translation model, the translation model including an encoder, a decoder, and a first object and a second object, the first object and the second object being obtained by splitting data from the encoder and decoder, the method including: Determine that the decoder is not running for the first time, obtain the current word to be translated in the original language text, generate the embedding vector corresponding to the current word using the first object, transmit the embedding vector to the encoder, and obtain the first encoder kv cache generated by the encoder corresponding to the embedding vector; The second object is used to process the first encoder kv cache to generate a second encoder kv cache, the second encoder kv cache including context information; The decoder processes the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache to generate the target decoder kv cache and the prediction matrix. The first object is used to process the prediction matrix to obtain the translation result of the current word. The historical decoder kv cache is stored in the decoder and is the target decoder kv cache corresponding to the previous word. The prediction matrix is ​​a logits matrix. The data from the encoder and decoder are split to generate a first object and a second object, including: Extract the embedding portion from the encoder and the decoder, as well as the language model header of the decoder, and save the embedding portion and the language model header as the first object, where the first object is a model file; Extract the portion of the decoder used to generate the encoder kv cache, and save the portion used to generate the encoder kv cache as a second object, which is a model file.

2. The translation method according to claim 1, characterized in that, The generation of the first object and the second object includes: The translation model is deployed to the conference terminal using a preset deep learning model deployment tool. During the deployment process, the data of the encoder and the decoder are split to generate a first object and a second object, wherein the first object and the second object contain different data.

3. The translation method according to claim 1, characterized in that, The method includes: If it is determined that the decoder is running for the first time, an initial decoder kv cache is generated using the second object, and the initial decoder kv cache is sent to the decoder so that the decoder can perform decoding operations using the initial decoder kv cache.

4. The translation method according to claim 1, characterized in that, The step of generating the target decoder kv cache and prediction matrix by processing the second encoder kv cache, the previous word corresponding to the current word, and the historical decoder kv cache through the decoder includes: The second encoder kv cache, the embedding vector of the previous word, and the history decoder cache are input into the decoder, wherein the embedding vector of the previous word is generated by the first object; Obtain the target decoder kv buffer and prediction matrix output by the decoder.

5. The translation method according to claim 1, characterized in that, The step of processing the prediction matrix using the first object to obtain the translation result of the current word includes: The prediction matrix is ​​processed using the first object to obtain the normalized probability distribution of the translation result, and the translation result of the current word is determined based on the normalized probability distribution.

6. The translation method according to claim 1, characterized in that, The method includes: Detect whether the word generated by the decoder is a preset word; If so, then the translation process is considered complete, and the invocation of the first object is stopped.

7. A video conferencing method, characterized in that, For a conference terminal equipped with a translation module, wherein the translation module performs text translation using the method described in any one of claims 1-6, the video conferencing method includes: In response to meeting instructions, obtain simultaneous interpretation information, which includes the original language and the target language; Receive the audio stream to be translated and obtain the original language text of the audio stream; The original language text is translated using the translation module to obtain the target language text corresponding to the original language text. Audio is played based on the target language text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as described in any one of claims 1-6.

Images