A multilingual speech translation model construction method and translation method

Abstract

The present invention relates to the technical field of intelligent translation, and discloses a multilingual speech translation model construction method and a translation method. The model construction method comprises: using target language text data to pre-train and fine-tune a preset speech translation model to obtain a language pre-training model of the target language, wherein the language pre-training model comprises a first encoder and a first decoder; using a preset multilingual speech recognition model as a teacher model to perform knowledge distillation to obtain a distillation model, wherein the distillation model comprises a second encoder and a second decoder; constructing a target speech translation model based on the second encoder and the first decoder; and using pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model. When the present invention trains the target speech translation model, only a small amount of data alignment is required for training to converge the model, thereby reducing the data required for training, and the multilingual speech translation model obtained by training is small in scale and suitable for the field of vehicle computers.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent translation, and in particular to a multilingual speech translation model construction method and a translation method. Background Art

[0002] As car companies begin to go global, there is a need for smart car control interaction in more languages. As the first step in the field of smart car control interaction, improving the accuracy of recognition and translation is particularly important for the interactive experience. How to support more languages ​​while using less resources and achieving higher translation accuracy is a problem that needs to be solved urgently.

[0003] Conventional solutions require separate modeling of the mapping from each source language to the target language. These methods not only consume a lot of resources, but also increase the complexity of user use. At the same time, for low-resource languages, there will be insufficient translation accuracy. The academic community has proposed the research field of automatic speech translation (AST), which translates multiple speech into a single speech text. However, the solutions of the academic community are difficult to implement at the industrial level, mainly because the models are too large and the training costs a lot of resources. Summary of the invention

[0004] In view of this, the present invention provides a multilingual speech translation model construction method and a translation method to solve the technical problems of existing multilingual translation methods, such as large models and high training resource consumption.

[0005] In a first aspect, the present invention provides a method for constructing a multilingual speech translation model, comprising: using target language text data to pre-train and fine-tune a preset speech translation model to obtain a language pre-training model of the target language, wherein the language pre-training model comprises a first encoder and a first decoder; using a preset multilingual speech recognition model as a teacher model to perform knowledge distillation to obtain a distillation model, wherein the distillation model comprises a second encoder and a second decoder; constructing a target speech translation model based on the second encoder and the first decoder; and using pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model, wherein the audio data comprises preset language audio and corresponding original language text and target language text.

[0006] A method for constructing a multilingual speech translation model in an embodiment of the present invention pre-trains and fine-tunes a preset speech translation model, performs knowledge distillation on a preset multilingual speech recognition model as a teacher model, and constructs a target speech translation model based on the obtained second encoder and first decoder, respectively. The second encoder and the first decoder are models whose parameters have been preliminarily trained, and the model parameters are small, so the target speech translation model can be controlled to an appropriate size, thereby reducing the scale of the multilingual speech translation model obtained after training, making it suitable for deployment on a resource-constrained vehicle system, and when training the target speech translation model, only less data alignment is required for training to make the model converge, thereby reducing the data required for training.

[0007] Optionally, constructing the target speech translation model based on the second encoder and the first decoder includes: using the first decoder as a translation decoder, and connecting the translation decoder to an output end of the second encoder; adding a fully connected layer as an audio recognition layer, and connecting the audio recognition layer to the output end of the second encoder to obtain the target speech translation model.

[0008] A target speech translation model is constructed based on the first decoder, the second encoder and the fully connected layer, so that it can perform automatic speech translation and automatic speech recognition simultaneously.

[0009] Optionally, the using of pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model includes: inputting the pre-collected audio data into the target speech translation model; performing automatic speech translation on the input audio data based on the second encoder and the translation decoder to obtain a first learning loss; performing automatic speech recognition on the input audio data based on the second encoder and the audio recognition layer to obtain a second learning loss; and adjusting parameters of the target speech translation model by combining the first learning loss and the second learning loss to obtain a multilingual speech translation model.

[0010] By combining the first learning loss and the second learning loss to adjust the parameters of the target speech translation model, the reasoning speed is accelerated, the translation effect is better, and it is more suitable for the field of vehicle computers.

[0011] Optionally, the preset multilingual speech recognition model is a whisper model; correspondingly, the preset multilingual speech recognition model is used as a teacher model for knowledge distillation to obtain a distilled model, including: using the whisper model as the teacher model, the first three layers and the last three layers initialized with the encoder of the whisper model as the encoder of the student model, and the student model and the teacher model share a decoder for knowledge distillation to obtain a distilled model.

[0012] The whisper model is used for knowledge distillation. The lightweight encoder after distillation can reduce the consumption of computing resources while maintaining high recognition accuracy.

[0013] In a second aspect, the present invention provides a multilingual speech translation method, comprising: inputting original speech into a multilingual speech translation model such as the first aspect or any corresponding embodiment thereof for translation to obtain a target language text.

[0014] In a third aspect, the present invention provides a device for constructing a multilingual speech translation model, comprising: a pre-training module, used to pre-train and fine-tune a preset speech translation model using target language text data to obtain a language pre-training model of the target language, wherein the language pre-training model comprises a first encoder and a first decoder; a distillation module, used to perform knowledge distillation using a preset multilingual speech recognition model as a teacher model to obtain a distillation model, wherein the distillation model comprises a second encoder and a second decoder; a target model construction module, used to construct a target speech translation model based on the second encoder and the first decoder; a model training module, used to train the target speech translation model using pre-collected audio data to obtain a multilingual speech translation model, wherein the audio data comprises preset language audio and corresponding original language text and target language text.

[0015] In a fourth aspect, the present invention provides a multilingual speech translation device, comprising: an input module for inputting original speech into a multilingual speech translation model such as the first aspect or any corresponding embodiment thereof for translation to obtain a target language text.

[0016] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multilingual speech translation model construction method as described in any one of claims 1 to 4 or the multilingual speech translation method as described in claim 5 by executing the computer instructions.

[0017] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute a multilingual speech translation model construction method as in the first aspect or any corresponding embodiment thereof, or a multilingual speech translation method as in the second aspect.

[0018] In a seventh aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the multilingual speech translation model construction method as in the first aspect or any corresponding embodiment thereof, or the multilingual speech translation method as in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a method for constructing a multilingual speech translation model according to an embodiment of the present invention;

[0021] Figure 2 is a flowchart of another method for constructing a multilingual speech translation model according to an embodiment of the present invention;

[0022] Figure 3 It is a structural block diagram of a multilingual speech translation model construction device according to an embodiment of the present invention;

[0023] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] A multilingual speech translation model construction method and a multilingual speech translation method provided by an embodiment of the present invention are applicable to the field of intelligent speech translation, and in particular to intelligent speech translation in the field of intelligent vehicle control interaction.

[0026] At present, multilingual speech translation consumes a lot of resources and increases the complexity of user use. At the same time, for low-resource languages, there may be insufficient translation accuracy. The embodiments of the present invention provide a multilingual speech translation model construction method and a multilingual speech translation method. While reducing resources, it optimizes the academic solution of automatic speech translation AST, improves the reasoning speed, and improves the translation accuracy in the vehicle computer field.

[0027] According to an embodiment of the present invention, an embodiment of a method for constructing a multilingual speech translation model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] In this embodiment, a multilingual speech translation model construction method is provided, which can be used in mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 is a flowchart of a method for constructing a multilingual speech translation model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0029] Step S101, pre-training and fine-tuning a preset speech translation model using target language text data to obtain a language pre-training model of the target language, wherein the language pre-training model includes a first encoder and a first decoder.

[0030] Specifically, the target language is the translated language, and text data of the target language is collected in advance. The text data of the target language includes general text data and vehicle-related text data.

[0031] The preset speech translation model is constructed based on the BART (Bidirectional and Auto-Regressive Transformers) framework, which is a Transformer neural network model with an encoder-decoder structure.

[0032] Step S102, performing knowledge distillation on the preset multilingual speech recognition model as a teacher model to obtain a distilled model, where the distilled model includes a second encoder and a second decoder.

[0033] The preset multilingual speech recognition model can adopt the existing large speech recognition model, adopt the encoder-decoder structure, and obtain a distillation model with relatively fewer parameters by performing knowledge distillation. The distillation model also has an encoder-decoder structure.

[0034] Step S103: constructing a target speech translation model based on the second encoder and the first decoder.

[0035] Specifically, the output of the second encoder is connected to the input of the first decoder, the second encoder encodes the input, and the encoded information is decoded by the first decoder to obtain a translation result.

[0036] Step S104, using the pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model, the audio data includes a preset language audio and corresponding original language text and target language text.

[0037] The preset language is the language before translation, including multiple languages. The preset language audio and the corresponding original language text and target language text are marked in the audio data. The audio data includes general open source data and vehicle-mounted field data, and the data is expanded through speech synthesis (text-to-speech, TTS) and other methods.

[0038] By inputting the collected audio data into the target speech translation model for model training, a multilingual speech translation model that can support multilingual speech recognition and translation is finally obtained.

[0039] A method for constructing a multilingual speech translation model in an embodiment of the present invention pre-trains and fine-tunes a preset speech translation model, performs knowledge distillation on a preset multilingual speech recognition model as a teacher model, and constructs a target speech translation model based on the obtained second encoder and first decoder, respectively. The second encoder and the first decoder are models whose parameters have been preliminarily trained, and the model parameters are small, so the target speech translation model can be controlled to an appropriate size, thereby reducing the scale of the multilingual speech translation model obtained after training, making it suitable for deployment on a resource-constrained vehicle system, and when training the target speech translation model, only less data alignment is required for training to make the model converge, thereby reducing the data required for training.

[0040] In one embodiment, the preset multilingual speech recognition model is a whisper model.

[0041] Correspondingly, in step S102, the preset multilingual speech recognition model is used as a teacher model for knowledge distillation to obtain a distillation model, including:

[0042] The whisper model is used as the teacher model, and the first three layers and the last three layers initialized by the whisper model's encoder are used as the encoder of the student model. The student model and the teacher model share a decoder for knowledge distillation to obtain a distilled model.

[0043] Specifically, the whisper model is an encoder-decoder architecture model, and its encoder and decoder are both 32-layer transformer structures; the distilled model selects a 6-layer transformer, uses the first three layers and the last three layers of the encoder in the whisper model as initial parameters, and starts distillation. Here, distillation needs to load two models, namely, the whisper model is used as the teacher model, the parameters are fixed, the model initialized by whisper is used as the student model, and the first three layers and the last three layers initialized by the whisper model encoder are used as the encoder of the student model. The teacher model and the student model share the same decoder. There are two training losses, one is the difference loss KLDIVLoss calculated by KL divergence between the teacher model encoder and the student model encoder, and the other is the loss LabelSmotthLoss that measures the model effect, which measures the audio recognition effect, and distillation training is performed by combining the two losses.

[0044] The whisper model is used for knowledge distillation. The lightweight encoder after distillation can reduce the consumption of computing resources while maintaining high recognition accuracy.

[0045] In this embodiment, a multilingual speech translation model construction method is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 2 is a flowchart of a method for constructing a multilingual speech translation model according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0046] Step S201, pre-train and fine-tune the preset speech translation model using the target language text data to obtain a language pre-training model of the target language, wherein the language pre-training model includes a first encoder and a first decoder. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0047] Step S202: Perform knowledge distillation on the preset multilingual speech recognition model as a teacher model to obtain a distillation model, which includes a second encoder and a second decoder. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0048] Step S203: construct a target speech translation model based on the second encoder and the first decoder.

[0049] Specifically, the above step S203 includes:

[0050] Step S2031, using the first decoder as a translation decoder, and connecting the translation decoder to the output end of the second encoder;

[0051] Step S2032: Add a fully connected layer as an audio recognition layer, and connect the audio recognition layer to the output end of the second encoder to obtain a target speech translation model.

[0052] The second encoder encodes feature data of the input audio data, and the feature data is respectively input to the translation decoder and the audio recognition layer for automatic speech translation AST and automatic speech recognition ASR.

[0053] Step S204: Use the pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model. The audio data includes the preset language audio and the corresponding original language text and target language text. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0054] The embodiment of the present invention constructs a target speech translation model based on the first decoder, the second encoder and the fully connected layer, so that it can perform automatic speech translation and automatic speech recognition at the same time.

[0055] In one embodiment, step S204, using the pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model, includes:

[0056] Step S2041, inputting the pre-collected audio data into the target speech translation model;

[0057] Step S2042, performing automatic speech translation on the input audio data based on the second encoder and the translation decoder to obtain a first learning loss;

[0058] Step S2043, performing automatic speech recognition on the input audio data based on the second encoder and the audio recognition layer to obtain a second learning loss;

[0059] Step S2044: adjusting parameters of the target speech translation model by combining the first learning loss and the second learning loss to obtain a multilingual speech translation model.

[0060] In the target speech translation model constructed in the embodiment of the present invention, all parameters have been trained and are easier to converge. When using this model for training, only the relationship between the two second encoders and the first decoder is learned, so that the output of the second encoder can adapt to the embedding layer of the first decoder, so that they can be connected accordingly to form a complete model link.

[0061] Through the first learning loss and the second learning loss, the automatic speech translation AST and automatic speech recognition ASR tasks are learned together. The inputs of the two are the same, the outputs are related, and they can be trained at the same time. In this process, the two tasks can promote each other, thereby improving the learning effect.

[0062] The embodiment of the present invention adjusts the parameters of the target speech translation model by combining the first learning loss and the second learning loss, thereby accelerating the reasoning speed, achieving better translation effect, and being more suitable for the field of vehicle computers.

[0063] A multilingual speech translation model construction method of an embodiment of the present invention trains speech texts in multiple languages ​​together, without the need to separately model the mapping of each source language to the target language. At the same time, it is iterated based on the open source whisper model, and the training consumes fewer resources. While reducing resources, the training scheme of the automatic speech translation AST is optimized through joint training, thereby improving the inference speed and the translation accuracy in the vehicle computer field.

[0064] The embodiment of the present invention also provides a multilingual speech translation method, comprising:

[0065] The original speech is input into the multilingual speech translation model as described in the above embodiment for translation to obtain the target language text.

[0066] The embodiment of the present invention also provides a multilingual speech translation model construction device, such as Figure 3 As shown, including:

[0067] A pre-training module 301 is used to pre-train and fine-tune a preset speech translation model using target language text data to obtain a language pre-training model of the target language, wherein the language pre-training model includes a first encoder and a first decoder;

[0068] A distillation module 302 is used to perform knowledge distillation on the preset multilingual speech recognition model as a teacher model to obtain a distillation model, wherein the distillation model includes a second encoder and a second decoder;

[0069] A target model construction module 303, used to construct a target speech translation model based on the second encoder and the first decoder;

[0070] The model training module 304 is used to train the target speech translation model using the pre-collected audio data to obtain a multilingual speech translation model. The audio data includes the preset language audio and the corresponding original language text and target language text.

[0071] A multilingual speech translation model construction device provided by an embodiment of the present invention pre-trains and fine-tunes a preset speech translation model, performs knowledge distillation on a preset multilingual speech recognition model as a teacher model, and constructs a target speech translation model based on the obtained second encoder and first decoder, respectively. The second encoder and the first decoder are models whose parameters have been preliminarily trained, and the model parameters are small. Therefore, the target speech translation model can be controlled to an appropriate size, reducing the model scale. When training the target speech translation model, only less data alignment is required for training to make the model converge, thereby reducing the data required for training.

[0072] Furthermore, the target model building module 303 includes:

[0073] A first connection module, used for using the first decoder as a translation decoder and connecting the translation decoder to an output end of the second encoder;

[0074] The second connection module is used to add a fully connected layer as an audio recognition layer, and connect the audio recognition layer and the output end of the second encoder to obtain a target speech translation model.

[0075] Furthermore, the model training module 304 includes:

[0076] An audio data input module, used to input pre-collected audio data into a target speech translation model;

[0077] A first loss acquisition module, configured to perform automatic speech translation on the input audio data based on the second encoder and the translation decoder to obtain a first learning loss;

[0078] A second loss acquisition module, used for performing automatic speech recognition on the input audio data based on the second encoder and the audio recognition layer to obtain a second learning loss;

[0079] The joint training module is used to adjust the parameters of the target speech translation model by combining the first learning loss and the second learning loss to obtain a multilingual speech translation model.

[0080] Furthermore, the preset multilingual speech recognition model is a whisper model; correspondingly, the distillation module 302 is used to:

[0081] The whisper model is used as the teacher model, and the first three layers and the last three layers initialized by the whisper model's encoder are used as the encoder of the student model. The student model and the teacher model share a decoder for knowledge distillation to obtain a distilled model.

[0082] The embodiment of the present invention further provides a multilingual speech translation device, comprising: an input module, used to input original speech into the multilingual speech translation model in the above embodiment for translation to obtain a target language text.

[0083] The embodiment of the present invention also provides a schematic diagram of the structure of a computer device, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0084] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0085] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0086] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0088] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0089] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0090] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0091] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0092] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing a multilingual speech translation model, characterized in that: include: Pre-training and fine-tuning a preset speech translation model using target language text data to obtain a language pre-training model of the target language, wherein the language pre-training model includes a first encoder and a first decoder; Performing knowledge distillation on a preset multilingual speech recognition model as a teacher model to obtain a distillation model, wherein the distillation model includes a second encoder and a second decoder; Building a target speech translation model based on the second encoder and the first decoder; Using pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model, the audio data including preset language audio and corresponding original language text and target language text; The step of constructing a target speech translation model based on the second encoder and the first decoder includes: Using the first decoder as a translation decoder, and connecting the translation decoder to an output terminal of the second encoder; Adding a fully connected layer as an audio recognition layer, and connecting the audio recognition layer and the output end of the second encoder to obtain a target speech translation model; The method of using the pre-collected audio data to train the target speech translation model to obtain a multilingual speech translation model includes: Inputting the pre-collected audio data into the target speech translation model; Performing automatic speech translation on the input audio data based on the second encoder and the translation decoder to obtain a first learning loss; Performing automatic speech recognition on the input audio data based on the second encoder and the audio recognition layer to obtain a second learning loss; The first learning loss and the second learning loss are combined to adjust the parameters of the target speech translation model to obtain a multilingual speech translation model.

2. The method for constructing a multilingual speech translation model according to claim 1, characterized in that: The preset multilingual speech recognition model is a whisper model; Correspondingly, the preset multilingual speech recognition model is used as a teacher model for knowledge distillation to obtain a distillation model, including: The whisper model is used as the teacher model, and the first three layers and the last three layers initialized by the whisper model's encoder are used as the encoder of the student model. The student model and the teacher model share a decoder for knowledge distillation to obtain a distilled model.

3. A multilingual speech translation method, characterized in that: include: The original speech is input into the multilingual speech translation model as described in any one of claims 1 to 2 for translation to obtain a text in the target language.

4. A multilingual speech translation model construction device, characterized in that: include: A pre-training module, used to pre-train and fine-tune a preset speech translation model using text data of a target language to obtain a language pre-training model of the target language, wherein the language pre-training model includes a first encoder and a first decoder; A distillation module, used to perform knowledge distillation on a preset multilingual speech recognition model as a teacher model to obtain a distillation model, wherein the distillation model includes a second encoder and a second decoder; A target model construction module, used for constructing a target speech translation model based on the second encoder and the first decoder; A model training module, used to train the target speech translation model using pre-collected audio data to obtain a multilingual speech translation model, wherein the audio data includes audio in a preset language and corresponding original language text and target language text; The target model building blocks include: A first connection module, used for using the first decoder as a translation decoder and connecting the translation decoder to an output end of the second encoder; A second connection module is used to add a fully connected layer as an audio recognition layer, and connect the audio recognition layer to the output end of the second encoder to obtain a target speech translation model; The model training module includes: An audio data input module, used to input pre-collected audio data into a target speech translation model; A first loss acquisition module, configured to perform automatic speech translation on the input audio data based on the second encoder and the translation decoder to obtain a first learning loss; A second loss acquisition module, used for performing automatic speech recognition on the input audio data based on the second encoder and the audio recognition layer to obtain a second learning loss; The joint training module is used to adjust the parameters of the target speech translation model by combining the first learning loss and the second learning loss to obtain a multilingual speech translation model.

5. A multilingual speech translation device, characterized in that: include: An input module is used to input the original speech into the multilingual speech translation model as described in any one of claims 1 to 2 for translation to obtain a text in a target language.

6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multilingual speech translation model construction method according to any one of claims 1 to 2 or the multilingual speech translation method according to claim 3 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multilingual speech translation model construction method according to any one of claims 1 to 2 or the multilingual speech translation method according to claim 3.

8. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the multilingual speech translation model construction method according to any one of claims 1 to 2 or the multilingual speech translation method according to claim 3.

Images