Method for building a multilingual speech recognition model

Abstract

The present invention relates to the technical field of intelligent translation, and discloses a method for constructing a multilingual speech recognition model, comprising: obtaining speech recognition training data, the speech recognition training data comprising: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data; inputting the audio data into a coding layer of a preset multilingual speech recognition model, the preset multilingual speech recognition model comprising several coding layers, obtaining intermediate data, obtaining language recognition data based on the intermediate data, obtaining first language encoding data based on the intermediate data and the language recognition data; obtaining second language encoding data based on the language data; obtaining first loss data and second loss data based on the text label data, the first language encoding data and the second language encoding data; calculating to obtain distillation loss data; constructing a loss function based on the first loss data, the second loss data and the distillation loss data, and constructing the multilingual speech recognition model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent translation technology, and in particular to a method for constructing a multilingual speech recognition model. Background Art

[0002] With the development of society, car companies are beginning to go global. In this process, there is a demand for intelligent vehicle control interaction in more languages. As the first step in the field of intelligent vehicle control interaction, improving the accuracy of recognition and translation is particularly important for the interactive experience. How to support more languages ​​while using less resources and having higher translation accuracy is a problem that needs to be solved urgently.

[0003] In the related art, separate modeling is performed through mapping from the source language to the target language, which consumes a lot of resources and increases the complexity of user use. At the same time, for low-resource languages, there may be insufficient recognition accuracy.

[0004] In the field of multilingual speech research in academia, the speech of multiple countries can be trained simultaneously. However, the solutions from academia are difficult to implement at the industrial level. The main problems are that the model is too large, the reasoning speed is slow, and the user experience is affected. At the same time, the effect in the field of car computers is poor. Summary of the invention

[0005] In view of this, the present invention provides a method for constructing a multilingual speech recognition model to solve the problem that the multilingual speech recognition model is too large and the inference speed is slow, affecting the user experience.

[0006] In a first aspect, the present invention provides a method for constructing a multilingual speech recognition model, the method comprising: obtaining speech recognition training data, the speech recognition training data comprising: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data; inputting the audio data into a coding layer of a preset multilingual speech recognition model, the preset multilingual speech recognition model comprising a plurality of the coding layers, obtaining intermediate data, the intermediate data being obtained by encoding the audio data through a target coding layer and a coding layer before the target coding layer, performing multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data , based on the intermediate data and the language recognition data, obtain first language encoding data; based on the language data, obtain language vector data, and based on the intermediate data and the language vector data, obtain second language encoding data; based on the text label data, the first language encoding data and the second language encoding data, perform encoding and decoding processing to obtain first loss data and second loss data; based on the language recognition data and the language vector data, calculate and obtain distillation loss data; based on the first loss data, the second loss data and the distillation loss data, construct a loss function, and construct a multilingual speech recognition model based on the loss function.

[0007] In an optional embodiment, encoding and decoding processing is performed based on the text label data, the first language encoding data and the second language encoding data to obtain first loss data and second loss data, including: inputting the first language encoding data into the encoding layer after the target encoding layer, and performing full connection based on the obtained encoding result to obtain first fully connected processed data; calculating the first loss data by connecting the first fully connected processed data and the text label data through a temporal classification algorithm; inputting the second language encoding data into the encoding layer after the target encoding layer, and performing full connection based on the obtained encoding result to obtain second fully connected processed data; calculating the second loss data by connecting the second fully connected processed data and the text label data through a temporal classification algorithm.

[0008] In an optional embodiment, before inputting the audio data into the encoding layer of a preset multilingual speech recognition model, it also includes: performing feature extraction based on the audio data to obtain audio feature data; performing one-dimensional convolution and Gaussian error linear unit processing on the audio feature data to obtain an audio data processing result, and inputting the audio data processing result into the preset multilingual speech recognition model.

[0009] In an optional implementation, constructing a loss function based on the first loss data, the second loss data, and the distillation loss data includes:

[0010] The loss function L is constructed by the following formula:

[0011] L = a*((ctc loss1 +ctc loss2 )+b*KL loss

[0012] Among them, a and b are the loss function coefficients, ctc loss1 is the first loss data, ctc loss2 For the second loss data, KL loss is the distillation loss data.

[0013] In an optional implementation, the preset multilingual speech recognition model is a Whisper Large V3 model.

[0014] In a second aspect, the present invention provides a method for multilingual speech recognition, comprising: inputting original speech into the aforementioned multilingual speech recognition model for recognition to obtain a target language text.

[0015] In a third aspect, the present invention provides a device for constructing a multilingual speech recognition model, the device comprising: an acquisition module, used to acquire training data, the training data comprising: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data; a first encoding module, used to input the audio data into an encoding layer of a preset multilingual speech recognition model, the preset multilingual speech recognition model comprising a plurality of the encoding layers, acquire intermediate data, the intermediate data being encoded by the audio data at a target encoding layer through an encoding layer before the target encoding layer, perform multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data, and based on the intermediate data The intermediate data and the language recognition data are used to obtain first language encoding data; a second encoding module is used to convert the language data into a vector to obtain language vector data, and obtain second language encoding data based on the intermediate data and the language vector data; a construction module is used to perform encoding and decoding processing based on the text label data, the first language encoding data and the second language encoding data to obtain first loss data and second loss data; based on the language recognition data and the language vector data, distillation loss data is calculated; based on the first loss data, the second loss data and the distillation loss data, a loss function is constructed, and a multilingual speech recognition model is constructed based on the loss function.

[0016] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for constructing a multilingual speech recognition model according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0017] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for constructing a multilingual speech recognition model according to the first aspect or any corresponding embodiment thereof.

[0018] In a sixth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for constructing a multilingual speech recognition model according to the first aspect or any corresponding embodiment thereof.

[0019] By introducing language data corresponding to audio data, there is no need to separately train a language classification model to determine the language model for speech recognition. It can support multi-language speech recognition of specified and unspecified languages. The multi-language speech recognition model is lightweight, which can reduce resource consumption, increase the inference speed of speech recognition, improve the accuracy of multi-language speech recognition, increase user satisfaction with multi-language speech recognition, and improve translation accuracy in the car computer field. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a process for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown;

[0022] Figure 2 Another schematic diagram of a process for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram showing the structure of a device for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown;

[0024] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] In multi-language speech recognition in the field of automotive computers, before speech recognition, a language classification model needs to be trained separately to determine which language model to select for speech recognition.

[0027] According to an embodiment of the present invention, a method embodiment for building a multilingual speech recognition model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] In this embodiment, a method for constructing a multilingual speech recognition model is provided, which can be used in mobile terminals, such as mobile phones, vehicle-mounted terminals equipped with processing systems, and tablet computers. Figure 1 A flow chart of a method for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown. Figure 1 As shown, the process includes the following steps:

[0029] Step S101, acquiring speech recognition training data, the speech recognition training data including: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data.

[0030] In this step, text label data refers to text information identified and transcribed from audio data. This information exists in the form of labels and is used to represent the key information or the overall text of the audio content. When generating text label data, certain specifications and standards need to be followed to ensure the accuracy and consistency of the labels. For example, for the text output by the speech recognition system, text normalization processing is required, including: removing redundant information, correcting grammatical errors, or unifying punctuation marks, etc.

[0031] Step S102: input the audio data into the encoding layer of a preset multilingual speech recognition model, where the preset multilingual speech recognition model includes several encoding layers, obtain intermediate data, where the intermediate data is obtained by encoding the audio data through a target encoding layer and an encoding layer before the target encoding layer, perform multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data, and obtain first language encoding data based on the intermediate data and the language recognition data.

[0032] In this step, the multilingual speech recognition model includes n coding layers, specifically including coding layer 1, coding layer 2...coding layer i, coding layer i+1...coding layer n, wherein the i-th coding layer is the target coding layer, and the intermediate data is derived from the target coding layer, and the intermediate data is downsampled, activated, upsampled and normalized to obtain the multi-layer perceptron processing result. The multi-layer perceptron processing result is subjected to mean pooling to obtain language identification data (Language Identification, referred to as LID).

[0033] Mean Pooling is used to reduce the dimension of the feature map output by the convolution layer. It reduces the amount of data by calculating the average value of feature points in the local area while retaining important background information. It can reduce the computational complexity, improve the generalization ability of the model, and reduce the risk of overfitting. It can enhance the robustness of the model to different frequency components in the intermediate data and improve the accuracy of recognition.

[0034] Based on the intermediate data and the language recognition data, the first language coded data is obtained. The first language coded data is obtained by splicing the intermediate data and the language recognition data. Specifically, the language recognition data is spliced ​​before the intermediate data to obtain the first language coded data. The time dimension of the intermediate data is n, and the time dimension of the first language coded data is n+1.

[0035] Step S103, converting the language data into a vector to obtain language vector data, and obtaining second language encoding data based on the intermediate data and the language vector data.

[0036] In this step, the language data can be converted into language vector data LE based on the Embedding in the neural network. Based on the intermediate data and the language vector data, the second language encoding data is obtained. The second language encoding data is obtained by splicing the intermediate data and the language vector data. Specifically, the language vector data is spliced ​​before the intermediate data to obtain the second language encoding data. The time dimension of the second language encoding data is n+1.

[0037] Step S104, performing encoding and decoding processing based on the text label data, the first language encoding data and the second language encoding data to obtain first loss data and second loss data; calculating the distillation loss data based on the language recognition data and the language vector data; constructing a loss function based on the first loss data, the second loss data and the distillation loss data, and constructing a multilingual speech recognition model based on the loss function.

[0038] In this step, the first language coded data is input into the coding layer i+1, and is coded by the remaining coding layers of the multilingual speech recognition model to obtain a first coding result, and the first coding result is decoded based on the text label data to obtain a first loss function. The second language coded data is input into the coding layer i+1, and is coded by the remaining coding layers of the multilingual speech recognition model to obtain a second coding result, and the second coding result is decoded based on the text label data to obtain a second loss function.

[0039] Distillation loss is mainly used to measure the difference between the teacher model and the student model. The teacher model is a large model with good performance, which refers to the language vector data in this invention. The student model is a small model or a simplified model that needs to be trained, which refers to the language recognition data in this invention. Distillation loss guides the learning of the student model by comparing the outputs of the teacher model and the student model. Specifically, distillation loss calculates the difference between the language vector data and the language recognition data, and uses this difference to update the parameters of the student model so that the student model can imitate the behavior of the teacher model.

[0040] You can use functions such as cross entropy or KL divergence to calculate the difference between language recognition data and language vector data.

[0041] The method for constructing a multilingual speech recognition model provided in this embodiment, by introducing language data corresponding to audio data, does not need to separately train a language classification model to determine the language model for speech recognition. It can support multilingual speech recognition of specified and unspecified languages. The multilingual speech recognition model is lightweight, which can reduce resource consumption, increase the inference speed of speech recognition, improve the accuracy of multilingual speech recognition, improve user satisfaction with multilingual speech recognition, and improve translation accuracy in the vehicle computer field.

[0042] In an optional embodiment, encoding and decoding processing is performed based on text label data, first language encoded data and second language encoded data to obtain first loss data and second loss data, including: inputting the first language encoded data into the encoding layer after the target encoding layer, and performing full connection based on the obtained encoding result to obtain first fully connected processed data; calculating the first loss data by connecting the first fully connected processed data and the text label data through a time series classification algorithm; inputting the second language encoded data into the encoding layer after the target encoding layer, and performing full connection based on the obtained encoding result to obtain second fully connected processed data; calculating the second loss data by connecting the second fully connected processed data and the text label data through a time series classification algorithm.

[0043] In this step, the fully connected (FC) integrates the features extracted from the previous layers and outputs a nonlinear feature space. The Connectionist Temporal Classification (CTC) algorithm predicts labels for unknown input sequences. In this way, the CTC decoding combined with the language model replaces the encoder-decoder structure, avoiding the frequent calls of the autoregressive model, saving resources and speeding up the inference speed.

[0044] In an optional embodiment, before inputting the audio data into the encoding layer of a preset multilingual speech recognition model, it also includes: performing feature extraction based on the audio data to obtain audio feature data; performing one-dimensional convolution and Gaussian error linear unit processing on the audio feature data to obtain an audio data processing result, and inputting the audio data processing result into the preset multilingual speech recognition model.

[0045] In this embodiment, the log-Mel Spectrogram feature of the audio data can be extracted, and the Log-Mel Spectrogram feature is the time-frequency representation feature of the audio data. The training speed of the model can be accelerated. The audio feature data is processed by one-dimensional convolution (Conv1D) and Gaussian Error Linear Units (GELU for short) to obtain the audio data processing result. It can help to improve the convergence speed and performance of the training process.

[0046] In an optional implementation, a loss function is constructed based on the first loss data, the second loss data, and the distillation loss data, including:

[0047] The loss function L is constructed by the following formula:

[0048] L = a*(ctc loss1 +ctcloss2 )+b*KL loss

[0049] Among them, a and b are the loss function coefficients, ctc loss1 is the first loss data, ctc loss2 is the second loss data, KL loss is the distillation loss data.

[0050] In this way, a clear optimization goal can be provided for the training of the multilingual speech recognition model. By calculating the loss value, the parameter update direction of the multilingual speech recognition model can be guided to reduce the prediction error. During the training process, the loss function prompts the model to continuously learn and improve, which can improve the prediction accuracy.

[0051] In an optional implementation, the preset multilingual speech recognition model is the Whisper Large V3 model.

[0052] In this embodiment, the audio data is input into the encoding layer of the Whisper Large V3 model for encoding, and CTC decoding is adopted to solve the problem of slow decoding speed in the autoregressive decoding process of the Whisper Large V3 model.

[0053] The embodiment of the present invention also provides a method for multilingual speech recognition, comprising: inputting original speech into the multilingual speech recognition model as in the above embodiment for translation to obtain a target language text.

[0054] Figure 2 Another flow chart of the method for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown. Figure 2As shown, input speech recognition training data; 201: extract the log-Mel Spectrogram features of audio data; 202: perform one-dimensional convolution (Conv1D) and Gaussian Error Linear Units (GELU) processing on the audio feature data; 203: input the audio data processing result into the preset multilingual speech recognition model, which is the Whisper Large V3 model, and extract the intermediate data in the i-th encoding layer; 204: the intermediate data is processed by a multi-layer perceptron to obtain the multi-layer perceptron processing result; 205: the multi-layer perceptron processing result is average pooled; 206: obtain language recognition data; 207: based on the intermediate data and the language recognition data, obtain the first language encoding data; 208: the language data is converted into language vector data through Embedding; 209: based on the intermediate data and the language vector data, obtain the second language encoding data; 210: input the first language encoding data into the encoding layer i+1, and pass the remaining encoding of the multilingual speech recognition model layer to obtain a first encoding result, input the second language encoding data into the encoding layer i+1, and obtain a second encoding result after encoding by the remaining encoding layers of the multilingual speech recognition model; 211: decode the first encoding result based on the text label data to obtain a first loss function, decode the second encoding result based on the text label data to obtain a second loss function; 212: calculate the distillation loss data based on the language recognition data and the language vector data, construct a loss function based on the first loss data, the second loss data and the distillation loss data, and construct the multilingual speech recognition model based on the loss function.

[0055] The method for constructing a multilingual speech recognition model provided in this embodiment, by introducing language data corresponding to audio data, does not need to separately train a language classification model to determine the language model for speech recognition. It can support multilingual speech recognition of specified and unspecified languages. The multilingual speech recognition model is lightweight, which can reduce resource consumption, increase the inference speed of speech recognition, improve the accuracy of multilingual speech recognition, improve user satisfaction with multilingual speech recognition, and improve translation accuracy in the vehicle computer field.

[0056] In the present embodiment, a device for building a multilingual speech recognition model is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0057] Figure 3The structure diagram of the device for constructing a multilingual speech recognition model provided by an embodiment of the present invention is shown. This embodiment provides a device for constructing a multilingual speech recognition model, such as Figure 3 As shown, including:

[0058] The acquisition module 301 is used to acquire training data, where the training data includes: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data.

[0059] The first encoding module 302 is used to input the audio data into the encoding layer of a preset multilingual speech recognition model, where the preset multilingual speech recognition model includes several encoding layers, obtain intermediate data, where the intermediate data is obtained by encoding the audio data in the encoding layer before the target encoding layer at the target encoding layer, perform multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data, and obtain first language encoding data based on the intermediate data and the language recognition data.

[0060] The second encoding module 303 is used to convert the language data into a vector to obtain language vector data, and obtain second language encoding data based on the intermediate data and the language vector data.

[0061] The construction module 304 is used to perform encoding and decoding processing based on the text label data, the first language encoding data and the second language encoding data to obtain the first loss data and the second loss data; calculate the distillation loss data based on the language recognition data and the language vector data; construct a loss function based on the first loss data, the second loss data and the distillation loss data, and construct a multilingual speech recognition model based on the loss function.

[0062] In some optional implementations, the construction module 304 includes:

[0063] The first construction unit is used to input the first language encoded data into the encoding layer after the target encoding layer, perform full connection based on the obtained encoding result, and obtain first fully connected processed data; the first fully connected processed data and text label data are calculated by connecting the time series classification algorithm to obtain the first loss data.

[0064] The second construction unit is used to input the second language encoded data into the encoding layer after the target encoding layer, perform full connection based on the obtained encoding result, and obtain second fully connected processed data; the second fully connected processed data and the text label data are calculated by connecting the time series classification algorithm to obtain the second loss data.

[0065] In some optional implementations, the aforementioned apparatus for constructing a multilingual speech recognition model further includes:

[0066] The speech data processing module is used to extract features based on the audio data to obtain audio feature data before inputting the audio data into the encoding layer of the preset multilingual speech recognition model; perform one-dimensional convolution and Gaussian error linear unit processing on the audio feature data to obtain an audio data processing result, and input the audio data processing result into the preset multilingual speech recognition model.

[0067] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0068] By introducing language data corresponding to audio data, there is no need to separately train a language classification model to determine the language model for speech recognition. It can support multi-language speech recognition of specified and unspecified languages. The multi-language speech recognition model is lightweight, which can reduce resource consumption, increase the inference speed of speech recognition, improve the accuracy of multi-language speech recognition, increase user satisfaction with multi-language speech recognition, and improve translation accuracy in the car computer field.

[0069] The device for constructing a multilingual speech recognition model in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0070] The embodiment of the present invention also provides a computer device having the above Figure 3 The device shown is used to build a multilingual speech recognition model.

[0071] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3A processor 10 is taken as an example.

[0072] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0073] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0074] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0076] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0077] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0078] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0079] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0080] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing a multilingual speech recognition model, characterized in that: The method comprises: Acquire speech recognition training data, the speech recognition training data comprising: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data; Inputting the audio data into a coding layer of a preset multilingual speech recognition model, wherein the preset multilingual speech recognition model includes a plurality of coding layers, obtaining intermediate data, wherein the intermediate data is obtained by encoding the audio data through a target coding layer and a coding layer before the target coding layer, performing multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data, and obtaining first language coded data based on the intermediate data and the language recognition data; Obtaining language vector data based on the language data, and obtaining second language encoding data based on the intermediate data and the language vector data; Based on the text label data, the first language encoding data and the second language encoding data, encoding and decoding processing is performed to obtain first loss data and second loss data; based on the language recognition data and the language vector data, distillation loss data is calculated; based on the first loss data, the second loss data and the distillation loss data, a loss function is constructed, and a multilingual speech recognition model is constructed based on the loss function.

2. The method according to claim 1, characterized in that Performing encoding and decoding based on the text label data, the first language coded data, and the second language coded data to obtain first loss data and second loss data includes: Inputting the first language encoded data into the encoding layer after the target encoding layer, performing full connection based on the obtained encoding result to obtain first fully connected processed data; calculating the first loss data by connecting the first fully connected processed data and the text label data through a temporal classification algorithm; The second language encoded data is input into the encoding layer after the target encoding layer, and a full connection is performed based on the obtained encoding result to obtain second fully connected processed data; the second fully connected processed data and the text label data are calculated by a connection time series classification algorithm to obtain the second loss data.

3. The method according to claim 1 or 2, characterized in that: Before inputting the audio data into the encoding layer of the preset multilingual speech recognition model, the method further includes: Perform feature extraction based on the audio data to obtain audio feature data; The audio feature data is processed by one-dimensional convolution and Gaussian error linear unit to obtain an audio data processing result, and the audio data processing result is input into the preset multilingual speech recognition model.

4. The method according to claim 1, characterized in that Constructing a loss function based on the first loss data, the second loss data, and the distillation loss data, including: The loss function L is constructed by the following formula: L=a*(ctc loss1 +ctc loss2 )+b*KL loss Among them, a and b are the loss function coefficients, ctc loss1 is the first loss data, ctc loss2 For the second loss data, KL loss is the distillation loss data.

5. The method according to claim 1, characterized in that The preset multilingual speech recognition model is the WhisperLarge V3 model.

6. A method for multilingual speech recognition, characterized in that: include: The original speech is input into the multilingual speech recognition model as described in any one of claims 1 to 5 for recognition to obtain the target language text.

7. A device for constructing a multilingual speech recognition model, characterized in that: The device comprises: An acquisition module, used to acquire training data, the training data including: audio data, text label data corresponding to the audio data, and language data corresponding to the audio data; A first encoding module is configured to input the audio data into an encoding layer of a preset multilingual speech recognition model, wherein the preset multilingual speech recognition model includes a plurality of the encoding layers, obtain intermediate data, wherein the intermediate data is obtained by encoding the audio data at a target encoding layer through an encoding layer before the target encoding layer, perform multi-layer perceptron and mean pooling processing on the intermediate data to obtain language recognition data, and obtain first language encoding data based on the intermediate data and the language recognition data; A second encoding module, configured to convert the language data into a vector to obtain language vector data, and obtain second language encoded data based on the intermediate data and the language vector data; A construction module is used to perform encoding and decoding processing based on the text label data, the first language encoding data and the second language encoding data to obtain first loss data and second loss data; calculate distillation loss data based on the language recognition data and the language vector data; construct a loss function based on the first loss data, the second loss data and the distillation loss data, and construct a multilingual speech recognition model based on the loss function.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for constructing a multilingual speech recognition model according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for constructing a multilingual speech recognition model according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for constructing a multilingual speech recognition model according to any one of claims 1 to 5.

Images