Multi-language speech recognition method and model, electronic equipment, storage medium and program product

Abstract

The invention discloses a multi-language speech recognition method. A multi-language speech recognition model comprises an encoder, a language classification model, a CTC module and a decoder. The method comprises the following steps: acquiring acoustic features of an audio to be identified, and inputting the acoustic features to the encoder to obtain an encoder output result containing acoustic feature local information and acoustic feature global information; a language classification model carries out preset processing on the encoder output result to obtain a language classification result; the CTC module is combined with a weighted finite state conversion machine, and an encoder output result is mapped to a streaming nbest identification result through a prefix beam search algorithm; and the decoder re-scores the output result of the encoder according to the streaming nbest identification result. The multi-language speech recognition model only shares an encoder at the bottom layer, and the language classification model, the CTC module and the decoder at the top layer are independent. The problem that homophones of different languages are confused in speech recognition is reduced through shared bottom layer parameters and a re-scoring process, and the language classification accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multilingual speech recognition method, model, electronic device, storage medium and program product. Background Art

[0002] The architecture of a general multilingual speech recognition system is the same as that of a single-language speech recognition system, the only difference being the use of mixed multilingual data or single-language data to train the acoustic and language models. This type of system can meet the application requirements of simple scenarios. However, in complex scenarios such as a large number of languages, small differences between languages, and non-standard user pronunciation, this type of system cannot achieve accurate recognition, and it is difficult to balance model complexity and performance. Summary of the invention

[0003] The applicant has tried the following solutions in the process of implementing this application: Solution 1: Only use the encoder + CTC module for streaming recognition, set up language-aware multi-head experts inside the encoder, the language classification model is consistent with this application, and there is no decoder. The advantages are: simple system structure, less resource consumption, lower latency, and higher concurrency; the disadvantages are: the results of the language classification model cannot be used in the speech recognition model; the encoder and CTC module are streaming structures, unlike the re-scoring scheme, which has a global receptive field, and the recognition rate is low.

[0004] Solution 2: The overall model structure is the same as that of this application, the only difference is that the decoder is autoregressive decoding instead of re-scoring. Advantages: higher recognition rate. Disadvantages: autoregressive decoding leads to a significant increase in latency and affects the number of concurrent connections.

[0005] The embodiments of the present application provide a multilingual speech recognition method, model, electronic device, storage medium and program product, which are used to solve at least one of the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a multilingual speech recognition method, characterized in that the multilingual speech recognition model includes: an encoder, a language classification model, a CTC module and a decoder; the method includes: Acquiring the acoustic features of the audio to be recognized and inputting them into the encoder to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; The language classification model performs preset processing on the encoder output result to obtain a language classification result; The CTC module combines a weighted finite state transition machine to map the encoder output results to streaming nbest recognition results through a prefix beam search algorithm; The decoder re-scores the encoder output result according to the streaming nbest recognition result.

[0007] In some embodiments, the encoder includes a convolution module and a self-attention module, the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features.

[0008] In some embodiments, the language classification model performs preset processing on the encoder output result to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain a language classification result.

[0009] In some embodiments, the decoder includes a decoder attention module, a plurality of expert modules and a gated network module; the decoder re-scoring the encoder output result according to the streaming nbest recognition result includes: The decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; The decoder attention module is respectively fed into multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

[0010] In some embodiments, the multilingual speech recognition method further includes: the gated network module calculates the weight of the corresponding expert module according to the given language label information, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result; the target expert module is an expert module selected according to the weight; and the streaming nbest recognition results are reordered according to the attention score to obtain the final recognition result.

[0011] In a second aspect, the embodiment of the present application provides a multilingual speech recognition model, characterized in that it includes: The encoder is configured to: encode the acoustic features of the audio to be recognized to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; The language classification model is configured as follows: performing a preset process on the encoder output result to obtain a language classification result; The CTC module is configured as follows: combining a weighted finite state transition machine to map the encoder output results to streaming nbest recognition results through a prefix beam search algorithm; The decoder is configured to re-score the encoder output result according to the streaming nbest recognition result.

[0012] In some embodiments, the encoder includes a convolution module and a self-attention module, the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features.

[0013] In some embodiments, the encoder output result is subjected to preset processing to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain the language classification result.

[0014] In some embodiments, the decoder includes a decoder attention module, a plurality of expert modules and a gated network module; the decoder re-scoring the encoder output result according to the streaming nbest recognition result includes: The decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; The decoder attention module is respectively fed into multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

[0015] In some embodiments, it also includes: the gated network module calculates the weight of the corresponding expert module according to the language classification result or the user-specified language, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result; the target expert module is an expert module selected according to the weight; the streaming nbest recognition results are reordered according to the attention score to obtain the final recognition result.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of any one of the multilingual speech recognition methods described in the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, characterized in that when the computer program / instruction is executed by a processor, the steps of any one of the multilingual speech recognition methods described in the present application are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of any one of the multilingual speech recognition methods described in the present application are implemented.

[0019] In this embodiment, the multilingual speech recognition model only shares the bottom-level encoder, and the top-level language classification model, CTC module, and decoder are independent. The shared bottom-level parameters can reduce the confusion of homophones in different languages ​​in speech recognition, and can also improve the accuracy of language classification. Independent top-level parameters can enable two tasks to be calculated in parallel, reducing latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an embodiment of a multilingual speech recognition model of the present application; Figure 2 A flowchart of an embodiment of the multilingual speech recognition method of the present application; Figure 3 is a schematic diagram of an embodiment of a decoder in the present application; Figure 4 A flowchart of another embodiment of the multilingual speech recognition method of the present application; Figure 5 A schematic diagram of an embodiment of the input and output structure of the decoder in the present application; Figure 6 A flowchart of another embodiment of the multilingual speech recognition method of the present application; Figure 7 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0023] It should also be noted that, in this article, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0024] In order to solve at least one problem in the prior art, the present application proposes a multilingual speech recognition model, which can be used to implement a multilingual speech recognition method, specifically a multilingual streaming speech recognition method based on an improved hybrid expert model and attention rescoring. The method uses a hybrid expert model to enhance generalization and avoid parameter expansion during model deployment; introduces prior knowledge into the hybrid expert model, and uses prior information such as language to assist expert selection; achieves high-precision streaming recognition based on the CTC+attention rescoring scheme; the language classification model and the recognition model only share the underlying encoder, and the high-level is independent.

[0025] In order to solve at least one problem in the prior art, the present application proposes a multilingual speech recognition model, the model architecture of which is based on a hybrid CTC / attention neural network of an improved hybrid expert model. Figure 1 The figure shows a schematic diagram of an embodiment of a multilingual speech recognition model of the present application, which includes four parts: an encoder, a decoder, a CTC module and a language classification model.

[0026] like Figure 2 As shown, an embodiment of the present application provides a multilingual speech recognition method, the method comprising: S10, obtaining the acoustic features of the audio to be recognized and inputting them into the encoder to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; S20, the language classification model performs preset processing on the output result of the encoder to obtain a language classification result. In the training stage of the multilingual speech recognition model, the audio to be recognized can be replaced with a sample audio, and the language classification cross entropy is determined in combination with the language label information of the sample audio to optimize the language classification model; S30, CTC module combined with weighted finite state transition machine (WFST), through the prefix beam search algorithm to map the encoder output results to the streaming nbest recognition results. In the training stage of the multilingual speech recognition model, the CTC module can be optimized by combining the weighted finite state transition machine (WFST) with the prefix beam search algorithm according to the CTC loss; in addition, in the training stage, the CTC loss will be back-propagated to the encoder for training the encoder.

[0027] S40, the decoder re-scores the encoder output result according to the streaming nbest recognition result. In the training stage of the multilingual speech recognition model, the audio to be recognized can be replaced with a sample audio, and the decoder cross entropy is calculated according to the language label information of the sample audio to achieve the optimization of the decoder.

[0028] In this embodiment, the multilingual speech recognition model only shares the bottom-level encoder, and the top-level language classification model, CTC module, and decoder are independent. The shared bottom-level parameters can reduce the confusion of homophones in different languages ​​in speech recognition, and can also improve the accuracy of language classification. Independent top-level parameters can enable two tasks to be calculated in parallel, reducing latency.

[0029] In some embodiments, the encoder includes a convolution module and a self-attention module, the convolution module is used to count the local information of acoustic features, and the self-attention module is used to count the global information of acoustic features. Exemplarily, the acoustic features are input into the encoder, and the local and global information of the acoustic features are counted by the convolution module and the self-attention module respectively to obtain a multi-frame encoding output. The local information of acoustic features refers to the time-frequency correlation in the speech signal in a short period of time, which is mainly completed by the convolution module. Extracting local timing patterns can enhance the sensitivity of the model to short-term changes, such as rapid acoustic changes. The global information of acoustic features refers to the long-term contextual dependencies and long-distance relationships in the speech signal, which is mainly completed by the self-attention mechanism. The model has the ability to capture both fine-grained local patterns and long-distance contextual dependencies.

[0030] In some embodiments, the language classification model performs preset processing on the encoder output result to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain a language classification result.

[0031] For example, assume that the input feature is a two-dimensional vector: X∈R T×D in, T is the number of time steps, D is the feature dimension, R is the set of real numbers.

[0032] 1. Attention weight calculation. The attention mechanism is for each time step t Generate an attention weight: in, W∈R d×D , b∈R d , w∈R d are learnable weights, d It is the attention dimension.

[0033] 2. Weighted mean calculation. Using attention weights α t Perform weighted summation on the features to obtain the mean of the global representation: 3. Weighted standard deviation calculation: 4. Mean and standard deviation concatenation: 5. Fully connected layer + Softmax activation. The fully connected layer maps the high-dimensional feature vector to the dimension of the number of categories, and the Softmax activation function converts the output of the linear layer o Convert to probability distribution p : in, C is the total number of categories.

[0034] In some embodiments, the multilingual speech recognition model of the present application can be implemented as a hybrid CTC / attention neural network based on an improved hybrid expert model. Figure 3 , which is a schematic diagram of an embodiment of a decoder in the present application. In this embodiment, the decoder includes a decoder attention module, multiple expert modules 1-K and a gated network module. Among them, each language corresponds to multiple expert modules (for example, Top-1 or Top-2 expert modules). In the training stage, an expert head is selected according to a given language label, the attention score of the target token sequence for the encoder output is calculated, and the final recognition result is output.

[0035] For example, the attention module calculates the time alignment information of the token sequence and the encoder output. The attention output is sent to K expert modules respectively, and the gated network calculates the weight of the corresponding expert according to the given language prior information, thereby summarizing the expert results.

[0036] For example, in a decoder based on a hybrid expert model, one or more expert modules are assigned to each language, and only a small number of expert modules (usually Top-1 or Top-2) are activated for each input token. Each expert module can learn a specific type of input features, and this distributed learning can improve generalization capabilities. Although the hybrid expert model has a large number of total parameters, only one language expert module is activated each time, and only a small number of parameters are calculated. This sparse activation mechanism ensures that the computational efficiency and the number of parameters actually used do not increase linearly with the number of experts.

[0037] In the hybrid expert model, the core task of the gating network is to select the appropriate expert module for each input token. The gating network generates a weight distribution based on the language and input to control which expert modules participate in the calculation and the contribution of each expert module. The following is the working principle and calculation steps of the gating network: 1. Linear transformation. The gating network transforms the input x Apply a linear transformation to map the input features to K Dimensions of expert modules: in, W gate ∈R n×d is the weight matrix of the gating network, b gate ∈R n is the bias vector.

[0038] 2. Apply a language mask. Use the mask to filter the expert scores and keep only the experts that are allowed to be activated: 3. Softmax activation. Calculate the weight of the masked expert module: in, g i ∈ [0,1] represents the i The weights of the expert modules, and .

[0039] 4. Sparseness. Limit the maximum number of activations m Experts can use Top-m Sparse, and renormalize the sparse weights: like Figure 4FIG. 1 is a flow chart of another embodiment of the multilingual speech recognition method of the present application. In this embodiment, the decoder re-scores the encoder output result according to the streaming nbest recognition result (for example, a token sequence), including: S41, the decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; S42, the decoder attention module is sent to multiple expert modules respectively to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

[0040] In some embodiments, the multilingual speech recognition method further includes: the gated network module receives the given language tag information (such as Figure 3 As shown, the weight of the corresponding expert module can be calculated by the language classification prediction value or the language tag information determined by the user setting the language, and the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result is output; the target expert module is the expert module selected according to the weight; the streaming nbest recognition results are reordered according to the attention score to obtain the final recognition result. Exemplarily, the weight value of the expert module is 0 or g i , when the current speech is a certain language, the weight value of the expert module corresponding to the certain language is g i , the weight values ​​of other expert modules are 0.

[0041] Exemplarily, in the decoding phase, the encoder and language classification model work in the same way as in the training phase. CTC outputs nbest streaming decoding results based on WFST and beam search algorithms. The decoder can select an expert head based on the language prior information given by the user or the prediction results of the language classification model, calculate the attention score of each CTC decoding result for the encoder output, reorder the nbest results based on the score, and obtain the result with the highest score as the decoder recognition output.

[0042] For example, since the expert modules in the decoder are bound to the language, in a scenario where the language used by the user is known, the expert modules corresponding to the language not used by the user can be trimmed to reduce the number of parameters when the model is deployed.

[0043] like Figure 5 , which is a schematic diagram of an embodiment of the input and output structure of the decoder in the present application. The input of the decoder includes the encoder output, nbest results 1-N, language classification prediction value or user-set language, and the output of the decoder includes scores 1 to N, and the final recognition result is output after reordering the scores 1 to N.

[0044] In this embodiment, the streaming reasoning of the CTC module is combined with the decoder re-scoring. Since the language of the decoder is known through the language label information, errors in the nbest result of the CTC module can be corrected, thus overcoming the problem of language confusion in the streaming reasoning stage of the CTC module due to the lack of language prior information.

[0045] In this embodiment, a hybrid expert model is used to enhance generalization and avoid parameter expansion during model deployment; prior knowledge (language label information) is introduced into the hybrid expert model, and prior information such as language is used to assist expert selection, thereby obtaining more accurate recognition results.

[0046] like Figure 6 The flowchart of another embodiment of the multilingual speech recognition method of the present application is shown, and the embodiment includes the following steps: Step 1: Prepare the audio to be recognized. Frame the audio, add windows, and extract features. Obtain n-dimensional acoustic features for each frame. The features here can be fbank (FilterBank), mfcc (Mel Frequency Cepstrum Coefficients), or plp (perceptual linear prediction), etc. The frame length can be 20-40ms, and the frame shift can be 10-20ms.

[0047] Step 2: Under the premise that the language is unknown, the CTC module of the multilingual speech recognition model calculates the streaming nbest recognition results.

[0048] Step 3: If the user specifies a language, the user-specified language is directly used as the language corresponding to the audio to be recognized; otherwise, the language classification model is used to predict the language corresponding to the audio to be recognized.

[0049] Step 4: According to the language corresponding to the audio to be recognized, activate the specified language expert module of the decoder and re-score the CTC streaming nbest recognition results.

[0050] Step 5: Sort the CTC streaming nbest recognition results according to the decoder re-scores, and output the recognition result with the highest score.

[0051] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0052] In a second aspect, the embodiment of the present application provides a multilingual speech recognition model, characterized in that it includes: The encoder is configured to: encode the acoustic features of the audio to be recognized to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; The language classification model is configured as follows: performing a preset process on the encoder output result to obtain a language classification result; The CTC module is configured as follows: combining a weighted finite state transition machine (WFST) to map the encoder output results to streaming nbest recognition results through a prefix beam search algorithm; The decoder is configured to re-score the encoder output result according to the streaming nbest recognition result.

[0053] In some embodiments, the encoder includes a convolution module and a self-attention module, the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features.

[0054] In some embodiments, the encoder output result is subjected to preset processing to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain the language classification result.

[0055] In some embodiments, the decoder includes a decoder attention module, a plurality of expert modules and a gated network module; the decoder re-scoring the encoder output result according to the streaming nbest recognition result includes: The decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; The decoder attention module is respectively fed into multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

[0056] In some embodiments, it also includes: the gated network module calculates the weight of the corresponding expert module according to the language classification result or the user-specified language, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result; the target expert module is an expert module selected according to the weight; the streaming nbest recognition results are reordered according to the attention score to obtain the final recognition result.

[0057] In some embodiments, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of any one of the multilingual speech recognition methods described in the present application.

[0058] In some embodiments, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of any one of the multilingual speech recognition methods described in the present application.

[0059] In some embodiments, an embodiment of the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of any multilingual speech recognition method described in the present application are implemented.

[0060] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for executing a multilingual speech recognition method provided by another embodiment of the present application. Figure 7 As shown, the device includes: One or more processors 710 and memory 720, Figure 7 A processor 710 is taken as an example.

[0061] The apparatus for executing the multi-language speech recognition method may further include: an input device 730 and an output device 740 .

[0062] The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0063] The memory 720 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multi-language speech recognition method in the embodiment of the present application. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the multi-language speech recognition method in the above method embodiment is implemented.

[0064] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the multilingual speech recognition device, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the multilingual speech recognition device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The input device 730 can receive input digital or character information and generate signals related to user settings and function control of the multi-language speech recognition device. The output device 740 can include display devices such as a display screen.

[0066] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, the multilingual speech recognition method in any of the above method embodiments is executed.

[0067] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0068] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0069] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0070] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0071] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to the general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.

[0072] (5) Other electronic devices with data interaction functions.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multilingual speech recognition method, used for a multilingual speech recognition model, characterized in that: The multilingual speech recognition model includes: an encoder, a language classification model, a CTC module and a decoder; the method includes: Acquiring the acoustic features of the audio to be recognized and inputting them into the encoder to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; The language classification model performs preset processing on the encoder output result to obtain a language classification result; The CTC module combines a weighted finite state transition machine to map the encoder output results to streaming nbest recognition results through a prefix beam search algorithm; The decoder re-scores the encoder output result according to the streaming nbest recognition result.

2. The method according to claim 1, characterized in that The encoder includes a convolution module and a self-attention module, wherein the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features.

3. The method according to claim 1, characterized in that: The language classification model performs preset processing on the output result of the encoder to obtain a language classification result, including: The language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain a language classification result.

4. The method according to claim 1, characterized in that: The decoder includes a decoder attention module, a plurality of expert modules and a gated network module; The decoder re-scoring the encoder output result according to the streaming nbest recognition result includes: The decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; The decoder attention module is respectively fed into multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

5. The method according to claim 4, characterized in that Also includes: The gated network module calculates the weight of the corresponding expert module according to the given language label information, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result; The target expert module is an expert module selected according to the weight; The streaming nbest recognition results are reordered according to the attention scores to obtain a final recognition result.

6. A multilingual speech recognition model, characterized in that: include: The encoder is configured to: encode the acoustic features of the audio to be recognized to obtain an encoder output result including local information of the acoustic features and global information of the acoustic features; The language classification model is configured as follows: performing a preset process on the encoder output result to obtain a language classification result; The CTC module is configured as follows: combining a weighted finite state transition machine to map the encoder output results to streaming nbest recognition results through a prefix beam search algorithm; The decoder is configured to re-score the encoder output result according to the streaming nbest recognition result.

7. The model according to claim 6, characterized in that The encoder includes a convolution module and a self-attention module, wherein the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features.

8. The model according to claim 6, characterized in that The encoder output result is subjected to preset processing to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operations on the encoder output result to obtain the language classification result.

9. The model according to claim 6, characterized in that The decoder includes a decoder attention module, a plurality of expert modules and a gated network module; The decoder re-scoring the encoder output result according to the streaming nbest recognition result includes: The decoder attention module calculates the time alignment information of the streaming nbest recognition result and the encoder output result; The decoder attention module is respectively fed into multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

10. The model according to claim 9, characterized in that Also includes: The gated network module calculates the weight of the corresponding expert module according to the language classification result or the language specified by the user, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result; The target expert module is an expert module selected according to the weight; The streaming nbest recognition results are reordered according to the attention scores to obtain a final recognition result.

11. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images