Multilingual speech recognition method, model, electronic device, storage medium, and program product

Abstract

The application discloses a multilingual speech recognition method, and a multilingual speech recognition model comprises an encoder, a language classification model, a CTC module and a decoder; the method comprises the following steps: obtaining acoustic feature of audio to be recognized and inputting the acoustic feature into the encoder to obtain an encoder output result containing local information of the acoustic feature and global information of the acoustic feature; the language classification model performs preset processing on the encoder output result to obtain a language classification result; the CTC module combines a weighted finite state transducer, and maps the encoder output result to a streaming nbest recognition result through a prefix beam search algorithm; and the decoder re-scores the encoder output result according to the streaming nbest recognition result. The multilingual speech recognition model only shares the bottom-layer encoder, and the top-layer language classification model, the CTC module and the decoder are independent. Through the shared bottom-layer parameters and the re-scoring process, the problem of confusing homophonic characters of different languages in speech recognition is reduced, and the language classification accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-language speech recognition method and model, an electronic device, a storage medium and a program product. BACKGROUND

[0002] The general multi-language speech recognition system architecture is the same as the single-language speech recognition system, and the difference is only that the acoustic and language models are trained using mixed multi-language data or single-language data. Such a system can meet the application requirements of simple scenarios. However, in complex scenarios with a large number of languages, small differences between languages, and non-standard pronunciation of users, such a system cannot achieve accurate recognition, and it is also difficult to balance the model complexity and performance. SUMMARY

[0003] The applicant tried the following schemes in the process of implementing the present application:

[0004] Scheme 1: only using an encoder + CTC module for streaming recognition, setting a language-aware multi-head expert inside the encoder, and using a language classification model consistent with the present application without a decoder. The advantages are: simple system structure, less resource occupation, lower latency, and higher concurrency. The disadvantages are: the language classification model results cannot be applied to the speech recognition model; and the encoder and CTC module are in a streaming structure, which does not have a global receptive field like the re-scoring scheme, and the recognition rate is low.

[0005] Scheme 2: the overall model structure is the same as the present application, and the difference is only that the decoder is autoregressive decoding instead of re-scoring. The advantage is: higher recognition rate. The disadvantage is: autoregressive decoding significantly increases the latency and affects the concurrency.

[0006] The embodiments of the present application provide a multi-language speech recognition method, model, electronic device, storage medium and program product, which are used to at least solve one of the above technical problems.

[0007] In a first aspect, the embodiments of the present application provide a multi-language speech recognition method, characterized in that the multi-language speech recognition model comprises an encoder, a language classification model, a CTC module and a decoder; and the method comprises:

[0008] obtaining the acoustic features of the audio to be recognized as input to the encoder to obtain an encoder output result containing local acoustic feature information and global acoustic feature information;

[0009] the language classification model performs a preset processing on the encoder output result to obtain a language classification result;

[0010] the CTC module combines a weighted finite state transducer to map the encoder output result to a streaming nbest recognition result through a prefix beam search algorithm;

[0011] The decoder re-scores the encoder output result according to the streaming nbest recognition result.

[0012] In some embodiments, the encoder comprises a convolution module for counting local information of acoustic features and a self-attention module for counting global information of acoustic features.

[0013] In some embodiments, the language classification model performs preset processing on the encoder output result to obtain a language classification result, comprising: the language classification model performs mean calculation, variance calculation and pooling operation on the encoder output result to obtain the language classification result.

[0014] In some embodiments, the decoder comprises a decoder attention module, a plurality of expert modules and a gating network module; the decoder re-scores the encoder output result according to the streaming nbest recognition result, comprising:

[0015] The decoder attention module calculates time alignment information of the streaming nbest recognition result and the encoder output result.

[0016] The decoder attention module is respectively sent into a plurality of expert modules to jointly calculate attention scores of the streaming nbest recognition result for the encoder output result.

[0017] In some embodiments, the multi-language speech recognition method further comprises: the gating network module calculates a weight of a corresponding expert module according to a given language label information, and outputs an attention score of the streaming nbest recognition result for the encoder output result calculated by a target expert module; the target expert module is an expert module selected according to the weight; and the streaming nbest recognition result is re-ordered according to the attention score to obtain a final recognition result.

[0018] In a second aspect, the embodiments of the present application provide a multi-language speech recognition model, characterized in that comprising:

[0019] The encoder is configured to encode acoustic features of the audio to be recognized to obtain an encoder output result containing local information of acoustic features and global information of acoustic features.

[0020] The language classification model is configured to perform preset processing on the encoder output result to obtain a language classification result.

[0021] The CTC module is configured to combine a weighted finite state transducer and map the encoder output result to a streaming nbest recognition result through a prefix beam search algorithm.

[0022] The decoder is configured to re-score the encoder output result according to the streaming nbest recognition result.

[0023] In some embodiments, the encoder comprises a convolution module and a self-attention module, the convolution module is configured to count local information of acoustic features, and the self-attention module is configured to count global information of acoustic features.

[0024] In some embodiments, the preset processing of the encoder output result obtains a language classification result, comprising: the language classification model performs mean calculation, variance calculation and pooling operation on the encoder output result to obtain the language classification result.

[0025] In some embodiments, the decoder comprises a decoder attention module, a plurality of expert modules and a gating network module; the decoder re-scores the encoder output result according to the streaming nbest recognition result, comprising:

[0026] The decoder attention module calculates time alignment information of the streaming nbest recognition result and the encoder output result.

[0027] The decoder attention module is respectively sent into a plurality of expert modules, and the streaming nbest recognition result and the encoder output result are jointly calculated to obtain attention scores.

[0028] In some embodiments, the gating network module calculates the weight of the corresponding expert module according to the language classification result or the user-specified language, and outputs the attention score of the streaming nbest recognition result for the encoder output result calculated by the target expert module; the target expert module is the expert module selected according to the weight; and the final recognition result is obtained by reordering the streaming nbest recognition result according to the attention score.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the multilingual speech recognition method in any one of the embodiments of the present application.

[0030] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the multilingual speech recognition method in any one of the embodiments of the present application.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer programs / instructions, characterized by that the computer programs / instructions, when executed by a processor, implement the steps of the multilingual speech recognition method according to any one of the embodiments of the present application.

[0032] In the multilingual speech recognition model in the embodiment, only the underlying encoder is shared, and the top language classification model, the CTC module and the decoder are independent. The underlying shared parameters can reduce the problem of confusing homophonic characters of different languages in speech recognition, and can also improve the language classification accuracy. The independent top parameters can enable parallel computing of the two tasks, and reduce the delay. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0034] Figure 1 FIG. 1 is a schematic diagram of an embodiment of the multilingual speech recognition model of the present application;

[0035] Figure 2 FIG. 2 is a flowchart of an embodiment of the multilingual speech recognition method of the present application;

[0036] Figure 3 FIG. 3 is a schematic diagram of an embodiment of the decoder in the present application;

[0037] Figure 4 FIG. 4 is a flowchart of another embodiment of the multilingual speech recognition method of the present application;

[0038] Figure 5 FIG. 5 is a schematic diagram of an embodiment of the input and output structure of the decoder in the present application;

[0039] Figure 6 FIG. 6 is a flowchart of another embodiment of the multilingual speech recognition method of the present application;

[0040] Figure 7 FIG. 7 is a structural schematic diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0041] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0042] It should also be noted that, in this document, the terms "comprising" and "including" not only include those elements, but also include other elements not explicitly listed, or include elements inherent to the process, method, article or device. Without more limitations, the elements defined by the statement "comprising" do not exclude the presence of other identical elements in the process, method, article or device comprising the elements.

[0043] In order to solve at least one problem in the prior art, the present application provides a multi-language speech recognition model, which can be used to implement a multi-language speech recognition method, and can be specifically implemented as a multi-language streaming speech recognition method based on an improved hybrid expert model and attention re-scoring. The method uses a hybrid expert model to enhance generalization and avoid parameter expansion when deploying the model; prior knowledge is introduced into the hybrid expert model to assist expert selection using prior information such as language; a high-precision streaming recognition is realized based on a CTC+attention re-scoring scheme; the language classification model and the recognition model only share the underlying encoder, and the upper layers are independent.

[0044] In order to solve at least one problem in the prior art, the present application provides a multi-language speech recognition model, which is based on a hybrid CTC / attention (Hybrid CTC / attention) neural network of an improved hybrid expert model. As shown in Figure 1 The multi-language speech recognition model of an embodiment of the present application includes four parts: an encoder, a decoder, a CTC module and a language classification model.

[0045] As shown in Figure 2 The embodiment of the present application provides a multi-language speech recognition method, which includes:

[0046] S10, obtaining the acoustic feature of the audio to be recognized and inputting the acoustic feature into the encoder to obtain an encoder output result containing local acoustic feature information and global acoustic feature information;

[0047] S20, the language classification model performs a preset processing on the encoder output result to obtain a language classification result. In the training phase of the multi-language speech recognition model, the to-be-recognized audio can be replaced by a sample audio, and a language classification cross-entropy is determined in combination with language label information of the sample audio to optimize the language classification model.

[0048] S30, a CTC module combines a weighted finite state transducer (WFST) to map the encoder output result to a streaming nbest recognition result through a prefix beam search algorithm. In the training phase of the multi-language speech recognition model, the CTC module can be optimized through the prefix beam search algorithm in combination with the weighted finite state transducer (WFST) according to a CTC loss. In addition, the CTC loss is back-propagated to the encoder for training the encoder in the training phase.

[0049] S40, the decoder re-scores the encoder output result according to the streaming nbest recognition result. In the training phase of the multi-language speech recognition model, the to-be-recognized audio can be replaced by a sample audio, and a decoder cross-entropy is calculated according to language label information of the sample audio to optimize the decoder.

[0050] In the embodiment, the multi-language speech recognition model only shares the bottom-layer encoder, and the top-layer language classification model, the CTC module and the decoder are independent. The shared bottom-layer parameters can reduce the problem of confusing homophonic characters of different languages in speech recognition, and can also improve the language classification accuracy. The independent top-layer parameters can enable parallel computing of the two tasks and reduce the delay.

[0051] In some embodiments, the encoder includes a convolution module and a self-attention module, the convolution module is used to count local information of acoustic features, and the self-attention module is used to count global information of acoustic features. Illustratively, the acoustic feature input encoder uses the convolution module and the self-attention module to count local and global information of acoustic features respectively to obtain multi-frame encoding output. The local information of acoustic features refers to the time-frequency correlation in a short time in the speech signal, which is mainly completed by the convolution module. Extracting local timing patterns can improve the sensitivity of the model to short-term changes, such as rapid acoustic changes. The global information of acoustic features refers to the context dependence and long-distance relationship in a long time range in the speech signal, which is mainly completed by the self-attention mechanism. The model has the ability to capture fine-grained local patterns and long-distance context dependence.

[0052] In some embodiments, the language classification model performs a preset processing on the encoder output result to obtain a language classification result, including: the language classification model performs mean calculation, variance calculation and pooling operation on the encoder output result to obtain a language classification result.

[0053] Exemplarily, assume the input feature is a two-dimensional vector: X e R T×D

[0054] where, T is the number of time steps, D is the feature dimension, R is the real number set.

[0055] 1. Attention weight calculation. The attention mechanism generates an attention weight for each time step t :

[0056]

[0057] where, W e R d×D , b e R d , w e R d is a learnable weight, d is the attention dimension.

[0058] 2. Weighted mean calculation. Using the attention weight α t , the feature is weighted and summed to obtain the mean of the global representation:

[0059]

[0060] 3. Weighted standard deviation calculation:

[0061]

[0062] 4. Mean and standard deviation concatenation:

[0063]

[0064] 5. Fully connected layer + Softmax activation. The fully connected layer maps the high-dimensional feature vector to the dimension of the number of categories, and the Softmax activation function converts the output of the linear layer o to a probability distribution p :

[0065]

[0066]

[0067] where, C is the total number of categories.

[0068] In some embodiments, the multilingual speech recognition model of the present application can be implemented as a hybrid CTC / attention neural network based on an improved hybrid expert model. As shown in Figure 3 FIG. 1 is a schematic diagram of an embodiment of the decoder in the present application. The decoder in this embodiment includes a decoder attention module, a plurality of expert modules 1-K, and a gating network module. Each language corresponds to a plurality of expert modules (e.g., Top-1 or Top-2 expert modules). In the training phase, the expert head is selected according to the given language label, the attention score of the target token sequence for the encoder output is calculated, and the final recognition result is output.

[0069] For example, the attention module calculates the time alignment information of the token sequence and the encoder output. The attention output is sent into K expert modules, respectively, and the gating network calculates the weight of the corresponding expert according to the given language prior information, thereby summarizing the expert results.

[0070] For example, in the decoder based on the hybrid expert model, one or more expert modules are allocated for each language, and each input token only activates a small number of expert modules (usually Top-1 or Top-2). Each expert module can learn specific types of input features, and this distributed learning can improve the generalization ability. Although the total parameter amount of the hybrid expert model is large, only the expert modules of one language are activated each time, and only a small part of the parameters are calculated. This sparse activation mechanism ensures that the calculation efficiency and the actual parameter amount used do not increase linearly with the number of experts.

[0071] In the hybrid expert model, the core task of the gating network is to select the appropriate expert module for each input token. The gating network generates a weight distribution according to the language and the input, which is used to control which expert modules participate in the calculation and the contribution degree of each expert module. The working principle and calculation steps of the gating network are as follows:

[0072] 1. Linear transformation. The gating network applies linear transformation to the input x to map the input features to the dimension of the K expert modules:

[0073]

[0074] wherein, W gate ∈R n×d is the weight matrix of the gating network, b gate ∈R n is the bias vector.

[0075] 2. Apply a language mask. Filter expert scores using a mask, retaining only those experts who are allowed to activate:

[0076]

[0077] 3. Softmax activation. Calculate the weights of the masked expert modules:

[0078]

[0079] in, g i ∈ [0,1] represents the first... i The weights of each expert module, and .

[0080] 4. Sparsity. Limit the maximum number of activations. m An expert can use Top-m Sparsification is performed, and the sparsified weights are then renormalized:

[0081]

[0082] like Figure 4 The diagram shown is a flowchart of another embodiment of the multilingual speech recognition method of this application. In this embodiment, the decoder re-scores the encoder output based on the streaming nbest recognition result (e.g., a token sequence), including:

[0083] S41, The decoder attention module calculates the time alignment information between the streaming nbest recognition result and the encoder output result;

[0084] S42. The decoder attention module sends the data to multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result.

[0085] In some embodiments, the multilingual speech recognition method further includes: the gating network module classifying speech based on given language label information (such as...). Figure 3 As shown, the weights of the corresponding expert modules can be calculated using language classification prediction values ​​or user-defined language label information, and the attention score of the streaming nbest recognition result calculated by the target expert module relative to the encoder output is output. The target expert module is the expert module selected based on the weights. The streaming nbest recognition results are reordered based on the attention score to obtain the final recognition result. For example, the weight of the expert module can be 0 or... g i, the weight value of the expert module corresponding to the certain language is g i the weight value of the other expert modules is 0.

[0086] Exemplarily, in the decoding stage, the encoder and the language classification model work in the same way as in the training stage. The CTC outputs nbest streaming decoding results based on the WFST and the beam search algorithm. The decoder can select the expert head according to the language prior information given by the user or according to the prediction result of the language classification model, calculate the attention score of each CTC decoding result for the encoder output, reorder the nbest results according to the scores, and obtain the result with the highest score as the recognition output of the decoder.

[0087] Exemplarily, since the expert modules in the decoder are bound to the language, in the scenario where the user uses a known language, the expert modules corresponding to the languages not used by the user can be pruned, thereby reducing the parameter amount in model deployment.

[0088] As shown in Figure 5 , it is a schematic diagram of an embodiment of the input and output structure of the decoder in the present application. The input of the decoder includes the encoder output, nbest results 1-N, language classification prediction values or user-set languages, and the output of the decoder includes scores 1 to N. After reordering the scores 1 to N, the final recognition result is output.

[0089] In the present embodiment, the CTC module streaming inference is combined with the decoder rescoring. Since the decoder is known to be of a certain language through the language label information, the nbest results of the CTC module can be corrected, thereby overcoming the problem of language confusion in the streaming inference stage of the CTC module without language prior information.

[0090] In the present embodiment, the mixed expert model is used to enhance the generalization and avoid the parameter amount inflation in model deployment. The prior knowledge (language label information) is introduced into the mixed expert model, and the language and other prior information is used to assist the expert selection, thereby obtaining more accurate recognition results.

[0091] As shown in Figure 6 , it is a flowchart of another embodiment of the multilingual speech recognition method of the present application. The embodiment includes the following steps:

[0092] Step 1: Prepare the audio to be recognized. Frame the audio, window it, and extract features, and obtain n-dimensional acoustic features for each frame. Here, the features can be fbank (Filter Bank), mfcc (Mel Frequency Cepstrum Coefficients), or plp (perceptual linear prediction), etc., the frame length can be 20-40 ms, and the frame shift can be 10-20 ms.

[0093] Step 2: Under the premise that the language is unknown, the CTC module of the multi-language speech recognition model calculates the streaming nbest recognition result.

[0094] Step 3: If the user specifies the language, then directly use the user-specified language as the corresponding language of the audio to be recognized, otherwise use the language classification model to predict the corresponding language of the audio to be recognized.

[0095] Step 4: According to the corresponding language of the audio to be recognized, activate the specified language expert module of the decoder, and re-score the CTC streaming nbest recognition result.

[0096] Step 5: Sort the CTC streaming nbest recognition result according to the re-scoring of the decoder, and output the recognition result with the highest score.

[0097] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0098] In a second aspect, the embodiments of the present application provide a multi-language speech recognition model, characterized in that it comprises:

[0099] The encoder is configured to encode the acoustic features of the audio to be recognized to obtain an encoder output result containing local information of the acoustic features and global information of the acoustic features;

[0100] The language classification model is configured to perform a preset processing on the encoder output result to obtain a language classification result;

[0101] The CTC module is configured to map the encoder output result to a streaming nbest recognition result through a prefix beam search algorithm in combination with a weighted finite state transducer (WFST).

[0102] The decoder is configured to re-score the encoder output result according to the streaming nbest recognition result.

[0103] In some embodiments, the encoder comprises a convolution module for counting local information of acoustic features and a self-attention module for counting global information of acoustic features.

[0104] In some embodiments, the preset processing of the encoder output result obtains a language classification result, comprising: the language classification model performs mean calculation, variance calculation and pooling operation on the encoder output result to obtain the language classification result.

[0105] In some embodiments, the decoder comprises a decoder attention module, a plurality of expert modules and a gating network module; the decoder re-scores the encoder output result according to the streaming nbest recognition result, comprising:

[0106] The decoder attention module calculates time alignment information of the streaming nbest recognition result and the encoder output result.

[0107] The decoder attention module is respectively fed into a plurality of expert modules to jointly calculate attention scores of the streaming nbest recognition result on the encoder output result.

[0108] In some embodiments, the gating network module calculates a weight of a corresponding expert module according to the language classification result or a user-specified language, and outputs an attention score of the streaming nbest recognition result on the encoder output result calculated by a target expert module; the target expert module is an expert module selected according to the weight; and the streaming nbest recognition result is reordered according to the attention score to obtain a final recognition result.

[0109] In some embodiments, the present application provides an electronic device, comprising: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of any one of the multi-language speech recognition methods.

[0110] In some embodiments, the present application provides a computer-readable storage medium having stored thereon computer programs / instructions, characterized in that the computer programs / instructions are executed by a processor to implement the steps of any one of the multi-language speech recognition methods.

[0111] In some embodiments, the present application provides a computer program product comprising computer programs / instructions, characterized in that the computer programs / instructions, when executed by a processor, implement the steps of the multilingual speech recognition method according to any one of the embodiments of the present application.

[0112] Figure 7 is a schematic diagram of the hardware structure of an electronic device for performing the multilingual speech recognition method according to another embodiment of the present application, as shown in Figure 7 The device comprises:

[0113] one or more processors 710 and a memory 720, Figure 7 In the above description, the processor 710 is taken as an example.

[0114] The device for performing the multilingual speech recognition method can further comprise an input device 730 and an output device 740.

[0115] The processor 710, the memory 720, the input device 730 and the output device 740 can be connected by a bus or other means, Figure 7 In the above description, the connection by the bus is taken as an example.

[0116] The memory 720, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the multilingual speech recognition method according to the embodiments of the present application. The processor 710 performs various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, implements the multilingual speech recognition method according to the method embodiment.

[0117] The memory 720 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function; the data storage area can store data created according to the use of the multilingual speech recognition device, etc. In addition, the memory 720 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 can optionally include a memory remotely arranged with respect to the processor 710, and these remote memories can be connected to the multilingual speech recognition device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0118] The input device 730 can receive input digital or character information, and generate signals related to the user settings and function control of the multilingual speech recognition device. The output device 740 can include a display device such as a display screen.

[0119] The one or more modules are stored in the memory 720 and, when executed by the one or more processors 710, perform the multilingual speech recognition method of any of the method embodiments described above.

[0120] The product described above can perform the method provided in the embodiments of the present application, has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided in the embodiments of the present application.

[0121] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0122] (1) Mobile communication device: The feature of this kind of device is to have mobile communication function, and to provide voice and data communication as the main target. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0123] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has the functions of calculation and processing, and generally also has the characteristics of mobile Internet. This kind of terminal includes: PDA, MID and UMPC device, such as iPad.

[0124] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player (such as iPod), palm game machine, electronic book, and smart toy and portable car navigation device.

[0125] (4) Server: The device providing calculation service, the constitution of server includes processor, hard disk, memory, system bus, etc. The server and general computer architecture are similar, but since it needs to provide high reliable service, the requirements in processing capacity, stability, reliability, security, scalability, manageability, etc. are higher.

[0126] (5) Other electronic devices with data interaction function.

[0127] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0128] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software plus a general hardware platform from the above description of the embodiments, and of course, the various embodiments can also be implemented by hardware. Based on such an understanding, the above technical solutions, essentially or in other words, the part that contributes to the related art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A multilingual speech recognition method for a multilingual speech recognition model, characterized in that, The multilingual speech recognition model includes: an encoder, a language classification model, a CTC module, and a decoder; the decoder includes a decoder attention module, multiple expert modules, and a gating network module; the method includes: Acoustic features of the audio to be recognized are input into the encoder to obtain encoder output containing local and global acoustic feature information; The language classification model performs preset processing on the encoder output to obtain the language classification result; The CTC module, combined with a weighted finite state transition machine, maps the encoder output to the streaming nbest recognition result through a prefix beam search algorithm. The decoder attention module calculates the time alignment information between the streaming nbest recognition result and the encoder output result; The decoder attention module sends the data to multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result. The gating network module calculates the weights of the corresponding expert modules based on the given language classification results, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module relative to the encoder output result; the target expert module is the expert module selected based on the weights. The streaming nbest recognition results are reordered based on the attention scores to obtain the final recognition result.

2. The method according to claim 1, characterized in that, The encoder includes a convolution module and a self-attention module. The convolution module is used to collect local acoustic feature information, and the self-attention module is used to collect global acoustic feature information.

3. The method according to claim 1, characterized in that, The language classification model performs pre-defined processing on the encoder output to obtain the language classification result, including: The language classification model performs mean calculation, variance calculation, and pooling operations on the encoder output to obtain the language classification result.

4. A multilingual speech recognition model, characterized in that, include: The encoder is configured to encode the acoustic features of the audio to be recognized to obtain an encoder output that includes both local and global acoustic feature information. The language classification model is configured as follows: the language classification result is obtained by performing preset processing on the encoder output result; The CTC module is configured to combine a weighted finite state transition machine and use a prefix beam search algorithm to map the encoder output to the streaming nbest recognition result. The decoder includes a decoder attention module, multiple expert modules, and a gating network module; the decoder is configured as follows: The decoder attention module calculates the time alignment information between the streaming nbest recognition result and the encoder output result; The decoder attention module sends the data to multiple expert modules to jointly calculate the attention score of the streaming nbest recognition result for the encoder output result. The gating network module calculates the weight of the corresponding expert module based on the language classification result or the user-specified language, and outputs the attention score of the streaming nbest recognition result calculated by the target expert module for the encoder output result. The target expert module is an expert module selected based on the weights; The streaming nbest recognition results are reordered based on the attention scores to obtain the final recognition result.

5. The model according to claim 4, characterized in that, The encoder includes a convolution module and a self-attention module. The convolution module is used to collect local acoustic feature information, and the self-attention module is used to collect global acoustic feature information.

6. The model according to claim 4, characterized in that, The language classification result is obtained by performing preset processing on the encoder output result, including: the language classification model performs mean calculation, variance calculation and pooling operation on the encoder output result to obtain the language classification result.

7. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-3.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-3.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-3.

Images