Multi-dialect speech recognition model training method and device and electronic equipment

Abstract

The invention relates to a multi-dialect speech recognition model training method and device and electronic equipment, and particularly relates to the technical field of speech recognition. Comprising the steps that sample voice and labeled voice are acquired, the sample voice comprises mandarin voice and dialect voice of different types, and the labeled voice is real text content and real language type corresponding to the sample voice; the sample voice and the labeled voice are input into an initial voice recognition model, and the initial voice recognition model comprises an initial voice recognition module and an initial dialect classification module; and training model parameters of the initial speech recognition model according to output results of the initial speech recognition module and the initial dialect classification module to obtain a convergent multi-dialect speech recognition model. According to the embodiment of the invention, the robustness of the multi-dialect speech recognition model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition, and in particular to a training method, device and electronic equipment for a multi-dialect speech recognition model. Background Art

[0002] Speech recognition technology has been widely used in our daily life. In the actual speech interaction process, dialects are still a technical difficulty that cannot be avoided in speech recognition. At present, there are many dialects in my country, and it is more difficult to obtain dialect data than Mandarin. Therefore, the amount of dialect data is small, which leads to the problem of data sparsity in training multi-dialect speech recognition models. The trained models have poor robustness and poor recognition effect. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a training method, device and electronic device for a multi-dialect speech recognition model, which can distinguish different language types and accurately recognize speech text content of different language types.

[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a first aspect, the present disclosure provides a method for training a multi-dialect speech recognition model, the method comprising:

[0006] Obtain sample speech and annotated speech, where the sample speech includes Mandarin speech and different types of dialect speech; the annotated speech is the real text content and real language type corresponding to the sample speech;

[0007] Inputting the sample speech and the annotated speech into an initial speech recognition model; wherein the initial speech recognition model includes an initial speech recognition module and an initial dialect classification module;

[0008] According to the output results of the initial speech recognition module and the initial dialect classification module, the model parameters of the initial speech recognition model are trained to obtain a converged multi-dialect speech recognition model.

[0009] As an optional implementation of the present disclosure, the initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer; the initial speech recognition module outputs the first loss function value, and the initial dialect classification module outputs the second loss function value;

[0010] According to the output results of the initial speech recognition module and the initial dialect classification module, the model parameters of the initial speech recognition model are trained to obtain a converged multi-dialect speech recognition model, including:

[0011] When the first loss function value and / or the second loss function value is greater than the preset loss threshold, return to adjust the model parameters of the initial speech recognition model; until the first loss function value and the second loss function value are both less than or equal to the preset loss threshold, a multi-dialect speech recognition model is trained.

[0012] As an optional implementation of the present disclosure, the initial speech recognition model further includes: an initial speech preprocessing module connected to the initial speech recognition module and the initial dialect classification module;

[0013] After the sample speech and the annotated speech are input into the initial speech recognition model, the model parameters of the initial speech recognition model are trained according to the output results of the initial speech recognition module and the initial dialect classification module. Before a converged multi-dialect speech recognition model is obtained, the following steps are included:

[0014] Preprocessing the sample speech through the initial speech preprocessing module to obtain a three-dimensional matrix, and the three-dimensional matrix represents the acoustic features of the sample speech;

[0015] The three-dimensional matrix is ​​decoded to obtain predicted text features corresponding to the sample speech, and the first loss function layer calculates the first loss function value according to the predicted text features and the real text content;

[0016] The three-dimensional matrix is ​​classified and processed to obtain the predicted language type features corresponding to the sample speech, and the second loss function layer calculates the second loss function value according to the predicted language type features and the true language type.

[0017] As an optional implementation of the present disclosure, the three-dimensional matrix is ​​classified to obtain the predicted language type features corresponding to the sample speech, including:

[0018] Performing pooling processing on the three-dimensional matrix, and converting the three-dimensional matrix into a first matrix;

[0019] Calculate the attention weight matrix corresponding to the first matrix, and multiply the attention weight matrix and the three-dimensional matrix to obtain the calculation result;

[0020] The calculation results are weighted and summed on the preset dimension of the three-dimensional matrix to obtain the language type features corresponding to each sentence in the sample speech;

[0021] The language type features corresponding to each sentence in the sample speech are decoded to determine the predicted language type features.

[0022] As an optional implementation of the present disclosure, after obtaining the sample speech and the annotated speech, before inputting the sample speech and the annotated speech into the initial speech recognition model, the method includes:

[0023] Build the pending speech recognition module;

[0024] Perform speech recognition on the sample speech based on the pending speech recognition module, and obtain the error rate of the pending speech recognition module, where the error rate represents the probability that the text content predicted by the pending speech recognition module is different from the actual speech text content of the sample speech;

[0025] When the error rate is greater than or equal to the preset error rate, determining a special output node of the pending speech recognition module corresponding to the error rate;

[0026] The output nodes of the pending speech recognition module are added according to the special output nodes to obtain the initial speech recognition module.

[0027] As an optional implementation of the present disclosure, the method further includes: obtaining speech to be recognized; inputting the speech to be recognized into a multi-dialect speech recognition model to obtain speech text content and language type output by the multi-dialect speech recognition model.

[0028] As an optional implementation of the present disclosure, the first loss function layer is a connection time series classification loss function layer, and the second loss function layer is a cross entropy loss function layer.

[0029] In a second aspect, the present disclosure provides a training device for a multi-dialect recognition model, comprising:

[0030] The acquisition module is used to: acquire sample speech and annotated speech, wherein the sample speech includes Mandarin speech and different types of dialect speech; the annotated speech is the real text content and real language type corresponding to the sample speech;

[0031] The processing module is used to: input the sample speech and the annotated speech into the initial speech recognition model; wherein the initial speech recognition model includes: an initial speech recognition module and an initial dialect classification module;

[0032] The training module is used to: train the model parameters of the initial speech recognition model according to the output results of the initial speech recognition module and the initial dialect classification module to obtain a converged multi-dialect speech recognition model.

[0033] As an optional implementation of the present disclosure, the initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer; the initial speech recognition module outputs the first loss function value, and the initial dialect classification module outputs the second loss function value;

[0034] The training module is specifically used to: when the first loss function value and / or the second loss function value is greater than a preset loss threshold, return to adjust the model parameters of the initial speech recognition model; until the first loss function value and the second loss function value are both less than or equal to the preset loss threshold, a multi-dialect speech recognition model is trained.

[0035] As an optional implementation of the present disclosure, the initial speech recognition model further includes: an initial speech preprocessing module connected to the initial speech recognition module and the initial dialect classification module;

[0036] After the sample speech and the annotated speech are input into the initial speech recognition model, the model parameters of the initial speech recognition model are trained according to the output results of the initial speech recognition module and the initial dialect classification module. Before a converged multi-dialect speech recognition model is obtained, the following steps are included:

[0037] Preprocessing the sample speech through the initial speech preprocessing module to obtain a three-dimensional matrix, and the three-dimensional matrix represents the acoustic features of the sample speech;

[0038] The three-dimensional matrix is ​​decoded to obtain predicted text features corresponding to the sample speech, and the first loss function layer calculates the first loss function value according to the predicted text features and the real text content;

[0039] The three-dimensional matrix is ​​classified and processed to obtain the predicted language type features corresponding to the sample speech, and the second loss function layer calculates the second loss function value according to the predicted language type features and the true language type.

[0040] As an optional implementation of the present disclosure, the three-dimensional matrix is ​​classified to obtain the predicted language type features corresponding to the sample speech, including:

[0041] Performing pooling processing on the three-dimensional matrix, and converting the three-dimensional matrix into a first matrix;

[0042] Calculate the attention weight matrix corresponding to the first matrix, and multiply the attention weight matrix and the three-dimensional matrix to obtain the calculation result;

[0043] The calculation results are weighted and summed on the preset dimension of the three-dimensional matrix to obtain the language type features corresponding to each sentence in the sample speech;

[0044] The language type features corresponding to each sentence in the sample speech are decoded to determine the predicted language type features.

[0045] As an optional implementation of the present disclosure, after obtaining the sample speech and the annotated speech, before inputting the sample speech and the annotated speech into the initial speech recognition model, the method includes:

[0046] Build the pending speech recognition module;

[0047] Perform speech recognition on the sample speech based on the pending speech recognition module, and obtain the error rate of the pending speech recognition module, where the error rate represents the probability that the text content predicted by the pending speech recognition module is different from the actual speech text content of the sample speech;

[0048] When the error rate is greater than or equal to the preset error rate, determining a special output node of the pending speech recognition module corresponding to the error rate;

[0049] The output nodes of the pending speech recognition module are added according to the special output nodes to obtain the initial speech recognition module.

[0050] As an optional implementation of the present disclosure, the device is also used to: obtain speech to be recognized; input the speech to be recognized into a multi-dialect speech recognition model to obtain the speech text content and language type output by the multi-dialect speech recognition model.

[0051] As an optional implementation of the present disclosure, the first loss function layer is a connection time series classification loss function layer, and the second loss function layer is a cross entropy loss function layer.

[0052] In a third aspect, the present disclosure provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for training a multi-dialect speech recognition model as described in the first aspect or any one of its optional embodiments.

[0053] In a fourth aspect, the present disclosure provides a vehicle, comprising a training device for a multi-speech recognition model as described in the second aspect or any optional embodiment thereof, or an electronic device as described in the third aspect.

[0054] In a fifth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the training method of a multi-dialect speech recognition model as described in the first aspect or any one of its optional embodiments is implemented.

[0055] In a sixth aspect, the present disclosure provides a computer program product, comprising: when the computer program product is run on a computer, the computer is enabled to implement the training method for a multi-dialect speech recognition model as described in the first aspect or any one of its optional embodiments.

[0056] The training method of the multi-dialect speech recognition model provided by the embodiment of the present disclosure obtains sample speech including Mandarin speech and different types of dialect speech, and sample annotated speech, wherein the sample annotated speech includes the real text content and the real language type corresponding to the sample speech. During the training process, the sample speech and the annotated speech are input into the initial speech recognition model, wherein the initial speech recognition model includes an initial speech recognition module and an initial dialect classification module, so as to obtain the output results of the initial speech recognition module and the initial dialect classification module, and then the model parameters of the initial speech recognition module are trained according to the output results to obtain a converged multi-dialect speech recognition model. The present disclosure uses Mandarin speech and different types of dialect speech as samples for model training, solves the data sparsity problem existing in the model training process, and is conducive to improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 A flowchart of a method for training a multi-dialect speech recognition model provided in an embodiment of the present disclosure;

[0060] Figure 2 A schematic diagram of the structure of an initial speech recognition module provided in an embodiment of the present disclosure;

[0061] Figure 3 A schematic diagram of the structure of a multi-dialect speech recognition model provided in an embodiment of the present disclosure;

[0062] Figure 4 A schematic diagram of the structure of a training device for a multi-speech recognition model is provided for an embodiment of the present disclosure;

[0063] Figure 5 A schematic structural diagram of an electronic device is provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0066] In order to solve the technical problems existing in the related art, the embodiment of the present disclosure provides a training method for a multi-dialect speech recognition model, the method obtains sample speech including Mandarin speech and different types of dialect speech, and sample annotated speech, during the training process, the sample speech and the annotated speech are input into an initial speech recognition model, wherein the initial speech recognition model includes an initial speech recognition module and an initial dialect classification module, to obtain the output results of the initial speech recognition module and the initial dialect classification module, and then the model parameters of the initial speech recognition module are trained according to the output results to obtain a converged multi-dialect speech recognition model. The present disclosure uses Mandarin speech and different types of dialect speech as samples for model training, solves the data sparsity problem existing in the model training process, and is conducive to improving the robustness of the model.

[0067] The training method of the multi-dialect speech recognition model provided in the embodiments of the present disclosure can be implemented by a training device or electronic device for the multi-dialect speech recognition model. The electronic device may include: a server, a personal computer, a laptop computer, a tablet computer, a smart phone, a car terminal, etc. The operating system of the electronic device may include Android, a mobile operating system developed by Apple (iOS), an operating system developed by Microsoft Corporation of the United States (Windows), etc., and the embodiments of the present disclosure are not limited to this.

[0068] like Figure 1 As shown, Figure 1 A flowchart of a method for training a multi-dialect speech recognition model provided in an embodiment of the present disclosure includes the following steps S101 to S103:

[0069] S101: Obtain sample speech and annotated speech.

[0070] The sample speech includes Mandarin speech and different types of dialect speech. The annotated speech is to annotate the real text content and the real speech type based on the sample speech. The present disclosure supports that the number of different types of dialect speech in the sample speech is less than the number of Mandarin speech.

[0071] In some embodiments, the sample speech includes Mandarin speech and dialect speech. The dialect speech may be dialect speech of different regions and types. After obtaining the sample speech, the sample speech is annotated to obtain annotated speech. Annotating the sample speech includes annotating the real speech text content and the real speech type of the sample speech.

[0072] The present invention ensures that the data scale of the training set is large by using both Mandarin and dialects as the training sets of the multi-dialect speech recognition model, so as to avoid the problem of data sparsity in model training, which is beneficial to improving the robustness of the model and further improving the recognition effect of the multi-dialect speech recognition model.

[0073] S102: Input the sample speech data and the annotated speech into an initial speech recognition model.

[0074] The initial speech recognition model includes an initial speech recognition module and an initial dialect classification module. The initial speech recognition module is used to recognize the text content of the speech, and the initial dialect classification module is used to recognize the language type of the speech.

[0075] In some embodiments, the initial speech recognition model further includes an initial speech preprocessing module connected to the initial speech recognition module and the initial dialect classification module.

[0076] Specifically, the initial speech recognition model is based on a conformer structure, including an initial speech preprocessing module, an initial speech recognition module and an initial dialect classification module connected to the initial speech preprocessing module. The initial preprocessing module is used to extract acoustic features of the sample speech. The acoustic features include: energy concentration area, resonance peak frequency, resonance peak intensity and bandwidth representing timbre, and duration, fundamental frequency, average voice power, etc. representing speech rhythmic characteristics, which are not limited in the present disclosure.

[0077] In some embodiments, the initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer. The initial speech recognition module outputs the first loss function value, and the initial dialect classification module outputs the second loss function value.

[0078] Specifically, the first loss function layer included in the initial speech recognition module adopts a first loss function, and the first loss function may be a connectionist temporal classification (CTC) loss function, which is not specifically limited in the present disclosure. The second loss function layer included in the initial dialect classification module adopts a second loss function different from the first loss function, and the second loss function may be a cross entropy loss function, which is not specifically limited in the present disclosure.

[0079] like Figure 2 As shown, Figure 2 This is a structural schematic diagram of an initial speech recognition module provided in an embodiment of the present disclosure. The figure shows that the initial speech recognition model includes: an initial speech preprocessing module 201, an initial speech recognition module 202 and an initial dialect classification module 203.

[0080] The initial speech preprocessing module 201 includes an embedding layer and multiple encoders, each encoder including: a first feed forward layer, a first residual connection and a normalization layer, a multi-head self-attention layer, a second residual connection and a normalization layer, a convolution layer, a third residual connection and a normalization layer, a second feed forward layer, and a fourth residual connection and a normalization layer. The embedding layer includes an input embedding layer and a positional embedding layer.

[0081] The processing of the input speech by the initial speech preprocessing module includes but is not limited to: denoising, voice extraction, enhancement, and encoding, which are not specifically limited in the present disclosure.

[0082] The initial speech recognition module 202 includes: a first fully connected layer 21 and a first loss function layer 22 .

[0083] The initial dialect classification module 203 includes: a pooling layer based on a self-attention mechanism, referred to as a self-attention pooling layer (SAP Layer) 31, a second fully connected layer 32 and a second loss function layer 33.

[0084] Among them, the self-attention pooling layer 31 includes: a matrix transformation layer, a normalized exponential function layer, a matrix multiplication layer, and a weighted summation layer. The matrix transformation layer uses a parameter-learnable change matrix to convert the three-dimensional matrix into a first matrix; the normalized exponential function layer is used to calculate the attention weight matrix corresponding to the first matrix; the matrix multiplication layer is used to multiply the attention weight matrix with the three-dimensional matrix to obtain a calculation result; the weighted summation layer is used to perform weighted summation on the calculation result on the preset dimension of the three-dimensional matrix to obtain the language type of each sentence in the speech to be recognized. The preset dimension is a specific dimension in the pre-set three-dimensional matrix. In the embodiment of the present disclosure, the dimensions of the three-dimensional matrix include: N, T, H, where N represents the number of sentences of speech, T represents the number of frames of speech, H represents the feature dimension, and the preset dimension is T.

[0085] In some embodiments, after executing step S101 (obtaining sample speech and annotated speech) and before executing step S102 (inputting the sample speech and annotated speech into the initial speech recognition model), the method further includes:

[0086] Build the pending speech recognition module;

[0087] Perform speech recognition on the sample speech based on the pending speech recognition module, and obtain the error rate of the pending speech recognition module, where the error rate represents the probability that the text content predicted by the pending speech recognition module is different from the actual speech text content of the sample speech;

[0088] When the error rate is greater than or equal to the preset error rate, determining a special output node of the pending speech recognition module corresponding to the error rate;

[0089] The output nodes of the pending speech recognition module are added according to the special output nodes to obtain the initial speech recognition module.

[0090] An initial speech recognition module is obtained through the above embodiment. Specifically, after obtaining the sample speech and the annotated speech, a pending speech recognition module is constructed. The pending speech recognition module is connected to the initial speech preprocessing module. Optionally, the pending speech recognition module is a Mandarin speech recognition module in the prior art, which can accurately recognize Mandarin speech. Then, the sample speech is input into the initial speech preprocessing module and the pending speech recognition module connected thereto, and the sample speech is processed by these two modules. It can be understood that the sample speech is recognized in the manner of recognizing Mandarin, and the error rate calculated by the pending speech recognition module is obtained. The error rate can represent the error rate of recognizing the sample speech in the manner of recognizing Mandarin. For example, as shown in Table 1, Table 1 is a comparison table of the sample speech and the Mandarin speech.

[0091] Table 1

[0092] Sample Voice dad dad you close rise sound happy Mandarin Voice dad dad you close superior sound happy

[0093] According to the following formula (1):

[0094] Error rate = number of incorrect words / total number of words (1)

[0095] The error rate corresponding to the sample speech is calculated to be

[0096] Further, compare whether the error rate is greater than or equal to a preset error rate. The preset error rate represents the minimum probability of speech recognition errors. If not, that is, the error rate is less than the preset error rate, indicating that the sample speech is Mandarin speech, then directly use the pending speech recognition module as the initial speech recognition module for subsequent model training. If so, that is, the error rate is greater than or equal to the preset error rate, indicating that the sample speech is not Mandarin speech, then determine the special output node of the pending speech recognition module corresponding to the error rate, and then increase the output nodes of the pending speech recognition module according to the special output node to obtain the initial speech recognition module. It can be understood that since the initial preset speech preprocessing module extracts the acoustic features of the sample speech, the acoustic features pass through the speech recognition module and are decoded to obtain the recognized text content. Then, the recognition error rate is statistically calculated according to the annotated text of the speech. It should be emphasized that the pending speech recognition module is essentially a classification module. The speech module to be recognized processes the acoustic features to classify and obtain the probability of the text category in the sample speech, and then the specific text content corresponding to the sample speech can be obtained through decoding. When the error rate is greater than or equal to the preset error rate, it means that the pending speech recognition module recognizes a large number of incorrect dialect texts. Then, these dialect texts (i.e., text categories) are separately modeled, and the output nodes of the pending recognition module are increased according to the number of incorrect dialect texts indicated by the error rate to obtain the initial speech recognition module.

[0097] Continuing with the above example, assume that the word "起" appears 100 times in the entire sample speech and is recognized incorrectly 30 times. Then the error rate of "起" is 30%. If the preset error rate is 30%, then "起" is separately modeled, that is, a special output node corresponding to "起" is added to the pending speech recognition module. Another example, assume that the word "你" appears 100 times in the entire sample speech and is recognized incorrectly 5 times. Then the error rate of "你" is 5%, and "你" is not separately modeled.

[0098] Exemplarily, the number of original output nodes of the pending speech recognition model is K, and the number of special output nodes corresponding to the error rate is M. The number of output nodes of the pending speech recognition model is changed from the original K to K + M. This realizes separate modeling of the sample speech with a high error rate of text based on the sample speech extracted by the initial speech preprocessing module, avoids mutual interference between Mandarin speech and dialect speech, and thus improves the recognition effect of the model.

[0099] The above-mentioned initial speech recognition module with added output nodes is used as a part of the structure of the initial speech recognition model to be trained, thereby completing the basic construction of the initial speech recognition model, realizing the use of Chinese characters for modeling, and solving the problem of mutual interference between modeling units in the existing dialect recognition model training based on the data-driven method. It needs to be explained that data-driven is to collect massive amounts of data through mobile Internet or other related software, organize the data into information, and then integrate and refine the relevant information, and form an automated decision-making model through training and fitting based on the data. In short, it is to make decisions and actions based on data.

[0100] The following further introduces the processing of the sample speech and the annotated speech by the initial speech recognition module after the sample speech and the annotated speech are input into the initial speech recognition module:

[0101] In some embodiments, after the sample speech and the annotated speech are input into the initial speech recognition model, the model parameters of the initial speech recognition model are trained according to the output results of the initial speech recognition module and the initial dialect classification module, and before the converged multi-dialect speech recognition model is obtained, the following steps are included:

[0102] Preprocessing the sample speech through the initial speech preprocessing module to obtain a three-dimensional matrix, and the three-dimensional matrix represents the acoustic features of the sample speech;

[0103] The three-dimensional matrix is ​​decoded to obtain predicted text features corresponding to the sample speech, and the first loss function layer calculates the first loss function value according to the predicted text features and the real text content;

[0104] The three-dimensional matrix is ​​classified and processed to obtain the predicted language type features corresponding to the sample speech, and the second loss function layer calculates the second loss function value according to the predicted language type features and the true language type.

[0105] Specifically, after the sample speech is input into the initial speech recognition model, it is preprocessed by the initial speech preprocessing module of the initial speech recognition model to obtain a three-dimensional matrix, which represents the acoustic features of the sample speech.

[0106] refer to Figure 2 As shown in the structure of the initial speech recognition module, after the sample speech is input into the initial speech recognition model, the sample speech is first preprocessed by the initial speech preprocessing module 201, including but not limited to: denoising, voice extraction, enhancement and encoding, so as to obtain a three-dimensional matrix of the sample speech, which represents the acoustic features extracted from the sample speech.

[0107] After obtaining the three-dimensional matrix, on the one hand, the three-dimensional matrix is ​​decoded by the first fully connected layer in the initial speech recognition module of the initial speech recognition model to obtain the predicted text features. Specifically, the first fully connected layer 21 of the initial speech recognition module 202 decodes the three-dimensional matrix to obtain the predicted text features corresponding to the sample speech, and further, the first loss function layer 22 of the initial speech recognition module 202 calculates the first loss function value according to the predicted text features and the real text content.

[0108] Exemplarily, the first loss function is a CTC loss function, and the first loss function value is calculated by the following formula (2):

[0109] loss1=CTCLoss(x,label) (2)

[0110] Among them, x represents the predicted text features and label represents the actual text content.

[0111] On the other hand, the three-dimensional matrix is ​​processed by the initial dialect classification module. Specifically, the initial dialect classification module classifies the three-dimensional matrix to obtain the predicted language type features corresponding to the sample speech, and then the second loss function layer calculates the second loss function value based on the predicted speech type features and the actual language type.

[0112] In some embodiments, the three-dimensional matrix is ​​classified to obtain predicted language type features corresponding to the sample speech, including:

[0113] Performing pooling processing on the three-dimensional matrix, and converting the three-dimensional matrix into a first matrix;

[0114] Calculate the attention weight matrix corresponding to the first matrix, and multiply the attention weight matrix and the three-dimensional matrix to obtain the calculation result;

[0115] The calculation results are weighted and summed on the preset dimension of the three-dimensional matrix to obtain the language type features corresponding to each sentence in the sample speech;

[0116] The language type features corresponding to each sentence in the sample speech are decoded to determine the predicted language type features.

[0117] Specifically, the three-dimensional matrix is ​​first converted into a first matrix through a pooling layer based on a self-attention mechanism, and then the attention weight matrix corresponding to the first matrix is ​​calculated, and the attention weight matrix and the three-dimensional matrix are multiplied to obtain the calculation result, and then the calculation result is weighted and summed on the preset dimension of the three-dimensional matrix to obtain the speech type feature corresponding to each sentence in the sample speech. Furthermore, the speech type feature corresponding to each sentence of speech is decoded through the second fully connected layer to determine the predicted language type feature.

[0118] Combination Figure 2 The specific structure of the self-attention pooling layer in the initial dialect classification module shown is further explained: the self-attention pooling layer 31 performs the following operations on the three-dimensional matrix output by the initial speech preprocessing module 201: first, the three-dimensional matrix is ​​passed through a layer of parameter-learnable transformation matrix to obtain a first matrix, and then passed through a normalized exponential function (Softmax) layer to obtain an attention weight matrix for each frame in the sample speech, and then the attention weight matrix is ​​multiplied by the three-dimensional matrix, and further, the weighted summation is performed on the preset dimensions of the three-dimensional matrix to obtain the language type features of each sentence in the sample speech.

[0119] Exemplarily, the three-dimensional matrix is ​​A: [N, T, N], and the change matrix B: [H, 1]; the three-dimensional matrix A is passed through the change matrix B to obtain the first matrix C: [N, T], and then passed through the Softmax layer to obtain the attention weight matrix D: [N, T] of each frame, the attention weight matrix D is multiplied by the three-dimensional matrix A, and finally the weighted sum is performed on the preset dimension T to obtain the output E: [N, H], and the output E represents the predicted language type features of each sentence in the sample speech.

[0120] Referring to the above example, the Pytorch[5] pseudo code is as follows:

[0121] def SAP:

[0122] C=A@B

[0123] D=torch.softmax(C.squeeze(-1)).unsqueeze(-1)

[0124] E = torch.sum(A*D, dim=1)

[0125] The above embodiment constructs an initial speech recognition model based on the conformer structure. In particular, in the dialect classification module, a self-attention pooling layer is set to perform a pooling operation of the self-attention mechanism on the three-dimensional matrix output by the initial speech preprocessing module, thereby improving the classification effect of the initial dialect classification module.

[0126] Furthermore, the second loss function layer in the initial dialect classification module calculates the second loss function value according to the predicted language type feature and the actual language type. Exemplarily, the second loss function is a cross entropy loss function, and the second loss function value is calculated according to the following formula (3):

[0127]

[0128] Where N is the number of sample speech, M is the number of language types, and y ic Indicates the real language type, p icRepresents the predicted language type feature.

[0129] After the first loss function value and the second loss function value are calculated, the first loss function value is compared with a preset loss threshold, where the preset loss threshold is a preset minimum loss function value indicating model convergence.

[0130] In some embodiments, according to the output results of the initial speech recognition module and the initial dialect classification module, the model parameters of the initial speech recognition model are trained to obtain a converged multi-dialect speech recognition model, including:

[0131] When the first loss function value and / or the second loss function value is greater than the preset loss threshold, return to adjust the model parameters of the initial speech recognition model; until the first loss function value and the second loss function value are both less than or equal to the preset loss threshold, a multi-dialect speech recognition model is trained.

[0132] Specifically, when the first loss function value and / or the second loss function value is greater than the preset loss threshold, the model parameters of the initial speech recognition model are returned to be adjusted, wherein the model parameters of the initial language recognition model include but are not limited to: weights in the neural network. It should be emphasized that during the model training process, the model parameters corresponding to each module in the initial speech recognition module are returned to be adjusted, that is, the model parameters corresponding to the initial speech preprocessing module are adjusted, the model parameters corresponding to the initial speech recognition module are adjusted, and the model parameters corresponding to the initial dialect classification module are adjusted.

[0133] When the first loss function value is less than the preset loss threshold, and the second loss function value is less than the preset loss threshold, it indicates that the initial speech recognition model has been trained to convergence, and other structures except the first loss function layer and the second loss function layer in the initial speech recognition model after the model parameters are adjusted are retained, such as Figure 3 As shown, Figure 3 This is a structural diagram of a multi-dialect speech recognition model provided in an embodiment of the present disclosure. The figure shows that the multi-dialect speech recognition model includes: a speech preprocessing module 301, a speech recognition module 302 and a dialect classification module 303.

[0134] Among them, the speech preprocessing module 301 includes an embedding layer and N encoders, N is a positive integer, and each encoder includes: a first feedforward layer, a first residual connection and normalization layer, a multi-head self-attention layer, a second residual connection and normalization layer, a convolution layer, a third residual connection and normalization layer, a second feedforward layer, and a fourth residual connection and normalization layer. The embedding layer includes an input embedding layer and a position embedding layer. The speech recognition module 302 includes a first fully connected layer 21. The dialect classification module 303 includes a self-attention pooling layer 31 and a second fully connected layer 32.

[0135] It should be noted that the multi-dialect speech recognition model and the initial speech recognition model have the same structure and function, but the model parameters of the structure are different. The first fully connected layer 21 outputs text content, and the second fully connected layer 32 outputs language type.

[0136] Therefore, the converged initial speech recognition model is used as a multi-dialect speech recognition model to recognize various types of dialect speech. It can accurately distinguish between Mandarin and dialects, as well as different dialects, and recognize the text content corresponding to speech of various language types.

[0137] During the training process of the multi-dialect speech recognition model, the second loss function layer is introduced as the dialect classification loss function, which increases the robustness of the multi-dialect speech recognition model and helps improve the recognition effect of the model. In addition, through the data-driven method, the recognized erroneous speech text content is modeled separately to reduce the mutual interference between the various voices in the multi-dialect recognition scenario, thereby increasing the types of dialects that can be recognized by the multi-dialect speech recognition model while ensuring the recognition effect.

[0138] Based on the above training process, the converged multi-dialect speech recognition model obtained through training can be used for multi-dialect speech recognition. Optionally, the speech to be recognized is obtained, and the speech to be recognized is input into the trained multi-dialect speech recognition model. The speech to be recognized is calculated and processed within the model to output the speech text content and language type of the model to be recognized.

[0139] In some embodiments, the speech to be recognized is input into a multi-dialect speech recognition model, and is preprocessed by a speech preprocessing module of the multi-dialect speech recognition model to obtain a three-dimensional matrix. The three-dimensional matrix is ​​then decoded by the speech recognition module of the multi-dialect speech recognition model to obtain speech text content, and the three-dimensional matrix is ​​classified by a dialect classification module to obtain a language type.

[0140] In some embodiments, in the process of classifying the three-dimensional matrix through the dialect classification module to obtain the language type, the three-dimensional matrix is ​​first converted into a first matrix through the matrix transformation layer, and then the attention weight matrix corresponding to the first matrix is ​​calculated by the normalized exponential function layer, and then the attention weight matrix is ​​multiplied by the three-dimensional matrix by the matrix multiplication layer to obtain the calculation result, and further, the calculation result is weighted and summed on the preset dimension of the three-dimensional matrix to obtain the language type corresponding to each sentence in the speech to be recognized. After that, the language type corresponding to each sentence of speech is classified through the second fully connected layer to obtain the predicted language type of the speech to be recognized.

[0141] In summary, the embodiments of the present disclosure provide a training method for a multi-dialect speech recognition model, the method obtaining sample speech including Mandarin speech and different types of dialect speech, and sample annotated speech. During the training process, the sample speech and the annotated speech are input into the initial speech recognition model to obtain a first loss function value and a second loss function value output by the initial speech recognition model, wherein the model includes: an initial speech preprocessing module, and an initial speech recognition module and an initial dialect recognition module connected to the initial speech preprocessing module, the initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer; when the first loss function value and / or the second loss function value is greater than a preset loss threshold, the model parameters of the initial speech recognition model are returned to adjust, and the number of output nodes of the initial speech recognition module and the initial dialect classification module can be adjusted, so as to achieve modeling based on the text extracted by the initial speech preprocessing module. Therefore, the embodiments of the present disclosure can use Mandarin speech and different types of dialect speech as sample speech for model training, solve the problem of sparse dialect data in the related art, and help improve the robustness of the model; during the model training process, modeling is performed in the form of text to avoid interference existing in training the model in the form of speech, thereby improving the training effect of the model.

[0142] like Figure 4 As shown, Figure 4 A schematic diagram of the structure of a multi-speech recognition device is provided for an embodiment of the present disclosure, and the device includes:

[0143] The acquisition module 401 is used to: acquire sample speech and annotated speech, wherein the sample speech includes Mandarin speech and different types of dialect speech; the annotated speech is the real text content and real language type corresponding to the sample speech;

[0144] The processing module 402 is used to input the sample speech and the annotated speech into an initial speech recognition model; wherein the initial speech recognition model includes an initial speech recognition module and an initial dialect classification module;

[0145] The training module 403 is used to train the model parameters of the initial speech recognition model according to the output results of the initial speech recognition module and the initial dialect classification module to obtain a converged multi-dialect speech recognition model.

[0146] As an optional implementation of the present disclosure, the initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer; the initial speech recognition module outputs the first loss function value, and the initial dialect classification module outputs the second loss function value;

[0147] The training module 403 is specifically used to: when the first loss function value and / or the second loss function value is greater than a preset loss threshold, return to adjust the model parameters of the initial speech recognition model; until the first loss function value and the second loss function value are both less than or equal to the preset loss threshold, a multi-dialect speech recognition model is obtained by training.

[0148] As an optional implementation of the present disclosure, the initial speech recognition model further includes: an initial speech preprocessing module connected to the initial speech recognition module and the initial dialect classification module;

[0149] After the sample speech and the annotated speech are input into the initial speech recognition model, the model parameters of the initial speech recognition model are trained according to the output results of the initial speech recognition module and the initial dialect classification module. Before obtaining a converged multi-dialect speech recognition model, the training module is used to:

[0150] Preprocessing the sample speech through the initial speech preprocessing module to obtain a three-dimensional matrix, and the three-dimensional matrix represents the acoustic features of the sample speech;

[0151] The three-dimensional matrix is ​​decoded to obtain predicted text features corresponding to the sample speech, and the first loss function layer calculates the first loss function value according to the predicted text features and the real text content;

[0152] The three-dimensional matrix is ​​classified and processed to obtain the predicted language type features corresponding to the sample speech, and the second loss function layer calculates the second loss function value according to the predicted language type features and the true language type.

[0153] As an optional implementation of the present disclosure, the training module 403 performs classification processing on the three-dimensional matrix to obtain predicted language type features corresponding to the sample speech, specifically for:

[0154] Performing pooling processing on the three-dimensional matrix, and converting the three-dimensional matrix into a first matrix;

[0155] Calculate the attention weight matrix corresponding to the first matrix, and multiply the attention weight matrix and the three-dimensional matrix to obtain the calculation result;

[0156] The calculation results are weighted and summed on the preset dimension of the three-dimensional matrix to obtain the language type features corresponding to each sentence in the sample speech;

[0157] The language type features corresponding to each sentence in the sample speech are decoded to determine the predicted language type features.

[0158] As an optional implementation of the present disclosure, after obtaining the sample speech and the annotated speech, and before inputting the sample speech and the annotated speech into the initial speech recognition model, the acquisition module 401 is used to:

[0159] Build the pending speech recognition module;

[0160] Perform speech recognition on the sample speech based on the pending speech recognition module, and obtain the error rate of the pending speech recognition module, where the error rate represents the probability that the text content predicted by the pending speech recognition module is different from the actual speech text content of the sample speech;

[0161] When the error rate is greater than or equal to the preset error rate, determining a special output node of the pending speech recognition module corresponding to the error rate;

[0162] The output nodes of the pending speech recognition module are added according to the special output nodes to obtain the initial speech recognition module.

[0163] As an optional implementation of the present disclosure, the device is also used to: obtain speech to be recognized; input the speech to be recognized into a multi-dialect speech recognition model to obtain the speech text content and language type output by the multi-dialect speech recognition model.

[0164] As an optional implementation of the present disclosure, the first loss function layer is a connection time series classification loss function layer, and the second loss function layer is a cross entropy loss function layer.

[0165] like Figure 5 As shown, Figure 5 The present disclosure provides a schematic diagram of the structure of an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, each process of the method for training a multi-dialect speech recognition model in the above method embodiment is implemented. The same technical effect can be achieved, and it will not be described here to avoid repetition.

[0166] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the training method for a multi-dialect speech recognition model in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0167] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0168] An embodiment of the present invention provides a computer program product, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the training method for a multi-dialect speech recognition model in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0169] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0170] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0171] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0172] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0173] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0174] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a multi-dialect speech recognition model, characterized in that: include: Acquire sample speech and annotated speech, wherein the sample speech includes Mandarin speech and different types of dialect speech; The annotated speech is the real text content and real language type corresponding to the sample speech; Inputting the sample speech and the annotated speech into an initial speech recognition model; wherein the initial speech recognition model includes an initial speech recognition module and an initial dialect classification module; According to the output results of the initial speech recognition module and the initial dialect classification module, the model parameters of the initial speech recognition model are trained to obtain a converged multi-dialect speech recognition model.

2. The method according to claim 1, characterized in that: The initial speech recognition module includes a first loss function layer, and the initial dialect classification module includes a second loss function layer; the initial speech recognition module outputs a first loss function value, and the initial dialect classification module outputs a second loss function value; The method of training the model parameters of the initial speech recognition model according to the output results of the initial speech recognition module and the initial dialect classification module to obtain a converged multi-dialect speech recognition model comprises: When the first loss function value and / or the second loss function value is greater than a preset loss threshold, return to adjust the model parameters of the initial speech recognition model; until the first loss function value and the second loss function value are both less than or equal to the preset loss threshold, a multi-dialect speech recognition model is trained.

3. The method according to claim 2, characterized in that The initial speech recognition model further includes: an initial speech preprocessing module connected to the initial speech recognition module and the initial dialect classification module; After the sample speech and the annotated speech are input into the initial speech recognition model, and before the model parameters of the initial speech recognition model are trained according to the output results of the initial speech recognition module and the initial dialect classification module to obtain a converged multi-dialect speech recognition model, the method includes: Preprocessing the sample speech by the initial speech preprocessing module to obtain a three-dimensional matrix, wherein the three-dimensional matrix represents the acoustic features of the sample speech; Decoding the three-dimensional matrix to obtain predicted text features corresponding to the sample speech, and calculating the first loss function value by the first loss function layer according to the predicted text features and the real text content; The three-dimensional matrix is ​​classified to obtain predicted language type features corresponding to the sample speech, and the second loss function layer calculates the second loss function value according to the predicted language type features and the true language type.

4. The method according to claim 3, characterized in that: The classifying process of the three-dimensional matrix to obtain the predicted language type feature corresponding to the sample speech includes: Performing pooling processing on the three-dimensional matrix to convert the three-dimensional matrix into a first matrix; Calculating an attention weight matrix corresponding to the first matrix, and multiplying the attention weight matrix and the three-dimensional matrix to obtain a calculation result; Performing weighted summation on the calculation results on a preset dimension of the three-dimensional matrix to obtain a language type feature corresponding to each sentence of the sample speech; The language type feature corresponding to each sentence in the sample speech is decoded to determine the predicted language type feature.

5. The method according to claim 1, characterized in that: After acquiring the sample speech and the annotated speech, and before inputting the sample speech and the annotated speech into the initial speech recognition model, the method includes: Build the pending speech recognition module; Performing speech recognition on the sample speech based on the pending speech recognition module, obtaining an error rate of recognition by the pending speech recognition module, wherein the error rate indicates a probability that the text content predicted by the pending speech recognition module is different from the actual speech text content of the sample speech; When the error rate is greater than or equal to a preset error rate, determining a special output node of the pending speech recognition module corresponding to the error rate; The output nodes of the pending speech recognition module are added according to the special output nodes to obtain the initial speech recognition module.

6. The method according to claim 1, characterized in that The method further comprises: Get the speech to be recognized; The speech to be recognized is input into the multi-dialect speech recognition model to obtain the speech text content and language type output by the multi-dialect speech recognition model.

7. The method according to claim 2, characterized in that: The first loss function layer is a connection temporal classification loss function layer, and the second loss function layer is a cross entropy loss function layer.

8. A training device for a multi-dialect speech recognition model, characterized in that: include: An acquisition module is used to: acquire sample speech and annotated speech, wherein the sample speech includes Mandarin speech and different types of dialect speech; The annotated speech is the real text content and real language type corresponding to the sample speech; An input module, used to: input the sample speech and the annotated speech into an initial speech recognition model; wherein the initial speech recognition model includes: an initial speech recognition module and an initial dialect classification module; The training module is used to: train the model parameters of the initial speech recognition model according to the output results of the initial speech recognition module and the initial dialect classification module to obtain a converged multi-dialect speech recognition model.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the training method for a multi-dialect speech recognition model as described in any one of claims 1 to 7.

10. A vehicle, characterized in that: include: The training device for a multi-dialect speech recognition model as claimed in claim 8, or the electronic device as claimed in claim 9.

11. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the training method of a multi-dialect speech recognition model as described in any one of claims 1 to 7 is implemented.

Images