Training method of accent speech recognition model, speech recognition method and device, and computer equipment

Abstract

The invention relates to a training method of an accent speech recognition model, a speech recognition method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a sample set; the sample set comprises a plurality of first sample accent voice data and spoken language voice tags corresponding to the first sample accent voice data; inputting the first sample accent voice data into a coding unit of a pre-trained fine tuning voice model, and outputting a first sample voice feature carrying accent information; inputting the first sample accent voice data into a coding unit of a to-be-trained accent voice recognition model, and outputting a second sample voice feature; fusing the first sample voice feature and the second sample voice feature to obtain a fused sample voice feature; and training a to-be-trained accent speech recognition model according to the fused sample speech features to obtain a pre-trained accent speech recognition model. By adopting the method, the accuracy of speech recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for an accent speech recognition model, a speech recognition method, an apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence and deep neural networks, speech recognition technology is experiencing widespread application and popularity. From voice assistants to customer service, from medical records to smart homes, this technology is constantly improving our daily lives and the way we work. Speech recognition technology is a computer application technology that converts spoken speech into text or instructions. It uses various algorithms and models to automatically recognize and understand the speaker's speech content.

[0003] In practical applications, automatic speech recognition technology often faces the challenge of input with heavy accents or dialects.

[0004] Due to the diversity of pronunciation changes between different accents and the scarcity of supervised training data for automatic speech recognition of specific accents, related technologies may still fail when processing speech with different accents, resulting in a decrease in the accuracy of speech recognition.

[0005] Therefore, there is a problem of low accuracy of speech recognition in the related art. Summary of the invention

[0006] Based on this, it is necessary to provide a training method for an accent speech recognition model, a speech recognition method, an apparatus, a computer device, a computer-readable storage medium and a computer program product that can improve the accuracy of speech recognition in order to address the above technical problems.

[0007] In a first aspect, the present application provides a method for training an accent speech recognition model, comprising:

[0008] Acquire a sample set; the sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data;

[0009] The first sample accent speech data is input into the encoding unit of the pre-trained fine-tuning speech model, and the first sample speech feature carrying the accent information is output; the pre-trained fine-tuning speech model is obtained by fine-tuning the pre-trained general speech model using the second sample accent speech data carrying the first accent category label;

[0010] Inputting the first sample accent speech data into an encoding unit of an accent speech recognition model to be trained, and outputting second sample speech features;

[0011] fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature;

[0012] The accent speech recognition model to be trained is trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on input speech data.

[0013] In one embodiment, the accent speech recognition model to be trained further includes a decoding unit, and the spoken speech tag includes real speech text content; the training of the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model includes:

[0014] Inputting the fused sample speech features into the decoding unit to obtain a sample speech recognition result;

[0015] According to the difference between the sample speech recognition result and the real speech text content, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end conditions, thereby obtaining the pre-trained accent speech recognition model.

[0016] In one embodiment, the training method of the accent speech recognition model to be trained adopts a transfer learning training method, the accent speech recognition model to be trained also includes an accent classification unit, and the spoken speech label also includes a second accent category label;

[0017] The step of training the to-be-trained accent speech recognition model according to the fused sample speech features to obtain a pre-trained accent speech recognition model comprises:

[0018] Inputting the fused sample speech features into the accent classification unit to obtain a sample accent recognition result;

[0019] Taking the difference between the sample speech recognition result and the real speech text content as the first difference;

[0020] taking the difference between the sample accent recognition result and the second accent category label as a second difference;

[0021] According to the first difference and the second difference, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining the pre-trained accent speech recognition model.

[0022] In one of the embodiments, the transfer learning training method includes a multi-task learning method or a domain adversarial training method.

[0023] In one embodiment, when the transfer learning training method is the domain adversarial training method, adjusting the model parameters of the to-be-trained accent speech recognition model according to the first difference and the second difference until the adjusted to-be-trained accent speech recognition model meets the preset training end condition, thereby obtaining the pre-trained accent speech recognition model, includes:

[0024] According to the first difference and the second difference, adjusting the model parameters of the accented speech recognition model to be trained by back propagation; wherein, in the back propagation process, the gradient of the model parameters is multiplied by a negative coefficient;

[0025] When the adjusted accent speech recognition model to be trained meets the preset training end condition, the pre-trained accent speech recognition model is obtained.

[0026] In a second aspect, the present application provides a speech recognition method, comprising:

[0027] Acquire voice data to be recognized;

[0028] The speech data is input into a pre-trained accent speech recognition model, and a speech recognition result is output; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described above.

[0029] In a third aspect, the present application also provides a training device for an accent speech recognition model, comprising:

[0030] A sample acquisition module, used to acquire a sample set; the sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data;

[0031] A first feature determination module is used to input the first sample accent speech data into an encoding unit of a pre-trained fine-tuning speech model, and output a first sample speech feature carrying accent information; the pre-trained fine-tuning speech model is obtained by fine-tuning a pre-trained general speech model using the second sample accent speech data carrying a first accent category label;

[0032] a second feature determination module, configured to input the first sample accent speech data into an encoding unit of an accent speech recognition model to be trained, and output a second sample speech feature;

[0033] A fusion module, used for fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature;

[0034] A training module is used to train the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on input speech data.

[0035] In a fourth aspect, the present application further provides a speech recognition device, comprising:

[0036] A data acquisition module, used to acquire voice data to be recognized;

[0037] A recognition module is used to input the speech data into a pre-trained accent speech recognition model and output a speech recognition result; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described above.

[0038] In a fifth aspect, the present application further provides an electronic device, wherein the electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0039] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0040] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0041] The training method, speech recognition method, device, computer equipment, computer-readable storage medium and computer program product of the above-mentioned accent speech recognition model are as follows: obtaining a sample set; the sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data; inputting the first sample accent speech data into the encoding unit of the pre-trained fine-tuning speech model, and outputting the first sample speech features carrying the accent information; the pre-trained fine-tuning speech model is obtained by fine-tuning the pre-trained general speech model using the second sample accent speech data carrying the first accent category label; inputting the first sample accent speech data into the encoding unit of the accent speech recognition model to be trained, and outputting the second sample speech features; fusing the first sample speech features with the second sample speech features to obtain the fused sample speech features; training the accent speech recognition model to be trained according to the fused sample speech features to obtain the pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on the input speech data.

[0042] In this way, the pre-trained general speech model can learn rich general speech knowledge. The pre-trained general speech model is fine-tuned by the second sample accent speech data carrying the first accent category label to obtain a pre-trained fine-tuned speech model, so that the pre-trained fine-tuned speech model can learn accent-related knowledge. Therefore, the first sample accent speech data is input into the encoding unit of the pre-trained fine-tuned speech model, and the first sample speech feature carrying the accent information is output. Then, the first sample speech feature and the second sample speech feature output by the accent speech recognition model to be trained for the first sample accent speech data are fused. The accent speech recognition model to be trained is trained using the fused sample speech features, so that the first sample speech feature encoded by the pre-trained fine-tuned speech model is introduced as an auxiliary feature into the training process of the accent speech recognition model to be trained. With the help of the pre-trained fine-tuned speech model's ability to encode the accent, the robustness of the accent speech recognition task to the accent is enhanced, and the recognition ability of the accent data is improved, so that the speech recognition accuracy of the pre-trained accent speech recognition model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a method for training an accent speech recognition model in one embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of steps for training an accent speech recognition model to be trained in one embodiment;

[0046] Figure 3 is a model structure diagram of an accent speech recognition model to be trained in an embodiment;

[0047] Figure 4 is a flow chart of a speech recognition method in one embodiment;

[0048] Figure 5 is a flow chart of a speech recognition method in another embodiment;

[0049] Figure 6 is a structural block diagram of a training device for an accent speech recognition model in one embodiment;

[0050] Figure 7 is a structural block diagram of a speech recognition device in one embodiment;

[0051] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0054] like Figure 1 As shown, a method for training an accent speech recognition model is provided. This embodiment uses the method applied to a computer device as an example. It can be understood that the computer device can be a server or a terminal. The method can be implemented by the server or the terminal alone, or through interaction between the server and the terminal. In this embodiment, the method includes the following steps:

[0055] Step S110, obtaining a sample set.

[0056] The sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data.

[0057] The multiple first-accent voice samples refer to voice data or audio samples with different accents.

[0058] The spoken speech tag is used to characterize the real speech text content corresponding to the first sample accent speech data.

[0059] The voice text content refers to the text content corresponding to the voice data.

[0060] In a specific implementation, a computer device may obtain a sample set. The sample set may be pre-stored in a storage unit of the computer device, and the computer device may directly obtain the sample set locally. The sample set may also be downloaded from a network by the computer device. This application does not specifically limit the method for obtaining the sample set.

[0061] Step S120: input the first sample accent speech data into the encoding unit of the pre-trained fine-tuning speech model, and output the first sample speech features carrying the accent information.

[0062] The pre-trained fine-tuning speech model is obtained by fine-tuning a pre-trained general speech model using second sample accent speech data carrying a first accent category label.

[0063] The second sample accented speech data may be speech data or audio samples having different accents.

[0064] Among them, the first accent category label is used to characterize the accent type of the second sample accent speech data, such as Shandong accent, Northeast accent, Sichuan accent, Guangdong accent, etc.

[0065] The pre-trained general speech model may be a speech model obtained through unsupervised training on large-scale corpus data. In this way, the pre-trained general speech model can learn rich general speech knowledge.

[0066] Among them, the speech model is a deep learning model with speech processing function, including but not limited to: Hubert model, Wav2vec 2.0 model and large language model with speech processing function.

[0067] Among them, after training to obtain a pre-trained general speech model, the pre-trained general speech model can be fine-tuned using second sample accent speech data carrying a first accent category label, and the fine-tuning task is accent classification to learn accent-related knowledge, thereby pre-training a fine-tuned speech model, so that the model encoding of the pre-trained fine-tuned speech model contains accent information.

[0068] In this way, by inputting the first sample accent speech data into the encoding unit of the pre-trained fine-tuning speech model for feature encoding, a sample speech feature carrying accent information can be output, and the sample speech feature output by the encoding unit of the pre-trained fine-tuning speech model can be used as the first sample speech feature.

[0069] Step S130: input the first sample accent speech data into the encoding unit of the accent speech recognition model to be trained, and output the second sample speech features.

[0070] The main model of the accented speech recognition model to be trained adopts the encoder-decoder architecture. The encoder adopts the conformer (a neural network architecture based on the self-attention mechanism), and the decoder adopts the tranformer (a deep learning model architecture used for natural language processing and other sequence-to-sequence tasks).

[0071] In a specific implementation, a computer device may input first sample accent speech data into an encoding unit of an accent speech recognition model to be trained for feature encoding, and the encoding unit of the accent speech recognition model to be trained may output sample speech features of the first sample accent speech data as second sample speech features.

[0072] Step S140: Fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature.

[0073] In a specific implementation, the computer device may fuse the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature.

[0074] Specifically, the first sample speech feature and the second sample speech feature may be fused by adding, concatenating, or the like.

[0075] Step S150, training the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model.

[0076] Among them, the pre-trained accent speech recognition model is used to perform speech recognition on the input speech data.

[0077] The input voice data may be voice data to be processed.

[0078] Among them, the voice data to be processed is the voice data to be processed in the voice task. The voice data is essentially an audio signal with specific audio features, including but not limited to: amplitude, frequency, timbre, period and wavelength. Voice tasks include but are not limited to: voice transcription tasks, voice generation tasks, voice reply tasks and voice translation tasks. For example, the voice data of the voice transcription task in the online meeting is the voice data of the speaker in the online meeting. Another example is the voice data of the voice reply task in the social software, that is, the voice data sent by the user in the social software. Another example is the voice data of the voice translation task in the translation software, that is, the voice data of the first language entered by the user in the translation software.

[0079] In a specific implementation, the computer device can train the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model. The pre-trained accent speech recognition model can be used to perform speech recognition on the input speech data to identify the text content corresponding to the input speech data.

[0080] In the training method of the above-mentioned accent speech recognition model, a sample set is obtained; the sample set includes multiple first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data; the first sample accent speech data is input into the encoding unit of the pre-trained fine-tuning speech model, and the first sample speech features carrying the accent information are output; the pre-trained fine-tuning speech model is obtained by fine-tuning the pre-trained general speech model using the second sample accent speech data carrying the first accent category label; the first sample accent speech data is input into the encoding unit of the accent speech recognition model to be trained, and the second sample speech features are output; the first sample speech features and the second sample speech features are fused to obtain the fused sample speech features; the accent speech recognition model to be trained is trained according to the fused sample speech features to obtain the pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on the input speech data.

[0081] In this way, the pre-trained general speech model can learn rich general speech knowledge. The pre-trained general speech model is fine-tuned by the second sample accent speech data carrying the first accent category label to obtain a pre-trained fine-tuned speech model, so that the pre-trained fine-tuned speech model can learn accent-related knowledge. Therefore, the first sample accent speech data is input into the encoding unit of the pre-trained fine-tuned speech model, and the first sample speech feature carrying the accent information is output. Then, the first sample speech feature and the second sample speech feature output by the accent speech recognition model to be trained for the first sample accent speech data are fused. The accent speech recognition model to be trained is trained using the fused sample speech features, so that the first sample speech feature encoded by the pre-trained fine-tuned speech model is introduced as an auxiliary feature into the training process of the accent speech recognition model to be trained. With the help of the pre-trained fine-tuned speech model's ability to encode the accent, the robustness of the accent speech recognition task to the accent is enhanced, and the recognition ability of the accent data is improved, so that the speech recognition accuracy of the pre-trained accent speech recognition model is improved.

[0082] In one embodiment, the accent speech recognition model to be trained also includes a decoding unit, and the spoken speech label includes real speech text content; the accent speech recognition model to be trained is trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model, including: inputting the fused sample speech features into the decoding unit to obtain a sample speech recognition result; according to the difference between the sample speech recognition result and the real speech text content, adjusting the model parameters of the accent speech recognition model to be trained until the adjusted accent speech recognition model to be trained meets the preset training end conditions, thereby obtaining a pre-trained accent speech recognition model.

[0083] The sample speech recognition result may be text content corresponding to the first sample accent speech data predicted by the decoding unit according to the fused sample speech features.

[0084] The preset training end condition may be that the number of training times reaches a preset number threshold, or the loss function value is less than or equal to a preset threshold.

[0085] In a specific implementation, when a computer device trains an accent speech recognition model to be trained based on fused sample speech features to obtain a pre-trained accent speech recognition model, the computer device can input the fused sample speech features into the decoding unit to obtain a sample speech recognition result; calculate the difference between the sample speech recognition result and the real speech text content, and adjust the model parameters of the accent speech recognition model to be trained based on the difference between the sample speech recognition result and the real speech text content, until the adjusted accent speech recognition model to be trained meets a preset training end condition, thereby obtaining a pre-trained accent speech recognition model.

[0086] In the technical solution of this embodiment, the accent speech recognition model to be trained also includes a decoding unit, and the spoken speech label includes real speech text content; by inputting the fused sample speech features into the decoding unit, a sample speech recognition result is obtained; according to the difference between the sample speech recognition result and the real speech text content, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining a pre-trained accent speech recognition model. In this way, the accent speech recognition model to be trained is quickly and efficiently supervised, so that the pre-trained accent speech recognition model obtained by training can accurately perform speech recognition on speech data with an accent.

[0087] In one embodiment, the training method of the accent speech recognition model to be trained adopts a transfer learning training method, the accent speech recognition model to be trained also includes an accent classification unit, and the spoken speech label also includes a second accent category label;

[0088] like Figure 2 As shown, the accent speech recognition model to be trained is trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model, including the following steps:

[0089] Step S210: input the fused sample speech features into the accent classification unit to obtain a sample accent recognition result.

[0090] Among them, the training task of the accent classification unit is to classify the accent of the input speech data.

[0091] Among them, the sample accent recognition result is the accent type predicted by the accent classification unit based on the input fused sample speech features.

[0092] Step S220: taking the difference between the sample speech recognition result and the real speech text content as the first difference.

[0093] Step S230: taking the difference between the sample accent recognition result and the second accent category label as the second difference.

[0094] The second accent category label is used to characterize the real accent type corresponding to the first sample accent speech data.

[0095] The computer device may calculate a difference between the sample accent recognition result and the second accent category label, and use the difference as a second difference.

[0096] Step S240, adjusting the model parameters of the accent speech recognition model to be trained according to the first difference and the second difference, until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining a pre-trained accent speech recognition model.

[0097] In a specific implementation, the computer device can adjust the model parameters of the accent speech recognition model to be trained according to the first difference and the second difference, until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining a pre-trained accent speech recognition model.

[0098] In some of these embodiments, for the convenience of understanding by those skilled in the art, Figure 3 As shown in FIG. , a model structure diagram of an accent speech recognition model to be trained is provided. Figure 3 As shown, the accent speech recognition model to be trained includes an encoder (encoding unit), a decoder (decoding unit) and an accent classifier (accent classification unit).

[0099] The first sample accent speech data is input to the encoder of the accent speech recognition model to be trained, and the second sample speech features are output. The second sample speech features are fused with the first sample speech features output by the encoding unit of the pre-trained fine-tuning speech model, and the fused sample speech features are input to the decoder and the accent classifier to calculate the first difference and the second difference.

[0100] The encoder part consists of 12 Conformer (a neural network architecture based on self-attention mechanism) modules. Each Conformer module consists of four network layers, namely, feedforward network, multi-head attention network, convolution network, and feedforward network. Each network uses residual connection, and then the result of adding residual connection is normalized. The Conformer structure combines the advantages of CNN (Convolutional Neural Network) and attention, retaining the ability to learn local features and having good context learning ability.

[0101] The decoder part is a bitranformer (a neural network architecture) composed of two transformer (a deep learning model architecture used for natural language processing and other sequence-to-sequence tasks) models that encode the input from left to right and from right to left respectively. Each transformer model consists of three transformer blocks. In each transformer block, the input is first self-attentioned, then the input and encoder output are cross-attentioned, and finally passed through a forward propagation network. Each part of the network uses residual connections, and then the results of the residual connection addition are layer normalized. CTC loss (Connectionist Temporal Classification loss) is used as the model loss.

[0102] Exemplarily, CTC loss may be used to calculate the first difference between the sample speech recognition result and the actual speech text content.

[0103] Among them, the transfer learning training method includes a multi-task learning method or a domain adversarial training method.

[0104] When multi-task learning is used, the output of the encoder is connected to two branches, one branch is the decoder, and the other branch is the accent classifier. The accent classifier consists of two linear layers in the model structure, each layer is followed by a Relu activation function, and finally a softmax (an activation function) layer. The output dimension is the same as the accent type, and its training task is to classify accents.

[0105] When training with domain adversarial training, the model structure is the same as that of multi-task learning, but gradient reversal is used between the encoder and accent classification branches. That is, during backpropagation, the gradient is multiplied by a negative coefficient.

[0106] Specifically, when the transfer learning training method is the domain adversarial training method, the computer device adjusts the model parameters of the accent speech recognition model to be trained according to the first difference and the second difference until the adjusted accent speech recognition model to be trained meets the preset training end condition, and obtains the pre-trained accent speech recognition model. In the process, the computer device can adjust the model parameters of the accent speech recognition model to be trained according to the first difference and the second difference by back propagation; wherein, in the back propagation process, the gradient of the model parameters is multiplied by a negative coefficient; when the adjusted accent speech recognition model to be trained meets the preset training end condition, the pre-trained accent speech recognition model is obtained.

[0107] In this way, the training objectives of the encoder part and the accent classifier can be made opposite and counteract each other, so that the encoder's encoding remains invariant to different accents.

[0108] In the embodiments of the present application, multi-task learning is a deep learning paradigm that enables a model to learn multiple related tasks at the same time, and uses the correlation between these tasks to improve the model effect. In the training of accent data recognition, by adding an accent recognition branch to the speech recognition model and optimizing it using classification losses such as cross entropy, the speech recognition model's learning of accent information can be enhanced. Domain adversarial training is another very effective method in transfer learning, which allows the features learned by the speech recognition model to retain the distinctiveness in the source domain, but remain invariant to domain migration, so as to improve the generalization performance in the target domain.

[0109] According to the technical solution of this embodiment, the training method of the accent speech recognition model to be trained adopts the transfer learning training method, the accent speech recognition model to be trained also includes an accent classification unit, and the spoken speech label also includes a second accent category label; a sample accent recognition result is obtained by inputting the fused sample speech features into the accent classification unit; the difference between the sample speech recognition result and the real speech text content is taken as the first difference; the difference between the sample accent recognition result and the second accent category label is taken as the second difference; according to the first difference and the second difference, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining a pre-trained accent speech recognition model; wherein, the transfer learning training method includes a multi-task learning method or a domain adversarial training method.

[0110] In this way, during the training process of the accented speech recognition model to be trained, the transfer learning training method is adopted, an additional accent classification branch is added to the model, and multi-task training or domain adversarial training methods are used, which can help the speech recognition model better handle the accent features in the encoded information and enhance the robustness of the encoder encoding to the accent information.

[0111] In one embodiment, Figure 4 As shown, a speech recognition method is provided. This embodiment takes the method applied to a computer device as an example for explanation. In this embodiment, the method includes the following steps:

[0112] Step S402: Acquire speech data to be recognized.

[0113] In a specific implementation, the computer device may obtain the voice data to be recognized. The voice data to be recognized may be the voice data to be recognized uploaded by the voice acquisition device, or may be the voice data to be recognized obtained from a database, which is not limited here.

[0114] Step S404, input the speech data into the pre-trained accent speech recognition model, and output the speech recognition result.

[0115] The pre-trained accent speech recognition model is obtained by training according to the training method of the accent speech recognition model as described above.

[0116] In a specific implementation, the computer device can input the voice data to be recognized into a pre-trained accent speech recognition model and output the speech recognition result.

[0117] Among them, the speech recognition result includes the corresponding text content predicted by the pre-trained accent speech recognition model based on the input speech data.

[0118] In some embodiments, the speech recognition result may also include a predicted accent type corresponding to the input speech data.

[0119] The technical solution of this embodiment is to obtain speech data to be recognized; input the speech data into a pre-trained accent speech recognition model, and output a speech recognition result; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described above.

[0120] In this way, the pre-trained general speech model can learn rich general speech knowledge. The pre-trained general speech model is fine-tuned by the second sample accent speech data carrying the first accent category label to obtain a pre-trained fine-tuned speech model, so that the pre-trained fine-tuned speech model can learn accent-related knowledge. Therefore, the first sample accent speech data is input into the encoding unit of the pre-trained fine-tuned speech model, and the first sample speech feature carrying the accent information is output. Then, the first sample speech feature and the second sample speech feature output by the accent speech recognition model to be trained for the first sample accent speech data are fused. The accent speech recognition model to be trained is trained using the fused sample speech features, so that the first sample speech feature encoded by the pre-trained fine-tuned speech model is introduced as an auxiliary feature into the training process of the accent speech recognition model to be trained. With the help of the pre-trained fine-tuned speech model's ability to encode the accent, the robustness of the accent speech recognition task to the accent is enhanced, and the recognition ability of the pre-trained speech recognition model to the accent data is improved, so that the speech recognition accuracy of the pre-trained accent speech recognition model is improved.

[0121] In another embodiment, Figure 5 As shown, a speech recognition method is provided, which is described by taking the method applied to a computer device as an example, and includes the following steps:

[0122] Step S502: Obtain a sample set.

[0123] Step S504: input the first sample accent speech data into the encoding unit of the pre-trained fine-tuning speech model, and output the first sample speech features carrying the accent information.

[0124] Step S506: input the first sample accent speech data into the encoding unit of the accent speech recognition model to be trained, and output the second sample speech features.

[0125] Step S508: Fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature.

[0126] Step S510: input the fused sample speech features into a decoding unit to obtain a sample speech recognition result.

[0127] Step S512: input the fused sample speech features into the accent classification unit to obtain a sample accent recognition result.

[0128] Step S514: taking the difference between the sample speech recognition result and the real speech text content as the first difference.

[0129] Step S516: taking the difference between the sample accent recognition result and the second accent category label as the second difference.

[0130] Step S518, adjusting the model parameters of the accent speech recognition model to be trained according to the first difference and the second difference, until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining a pre-trained accent speech recognition model.

[0131] Step S520, obtaining speech data to be recognized, inputting the speech data into a pre-trained accent speech recognition model, and outputting a speech recognition result.

[0132] It should be noted that the specific limitations of the above steps can be found in the above-mentioned specific limitations of a training method for an accent speech recognition model and a speech recognition method.

[0133] In this way, the present application enhances the learning ability of the speech recognition model for accent data by using auxiliary training tasks such as multi-task learning or domain adversarial training in transfer learning. And with the help of the pre-trained speech model, the speech recognition effect of the speech recognition model is improved by using the general information learned on large-scale unsupervised corpus when the training data is limited. In addition, the present method has good portability, and when different subject models are used, the solutions used in the technology can be used as auxiliary.

[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiment of the present application also provides a training device for an accent speech recognition model for implementing the training method for the accent speech recognition model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more accent speech recognition models provided below can refer to the limitations of the training method for the accent speech recognition model above, and will not be repeated here.

[0136] In an exemplary embodiment, Figure 6As shown, a training device for an accent speech recognition model is provided, comprising: a sample acquisition module 610, a first feature determination module 620, a second feature determination module 630, a fusion module 640 and a training module 650, wherein:

[0137] The sample acquisition module 610 is used to acquire a sample set; the sample set includes a plurality of first sample accent speech data and spoken language speech tags corresponding to the first sample accent speech data.

[0138] The first feature determination module 620 is used to input the first sample accent speech data into the encoding unit of the pre-trained fine-tuning speech model, and output the first sample speech feature carrying the accent information; the pre-trained fine-tuning speech model is obtained by fine-tuning the pre-trained general speech model using the second sample accent speech data carrying the first accent category label.

[0139] The second feature determination module 630 is used to input the first sample accent speech data into the encoding unit of the accent speech recognition model to be trained, and output the second sample speech feature.

[0140] The fusion module 640 is used to fuse the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature.

[0141] The training module 650 is used to train the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on the input speech data.

[0142] In one embodiment, the accent speech recognition model to be trained also includes a decoding unit, and the spoken speech label includes real speech text content; the training module 650 is specifically used to input the fused sample speech features into the decoding unit to obtain a sample speech recognition result; according to the difference between the sample speech recognition result and the real speech text content, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end conditions, thereby obtaining the pre-trained accent speech recognition model.

[0143] In one embodiment, the training method of the accent speech recognition model to be trained adopts a transfer learning training method, the accent speech recognition model to be trained also includes an accent classification unit, and the spoken speech label also includes a second accent category label; the training module 650 is specifically used to input the fused sample speech features into the accent classification unit to obtain a sample accent recognition result; the difference between the sample speech recognition result and the real speech text content is used as the first difference; the difference between the sample accent recognition result and the second accent category label is used as the second difference; according to the first difference and the second difference, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining the pre-trained accent speech recognition model.

[0144] In one of the embodiments, the transfer learning training method includes a multi-task learning method or a domain adversarial training method.

[0145] In one of the embodiments, when the transfer learning training method is the domain adversarial training method, the training module 650 is specifically used to adjust the model parameters of the accent speech recognition model to be trained by back propagation according to the first difference and the second difference; wherein, during the back propagation process, the gradient of the model parameters is multiplied by a negative coefficient; after the adjustment, the accent speech recognition model to be trained meets the preset training end condition, and the pre-trained accent speech recognition model is obtained.

[0146] Each module in the training device for the accent speech recognition model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0147] Based on the same inventive concept, the embodiment of the present application also provides a speech recognition device for implementing the speech recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more speech recognition device embodiments provided below can refer to the limitations on the speech recognition method above, and will not be repeated here.

[0148] In an exemplary embodiment, Figure 7 As shown, a speech recognition device is provided, including: a data acquisition module 710 and a recognition module 720, wherein:

[0149] The data acquisition module 710 is used to acquire the speech data to be recognized.

[0150] The recognition module 720 is used to input the speech data into a pre-trained accent speech recognition model and output a speech recognition result; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described above.

[0151] Each module in the above-mentioned speech recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0152] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store model parameter data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a training and / or speech recognition method of an accent speech recognition model is implemented.

[0153] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0155] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0156] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0158] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0160] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training an accent speech recognition model, characterized in that: The method comprises: Acquire a sample set; the sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data; The first sample accent speech data is input into the encoding unit of the pre-trained fine-tuning speech model, and the first sample speech feature carrying the accent information is output; the pre-trained fine-tuning speech model is obtained by fine-tuning the pre-trained general speech model using the second sample accent speech data carrying the first accent category label; Inputting the first sample accent speech data into an encoding unit of an accent speech recognition model to be trained, and outputting second sample speech features; fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature; The accent speech recognition model to be trained is trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on input speech data.

2. The method according to claim 1, characterized in that The accent speech recognition model to be trained further includes a decoding unit, and the spoken speech tag includes real speech text content; the accent speech recognition model to be trained is trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model, including: Inputting the fused sample speech features into the decoding unit to obtain a sample speech recognition result; According to the difference between the sample speech recognition result and the real speech text content, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end conditions, thereby obtaining the pre-trained accent speech recognition model.

3. The method according to claim 2, characterized in that The training method of the accent speech recognition model to be trained adopts a transfer learning training method, the accent speech recognition model to be trained also includes an accent classification unit, and the spoken speech label also includes a second accent category label; The step of training the to-be-trained accent speech recognition model according to the fused sample speech features to obtain a pre-trained accent speech recognition model comprises: Inputting the fused sample speech features into the accent classification unit to obtain a sample accent recognition result; Taking the difference between the sample speech recognition result and the real speech text content as the first difference; taking the difference between the sample accent recognition result and the second accent category label as a second difference; According to the first difference and the second difference, the model parameters of the accent speech recognition model to be trained are adjusted until the adjusted accent speech recognition model to be trained meets the preset training end condition, thereby obtaining the pre-trained accent speech recognition model.

4. The method according to claim 3, characterized in that The transfer learning training method includes a multi-task learning method or a domain adversarial training method.

5. The method according to claim 4, characterized in that In the case where the transfer learning training mode is the domain adversarial training mode, the model parameters of the to-be-trained accent speech recognition model are adjusted according to the first difference and the second difference until the adjusted to-be-trained accent speech recognition model meets the preset training end condition, thereby obtaining the pre-trained accent speech recognition model, including: According to the first difference and the second difference, adjusting the model parameters of the accented speech recognition model to be trained by back propagation; wherein, in the back propagation process, the gradient of the model parameters is multiplied by a negative coefficient; When the adjusted accent speech recognition model to be trained meets the preset training end condition, the pre-trained accent speech recognition model is obtained.

6. A speech recognition method, characterized in that: The method comprises: Acquire voice data to be recognized; The speech data is input into a pre-trained accent speech recognition model, and a speech recognition result is output; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described in any one of claims 1 to 5.

7. A training device for an accent speech recognition model, characterized in that: The device comprises: A sample acquisition module, used to acquire a sample set; the sample set includes a plurality of first sample accent speech data and spoken speech labels corresponding to the first sample accent speech data; A first feature determination module is used to input the first sample accent speech data into an encoding unit of a pre-trained fine-tuning speech model, and output a first sample speech feature carrying accent information; the pre-trained fine-tuning speech model is obtained by fine-tuning a pre-trained general speech model using the second sample accent speech data carrying a first accent category label; a second feature determination module, configured to input the first sample accent speech data into an encoding unit of an accent speech recognition model to be trained, and output a second sample speech feature; A fusion module, used for fusing the first sample speech feature and the second sample speech feature to obtain a fused sample speech feature; A training module is used to train the accent speech recognition model to be trained according to the fused sample speech features to obtain a pre-trained accent speech recognition model; the pre-trained accent speech recognition model is used to perform speech recognition on input speech data.

8. A speech recognition device, characterized in that: The device comprises: A data acquisition module, used to acquire voice data to be recognized; A recognition module, used to input the speech data into a pre-trained accent speech recognition model and output a speech recognition result; the pre-trained accent speech recognition model is trained according to the training method of the accent speech recognition model as described in any one of claims 1 to 5.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images