Speech model training method, speech recognition method, device and electronic equipment

Abstract

The application discloses a speech model training method, a speech recognition method, a device and an electronic device, and relates to the technical field of speech recognition. The method is used to solve the problem of low accuracy of the existing accent speech recognition method. The method comprises the following steps: obtaining a speech sample set, each speech sample in the speech sample set is labeled with a word label, and each speech frame is labeled with an accent label; training an accent speech recognition model by using the speech sample set, an acoustic feature extraction module in the accent speech recognition model is used to extract acoustic features of input speech, an encoder is used to encode the acoustic features into acoustic representation vectors, a decoder is used to predict a spelling sequence according to the acoustic representation vectors, an accent classifier is used to output frame-level accent category information according to the acoustic representation vectors, and a translator is used to translate the spelling sequence according to the frame-level accent category information to obtain a word sequence. The embodiment of the application can improve the accuracy of accent speech recognition.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech model training method, speech recognition method, device and electronic device. Background Technology

[0002] Speech recognition models often experience significant performance degradation when recognizing speech with accents, making the development of speech recognition models capable of recognizing multiple accents a necessity in the industry. Current mainstream solutions mostly employ a multi-task structure, primarily comprising two task branches: speech recognition and accent classification. The accent classification branch provides accent information to the speech recognition branch, which then makes corresponding predictions based on this information. Such multi-accent speech recognition solutions are highly dependent on the accuracy of the accent classification branch; insufficient accuracy can lead to misleading predictions from the speech recognition branch. Currently, the accent classification branch primarily outputs sentence-level predictions, which is unsuitable for special scenarios such as multiple accents within a single sentence, causing a decrease in the accuracy of the accent classification branch and consequently affecting the accuracy of speech recognition. Summary of the Invention

[0003] This application provides a speech model training method, a speech recognition method, a device, and an electronic device to solve the problem of low accuracy in existing accent speech recognition methods.

[0004] In a first aspect, embodiments of this application provide a speech model training method, including:

[0005] A speech sample set is obtained, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag.

[0006] The initial accent speech recognition model is trained using the aforementioned speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into acoustic representation vectors. The decoder predicts a pinyin sequence based on the acoustic representation vectors output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vectors output by the encoder, and this frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech.

[0007] Optionally, training the initially established accent speech recognition model using the speech sample set includes:

[0008] Input the first speech sample from the speech sample set into the accent speech recognition model;

[0009] The acoustic feature extraction module extracts acoustic features from the first speech sample to obtain a first acoustic feature; the encoder encodes the first acoustic feature to obtain a first acoustic representation vector; the decoder decodes the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier classifies the first acoustic representation vector by accent to obtain a first accent classification result, which includes the accent category of each speech frame in the first speech sample; the translator translates the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample.

[0010] Based on the first pinyin sequence and the pinyin tags of the first speech sample, a first pinyin level loss is determined. The pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary.

[0011] The first accent classification loss is determined based on the first accent classification result and the accent label of each speech frame in the first speech sample.

[0012] Based on the first character sequence and the character labels of the first speech sample, determine the first character-level loss;

[0013] The structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0014] Optionally, the accent classifier also outputs a first accent embedding vector of the first speech sample;

[0015] The step of translating the first pinyin sequence according to the first accent classification result using the translator to obtain the first character sequence corresponding to the first speech sample includes:

[0016] The first accent embedding vector is downsampled, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence;

[0017] The first pinyin sequence is embedded and encoded to obtain the first pinyin embedding vector;

[0018] The first pinyin embedding vector is concatenated with the downsampled first accent embedding vector to obtain the first concatenated vector;

[0019] The first concatenated vector is translated by the translator to obtain the first character sequence.

[0020] Optionally, before inputting the first speech sample from the speech sample set into the accent speech recognition model, the method further includes:

[0021] The second speech sample in the speech sample set is input into the accent speech recognition model;

[0022] The acoustic feature extraction module extracts acoustic features from the second speech sample to obtain second acoustic features; the encoder encodes the second acoustic features to obtain a second acoustic representation vector; the decoder decodes the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier classifies the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category of each speech frame in the second speech sample; the translator translates the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample.

[0023] Based on the second pinyin sequence and the pinyin labels of the second speech sample, a second pinyin level loss is determined. The pinyin labels of the second speech sample are obtained by mapping the character labels of the second speech sample through a pronunciation dictionary.

[0024] The second accent classification loss is determined based on the second accent classification result and the accent label of each speech frame in the second speech sample.

[0025] The second character-level loss is determined based on the second character sequence and the character labels of the second speech sample;

[0026] Based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0027] The step of training the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss includes:

[0028] Based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training are trained in the second stage.

[0029] Optionally, the step of translating the pinyin tags of the second speech sample using the translator to obtain the second character sequence corresponding to the second speech sample includes:

[0030] The pinyin tags of the second speech sample are embedded and encoded to obtain the second pinyin embedding vector;

[0031] The first pinyin embedding vector is concatenated with the all-zero vector to obtain the second concatenated vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector;

[0032] The second concatenated vector is translated by the translator to obtain the second character sequence.

[0033] Optionally, the first-stage training of the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss includes:

[0034] The second pinyin-level loss, the second accent classification loss, and the second character-level loss are weighted and summed to obtain the first total loss;

[0035] Based on the first total loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0036] The second stage of training, based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, involves training the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training. This includes:

[0037] The first pinyin-level loss, the first accent classification loss, and the first character-level loss are weighted and summed to obtain the second total loss.

[0038] Based on the second total loss, the structural parameters of the encoder, decoder, accent classifier, and translator trained in the first stage are trained in the second stage.

[0039] Optionally, the accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network DNN; the penultimate layer of the accent classifier DNN is used to output the accent embedding vector, and the last layer of the accent classifier DNN is used to output the frame-level accent category.

[0040] Secondly, embodiments of this application also provide a speech recognition method, including:

[0041] The speech to be recognized is input into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator;

[0042] The acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature; the encoder encodes the third acoustic feature to obtain a third acoustic representation vector; the decoder decodes the third acoustic representation vector to obtain a third pinyin sequence; the accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector; and the translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain the third character sequence corresponding to the speech to be recognized.

[0043] Output the third character sequence.

[0044] Optionally, the accent speech recognition model is trained using the speech model training method described in the first aspect.

[0045] Thirdly, embodiments of this application also provide a speech model training device, comprising:

[0046] The acquisition module is used to acquire a speech sample set, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag.

[0047] A training module is used to train an initially established accent speech recognition model using the speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into acoustic representation vectors. The decoder predicts a pinyin sequence based on the acoustic representation vectors output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vectors output by the encoder, and the frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech.

[0048] Optionally, the training module includes:

[0049] The first input unit is used to input the first speech sample from the speech sample set into the accent speech recognition model;

[0050] The acoustic feature extraction module is used to extract acoustic features from the first speech sample to obtain a first acoustic feature; the encoder is used to encode the first acoustic feature to obtain a first acoustic representation vector; the decoder is used to decode the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier is used to classify the first acoustic representation vector by accent to obtain a first accent classification result, the first accent classification result including the accent category to which each speech frame in the first speech sample belongs; and the translator is used to translate the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample.

[0051] The first determining unit is configured to determine a first pinyin level loss based on the first pinyin sequence and the pinyin tags of the first speech sample, wherein the pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary;

[0052] The second determining unit is used to determine the first accent classification loss based on the first accent classification result and the accent label of each speech frame in the first speech sample.

[0053] The third determining unit is used to determine the first character-level loss based on the first character sequence and the character labels of the first speech sample;

[0054] The first training unit is used to train the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0055] Optionally, the accent classifier also outputs a first accent embedding vector of the first speech sample;

[0056] The accent speech recognition model also includes a downsampling module, an embedded encoder, and a splicing module;

[0057] The downsampling module is used to downsample the first accent embedding vector, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence.

[0058] The embedding encoder is used to embed the first pinyin sequence to obtain a first pinyin embedding vector;

[0059] The splicing module is used to splice the first pinyin embedding vector with the downsampled first accent embedding vector to obtain the first spliced ​​vector;

[0060] The translator is used to translate the first concatenated vector to obtain the first character sequence.

[0061] Optionally, the training module further includes:

[0062] The second input unit is used to input the second speech sample from the speech sample set into the accent speech recognition model;

[0063] The acoustic feature extraction module is used to extract acoustic features from the second speech sample to obtain a second acoustic feature; the encoder is used to encode the second acoustic feature to obtain a second acoustic representation vector; the decoder is used to decode the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier is used to classify the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category to which each speech frame in the second speech sample belongs; and the translator is used to translate the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample.

[0064] The fourth determining unit is used to determine the second pinyin level loss based on the second pinyin sequence and the pinyin tags of the second speech sample, wherein the pinyin tags of the second speech sample are obtained by mapping the character tags of the second speech sample through a pronunciation dictionary;

[0065] The fifth determining unit is used to determine the second accent classification loss based on the second accent classification result and the accent label of each speech frame in the second speech sample;

[0066] The sixth determining unit is used to determine the second character-level loss based on the second character sequence and the character labels of the second speech sample;

[0067] The second training unit is used to perform a first-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss.

[0068] The first training unit is used to perform a second-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first-stage training, based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0069] Optionally, the accent speech recognition model further includes an embedded encoder and a splicing module;

[0070] The embedding encoder is used to embed and encode the pinyin tags of the second speech sample to obtain the second pinyin embedding vector;

[0071] The splicing module is used to splice the first pinyin embedding vector with the all-zero vector to obtain a second spliced ​​vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector;

[0072] The translator is used to translate the second concatenated vector to obtain the second character sequence.

[0073] Optionally, the training module further includes:

[0074] The first calculation unit is used to perform a weighted summation of the second pinyin-level loss, the second accent classification loss, and the second character-level loss to obtain the first total loss.

[0075] The second calculation unit is used to perform a weighted summation of the first pinyin-level loss, the first accent classification loss, and the first character-level loss to obtain the second total loss.

[0076] The second training unit is used to perform a first-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first total loss;

[0077] The first training unit is used to perform a second-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first-stage training, based on the second total loss.

[0078] Optionally, the accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network DNN; the penultimate layer of the accent classifier DNN is used to output the accent embedding vector, and the last layer of the accent classifier DNN is used to output the frame-level accent category.

[0079] Fourthly, embodiments of this application also provide a voice recognition device, including:

[0080] The input module is used to input the speech to be recognized into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator;

[0081] The speech recognition module is used to extract acoustic features from the speech to be recognized by the acoustic feature extraction module to obtain a third acoustic feature; to encode the third acoustic feature by the encoder to obtain a third acoustic representation vector; to decode the third acoustic representation vector by the decoder to obtain a third pinyin sequence; to output the accent category information of each speech frame in the speech to be recognized by the accent classifier based on the third acoustic representation vector; and to translate the third pinyin sequence by the translator based on the accent category information of each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized.

[0082] The output module is used to output the third character sequence.

[0083] Optionally, the accent speech recognition model is trained using the speech model training method described in the first aspect.

[0084] Fifthly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the speech model training method as described in the first aspect; or to implement the steps in the speech recognition method as described in the second aspect.

[0085] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the speech model training method as described in the first aspect; or implements the steps in the speech recognition method as described in the second aspect.

[0086] In this embodiment, a speech sample set is obtained, where each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag. The speech sample set is used to train an initially established accent speech recognition model to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module is used to extract acoustic features from the input speech. The encoder is used to encode the acoustic features output by the acoustic feature extraction module into an acoustic representation vector. The decoder is used to predict a pinyin sequence based on the acoustic representation vector output by the encoder. The accent classifier is used to output frame-level accent category information based on the acoustic representation vector output by the encoder. The frame-level accent category information includes the accent category information of each speech frame in the input speech. The translator is used to translate the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech. In this way, by training an accent-based speech recognition model capable of identifying frame-level accent categories, the accuracy of accent classification can be improved, thereby ensuring the accuracy of accent-based speech recognition. This approach is suitable for scenarios where a sentence contains multiple accents. Furthermore, by employing a pinyin-character dual-granularity modeling method, the consistent sequence length of both avoids the need for a sub-encoder-decoder structure, effectively simplifying the model structure and improving inference speed and accuracy. Attached Figure Description

[0087] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a flowchart of the speech model training method provided in the embodiments of this application;

[0089] Figure 2 This is a schematic diagram of the accent speech recognition model and training process provided in the embodiments of this application;

[0090] Figure 3 This is a schematic diagram illustrating the generation process of pinyin tags and frame-level accent tags provided in the embodiments of this application;

[0091] Figure 4 This is a flowchart of the speech recognition method provided in the embodiments of this application;

[0092] Figure 5This is a schematic diagram of the reasoning process of the accent speech recognition model provided in the embodiments of this application;

[0093] Figure 6 This is a structural diagram of the speech model training device provided in the embodiments of this application;

[0094] Figure 7 This is a structural diagram of the speech recognition device provided in the embodiments of this application;

[0095] Figure 8 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0096] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0097] To make the embodiments of this application clearer, the relevant technical knowledge involved in the embodiments of this application will be introduced as follows:

[0098] Automatic speech recognition (ASR) is a technology that can convert speech into text. End-to-end (E2E) speech recognition is currently the mainstream speech recognition solution. It is based on neural network methods and typically includes three parts: an encoder, a Connectionist Temporal Classification (CTC) decoder, and an attention decoder. Compared to traditional Gaussian Mixture Models (GMM)-Hidden Markov Model (HMM) or Deep-Learning Neural Network (DNN)-HMM solutions, E2E speech recognition solutions have a simpler structure, are easier to train, and all parts are trained uniformly at once, without the need to train different sub-modules separately and then combine them.

[0099] Accents are regional variations of pronunciation relative to the standard pronunciation, such as Sichuan, Hunan, or Northeastern accents. People from different countries also speak a second language with different accents. Accent classification tasks aim to identify the type of accent in a speech segment. Accent classifiers can be used to provide accent information for speech recognition models or used independently to extract the speaker's possible regional location. Current mainstream accent classification tasks typically use classification models such as Residual Networks (ResNet), X-Vectors, and Emphasized Channel Attention-Time Delay Neural Networks (ECAPA-TDNN).

[0100] Current standard end-to-end speech recognition models often experience significant performance degradation when recognizing speech with accents. Therefore, there is a widespread demand in practical applications for developing end-to-end speech recognition models that can adapt to multiple accents simultaneously. Multi-accent speech recognition models typically rely on accent information such as accent labels or accent embeddings for accent adaptation. This accent information can be provided in advance or identified by an accent classifier. Specifically, mainstream end-to-end multi-accent speech recognition models often employ a multi-task structure, primarily consisting of two task branches: speech recognition and accent classification, which are trained simultaneously. Furthermore, the accent classification branch provides accent information to the speech recognition branch, which then responds and adapts accordingly. However, such end-to-end multi-accent speech recognition solutions are highly dependent on the accuracy of the accent classification branch; insufficient accuracy can lead to misleading of the speech recognition branch. Furthermore, the accent classification branch of the current multi-task structure mainly outputs sentence-level prediction results. However, considering that in some scenarios (such as dialogue scenarios), there may be multiple accents in a single sentence, it is necessary to provide more granular accent prediction results to ensure the accuracy of the speech recognition branch.

[0101] In traditional GMM-HMM and DNN-HMM speech recognition schemes, the acoustic information of phoneme-level modeling units is typically recognized by the GMM and DNN acoustic models, and then a decoder based on a finite-weighted state transition machine (WFST) converts the phoneme information into a text sequence. However, in current mainstream end-to-end speech recognition, to simplify the process, only word-level modeling units are usually used, which have the advantage of directly outputting text sequences usable by the user. In the few end-to-end speech recognition schemes that use both phoneme and word modeling units, considering the inconsistency in sequence length between phoneme and word units, these schemes usually include a phoneme-to-word (P2W) translation model composed of an encoder-decoder structure. Since the end-to-end speech recognition model already includes a set of encoder-decoder structures, this P2W scheme is structurally redundant.

[0102] This application proposes an accent-based speech recognition model and method based on a pinyin-character dual-granularity modeling unit. The model comprises four main modules: an encoder, a decoder, an accent classifier, and a translator. The modeling unit of the decoder is pinyin, while the modeling unit of the translator is characters. Using pinyin units in the accent classification branch and retaining character units in the speech recognition branch improves accent classification accuracy while preserving the advantage of end-to-end models that can directly output text sequences. Furthermore, this application utilizes frame-level accent category information, which is beneficial for adapting to scenarios where a sentence contains multiple accents. This application also designs a non-traditional decoder scheme to perform translation tasks, and in the translation task, it also requires accent information as a reference, adapting to the correspondence between pinyin and characters for different accent pronunciations. In addition, this application uses pinyin as a fine-grained unit and characters as a coarse-grained unit, leveraging their consistent sequence length to avoid the sub-encoder-decoder structure used in existing schemes, effectively simplifying the model structure and improving inference speed.

[0103] The speech model training method and speech recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0104] See Figure 1 , Figure 1 This is a flowchart of the speech model training method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0105] Step 101: Obtain a speech sample set. Each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag.

[0106] To train an accent-based speech recognition model that can recognize speech with different accents, it is necessary to obtain a large set of speech samples with different accents, such as speech samples with different regional accents.

[0107] The acquisition of the speech sample set can be obtained from a speech database by acquiring a speech sample set pre-labeled with word tags and frame-level accent tags. Specifically, the speech sample set can be obtained by collecting a large number of speech samples with local accents entered by users from different places, or by generating a large number of speech data with different local accents by a speech generation model.

[0108] Each speech sample in the speech sample set can be manually labeled with the required tags, including word tags and frame-level accent tags. Specifically, each speech sample can be labeled with the correct word-level tags, and each speech sample can be labeled with the correct accent tags frame by frame, that is, each speech frame in each speech sample can be labeled with the correct accent tags. Alternatively, the accent tags provided are usually at the sentence level, that is, each sentence in each speech sample with the same accent is labeled with the correct accent tags. To obtain frame-level accent tags, the accent tags of each sentence can be expanded by the number of frames to obtain the accent tags of each speech frame. For example, if a speech sentence includes 30 frames, the accent tags of that speech sentence can be expanded by the number of frames to obtain the accent tags of the 30 speech frames in that speech sentence. Figure 3 As shown, the correct frame-level accent tags required in the embodiments of this application can be obtained by copying sentence-level accent tags frame by frame.

[0109] In this embodiment of the application, by providing frame-level accent labels, it is possible to train an accent classifier to predict frame-level accent classification, that is, to accurately identify multiple different accents in a sentence of speech.

[0110] Step 102: Train the initially established accent speech recognition model using the speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into an acoustic representation vector. The decoder predicts a pinyin sequence based on the acoustic representation vector output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vector output by the encoder. The frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech.

[0111] In this embodiment of the application, an initial accent speech recognition model to be trained can be constructed, such as... Figure 2 As shown, the model mainly includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module is used to extract acoustic features from the input speech, and it adopts conventional audio feature extraction technology. The acoustic feature extraction module can select Mel Frequency Cepstrum Coefficient (MFCC) or Filter Bank (Fbank).

[0112] The encoder is responsible for extracting language information and accent information from the input acoustic features and encoding them into the same acoustic representation vector, which is then input into the decoder and the accent classifier, respectively. The encoder can select network structures such as Transformer and Conformer.

[0113] The decoder is used to decode the acoustic representation vector output by the encoder to obtain the predicted pinyin sequence. The decoder can be a CTC decoder, which includes a fully connected layer and a normalized exponential function (softmax) layer. Through CTC, the frame sequence of speech in time and the corresponding transcribed pinyin sequence can be automatically aligned during the model training process without the need to label the start and end time periods of each character or phoneme. This enables direct classification on the time series and eliminates the relatively complex alignment operation of traditional algorithms.

[0114] The accent classifier is used to predict the accent category of the acoustic representation vector output by the encoder. It can output frame-level accent category information, that is, for each speech frame in the input speech, it outputs the corresponding accent category. The accent classifier can be a network structure such as ECAPA-TDNN, ResNet or X-Vector.

[0115] The translator is used to translate the pinyin sequence predicted by the decoder from pinyin to characters by referring to the frame-level accent category information output by the accent classifier, thereby obtaining the character sequence corresponding to the input speech; the network structure of the translator can be consistent with the network structure of the encoder, and network structures such as Transformer and Conformer can be selected.

[0116] After constructing the initial accent speech recognition model, the speech sample set can be used to train the accent speech recognition model. Specifically, the speech samples in the speech sample set can be input into the accent speech recognition model. The acoustic feature extraction module, the encoder, the decoder, the accent classifier, and the translator in the accent speech recognition model can be used to perform accent and speech recognition on the speech samples. The decoder outputs the corresponding pinyin sequence, the accent classifier outputs the corresponding frame-level accent category information, and the translator outputs the corresponding word sequence.

[0117] To calculate the model loss value, the pinyin label of each speech sample in the speech sample set can also be obtained. Specifically, the pinyin label can be obtained by pre-annotating each speech sample with pinyin labels, or by mapping the characters to pinyin in the pronunciation dictionary based on the character labels of each speech sample.

[0118] Then, based on the character labels, frame-level accent labels, and pinyin labels annotated for each speech sample in the speech sample set, and the corresponding character sequences, frame-level accent category information, and pinyin sequences output by the accent speech recognition model, the model loss value is calculated. The structural parameters of each module in the accent speech recognition model, such as the encoder, the decoder, the accent classifier, and the translator, are adjusted according to the model loss value. Through repeated iterative training, a converged accent speech recognition model is obtained.

[0119] Optionally, step 102 includes:

[0120] Input the first speech sample from the speech sample set into the accent speech recognition model;

[0121] The acoustic feature extraction module extracts acoustic features from the first speech sample to obtain a first acoustic feature; the encoder encodes the first acoustic feature to obtain a first acoustic representation vector; the decoder decodes the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier classifies the first acoustic representation vector by accent to obtain a first accent classification result, which includes the accent category of each speech frame in the first speech sample; the translator translates the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample.

[0122] Based on the first pinyin sequence and the pinyin tags of the first speech sample, a first pinyin level loss is determined. The pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary.

[0123] The first accent classification loss is determined based on the first accent classification result and the accent label of each speech frame in the first speech sample.

[0124] Based on the first character sequence and the character labels of the first speech sample, determine the first character-level loss;

[0125] The structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0126] In one implementation, during model training, a first speech sample from the speech sample set can be input into the accent speech recognition model. The first speech sample can be any number of speech samples from the speech sample set. The first speech sample first enters the acoustic feature extraction module for acoustic feature extraction and outputs a first acoustic feature. The first acoustic feature is input into the encoder, which extracts language information and accent information from the first acoustic feature and encodes them into the same acoustic representation vector, outputting a first acoustic representation vector. The first acoustic representation vector is input into the decoder and the accent classifier, respectively. The decoder decodes the first acoustic representation vector, predicts a pinyin sequence, and outputs a first pinyin sequence. The accent classifier performs accent classification on the first acoustic representation vector, predicts frame-level accent categories, and outputs a first accent classification result, including the accent category to which each speech frame in the first speech sample belongs. After certain processing, the first pinyin sequence and the first accent classification result are sent to the translator. The translator refers to the first accent classification result to perform pinyin-to-character translation on the first pinyin sequence and outputs a first character sequence corresponding to the first speech sample.

[0127] In this embodiment, the word tags of the first speech sample can also be pre-mapped to corresponding pinyin tags using a pronunciation dictionary, such as... Figure 3 As shown, the correct pinyin-level tags required in the embodiments of this application can be obtained from the correct character-level tags through a pronunciation dictionary, which is a character-to-pinyin mapping table compiled by language experts.

[0128] Based on the first pinyin sequence output by the decoder and combined with the pinyin-level labels of the first speech sample, the pinyin-level loss can be calculated, resulting in the first pinyin-level loss. During model training, the pinyin-level loss constrains the encoder, causing its learned acoustic representation vector to contain a large amount of pinyin-level information. Such acoustic representation vectors, when used in the accent classifier, can significantly improve its accuracy.

[0129] Based on the first accent classification result output by the accent classifier, and combined with the accent labels of each speech frame in the first speech sample, the frame-level accent classification cross-entropy loss can be calculated to obtain the first accent classification loss. The frame-level accent classification cross-entropy loss can constrain the accent classifier, enabling it to accurately identify the accent of each speech frame.

[0130] Based on the first character sequence output by the translator and combined with the character labels of the first speech sample, the character-level cross-entropy loss can be calculated to obtain the first character-level loss. The translator primarily learns the ability to translate the erroneous pinyin sequence output by the CTC decoder into characters with the assistance of accent classification results. Some of these errors are caused by accents; therefore, with accent classification results provided, the translator can learn corresponding error correction capabilities based on different accents, significantly reducing the error rate of the final output character-level sequence.

[0131] Based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, the total model loss can be determined. For example, it can be a weighted sum of the three losses: the first pinyin-level loss, the first accent classification loss, and the first character-level loss. Based on the total model loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator can be adjusted. Through the above iterative training process, a converged accent speech recognition model can be obtained.

[0132] In this way, by training the accent speech recognition model through this implementation method, the encoder can learn an acoustic representation vector containing a large amount of pinyin-level information. Such an acoustic representation vector can significantly improve the accuracy of accent classification when used in an accent classifier, and can also enable the translator to learn corresponding pinyin error correction capabilities according to different accents, thereby improving the accuracy of speech recognition.

[0133] Optionally, before inputting the first speech sample from the speech sample set into the accent speech recognition model, the method further includes:

[0134] The second speech sample in the speech sample set is input into the accent speech recognition model;

[0135] The acoustic feature extraction module extracts acoustic features from the second speech sample to obtain second acoustic features; the encoder encodes the second acoustic features to obtain a second acoustic representation vector; the decoder decodes the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier classifies the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category of each speech frame in the second speech sample; the translator translates the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample.

[0136] Based on the second pinyin sequence and the pinyin labels of the second speech sample, a second pinyin level loss is determined. The pinyin labels of the second speech sample are obtained by mapping the character labels of the second speech sample through a pronunciation dictionary.

[0137] The second accent classification loss is determined based on the second accent classification result and the accent label of each speech frame in the second speech sample.

[0138] The second character-level loss is determined based on the second character sequence and the character labels of the second speech sample;

[0139] Based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0140] The step of training the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss includes:

[0141] Based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training are trained in the second stage.

[0142] In another embodiment, the accent speech recognition model can be trained in two stages to reduce the difficulty of model training, increase the training speed, and significantly improve the performance of the translator.

[0143] Specifically, during the first stage of training, a second speech sample from the speech sample set can be input into the accent speech recognition model. The second speech sample can be any number of speech samples from the speech sample set. The second speech sample first enters the acoustic feature extraction module for acoustic feature extraction and outputs a second acoustic feature. The second acoustic feature is input into the encoder, which extracts language information and accent information from the second acoustic feature and encodes them into the same acoustic representation vector, outputting a second acoustic representation vector. The second acoustic representation vector is input into the decoder and the accent classifier, respectively. The decoder decodes the second acoustic representation vector, predicts a pinyin sequence, and outputs a second pinyin sequence. The accent classifier performs accent classification on the second acoustic representation vector, predicts frame-level accent categories, and outputs the second accent classification result, including the accent category to which each speech frame in the second speech sample belongs.

[0144] The translator translates the pinyin tags of the second speech sample into characters and outputs the second character sequence corresponding to the second speech sample. That is, in the first stage of training, the correct pinyin level tags are used to train the translator so that it can first learn the ability to translate from pinyin to characters without accent information.

[0145] Based on the second pinyin sequence output by the decoder and combined with the pinyin-level labels of the second speech sample, the pinyin-level loss can be calculated to obtain the second pinyin-level loss.

[0146] Based on the second accent classification result output by the accent classifier, and combined with the accent label of each speech frame in the second speech sample, the frame-level accent classification cross-entropy loss can be calculated to obtain the second accent classification loss.

[0147] Based on the second character sequence output by the translator and combined with the character labels of the second speech sample, the character-level cross-entropy loss can be calculated to obtain the second character-level loss.

[0148] Based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss, the total model loss in the first training stage can be determined. For example, it can be a weighted sum of the two losses: the second pinyin-level loss, the second accent classification loss, and the second character-level loss. Based on the total model loss in this stage, the structural parameters of the encoder, the decoder, the accent classifier, and the translator can be adjusted. Through the above iterative training process, a converged accent speech recognition model can be obtained.

[0149] During the first stage of training, since the pinyin-level text features input to the translator only contain plain text information and all pinyin are correct, training the translator in this way allows it to focus on learning how to translate pinyin into characters without having to consider additional interference such as accents or incorrect input pinyin sequences. This reduces the training difficulty of the translator and speeds up the convergence.

[0150] In the second phase of training, the encoder, decoder, and accent classifier are trained using the same method as in the first phase. The main difference lies in the training of the translator. In the second phase, the input information for the translator differs from that in the first phase. Specifically, the correct pinyin-level labels input in the first phase are replaced with the pinyin sequence predicted by the encoder, and frame-level accent category information is added in the second phase. In the second phase, the training objective of the translator becomes learning the ability to translate the erroneous pinyin sequence output by the decoder into characters with the assistance of frame-level accent classification information.

[0151] In other words, the translator learns how to translate correct phonetic sequences into character sequences without considering accents in the first stage of training, a relatively simple task. The second stage builds upon this by learning to translate phonetic sequences with errors, some of which are due to accents. Therefore, with frame-level accent category information provided, the translator can learn corresponding error correction capabilities based on different accents, significantly reducing the error rate of the final output character-level sequence.

[0152] In this embodiment, a two-stage training scheme is used. In the first stage, the translator's translation function is trained using the correct pinyin sequence. In practical applications, the performance of the translator can be significantly improved by using additional text data in this stage. In the second stage, the predicted pinyin sequence with errors is used, and accent information is used as an aid to focus on training the translator's error correction function for different accents. This reduces the difficulty of model training and helps to reduce model training time and improve model performance.

[0153] Optionally, the first-stage training of the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss includes:

[0154] The second pinyin-level loss, the second accent classification loss, and the second character-level loss are weighted and summed to obtain the first total loss;

[0155] Based on the first total loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0156] The second stage of training, based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, involves training the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training. This includes:

[0157] The first pinyin-level loss, the first accent classification loss, and the first character-level loss are weighted and summed to obtain the second total loss.

[0158] Based on the second total loss, the structural parameters of the encoder, decoder, accent classifier, and translator trained in the first stage are trained in the second stage.

[0159] In one implementation, during the first and second training phases, the weighted sum of three losses—phonetic level loss, accent classification cross-entropy loss, and character-level cross-entropy loss—can be used as the total model loss. The weights of the three different losses can be flexibly adjusted according to actual needs. Thus, in each training phase, the structural parameters of the encoder, decoder, accent classifier, and translator are adjusted based on the total model loss, causing each module to converge.

[0160] In this way, by calculating the weighted sum of the pinyin-level loss, accent classification loss, and character-level loss, and adjusting the model's structural parameters based on the weighted sum of these three losses, the model's convergence speed can be accelerated while ensuring its accuracy.

[0161] Optionally, the step of translating the pinyin tags of the second speech sample using the translator to obtain the second character sequence corresponding to the second speech sample includes:

[0162] The pinyin tags of the second speech sample are embedded and encoded to obtain the second pinyin embedding vector;

[0163] The first pinyin embedding vector is concatenated with the all-zero vector to obtain the second concatenated vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector;

[0164] The second concatenated vector is translated by the translator to obtain the second character sequence.

[0165] In one implementation, during the first stage of training, the correct pinyin-level labels can be embedded and encoded, then concatenated with a set of zero-spaced vectors of the same time dimension, and then translated into a character-level sequence by the translator.

[0166] In the first stage of training, the second pinyin sequence output by the decoder and the frame-level accent category information output by the accent classifier have no practical use in subsequent processes; they mainly play a role in the second stage of training. To ensure that the input information of the translator in the first and second stages maintains consistency in the feature dimension, a placeholder vector of all zeros is used for placeholder operations in the first stage of training. Specifically, the correct pinyin-level labels are first embedded and encoded, keeping the time dimension unchanged, while the feature dimension is appropriately reduced to a feature dimension more suitable for model processing. Then, each pinyin in the time dimension is concatenated with the placeholder vector of all zeros in the feature dimension to obtain a pinyin-level text feature, which is used to train the translator.

[0167] In this way, the input information of the translator in the first and second stages of training can be kept consistent in the feature dimension, thereby ensuring the training effect of the translator.

[0168] Optionally, the accent classifier also outputs a first accent embedding vector of the first speech sample;

[0169] The step of translating the first pinyin sequence according to the first accent classification result using the translator to obtain the first character sequence corresponding to the first speech sample includes:

[0170] The first accent embedding vector is downsampled, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence;

[0171] The first pinyin sequence is embedded and encoded to obtain the first pinyin embedding vector;

[0172] The first pinyin embedding vector is concatenated with the downsampled first accent embedding vector to obtain the first concatenated vector;

[0173] The first concatenated vector is translated by the translator to obtain the first character sequence.

[0174] In one implementation, during the second training phase, the translator's input is replaced from the correct pinyin level labels and the all-zero placeholder vector with the pinyin sequence output by the decoder and the accent embedding vector output by the accent classifier, respectively. The accent embedding vector is downsampled to the same number of frames as the pinyin sequence. The dimension of the correct pinyin level labels is the same as the pinyin sequence, and the dimension of the all-zero placeholder vector is the same as the accent embedding vector. After the replacement, the translator's training objective becomes learning the ability to translate the erroneous pinyin sequence output by the decoder into a character sequence, aided by the accent embedding vector.

[0175] Specifically, the first pinyin sequence can be embedded and encoded to obtain a first pinyin embedding vector, keeping its time dimension unchanged while appropriately reducing its feature dimension to a more suitable feature dimension for model processing. The first accent embedding vector is then downsampled so that its time dimension is the same as that of the first pinyin sequence. Then, each pinyin in the time dimension of the first pinyin embedding vector is concatenated with the first accent embedding vector in the feature dimension to obtain a pinyin-level feature with accent, and the translator is trained accordingly.

[0176] In this way, the input information of the translator in the first and second training stages can be kept consistent in the feature dimension, and by providing accent embedding vectors, the translator can be trained to learn corresponding error correction capabilities according to different accents, which significantly reduces the error rate of the final output word-level sequence.

[0177] Optionally, the accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network DNN; the penultimate layer of the accent classifier DNN is used to output the accent embedding vector, and the last layer of the accent classifier DNN is used to output the frame-level accent category.

[0178] In one implementation, the accent classifier can be a combination of ResNet and multi-layer DNN, outputting accent embedding vectors through its penultimate DNN layer and frame-level accent categories through its final DNN layer. This allows the accent embedding vectors output by the accent classifier to be used to train the translator in the second training phase, enabling the translator to better learn corresponding error correction capabilities based on different accents, and allowing the translator to select the appropriate accent category during the model inference phase based on actual needs.

[0179] The following is combined with Figure 2 The diagram shows the accent speech recognition model and training process, with a detailed introduction to the training method for the accent speech recognition model:

[0180] In this embodiment, the acoustic feature extraction module uses Fbank, the encoder uses a Conformer structure, the decoder uses a CTC decoder, the accent classifier uses a combination of ResNet and several DNN layers, and the translator uses a Conformer structure. The Fbank acoustic features are obtained from the speech to be recognized using an audio feature extraction algorithm. Correct accent labels and correct word-level labels are manually labeled labels that must be prepared in the training dataset, such as... Figure 3 As shown, correct pinyin-level labels can be obtained from correct character-level labels through a pronunciation dictionary, while correct accent labels are frame-level and can be obtained by copying sentence-level labels frame by frame. The pronunciation dictionary is a character-to-pinyin mapping table compiled by language experts, not a neural network model.

[0181] like Figure 2 As shown, in the first stage of training, the Fbank acoustic features are input into the Conformer encoder to extract acoustic representation vectors. The Fbank acoustic features are three-dimensional tensors with dimensions denoted as B*T1*D1, where B is the number of speech samples, T1 is the number of frames, and D1 is the feature dimension. Typically, the Fbank feature dimension D1 = 71, while T1 is generally around several hundred. Subsequently, the acoustic representation vectors are input into the CTC decoder and the accent classifier, respectively. The CTC decoder, combined with the correct pinyin level label, calculates the pinyin level CTC loss and simultaneously outputs the predicted pinyin sequence. It is worth noting that both the correct pinyin level label and the predicted pinyin sequence have dimensions of B*T2*D2. T2 represents the length of the pinyin sequence, which is generally much shorter than T1, typically around tens. D2 represents the total number of pinyin modeling units, approximately 450 without tones and approximately 1200 with tones. This embodiment can choose the option without tones. For the accent classifier, the output of the penultimate DNN layer is used as the accent embedding vector, and the output of the last DNN layer is the predicted accent category. The accent classification cross-entropy loss is then calculated by combining this with the correct accent label. The accent classifier uses frame-level output, meaning it outputs an accent prediction result for each frame. Its time dimension is the same as Fbank's, T1. However, the accent labels provided by data companies are typically sentence-level, so they need to be extended to T1 frames (Fbank's frame count) before calculating the accent classification cross-entropy loss. Therefore, the dimensions of the predicted accent category and the correct accent label can be represented as B*T1*D3, where D3 is the total number of accent categories. The dimension D4 of the accent embedding vector depends on the output dimension of the penultimate DNN layer. Furthermore, Figure 2 Downsampling is an operation that compresses the time dimension. Specifically, first, the compression factor obtained by dividing T1 by T2 is calculated, assuming it is 3. Then, the zero-occupant vector or accent embedding vector is averaged over every 3 frames in the time dimension, thereby reducing the time dimension of these two vectors from T1 to T2.

[0182] In fact, the predicted pinyin sequence and accent embedding vector in the first stage of training have no practical effect in the subsequent process; they mainly play a role in the second stage of training. To ensure that the input information of the Conformer translator remains consistent in the feature dimension between the first and second stages, a placeholder vector of all zeros is used for placeholder operations in the first stage. Specifically, the correct pinyin-level labels are first embedded and encoded, keeping the time dimension T2 unchanged, while the feature dimension D2 is appropriately reduced to D5, which is more suitable for model processing. Then, each pinyin in the time dimension is concatenated with the placeholder vector of all zeros in the feature dimension to obtain a pinyin-level text feature of B*T2*(D4+D5), and the Conformer translator is trained with this feature. Since this pinyin-level text feature only contains pure text information and all pinyin is correct, training the Conformer translator with this feature allows it to focus on learning how to translate pinyin into characters without considering additional interference such as accents or incorrect input pinyin sequences. This reduces the training difficulty of the Conformer translator and speeds up the convergence speed. Finally, the Conformer translator outputs a character-level prediction sequence, which can be combined with the correct character-level labels to obtain the character-level cross-entropy loss. Since in Chinese, pinyin and characters are one-to-one and do not change in the time dimension, the dimensions of both the character-level prediction sequence and the correct character-level labels are B*T2*D6, where D6 is the total number of character modeling units.

[0183] In the second training phase, the encoder, CTC decoder, and accent classifier are trained in the same way as in the first phase, but the input information for the Conformer translator differs. Specifically, the correct pinyin-level labels are replaced with the predicted pinyin sequence, and the all-zero placeholder vectors are replaced with accent embedding vectors, all downsampled to the same number of frames as the pinyin sequence. This replacement process maintains the same dimensions for each corresponding tensor. After the replacement, the training objective of the Conformer translator becomes learning the ability to translate the erroneous predicted pinyin sequence output by the CTC decoder into a word sequence with the aid of the accent embedding vector. In other words, the Conformer translator learns how to translate a correct pinyin sequence into a word sequence without considering accents in the first training phase, a relatively simple task. The second phase builds upon this by learning to translate erroneous pinyin sequences, some of which are due to accents. Therefore, with the provided accent embedding vector, the Conformer translator can learn corresponding error correction capabilities based on different accents, significantly reducing the error rate of the final output word-level sequence.

[0184] In both training phases mentioned above, the total loss function is a weighted sum of three losses: pinyin-level CTC loss, accent-level cross-entropy loss, and character-level cross-entropy loss.

[0185] The accent speech recognition model trained in this embodiment is an end-to-end structure, with all modules trained together. This reduces the mismatch problem caused by optimizing each part separately and allows for direct output of word sequences, making it more convenient for users. Furthermore, the accent speech recognition model includes an accent classification branch, eliminating the need for specific accent labels during inference. This not only facilitates use but also allows for the recognition of accents not included in the training data to a certain extent. In addition, this embodiment uses frame-level accent embedding vectors, which is beneficial for adapting to scenarios where a sentence contains multiple accents.

[0186] This application embodiment effectively improves the accuracy of accent classification branches by using a pinyin-character dual-granularity modeling unit, reducing the negative impact of incorrect accent classification results on accent branches. Furthermore, this application embodiment uses pinyin as a fine-grained unit and characters as a coarse-grained unit, leveraging their consistent sequence length to avoid using a sub-encoder-decoder structure, effectively simplifying the model structure and improving inference speed.

[0187] This application embodiment significantly improves the accuracy of accent classification by introducing dual-granularity modeling into the accent speech recognition task, thereby effectively assisting the speech recognition task. Secondly, this application embodiment focuses on optimizing the difficulties brought to the decoder by dual-granularity modeling. Specifically: 1) This application embodiment adopts a translator based on the Conformer encoder structure, eliminating the acoustic embedding input and using only the pinyin text sequence and accent embedding vector, simplifying the input information; 2) The Conformer translator in this application embodiment does not need to learn to predict future information from known information, making the model more focused on the translation task and greatly reducing the training difficulty; 3) By using a two-stage training scheme, the first stage uses the correct pinyin sequence to train the translator's translation function, and in practical applications, this stage can significantly improve the translator's performance by using additional text data; the second stage uses a predicted pinyin sequence with errors and utilizes accent information as an aid, focusing on training the translator's error correction function for different accents, reducing the model training difficulty, helping to reduce training time and improve model performance.

[0188] The speech model training method of this application embodiment obtains a speech sample set, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag; the speech sample set is used to train an initially established accent speech recognition model to obtain a trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module is used to extract acoustic features from the input speech; the encoder is used to encode the acoustic features output by the acoustic feature extraction module into an acoustic representation vector; the decoder is used to predict a pinyin sequence based on the acoustic representation vector output by the encoder; the accent classifier is used to output frame-level accent category information based on the acoustic representation vector output by the encoder, the frame-level accent category information including the accent category information of each speech frame in the input speech; the translator is used to translate the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech. In this way, by training an accent-based speech recognition model capable of identifying frame-level accent categories, the accuracy of accent classification can be improved, thereby ensuring the accuracy of accent-based speech recognition. This approach is suitable for scenarios where a sentence contains multiple accents. Furthermore, by employing a pinyin-character dual-granularity modeling method, the consistent sequence length of both avoids the need for a sub-encoder-decoder structure, effectively simplifying the model structure and improving inference speed and accuracy.

[0189] See Figure 4 , Figure 4 This is a flowchart of the speech recognition method provided in the embodiments of this application, such as... Figure 4 As shown, it includes the following steps:

[0190] Step 401: Input the speech to be recognized into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator.

[0191] When it is necessary to recognize speech with accents, the speech to be recognized can be obtained, such as receiving speech input from a user, and then the speech to be recognized can be input into a pre-trained accent speech recognition model, so that the speech to be recognized can be recognized by the accent speech recognition model and converted into text.

[0192] The structure of the accent speech recognition model can be the same as described above. Figure 1 The accent speech recognition model described in the illustrated embodiments has a consistent structure, including an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator.

[0193] Step 402: The acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature; the encoder encodes the third acoustic feature to obtain a third acoustic representation vector; the decoder decodes the third acoustic representation vector to obtain a third pinyin sequence; the accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector; the translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized.

[0194] The speech to be recognized is first input into the acoustic feature extraction module for acoustic feature extraction to obtain a third acoustic feature. The third acoustic feature is then input into the encoder, which encodes the third acoustic feature to extract a third acoustic representation vector. The third acoustic representation vector is input into the decoder and the accent classifier, respectively. The decoder decodes the third acoustic representation vector to obtain a third pinyin sequence, and the accent classifier classifies the third acoustic representation vector by accent, outputting accent category information for each speech frame in the speech to be recognized. The accent category information output by the accent classifier is then processed and sent to the translator. The translator translates the third pinyin sequence from pinyin to characters based on the accent category information to obtain the third character sequence corresponding to the speech to be recognized.

[0195] Step 403: Output the third character sequence.

[0196] After recognizing the speech to be recognized using the accent speech recognition model, the translator's result can be output, namely, the third character sequence. Alternatively, the accent category can be output as needed; for example, when it is necessary to recognize a user's accent, the accent classification result of the accent classifier can be output.

[0197] Optionally, the accent speech recognition model is derived from... Figure 1 The speech model is trained using the method described in the embodiment shown.

[0198] That is, it can be adopted Figure 1 The method embodiment shown uses the trained accent speech recognition model to recognize speech, because Figure 1 The accent speech recognition model trained by the method embodiment shown has high accuracy and can accurately recognize speech with different accents, thus ensuring the accuracy of speech recognition.

[0199] To adopt Figure 2Taking the accented speech recognition model and training method shown as an example, the inference process of the accented speech recognition model is as follows: Figure 5 As shown, the Fbank features of the speech to be recognized are first extracted and then fed into the Conformer encoder to extract the acoustic representation vector. This acoustic representation vector is then input into the CTC decoder and the ResNet accent classifier, respectively. The CTC decoder outputs the predicted pinyin sequence, while the ResNet accent classifier outputs the accent embedding vector and the predicted accent category. The predicted accent category can be output or not as needed. Afterward, the predicted pinyin sequence, after embedding encoding, is concatenated with the downsampled accent embedding vector along the feature dimension. The resulting tensor is fed into the Conformer translator, which finally outputs the predicted word sequence.

[0200] The speech recognition method of this application embodiment inputs the speech to be recognized into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator; the acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature; the encoder encodes the third acoustic feature to obtain a third acoustic representation vector; the decoder decodes the third acoustic representation vector to obtain a third pinyin sequence; the accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector; the translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized; and the third character sequence is output. Thus, by employing an accent speech recognition model with pinyin-character dual-granularity modeling units, and by recognizing frame-level accent categories, the accuracy of accent classification can be improved, thereby ensuring the accuracy of accent speech recognition, and making it suitable for scenarios containing multiple accents in a single sentence.

[0201] This application also provides a speech model training device. See [link to relevant documentation]. Figure 6 , Figure 6 This is a structural diagram of the speech model training device provided in the embodiments of this application. Since the principle of the speech model training device in solving the problem is similar to that of the speech model training method in the embodiments of this application, the implementation of this speech model training device can refer to the implementation of the method, and the repeated parts will not be described again.

[0202] like Figure 6 As shown, the speech model training device 600 includes:

[0203] The acquisition module 601 is used to acquire a speech sample set, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag.

[0204] Training module 602 is used to train the initially established accent speech recognition model using the speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into an acoustic representation vector. The decoder predicts a pinyin sequence based on the acoustic representation vector output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vector output by the encoder, and the frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech.

[0205] Optionally, the training module 602 includes:

[0206] The first input unit is used to input the first speech sample from the speech sample set into the accent speech recognition model;

[0207] The acoustic feature extraction module is used to extract acoustic features from the first speech sample to obtain a first acoustic feature; the encoder is used to encode the first acoustic feature to obtain a first acoustic representation vector; the decoder is used to decode the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier is used to classify the first acoustic representation vector by accent to obtain a first accent classification result, the first accent classification result including the accent category to which each speech frame in the first speech sample belongs; and the translator is used to translate the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample.

[0208] The first determining unit is configured to determine a first pinyin level loss based on the first pinyin sequence and the pinyin tags of the first speech sample, wherein the pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary;

[0209] The second determining unit is used to determine the first accent classification loss based on the first accent classification result and the accent label of each speech frame in the first speech sample.

[0210] The third determining unit is used to determine the first character-level loss based on the first character sequence and the character labels of the first speech sample;

[0211] The first training unit is used to train the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0212] Optionally, the accent classifier also outputs a first accent embedding vector of the first speech sample;

[0213] The accent speech recognition model also includes a downsampling module, an embedded encoder, and a splicing module;

[0214] The downsampling module is used to downsample the first accent embedding vector, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence.

[0215] The embedding encoder is used to embed the first pinyin sequence to obtain a first pinyin embedding vector;

[0216] The splicing module is used to splice the first pinyin embedding vector with the downsampled first accent embedding vector to obtain the first spliced ​​vector;

[0217] The translator is used to translate the first concatenated vector to obtain the first character sequence.

[0218] Optionally, the training module 602 also includes:

[0219] The second input unit is used to input the second speech sample from the speech sample set into the accent speech recognition model;

[0220] The acoustic feature extraction module is used to extract acoustic features from the second speech sample to obtain a second acoustic feature; the encoder is used to encode the second acoustic feature to obtain a second acoustic representation vector; the decoder is used to decode the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier is used to classify the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category to which each speech frame in the second speech sample belongs; and the translator is used to translate the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample.

[0221] The fourth determining unit is used to determine the second pinyin level loss based on the second pinyin sequence and the pinyin tags of the second speech sample, wherein the pinyin tags of the second speech sample are obtained by mapping the character tags of the second speech sample through a pronunciation dictionary;

[0222] The fifth determining unit is used to determine the second accent classification loss based on the second accent classification result and the accent label of each speech frame in the second speech sample;

[0223] The sixth determining unit is used to determine the second character-level loss based on the second character sequence and the character labels of the second speech sample;

[0224] The second training unit is used to perform a first-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss.

[0225] The first training unit is used to perform a second-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first-stage training, based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0226] Optionally, the accent speech recognition model further includes an embedded encoder and a splicing module;

[0227] The embedding encoder is used to embed and encode the pinyin tags of the second speech sample to obtain the second pinyin embedding vector;

[0228] The splicing module is used to splice the first pinyin embedding vector with the all-zero vector to obtain a second spliced ​​vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector;

[0229] The translator is used to translate the second concatenated vector to obtain the second character sequence.

[0230] Optionally, the training module 602 includes:

[0231] The first calculation unit is used to perform a weighted summation of the second pinyin-level loss, the second accent classification loss, and the second character-level loss to obtain the first total loss.

[0232] The second calculation unit is used to perform a weighted summation of the first pinyin-level loss, the first accent classification loss, and the first character-level loss to obtain the second total loss.

[0233] The second training unit is used to perform a first-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first total loss;

[0234] The first training unit is used to perform a second-stage training on the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first-stage training, based on the second total loss.

[0235] Optionally, the accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network DNN; the penultimate layer of the accent classifier DNN is used to output the accent embedding vector, and the last layer of the accent classifier DNN is used to output the frame-level accent category.

[0236] The speech model training device 600 provided in this application embodiment can execute... Figure 1 The method embodiments shown are similar in principle and technical effect, and will not be described again here.

[0237] The speech model training device 600 of this application embodiment acquires a speech sample set, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag; the speech sample set is used to train an initially established accent speech recognition model to obtain a trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module is used to extract acoustic features from the input speech; the encoder is used to encode the acoustic features output by the acoustic feature extraction module into an acoustic representation vector; the decoder is used to predict a pinyin sequence based on the acoustic representation vector output by the encoder; the accent classifier is used to output frame-level accent category information based on the acoustic representation vector output by the encoder, the frame-level accent category information including the accent category information of each speech frame in the input speech; the translator is used to translate the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech. In this way, by training an accent-based speech recognition model capable of identifying frame-level accent categories, the accuracy of accent classification can be improved, thereby ensuring the accuracy of accent-based speech recognition. This approach is suitable for scenarios where a sentence contains multiple accents. Furthermore, by employing a pinyin-character dual-granularity modeling method, the consistent sequence length of both avoids the need for a sub-encoder-decoder structure, effectively simplifying the model structure and improving inference speed and accuracy.

[0238] This application also provides a speech recognition device. See [link to relevant documentation]. Figure 7 , Figure 7This is a structural diagram of the speech recognition device provided in the embodiments of this application. Since the principle of the speech recognition device in solving the problem is similar to that of the speech recognition method in the embodiments of this application, the implementation of this speech recognition device can refer to the implementation of the method, and the repeated parts will not be described again.

[0239] like Figure 7 As shown, the voice recognition device 700 includes:

[0240] The input module 701 is used to input the speech to be recognized into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator.

[0241] The speech recognition module 702 is used to extract acoustic features from the speech to be recognized by the acoustic feature extraction module to obtain a third acoustic feature; encode the third acoustic feature by the encoder to obtain a third acoustic representation vector; decode the third acoustic representation vector by the decoder to obtain a third pinyin sequence; output the accent category information of each speech frame in the speech to be recognized by the accent classifier based on the third acoustic representation vector; and translate the third pinyin sequence by the translator based on the accent category information of each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized.

[0242] Output module 703 is used to output the third character sequence.

[0243] Optionally, the accent speech recognition model is derived from... Figure 1 The speech model is trained using the method described in the embodiment shown.

[0244] The speech recognition device 700 provided in this application embodiment can perform... Figure 4 The method embodiments shown are similar in principle and technical effect, and will not be described again here.

[0245] The speech recognition device 700 of this application embodiment inputs the speech to be recognized into a pre-trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature. The encoder encodes the third acoustic feature to obtain a third acoustic representation vector. The decoder decodes the third acoustic representation vector to obtain a third pinyin sequence. The accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector. The translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized. The third character sequence is then output. Thus, by employing an accent speech recognition model with a pinyin-character dual-granularity modeling unit, and by recognizing frame-level accent categories, the accuracy of accent classification can be improved, thereby ensuring the accuracy of accent speech recognition. This model is suitable for scenarios where a sentence contains multiple accents.

[0246] This application also provides an electronic device. For example... Figure 8 As shown, the electronic device according to an embodiment of this application includes:

[0247] In one embodiment, processor 800 is configured to read a program from memory 820 and execute the following processes:

[0248] A speech sample set is obtained, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag.

[0249] The initial accent speech recognition model is trained using the aforementioned speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into acoustic representation vectors. The decoder predicts a pinyin sequence based on the acoustic representation vectors output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vectors output by the encoder, and this frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech.

[0250] In this embodiment, the principle by which the electronic device solves the problem is similar to the speech model training method in the embodiments of this application. Therefore, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0251] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 800 is responsible for managing the bus architecture and general processing, and memory 820 can store data used by processor 800 during operation.

[0252] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0253] Input the first speech sample from the speech sample set into the accent speech recognition model;

[0254] The acoustic feature extraction module extracts acoustic features from the first speech sample to obtain a first acoustic feature; the encoder encodes the first acoustic feature to obtain a first acoustic representation vector; the decoder decodes the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier classifies the first acoustic representation vector by accent to obtain a first accent classification result, which includes the accent category of each speech frame in the first speech sample; the translator translates the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample.

[0255] Based on the first pinyin sequence and the pinyin tags of the first speech sample, a first pinyin level loss is determined. The pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary.

[0256] The first accent classification loss is determined based on the first accent classification result and the accent label of each speech frame in the first speech sample.

[0257] Based on the first character sequence and the character labels of the first speech sample, determine the first character-level loss;

[0258] The structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss.

[0259] Optionally, the accent classifier also outputs a first accent embedding vector of the first speech sample;

[0260] The processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0261] The first accent embedding vector is downsampled, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence;

[0262] The first pinyin sequence is embedded and encoded to obtain the first pinyin embedding vector;

[0263] The first pinyin embedding vector is concatenated with the downsampled first accent embedding vector to obtain the first concatenated vector;

[0264] The first concatenated vector is translated by the translator to obtain the first character sequence.

[0265] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0266] The second speech sample in the speech sample set is input into the accent speech recognition model;

[0267] The acoustic feature extraction module extracts acoustic features from the second speech sample to obtain second acoustic features; the encoder encodes the second acoustic features to obtain a second acoustic representation vector; the decoder decodes the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier classifies the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category of each speech frame in the second speech sample; the translator translates the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample.

[0268] Based on the second pinyin sequence and the pinyin labels of the second speech sample, a second pinyin level loss is determined. The pinyin labels of the second speech sample are obtained by mapping the character labels of the second speech sample through a pronunciation dictionary.

[0269] The second accent classification loss is determined based on the second accent classification result and the accent label of each speech frame in the second speech sample.

[0270] The second character-level loss is determined based on the second character sequence and the character labels of the second speech sample;

[0271] Based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0272] Based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training are trained in the second stage.

[0273] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0274] The pinyin tags of the second speech sample are embedded and encoded to obtain the second pinyin embedding vector;

[0275] The first pinyin embedding vector is concatenated with the all-zero vector to obtain the second concatenated vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector;

[0276] The second concatenated vector is translated by the translator to obtain the second character sequence.

[0277] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0278] The second pinyin-level loss, the second accent classification loss, and the second character-level loss are weighted and summed to obtain the first total loss;

[0279] Based on the first total loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage.

[0280] The first pinyin-level loss, the first accent classification loss, and the first character-level loss are weighted and summed to obtain the second total loss.

[0281] Based on the second total loss, the structural parameters of the encoder, decoder, accent classifier, and translator trained in the first stage are trained in the second stage.

[0282] Optionally, the accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network DNN; the penultimate layer of the accent classifier DNN is used to output the accent embedding vector, and the last layer of the accent classifier DNN is used to output the frame-level accent category.

[0283] In another embodiment, the processor 800 reads the program from the memory 820 and performs the following steps:

[0284] The speech to be recognized is input into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator;

[0285] The acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature; the encoder encodes the third acoustic feature to obtain a third acoustic representation vector; the decoder decodes the third acoustic representation vector to obtain a third pinyin sequence; the accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector; and the translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain the third character sequence corresponding to the speech to be recognized.

[0286] Output the third character sequence.

[0287] Optionally, the accent speech recognition model is derived from... Figure 1 The speech model is trained using the method described in the embodiment shown.

[0288] In this embodiment, the principle by which the electronic device solves the problem is similar to the speech recognition method in the embodiments of this application. Therefore, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0289] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0290] Furthermore, the computer-readable storage medium of this application embodiment is used to store a computer program, which can be executed by a processor. Figure 1 or Figure 4 The steps in the method embodiment shown.

[0291] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0292] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0293] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0294] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A speech model training method, characterized in that, include: A speech sample set is obtained, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag. The initial accent speech recognition model is trained using the aforementioned speech sample set to obtain a trained accent speech recognition model. This model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into acoustic representation vectors. The decoder predicts a pinyin sequence based on the acoustic representation vectors output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vectors output by the encoder, including the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier, obtaining the character sequence corresponding to the input speech. The step of training the initially established accent speech recognition model using the speech sample set includes: Input the first speech sample from the speech sample set into the accent speech recognition model; The acoustic feature extraction module extracts acoustic features from the first speech sample to obtain a first acoustic feature; the encoder encodes the first acoustic feature to obtain a first acoustic representation vector; the decoder decodes the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier classifies the first acoustic representation vector by accent to obtain a first accent classification result, which includes the accent category of each speech frame in the first speech sample; the translator translates the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample. Based on the first pinyin sequence and the pinyin tags of the first speech sample, a first pinyin level loss is determined. The pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary. The first accent classification loss is determined based on the first accent classification result and the accent label of each speech frame in the first speech sample. Based on the first character sequence and the character labels of the first speech sample, determine the first character-level loss; The structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss. The accent classifier also outputs the first accent embedding vector of the first speech sample. The step of translating the first pinyin sequence according to the first accent classification result using the translator to obtain the first character sequence corresponding to the first speech sample includes: The first accent embedding vector is downsampled, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence; The first pinyin sequence is embedded and encoded to obtain the first pinyin embedding vector; The first pinyin embedding vector is concatenated with the downsampled first accent embedding vector to obtain the first concatenated vector; The first concatenated vector is translated by the translator to obtain the first character sequence.

2. The method according to claim 1, characterized in that, Before inputting the first speech sample from the speech sample set into the accent speech recognition model, the method further includes: The second speech sample in the speech sample set is input into the accent speech recognition model; The acoustic feature extraction module extracts acoustic features from the second speech sample to obtain second acoustic features; the encoder encodes the second acoustic features to obtain a second acoustic representation vector; the decoder decodes the second acoustic representation vector to obtain a second pinyin sequence; the accent classifier classifies the second acoustic representation vector by accent to obtain a second accent classification result, which includes the accent category of each speech frame in the second speech sample; the translator translates the pinyin tags of the second speech sample to obtain a second character sequence corresponding to the second speech sample. Based on the second pinyin sequence and the pinyin labels of the second speech sample, a second pinyin level loss is determined. The pinyin labels of the second speech sample are obtained by mapping the character labels of the second speech sample through a pronunciation dictionary. The second accent classification loss is determined based on the second accent classification result and the accent label of each speech frame in the second speech sample. The second character-level loss is determined based on the second character sequence and the character labels of the second speech sample; Based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage. The step of training the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss includes: Based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training are trained in the second stage.

3. The method according to claim 2, characterized in that, The step of translating the pinyin tags of the second speech sample using the translator to obtain the second character sequence corresponding to the second speech sample includes: The pinyin tags of the second speech sample are embedded and encoded to obtain the second pinyin embedding vector; The first pinyin embedding vector is concatenated with the all-zero vector to obtain the second concatenated vector, wherein the time dimension of the all-zero vector is the same as the time dimension of the first pinyin embedding vector; The second concatenated vector is translated by the translator to obtain the second character sequence.

4. The method according to claim 2, characterized in that, The first-stage training of the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the second pinyin-level loss, the second accent classification loss, and the second character-level loss includes: The second pinyin-level loss, the second accent classification loss, and the second character-level loss are weighted and summed to obtain the first total loss; Based on the first total loss, the structural parameters of the encoder, the decoder, the accent classifier, and the translator are trained in the first stage. The second stage of training, based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss, involves training the structural parameters of the encoder, the decoder, the accent classifier, and the translator after the first stage of training. This includes: The first pinyin-level loss, the first accent classification loss, and the first character-level loss are weighted and summed to obtain the second total loss. Based on the second total loss, the structural parameters of the encoder, decoder, accent classifier, and translator trained in the first stage are trained in the second stage.

5. The method according to any one of claims 1 to 4, characterized in that, The accent classifier adopts a structure combining a residual network ResNet and a multi-layer deep neural network (DNN); the penultimate DNN layer of the accent classifier is used to output the accent embedding vector, and the last DNN layer of the accent classifier is used to output the frame-level accent category.

6. A speech recognition method, characterized in that, include: The speech to be recognized is input into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator; The acoustic feature extraction module extracts acoustic features from the speech to be recognized to obtain a third acoustic feature; the encoder encodes the third acoustic feature to obtain a third acoustic representation vector; the decoder decodes the third acoustic representation vector to obtain a third pinyin sequence; the accent classifier outputs accent category information for each speech frame in the speech to be recognized based on the third acoustic representation vector; and the translator translates the third pinyin sequence based on the accent category information for each speech frame in the speech to be recognized to obtain the third character sequence corresponding to the speech to be recognized. Output the third word sequence; The accent speech recognition model is trained using any one of the methods in claims 1 to 5.

7. A speech model training device, characterized in that, include: The acquisition module is used to acquire a speech sample set, wherein each speech sample in the speech sample set is labeled with a character tag, and each speech frame of each speech sample is labeled with an accent tag. A training module is used to train an initially established accent speech recognition model using the speech sample set to obtain a trained accent speech recognition model. The accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator. The acoustic feature extraction module extracts acoustic features from the input speech. The encoder encodes the acoustic features output by the acoustic feature extraction module into acoustic representation vectors. The decoder predicts a pinyin sequence based on the acoustic representation vectors output by the encoder. The accent classifier outputs frame-level accent category information based on the acoustic representation vectors output by the encoder, and the frame-level accent category information includes the accent category information for each speech frame in the input speech. The translator translates the pinyin sequence predicted by the decoder based on the frame-level accent category information output by the accent classifier to obtain the character sequence corresponding to the input speech. The training module includes: The first input unit is used to input the first speech sample from the speech sample set into the accent speech recognition model; The acoustic feature extraction module is used to extract acoustic features from the first speech sample to obtain a first acoustic feature; the encoder is used to encode the first acoustic feature to obtain a first acoustic representation vector; the decoder is used to decode the first acoustic representation vector to obtain a first pinyin sequence; the accent classifier is used to classify the first acoustic representation vector by accent to obtain a first accent classification result, the first accent classification result including the accent category to which each speech frame in the first speech sample belongs; and the translator is used to translate the first pinyin sequence based on the first accent classification result to obtain a first character sequence corresponding to the first speech sample. The first determining unit is configured to determine a first pinyin level loss based on the first pinyin sequence and the pinyin tags of the first speech sample, wherein the pinyin tags of the first speech sample are obtained by mapping the character tags of the first speech sample through a pronunciation dictionary; The second determining unit is used to determine the first accent classification loss based on the first accent classification result and the accent label of each speech frame in the first speech sample. The third determining unit is used to determine the first character-level loss based on the first character sequence and the character labels of the first speech sample; The first training unit is used to train the structural parameters of the encoder, the decoder, the accent classifier, and the translator based on the first pinyin-level loss, the first accent classification loss, and the first character-level loss. The accent classifier also outputs the first accent embedding vector of the first speech sample. The accent speech recognition model also includes a downsampling module, an embedded encoder, and a splicing module; The downsampling module is used to downsample the first accent embedding vector, wherein the time dimension of the first accent embedding vector after downsampling is the same as the time dimension of the first pinyin sequence. The embedding encoder is used to embed the first pinyin sequence to obtain a first pinyin embedding vector; The splicing module is used to splice the first pinyin embedding vector with the downsampled first accent embedding vector to obtain the first spliced ​​vector; The translator is used to translate the first concatenated vector to obtain the first character sequence.

8. A voice recognition device, characterized in that, include: The input module is used to input the speech to be recognized into a pre-trained accent speech recognition model, wherein the accent speech recognition model includes an acoustic feature extraction module, an encoder, a decoder, an accent classifier, and a translator; The speech recognition module is used to extract acoustic features from the speech to be recognized by the acoustic feature extraction module to obtain a third acoustic feature; to encode the third acoustic feature by the encoder to obtain a third acoustic representation vector; to decode the third acoustic representation vector by the decoder to obtain a third pinyin sequence; to output the accent category information of each speech frame in the speech to be recognized by the accent classifier based on the third acoustic representation vector; and to translate the third pinyin sequence by the translator based on the accent category information of each speech frame in the speech to be recognized to obtain a third character sequence corresponding to the speech to be recognized. The output module is used to output the third character sequence; The accent speech recognition model is trained using any one of the methods in claims 1 to 5.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps in the speech model training method as described in any one of claims 1 to 5; or to implement the steps in the speech recognition method as described in claim 6.

10. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the speech model training method as described in any one of claims 1 to 5; or implements the steps in the speech recognition method as described in claim 6.

Images