Accent speech recognition method, device, equipment and storage medium

Abstract

The present invention relates to the field of artificial intelligence and discloses an accent speech recognition method, apparatus, device, and storage medium. The method comprises: obtaining multiple accent speech samples, and performing accent and phoneme annotation on each of the accent speech samples, thereby obtaining multiple accent labels and phoneme state sequences; extracting first acoustic features corresponding to each of the accent speech samples; inputting the first acoustic features, the accent labels, and the phoneme state sequences into a preset training model for training to obtain an accent speech recognition model; obtaining an accent speech to be recognized, and extracting a second acoustic feature of the accent speech; inputting the second acoustic feature into the accent speech recognition model for recognition, and outputting a second accent category and phoneme state sequence corresponding to the accent speech. The present invention achieves accurate recognition of multiple accented speech by introducing a gate mechanism and an accent recognition branch.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an accent speech recognition method, apparatus, device and storage medium. Background Art

[0002] As an important technology for human-computer interaction, speech recognition is becoming increasingly important in industrial production and daily life, such as smart home scenarios and smart car systems.

[0003] How to effectively recognize accented Mandarin and improve the recognition rate has become a hot topic in current research. However, traditional speech recognition methods have a very low recognition rate when recognizing Mandarin with a clear accent, resulting in many errors in the recognition results. Summary of the Invention

[0004] The main purpose of the present invention is to solve the technical problem of low recognition accuracy of existing accented speech.

[0005] A first aspect of the present invention provides an accent speech recognition method, the accent speech recognition method comprising:

[0006] Acquire multiple accented speech samples, and perform accent annotation and phoneme annotation on each of the accented speech samples, thereby obtaining multiple accent labels and phoneme state sequences;

[0007] Extracting first acoustic features corresponding to each of the accented speech samples respectively;

[0008] Inputting the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model;

[0009] Acquiring an accented speech to be recognized and extracting a second acoustic feature of the accented speech;

[0010] The second acoustic feature is input into the accent speech recognition model for recognition, and a second accent category and phoneme state sequence corresponding to the accent speech is output.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the training model includes: a Baseline model and an Xvector model, the Baseline model includes a first TDNN network, a gate mechanism layer, a second TDNN network, and a Softmax layer, and inputting the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model includes:

[0012] Inputting the first acoustic feature into the first TDNN network to extract acoustic information to obtain first valid information;

[0013] Inputting the first valid information into the Xvector model to perform accent category recognition to obtain a first accent category corresponding to the accented speech sample;

[0014] Inputting the first valid information and the first accent category into the gate mechanism layer to perform accent information fusion to obtain first accent fusion information;

[0015] Inputting the first accent fusion information into the second TDNN network to perform accent information adjustment to obtain first accent adjustment information;

[0016] Inputting the first accent adjustment information into the Softmax layer of the Baseline model to perform feature calculation, and outputting a phoneme state sequence corresponding to the accented speech sample;

[0017] According to the accent label and the phoneme state sequence, calling a preset loss function to calculate an error value of the first accent category and the phoneme state sequence corresponding to the accent speech sample;

[0018] According to the error value, parameters of the Baseline model and the Xvector model are tuned until the Baseline model and the Xvector model converge to obtain an accented speech recognition model.

[0019] Optionally, in a second implementation of the first aspect of the present invention, the Xvector model includes a third TDNN network, a statistics pooling layer, a fourth TDNN network, and a Softmax layer, and inputting the first valid information into the Xvector model for category recognition to obtain the first accent category corresponding to the accented speech sample includes:

[0020] Inputting the first valid information into the third TDNN network to perform acoustic feature segmentation to obtain first frame-level feature information;

[0021] Inputting the first frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain first average feature information;

[0022] Inputting the first average feature information into the fourth TDNN network to extract acoustic feature information to obtain first average effective information;

[0023] The first average effective information is input into the Softmax layer in the Xvector model to perform an accent classification probability operation to obtain a first accent category corresponding to the accented speech sample.

[0024] Optionally, in a third implementation of the first aspect of the present invention, extracting the first acoustic features corresponding to the accented speech samples includes:

[0025] Frame each of the accented speech samples to obtain a plurality of speech frames;

[0026] Pre-enhancement and windowing are performed on each of the speech frames to obtain a first acoustic feature.

[0027] Optionally, in a fourth implementation of the first aspect of the present invention, inputting the second acoustic feature into the accent speech recognition model for recognition, and outputting a second accent category and phoneme state sequence corresponding to the accent speech includes:

[0028] Inputting the second acoustic feature into the first TDNN network to extract acoustic information to obtain second effective information;

[0029] Inputting the second valid information into the Xvector model for category recognition, and outputting a second accent category corresponding to the accented speech;

[0030] Inputting the second valid information and the second accent category into the gate mechanism layer to perform accent information fusion to obtain second accent fusion information;

[0031] Inputting the second accent fusion information into the second TDNN network to perform accent information adjustment to obtain second accent adjustment information;

[0032] The second accent adjustment information is input into the Softmax layer of the Baseline model for feature calculation, and a phoneme state sequence corresponding to the accented speech is output.

[0033] Optionally, in a fifth implementation of the first aspect of the present invention, inputting the second valid information into the Xvector model for category recognition, and outputting the second accent category corresponding to the accented speech includes:

[0034] Inputting the second valid information into the third TDNN network to perform acoustic feature segmentation to obtain second frame-level feature information;

[0035] Inputting the second frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain second average feature information;

[0036] Inputting the second average feature information into the fourth TDNN network to extract acoustic feature information to obtain second average effective information;

[0037] The second average effective information is input into the Softmax layer in the Xvector model to perform accent probability calculation, and a second accent category corresponding to the accented speech is output.

[0038] Optionally, in a sixth implementation of the first aspect of the present invention, the acquiring of multiple accented speech samples, and performing accent tagging and phoneme tagging on each of the accented speech samples, to obtain corresponding multiple accent labels and phoneme state sequences includes:

[0039] Acquire multiple accented speech samples and convert each of the accented speech samples into text;

[0040] Each of the accented speech samples is labeled with an accent category to obtain a plurality of accent labels, and the text is transcribed into regular pinyin and labeled with word segmentation information to obtain a phoneme state sequence.

[0041] A second aspect of the present invention provides an accent speech recognition device, the accent speech recognition device comprising:

[0042] A sample acquisition module is used to acquire multiple accent speech samples and perform accent annotation and phoneme annotation on each of the accent speech samples to obtain multiple accent labels and phoneme state sequences;

[0043] A feature extraction module, configured to extract first acoustic features corresponding to each of the accented speech samples;

[0044] a model training module, configured to input the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model;

[0045] An accented speech preprocessing module, configured to obtain the accented speech to be recognized and extract the second acoustic feature of the accented speech;

[0046] The accent speech recognition module is used to input the second acoustic feature into the accent speech recognition model for recognition, and output a second accent category and phoneme state sequence corresponding to the accent speech.

[0047] Optionally, in a first implementation of the second aspect of the present invention, the training model includes: a Baseline model and an Xvector model, the Baseline model includes a first TDNN network, a gate mechanism layer, a second TDNN network and a Softmax layer, and the model training module includes:

[0048] A training extraction unit, configured to input the first acoustic feature into the first TDNN network to extract acoustic information and obtain first valid information;

[0049] A training accent recognition unit is configured to input the first valid information into the Xvector model to perform accent category recognition, and obtain a first accent category corresponding to the accent speech sample;

[0050] The training speech recognition unit is configured to input the first valid information and the first accent category into the gate mechanism layer for accent information fusion to obtain first accent fusion information; input the first accent fusion information into the second TDNN network for accent information adjustment to obtain first accent adjustment information; input the first accent adjustment information into the Softmax layer of the Baseline model for feature calculation, and output a phoneme state sequence corresponding to the accented speech sample;

[0051] A model optimization unit is configured to call a preset loss function based on the accent label and the phoneme state sequence to calculate an error value of the first accent category and the phoneme state sequence corresponding to the accent speech sample; and perform parameter tuning on the Baseline model and the Xvector model based on the error value until the Baseline model and the Xvector model converge to obtain an accent speech recognition model.

[0052] Optionally, in a second implementation of the second aspect of the present invention, the Xvector model includes a third TDNN network, a statistics pooling layer, a fourth TDNN network, and a Softmax layer, and the training accent recognition unit is specifically used to:

[0053] The first valid information is input into the third TDNN network for acoustic feature segmentation to obtain first frame-level feature information; the first frame-level feature information is input into the Statistics pooling layer for mean-variance calculation to obtain first average feature information; the first average feature information is input into the fourth TDNN network for acoustic feature information extraction to obtain first average valid information; the first average valid information is input into the Softmax layer in the Xvector model for accent classification probability calculation to obtain a first accent category corresponding to the accent speech sample.

[0054] Optionally, in a third implementation of the second aspect of the present invention, the feature extraction module is specifically configured to:

[0055] Frame each of the accented speech samples to obtain a plurality of speech frames;

[0056] Pre-enhancement and windowing are performed on each of the speech frames to obtain a first acoustic feature.

[0057] Optionally, in a fourth implementation of the second aspect of the present invention, the accent speech recognition module includes:

[0058] an information extraction unit, configured to input the second acoustic feature into the first TDNN network to extract acoustic information and obtain second valid information;

[0059] an accent recognition unit, configured to input the second valid information into the Xvector model for category recognition, and output a second accent category corresponding to the accented speech;

[0060] The speech recognition unit is configured to input the second valid information and the second accent category into the gate mechanism layer for accent information fusion to obtain second accent fusion information; input the second accent fusion information into the second TDNN network for accent information adjustment to obtain second accent adjustment information; input the second accent adjustment information into the Softmax layer in the Baseline model for feature calculation, and output a phoneme state sequence corresponding to the accented speech.

[0061] Optionally, in a fifth implementation of the second aspect of the present invention, the accent recognition unit is specifically configured to:

[0062] Inputting the second valid information into the third TDNN network to perform acoustic feature segmentation to obtain second frame-level feature information;

[0063] Inputting the second frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain second average feature information;

[0064] Inputting the second average feature information into the fourth TDNN network to extract acoustic feature information to obtain second average effective information;

[0065] The second average effective information is input into the Softmax layer in the Xvector model to perform accent probability calculation, and a second accent category corresponding to the accented speech is output.

[0066] Optionally, in a sixth implementation of the second aspect of the present invention, the sample acquisition module is specifically configured to:

[0067] Acquire multiple accented speech samples and convert each of the accented speech samples into text;

[0068] Each of the accented speech samples is labeled with an accent category to obtain a plurality of accent labels, and the text is transcribed into regular pinyin and labeled with word segmentation information to obtain a phoneme state sequence.

[0069] A third aspect of the present invention provides an accent speech recognition device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory to enable the accent speech recognition device to execute the above-mentioned accent speech recognition method.

[0070] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned accent speech recognition method.

[0071] The technical solution provided by the present invention addresses the problem that existing speech recognition cannot effectively recognize accented speech. Therefore, a gate mechanism and speech recognition branch are introduced on the basis of the original speech recognition. Specifically, the following steps are performed: first, accent and phoneme annotation are performed on a plurality of acquired accented speech samples to obtain a plurality of corresponding accent labels and phoneme state sequences, and then the acoustic features of the sample accented speech are extracted; the acoustic features of the sample, the accent labels, and the phoneme state sequences are input into a preset training model for training to obtain an accented speech recognition model. The training model adopted by the present invention adds an Xvector structure network and a gate mechanism to the baseline model, so that the model accent learns the differences between Mandarin with different accents, thereby achieving effective adaptive training for Mandarin with a specific accent; finally, the acoustic features extracted from the accented speech to be recognized are input into the accented speech recognition model for recognition, and the accent category and phoneme state sequence corresponding to the accented speech are output. The present invention has better adaptability to accented speech and greatly improves the recognition rate of accented speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a schematic diagram of a first embodiment of an accent speech recognition method according to an embodiment of the present invention;

[0073] Figure 2 2 is a schematic diagram of a second embodiment of the accent speech recognition method according to an embodiment of the present invention;

[0074] Figure 3 2 is a schematic diagram of a third embodiment of the accent speech recognition method according to an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of an embodiment of an accent speech recognition device according to an embodiment of the present invention;

[0076] Figure 5 FIG. 1 is a schematic diagram of an embodiment of an accent speech recognition device in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] Embodiments of the present invention provide an accented speech recognition method, apparatus, device, and storage medium. The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0078] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the accent speech recognition method in the embodiment of the present invention includes:

[0079] 101. Acquire multiple accented speech samples, and perform accent annotation and phoneme annotation on each of the accented speech samples, thereby obtaining multiple accent labels and phoneme state sequences.

[0080] It is understandable that the execution subject of the present invention may be an accent speech recognition device, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0081] In this example, accented speech samples were collected from Mandarin speakers from various regions, such as Guangdong, Guangxi, Jiangxi, and Xi'an. The collected speech was annotated by first determining the validity of the accented speech based on its starting time period and the corresponding speech text. If the speech was valid, the valid speech was annotated with multiple attributes, such as "Guangdong, Guangxi, Jiangxi, Xi'an, etc.", to obtain the accent-tagged speech.

[0082] Optionally, in one embodiment, the acquiring of multiple accented speech samples and performing accent tagging and phoneme tagging on each of the accented speech samples to obtain multiple accent labels and phoneme state sequences includes:

[0083] Acquire multiple accented speech samples and convert each of the accented speech samples into text;

[0084] Each of the accented speech samples is labeled with an accent category to obtain a plurality of accent labels, and the text is transcribed into regular pinyin and labeled with word segmentation information to obtain a phoneme state sequence.

[0085] In this embodiment, first, a speech-to-text tool is used to convert various accent speech samples into corresponding texts, and then the texts are converted into phonetic characters, that is, Chinese characters are converted into pinyin.

[0086] For example: The phoneme state sequence obtained from "Shanghai Restaurant" is "shanghaicanting". Among them, the phoneme corresponding to "上" is "shang", the phoneme corresponding to "海" is "hai", the phoneme corresponding to "餐" is "can", and the phoneme corresponding to "厅" is "ting".

[0087] 102. Respectively extract the first acoustic features corresponding to each of the accent speech samples;

[0088] In this embodiment, the Kaldi tool is used to extract features from the accent speech samples. The Kaldi framework is a script specifically for extracting speech features. For example, to extract the Pip pitch feature, Klp_pitch.sh is used. The input is the audio stream in the wav format of the training audio data, and the training data is audio data with an accent. The Kaldi tool can obtain the acoustic features of the accent speech by framing the accent speech and then performing pre-emphasis and windowing on the obtained frame-level speech.

[0089] Optionally, in one embodiment, the respectively extracting the first acoustic features corresponding to each of the accent speech samples includes:

[0090] Respectively frame each of the accent speech samples to obtain a plurality of speech frames;

[0091] Perform pre-emphasis and windowing on each of the speech frames to obtain the first acoustic features.

[0092] In this embodiment, the Kaldi tool first calculates the number of speech frames in an accent speech file (usually the frame length is 25ms and the frame shift is 10ms). Extract data for each frame, perform pre-emphasis and remove the DC offset, and it can also be multiplied by a window function, such as the Hamming window. Finally, perform feature-level channel length normalization to obtain the acoustic features of the accent speech samples.

[0093] 103. Input the first acoustic features, the accent labels, and the phoneme state sequences into a preset training model for training to obtain an accent speech recognition model;

[0094] In this embodiment, the training model includes a Baseline model and an Xvector model. The Baseline model consists of a four-layer TDNN network, a gate mechanism layer, a three-layer TDNN network, and a Softmax layer. The Xvector model consists of a one-layer TDNN network, a statistics pooling layer, two TDNN layers, and a Softmax layer. The first acoustic feature serves as the training speech feature, and the accent label and phoneme state sequence are used to determine the error rate of the first accent category and phoneme state sequence output during model training. Model training terminates when the error rate is less than a preset convergence threshold.

[0095] 104. Acquire an accented speech to be recognized, and extract a second acoustic feature of the accented speech;

[0096] In this embodiment, the Kaldi tool is used to extract features of the accented speech to be recognized. The Kaldi framework is a script specifically used to extract speech features. The Kaldi tool divides the accented speech to be recognized into frames and then pre-enhances and windows the obtained frame-level speech to obtain the acoustic features of the accented speech to be recognized.

[0097] 105. Input the second acoustic feature into the accent speech recognition model for recognition, and output a second accent category and phoneme state sequence corresponding to the accent speech.

[0098] In this embodiment, the Xvector model in the accent speech recognition model performs accent recognition on the acoustic features to obtain the second accent category of the accent speech. The second accent category and the acoustic features processed by the 4-layer TDNN network are then processed by a gate mechanism. The gate mechanism can control the flow of accent category information. The 3-layer TDNN network and Softmax operation in the Baseline model after the gate mechanism processing are combined to output the phoneme state sequence of the accent speech.

[0099] In an embodiment of the present invention, in order to solve the problem that existing speech recognition cannot effectively recognize accented speech, a gate mechanism and speech recognition branch are introduced on the basis of the original speech recognition. Specifically, the following steps are performed: first, the accent and phoneme annotations are performed on the acquired multiple accented speech samples to obtain multiple accent labels and phoneme state sequences, and then the acoustic features of the sample accented speech are extracted; the acoustic features of the sample, the accent labels, and the phoneme state sequences are input into a preset training model for training to obtain an accented speech recognition model. The training model used in the present invention adds an Xvector structure network and a gate mechanism to the baseline model, so that the model accent learns the differences between Mandarin with different accents, thereby achieving effective adaptive training for Mandarin with a specific accent; finally, the acoustic features extracted from the accented speech to be recognized are input into the accented speech recognition model for recognition, and the accent category and phoneme state sequence corresponding to the accented speech are output. The present invention has better adaptability to accented speech and greatly improves the recognition rate of accented speech.

[0100] See also Figure 2 The second embodiment of the accent speech recognition method in the embodiment of the present invention includes:

[0101] 201. Acquire multiple accented speech samples, and perform accent annotation and phoneme annotation on each of the accented speech samples, thereby obtaining multiple accent labels and phoneme state sequences.

[0102] 202. Extracting first acoustic features corresponding to each of the accented speech samples;

[0103] 203. Input the first acoustic feature into the first TDNN network to extract acoustic information to obtain first valid information;

[0104] In this embodiment, the first TDNN network is a 4-layer TDNN network. The TDNN network is multi-layered, and each layer has a strong abstract capability for features, including input layer, hidden layer and output layer. Therefore, the TDNN network is capable of expressing the temporal relationship of speech features. The 4-layer TDNN network structure is used to extract the expression of high-level acoustic information in acoustic features.

[0105] 204. Input the first valid information into the Xvector model to perform accent category recognition to obtain a first accent category corresponding to the accented speech sample;

[0106] Optionally, in one embodiment, inputting the first valid information into the Xvector model to perform accent category recognition to obtain the first accent category corresponding to the accented speech sample includes:

[0107] Inputting the first valid information into the third TDNN network to perform acoustic feature segmentation to obtain first frame-level feature information;

[0108] In this embodiment, the third TDNN network is a 1-layer TDNN network, and the third TDNN network divides the acoustic features processed by the 4-layer TDNN network into frame-level speech features.

[0109] Inputting the first frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain first average feature information;

[0110] In this embodiment, the statistics pooling layer calculates the mean and variance of the frame-level speech features according to the entire audio channel, that is, it averages all the frame-level speech features included in the audio and converts the frame-level speech features into the average speech features of the single-channel audio.

[0111] Inputting the first average feature information into the fourth TDNN network to extract acoustic feature information to obtain first average effective information;

[0112] The first average effective information is input into the Softmax layer in the Xvector model to perform an accent classification probability operation to obtain a first accent category corresponding to the accented speech sample.

[0113] In this embodiment, the fourth TDNN is a two-layer TDNN. The average speech features are processed through the output Softmax layer to output accent classification information. The Softmax layer can intuitively identify direct correlations between the average speech features, map the average speech features, and normalize all input data.

[0114] 205. Input the first valid information and the first accent category into the gate mechanism layer to perform accent information fusion to obtain first accent fusion information;

[0115] 206. Input the first accent fusion information into the second TDNN network to perform accent information adjustment to obtain first accent adjustment information;

[0116] 207. Input the first accent adjustment information into the Softmax layer of the Baseline model to perform feature calculation, and output a phoneme state sequence corresponding to the accented speech sample;

[0117] 208. Call a preset loss function based on the accent label and the phoneme state sequence to calculate an error value between the first accent category and the phoneme state sequence corresponding to the accented speech sample;

[0118] In this embodiment, the gate mechanism controls the weights of each accent output by the accent classification network, thereby controlling the input of different accent information into the speech recognition branch. The second TDNN network includes a three-layer TDNN network. The total loss function L is solved by weighted averaging the loss functions of different branches. The calculation formula is:

[0119] L=w1*Lpri+w2*Lsec

[0120] Among them, w1: represents the weight of the speech recognition branch (main branch); w2 represents the accent classification branch (secondary branch); Lpri represents the loss function of the speech recognition branch (main branch); Lsec represents the loss function of the accent classification branch (secondary branch); L represents the total loss function of the network.

[0121] 209. Optimize parameters of the Baseline model and the Xvector model according to the error value until the Baseline model and the Xvector model converge to obtain an accented speech recognition model.

[0122] 210. Acquire an accented speech to be recognized, and extract a second acoustic feature of the accented speech;

[0123] 211. Input the second acoustic feature into the accent speech recognition model for recognition, and output a second accent category and phoneme state sequence corresponding to the accent speech.

[0124] In an embodiment of the present invention, a multi-accent multi-task joint training network can realize the joint training of speech recognition tasks and accent classification tasks, including a speech recognition branch and an accent classification branch. The speech recognition branch uses phoneme state as a training target to realize automatic speech recognition, and the accent classification branch uses accent category as a training target to realize accent classification. In addition, a gate mechanism is introduced to effectively control the inflow of accent category information, so that the trained model has better adaptability to the task of accent speech recognition.

[0125] See also Figure 3 The third embodiment of the accent speech recognition method in the embodiment of the present invention includes:

[0126] 301. Acquire multiple accented speech samples, and perform accent annotation and phoneme annotation on each of the accented speech samples, thereby obtaining multiple accent labels and phoneme state sequences.

[0127] 302. Extracting first acoustic features corresponding to each of the accented speech samples;

[0128] 303. Input the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model;

[0129] 304. Acquire the accented speech to be recognized, and extract the second acoustic feature of the accented speech;

[0130] In this embodiment, the Kaldi tool is used to extract features of the accented speech to be recognized. The Kaldi framework is a script specifically used to extract speech features. The Kaldi tool divides the accented speech to be recognized into frames and then pre-enhances and windows the obtained frame-level speech to obtain the acoustic features of the accented speech to be recognized.

[0131] 305. Input the second acoustic feature into the first TDNN network to extract acoustic information to obtain second valid information;

[0132] 306. Input the second valid information into the Xvector model for category recognition, and output a second accent category corresponding to the accented speech;

[0133] Optionally, in one embodiment, inputting the second valid information into the Xvector model for category recognition, and outputting the second accent category corresponding to the accented speech includes:

[0134] Inputting the second valid information into the third TDNN network to perform acoustic feature segmentation to obtain second frame-level feature information;

[0135] Inputting the second frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain second average feature information;

[0136] Inputting the second average feature information into the fourth TDNN network to extract acoustic feature information to obtain second average effective information;

[0137] The second average effective information is input into the Softmax layer in the Xvector model to perform accent probability calculation, and a second accent category corresponding to the accented speech is output.

[0138] 307. Input the second valid information and the second accent category into the gate mechanism layer to perform accent information fusion to obtain second accent fusion information.

[0139] In this embodiment, the gate mechanism acts as a flow limiter, controlling the magnitude of values ​​through a dot product operation. The gate mechanism layer uses gates to control the flow of information about accent categories, connecting to the speech recognition branch network. This allows the speech recognition network to adjust the expressive power of the internal network based on specific accents. Different gate structures can be set depending on how the accent category label is combined with the speech recognition branch. Three different gate mechanisms are described here:

[0140] g(hi,va)=hi+Vva+b;

[0141] g(hi,va)=hi*Vva+b;

[0142] g(hi,va)=hi*(hi+Vva+b);

[0143] Among them, g(hi, va) represents the gate value, which needs to be passed to the (i+1)th layer, V represents the M*N matrix; hi represents the output of the i-th layer network; va represents the dimension N; and b is a vector.

[0144] 308. Input the second accent fusion information into the second TDNN network to perform accent information adjustment to obtain second accent adjustment information.

[0145] 309. Input the second accent adjustment information into the Softmax layer of the Baseline model to perform feature calculation, and output a phoneme state sequence corresponding to the accented speech.

[0146] In this embodiment of the present invention, the introduction of a gating mechanism enables the speech recognition branch to perform targeted learning on speech data with different accents, enabling recognition of multiple accents. The baseline model inputs the expression features corresponding to a text sequence. It then re-scores multiple sequences using a language model, selects the sequence with the highest score, and outputs multiple phoneme recognition sequences. Combined with the introduction of an accent recognition network, this results in more accurate recognition of accented speech.

[0147] The above describes the accent speech recognition method in the embodiment of the present invention. The following describes the accent speech recognition device in the embodiment of the present invention. Figure 4 In one embodiment of the present invention, an accent speech recognition device includes:

[0148] The sample acquisition module 401 is used to acquire multiple accented speech samples and perform accent annotation and phoneme annotation on each of the accented speech samples to obtain multiple accent labels and phoneme state sequences.

[0149] A feature extraction module 402 is configured to extract first acoustic features corresponding to each of the accented speech samples;

[0150] A model training module 403 is configured to input the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model;

[0151] The accented speech preprocessing module 404 is used to obtain the accented speech to be recognized and extract the second acoustic feature of the accented speech;

[0152] The accent speech recognition module 405 is configured to input the second acoustic feature into the accent speech recognition model for recognition, and output a second accent category and phoneme state sequence corresponding to the accent speech.

[0153] Optionally, in one embodiment, the training model includes: a Baseline model and an Xvector model, the Baseline model includes a first TDNN network, a gate mechanism layer, a second TDNN network and a Softmax layer, and the model training module 403 includes:

[0154] A training extraction unit 4031 is configured to input the first acoustic feature into the first TDNN network to extract acoustic information and obtain first valid information;

[0155] The training accent recognition unit 4032 is configured to input the first valid information into the Xvector model to perform accent category recognition and obtain a first accent category corresponding to the accent speech sample;

[0156] The training speech recognition unit 4033 is configured to input the first valid information and the first accent category into the gate mechanism layer for accent information fusion to obtain first accent fusion information; input the first accent fusion information into the second TDNN network for accent information adjustment to obtain first accent adjustment information; input the first accent adjustment information into the Softmax layer of the Baseline model for feature calculation, and output a phoneme state sequence corresponding to the accented speech sample;

[0157] The model optimization unit 4034 is used to call a preset loss function based on the accent label and the phoneme state sequence to calculate the error value of the first accent category and the phoneme state sequence corresponding to the accent speech sample; and according to the error value, perform parameter tuning on the Baseline model and the Xvector model until the Baseline model and the Xvector model converge to obtain an accent speech recognition model.

[0158] Optionally, in one embodiment, the Xvector model includes a third TDNN network, a Statisticspooling layer, a fourth TDNN network, and a Softmax layer, and the training accent recognition unit 4032 is specifically configured to:

[0159] The first valid information is input into the third TDNN network for acoustic feature segmentation to obtain first frame-level feature information; the first frame-level feature information is input into the Statistics pooling layer for mean-variance calculation to obtain first average feature information; the first average feature information is input into the fourth TDNN network for acoustic feature information extraction to obtain first average valid information; the first average valid information is input into the Softmax layer in the Xvector model for accent classification probability calculation to obtain a first accent category corresponding to the accent speech sample.

[0160] Optionally, in one embodiment, the feature extraction module 402 is specifically configured to:

[0161] Frame each of the accented speech samples to obtain a plurality of speech frames;

[0162] Pre-enhancement and windowing are performed on each of the speech frames to obtain a first acoustic feature.

[0163] Optionally, in one embodiment, the accent speech recognition module 405 includes:

[0164] An information extraction unit 4051 is configured to input the second acoustic feature into the first TDNN network to extract acoustic information and obtain second valid information;

[0165] The accent recognition unit 4052 is configured to input the second valid information into the Xvector model for category recognition, and output a second accent category corresponding to the accented speech;

[0166] The speech recognition unit 4053 is used to input the second valid information and the second accent category into the gate mechanism layer for accent information fusion to obtain second accent fusion information; input the second accent fusion information into the second TDNN network for accent information adjustment to obtain second accent adjustment information; input the second accent adjustment information into the Softmax layer in the Baseline model for feature calculation, and output the phoneme state sequence corresponding to the accented speech.

[0167] Optionally, in one embodiment, the accent recognition unit 4052 is specifically configured to:

[0168] Inputting the second valid information into the third TDNN network to perform acoustic feature segmentation to obtain second frame-level feature information;

[0169] Inputting the second frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain second average feature information;

[0170] Inputting the second average feature information into the fourth TDNN network to extract acoustic feature information to obtain second average effective information;

[0171] The second average effective information is input into the Softmax layer in the Xvector model to perform accent probability calculation, and a second accent category corresponding to the accented speech is output.

[0172] Optionally, in one embodiment, the sample acquisition module 401 is specifically configured to:

[0173] Acquire multiple accented speech samples and convert each of the accented speech samples into text;

[0174] Each of the accented speech samples is labeled with an accent category to obtain a plurality of accent labels, and the text is transcribed into regular pinyin and labeled with word segmentation information to obtain a phoneme state sequence.

[0175] In an embodiment of the present invention, in order to solve the problem that existing speech recognition cannot effectively recognize accented speech, a gate mechanism and speech recognition branch are introduced on the basis of the original speech recognition. Specifically, the following steps are performed: first, the accent and phoneme annotations are performed on the acquired multiple accented speech samples to obtain multiple accent labels and phoneme state sequences, and then the acoustic features of the sample accented speech are extracted; the acoustic features of the sample, the accent labels, and the phoneme state sequences are input into a preset training model for training to obtain an accented speech recognition model. The training model used in the present invention adds an Xvector structure network and a gate mechanism to the baseline model, so that the model accent learns the differences between Mandarin with different accents, thereby achieving effective adaptive training for Mandarin with a specific accent; finally, the acoustic features extracted from the accented speech to be recognized are input into the accented speech recognition model for recognition, and the accent category and phoneme state sequence corresponding to the accented speech are output. The present invention has better adaptability to accented speech and greatly improves the recognition rate of accented speech.

[0176] above Figure 4 The accent speech recognition apparatus in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The following describes the accent speech recognition device in the embodiment of the present invention in detail from the perspective of hardware processing.

[0177] Figure 5Figure 5 is a schematic diagram of the structure of an accent speech recognition device provided by an embodiment of the present invention. The accent speech recognition device 500 may vary significantly depending on configuration or performance. It may include one or more processors (central processing units, CPUs) 510 (e.g., one or more processors), memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing application programs 533 or data 532. The memory 520 and storage media 530 may be either transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instructions for operating the accent speech recognition device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute the series of instructions stored in the storage medium 530 on the accent speech recognition device 500.

[0178] The accent speech recognition device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 5 The structure of the accented speech recognition device shown does not limit the accented speech recognition device, and the device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] The present invention also provides an accent speech recognition device, comprising a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor performs the steps of the accent speech recognition method described in each of the above embodiments. The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer performs the steps of the accent speech recognition method.

[0180] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0182] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for accent speech recognition, characterized in that: The accent speech recognition method comprises: Acquire multiple accented speech samples, and perform accent annotation and phoneme annotation on each of the accented speech samples, thereby obtaining multiple accent labels and phoneme state sequences; Extracting first acoustic features corresponding to each of the accented speech samples respectively; Inputting the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model; Acquiring an accented speech to be recognized and extracting a second acoustic feature of the accented speech; inputting the second acoustic feature into the accent speech recognition model for recognition, and outputting a second accent category and phoneme state sequence corresponding to the accent speech; The training model includes: a Baseline model and an Xvector model, the Baseline model includes a first TDNN network, a gate mechanism layer, a second TDNN network and a Softmax layer, the first acoustic feature, the accent label and the phoneme state sequence are input into a preset training model for training to obtain an accent speech recognition model, which includes: inputting the first acoustic feature into the first TDNN network for acoustic information extraction to obtain first valid information; inputting the first valid information into the Xvector model for accent category recognition to obtain a first accent category corresponding to the accent speech sample; and inputting the first valid information and the first accent category into the gate mechanism layer for accent information fusion. Obtain first accent fusion information; input the first accent fusion information into the second TDNN network to adjust the accent information to obtain first accent adjustment information; input the first accent adjustment information into the Softmax layer in the Baseline model to perform feature calculation, and output the phoneme state sequence corresponding to the accent speech sample; based on the accent label and the phoneme state sequence, call a preset loss function to calculate the error value of the first accent category and the phoneme state sequence corresponding to the accent speech sample; based on the error value, perform parameter tuning on the Baseline model and the Xvector model until the Baseline model and the Xvector model converge to obtain an accent speech recognition model.

2. The accent speech recognition method according to claim 1, characterized in that: The Xvector model includes a third TDNN network, a Statistics pooling layer, a fourth TDNN network, and a Softmax layer. Inputting the first valid information into the Xvector model for category recognition to obtain the first accent category corresponding to the accented speech sample includes: Inputting the first valid information into the third TDNN network to perform acoustic feature segmentation to obtain first frame-level feature information; Inputting the first frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain first average feature information; Inputting the first average feature information into the fourth TDNN network to extract acoustic feature information to obtain first average effective information; The first average effective information is input into the Softmax layer in the Xvector model to perform an accent classification probability operation to obtain a first accent category corresponding to the accented speech sample.

3. The accent speech recognition method according to claim 1, characterized in that: The extracting of the first acoustic features corresponding to the accented speech samples comprises: Frame each of the accented speech samples to obtain a plurality of speech frames; Pre-enhancement and windowing are performed on each of the speech frames to obtain a first acoustic feature.

4. The method for accented speech recognition according to claim 2, wherein: Inputting the second acoustic feature into the accent speech recognition model for recognition, and outputting a second accent category and a phoneme state sequence corresponding to the accent speech includes: Inputting the second acoustic feature into the first TDNN network to extract acoustic information to obtain second effective information; Inputting the second valid information into the Xvector model for category recognition, and outputting a second accent category corresponding to the accented speech; Inputting the second valid information and the second accent category into the gate mechanism layer to perform accent information fusion to obtain second accent fusion information; Inputting the second accent fusion information into the second TDNN network to perform accent information adjustment to obtain second accent adjustment information; The second accent adjustment information is input into the Softmax layer of the Baseline model for feature calculation, and a phoneme state sequence corresponding to the accented speech is output.

5. The method for accented speech recognition according to claim 4, wherein: Inputting the second valid information into the Xvector model for category recognition, and outputting the second accent category corresponding to the accented speech includes: Inputting the second valid information into the third TDNN network to perform acoustic feature segmentation to obtain second frame-level feature information; Inputting the second frame-level feature information into the Statistics pooling layer to perform mean-variance calculation to obtain second average feature information; Inputting the second average feature information into the fourth TDNN network to extract acoustic feature information to obtain second average effective information; The second average effective information is input into the Softmax layer in the Xvector model to perform accent probability calculation, and a second accent category corresponding to the accented speech is output.

6. The method for accented speech recognition according to any one of claims 1 to 5, characterized in that: The acquiring of multiple accented speech samples and performing accent annotation and phoneme annotation on each of the accented speech samples to obtain multiple accent labels and phoneme state sequences includes: Acquire multiple accented speech samples and convert each of the accented speech samples into text; Each of the accented speech samples is labeled with an accent category to obtain a plurality of accent labels, and the text is transcribed into regular pinyin and labeled with word segmentation information to obtain a phoneme state sequence.

7. An accent speech recognition device, characterized in that: The accent speech recognition device comprises: A sample acquisition module is used to acquire multiple accent speech samples and perform accent annotation and phoneme annotation on each of the accent speech samples to obtain multiple accent labels and phoneme state sequences; A feature extraction module, configured to extract first acoustic features corresponding to each of the accented speech samples; a model training module, configured to input the first acoustic feature, the accent label, and the phoneme state sequence into a preset training model for training to obtain an accent speech recognition model; An accented speech preprocessing module, configured to obtain the accented speech to be recognized and extract the second acoustic feature of the accented speech; an accent speech recognition module, configured to input the second acoustic feature into the accent speech recognition model for recognition, and output a second accent category and a phoneme state sequence corresponding to the accent speech; The training model includes: a Baseline model and an Xvector model, the Baseline model includes a first TDNN network, a gate mechanism layer, a second TDNN network and a Softmax layer, and the model training module includes: a training extraction unit for inputting the first acoustic feature into the first TDNN network for acoustic information extraction to obtain first valid information; a training accent recognition unit for inputting the first valid information into the Xvector model for accent category recognition to obtain the first accent category corresponding to the accent speech sample; a training speech recognition unit for inputting the first valid information and the first accent category into the gate mechanism layer for accent information fusion to obtain first accent fusion information ; Input the first accent fusion information into the second TDNN network to adjust the accent information to obtain first accent adjustment information; Input the first accent adjustment information into the Softmax layer in the Baseline model to perform feature calculation, and output the phoneme state sequence corresponding to the accent speech sample; A model optimization unit is used to call a preset loss function according to the accent label and the phoneme state sequence, and calculate the error value of the first accent category and phoneme state sequence corresponding to the accent speech sample; According to the error value, the parameters of the Baseline model and the Xvector model are tuned until the Baseline model and the Xvector model converge to obtain an accent speech recognition model.

8. An accent speech recognition device, characterized in that: The accent speech recognition device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the accent speech recognition device to perform the accent speech recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the accented speech recognition method according to any one of claims 1 to 6 is implemented.

Images