Training method and device of speech recognition system, speech recognition method and device

Abstract

The embodiments of the present application provide a training method and device for a speech recognition system, and a speech recognition method and device, which are applied to the field of speech recognition technology. The method includes: extracting key features of the audio data of the target speaker to obtain the acoustic features of the audio data; extracting the voiceprint features of the training sample set according to the acoustic features and the preset training sample set; merging the acoustic features and the voiceprint features to obtain the fused voiceprint features; training the convolutional neural network module of the acoustic model based on the training sample set and the fused voiceprint features to obtain the trained acoustic model; training the constructed language model according to the training sample set to obtain the trained language model. The embodiments of the present application realize the training of the speech recognition system with adaptive voiceprint features, which solves the problem of low speech recognition accuracy in the prior art.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular, to a training method and device for a speech recognition system, and a speech recognition method and device. Background Art

[0002] Speech recognition is to let the machine complete the conversion of speech into text through the process of recognition and understanding. When people communicate with each other, they often guess which word the other party is saying based on the information provided by the context. They can also get a lot of information from the other party's tone, facial expressions and gestures. In particular, speakers often correct what they have said and use different words to repeat certain information. Obviously, it is very difficult to make machines recognize speech like humans.

[0003] A complete statistical speech recognition system can be roughly divided into three parts: speech signal preprocessing and feature extraction, acoustic model and pattern matching, language model and language processing. Among them, acoustic model training relies on a large amount of audio corpus, and the corpus needs to cover as many people as possible and as much audio material as possible to improve the recognition effect.

[0004] In the prior art, only the MFCC or FBANK features of the audio corpus are obtained to train the acoustic model. However, due to the differences in speakers in the training sample set, the corresponding features are also very different. The above differences have a great interference on the training process and affect the accuracy of speech recognition. Summary of the invention

[0005] The present application provides a training method and device for a speech recognition system, and a speech recognition method and device, which are used to solve the problem of low speech recognition accuracy.

[0006] In a first aspect, the present application provides a method for training a speech recognition system, the speech recognition system comprising an acoustic model and a language model, the method comprising:

[0007] Extract key features from the audio data of the target speaker to obtain the acoustic features of the audio data;

[0008] Extracting voiceprint features of the training sample set according to the acoustic features and the preset training sample set;

[0009] The acoustic features and the voiceprint features are combined to obtain the fused voiceprint features;

[0010] A convolutional neural network module for training an acoustic model based on a training sample set and fused voiceprint features to obtain a trained acoustic model;

[0011] The constructed language model is trained according to the training sample set to obtain a trained language model.

[0012] In the embodiment of the present application, the voiceprint features of the training sample set are obtained according to the acoustic features and the preset training sample set, including:

[0013] According to the audio sample data of each speaker in the training sample set, a GMM model is constructed, and the GMM model is trained to obtain an initialized GMM-UBM model. The GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set;

[0014] According to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker.

[0015] A first feature vector is obtained according to the initialized GMM-UBM model, and a second feature vector is obtained according to the GMM-UBM model of the target speaker;

[0016] The voiceprint features of the training sample set are obtained according to the first eigenvector and the second eigenvector.

[0017] In the embodiment of the present application, a convolutional neural network module of an acoustic model is trained based on a training sample set and a fusion voiceprint feature to obtain a trained acoustic model, including:

[0018] According to the text content corresponding to the audio sample data in the training sample set, the phonemes of the training sample set are extracted as acoustic labels;

[0019] Construct a dictionary table based on the relationship between the text content and the phonemes;

[0020] Numerically encode the acoustic labels according to the dictionary table to obtain label codes;

[0021] Input the label code and the fused voiceprint feature into the convolutional neural network module to obtain the first phoneme corresponding to the fused voiceprint feature;

[0022] The CTC loss function is used to iteratively update the parameters of the convolutional neural network module until the acoustic model converges to obtain the trained acoustic model;

[0023] Among them, the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label encoding.

[0024] In an embodiment of the present application, the constructed language model is trained according to the training sample set, and before obtaining the trained language model, the method further includes:

[0025] The corpus is constructed based on the text corpus, which includes Wikipedia and news corpus.

[0026] In the embodiment of the present application, a corpus set is constructed based on text corpus, and the text corpus includes Wikipedia and news corpus, including:

[0027] Acquire first text information corresponding to the audio sample data of the training sample set, and mark first phoneme information corresponding to the first text information;

[0028] Obtaining second text information corresponding to the Wikipedia and news corpus, and marking second phoneme information corresponding to the second text information;

[0029] The first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information are used as a corpus.

[0030] In an embodiment of the present application, the constructed language model is trained according to the training sample set to obtain a trained language model, including:

[0031] The first text information and the second text information are used as language labels, the first phoneme information, the second phoneme information and the language label are used as inputs of the language model, the language model is trained using the Self-attention algorithm, and the parameters of the language model are iteratively updated based on the cross entropy loss function until the language model converges, thereby obtaining a trained language model;

[0032] Among them, the cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

[0033] In a second aspect, the present application provides a speech recognition method, the method comprising:

[0034] Obtain audio data to be recognized from the target speaker;

[0035] A speech recognition system trained by the method shown in any embodiment of the first aspect performs speech recognition on the audio data to be recognized to obtain a recognition result;

[0036] Among them, speech recognition is performed on the audio data to be recognized, and the recognition result is obtained, including:

[0037] Input the audio data to be recognized into the trained acoustic model to obtain the phoneme sequence corresponding to the audio data to be recognized;

[0038] The phoneme sequence is input into the trained language model to obtain the text recognition result of the audio data to be recognized.

[0039] In a third aspect, the present application provides a training device for a speech recognition system, the device comprising:

[0040] An acoustic feature extraction module is used to extract key features from the audio data of the target speaker to obtain the acoustic features of the audio data;

[0041] A voiceprint feature extraction module, used to extract voiceprint features of a training sample set based on acoustic features and a preset training sample set;

[0042] A feature merging module is used to merge acoustic features and voiceprint features to obtain fused voiceprint features;

[0043] An acoustic model training module is used to train a convolutional neural network module of an acoustic model based on a training sample set and fused voiceprint features to obtain a trained acoustic model;

[0044] The language model training module is used to train the constructed language model based on the training sample set to obtain a trained language model.

[0045] In the embodiment of the present application, the voiceprint feature extraction module is specifically used for:

[0046] According to the audio sample data of each speaker in the training sample set, a GMM model is constructed, and the GMM model is trained to obtain an initialized GMM-UBM model. The GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set;

[0047] According to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker.

[0048] A first feature vector is obtained according to the initialized GMM-UBM model, and a second feature vector is obtained according to the GMM-UBM model of the target speaker;

[0049] The voiceprint features of the training sample set are obtained according to the first eigenvector and the second eigenvector.

[0050] In the embodiments of the present application, the acoustic model training module is specifically used for:

[0051] According to the text content corresponding to the audio sample data in the training sample set, the phonemes of the training sample set are extracted as acoustic labels;

[0052] Construct a dictionary table based on the relationship between the text content and the phonemes;

[0053] Numerically encode the acoustic labels according to the dictionary table to obtain label codes;

[0054] Input the label code and the fused voiceprint feature into the convolutional neural network module to obtain the first phoneme corresponding to the fused voiceprint feature;

[0055] The CTC loss function is used to iteratively update the parameters of the convolutional neural network module until the acoustic model converges to obtain the trained acoustic model;

[0056] Among them, the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label encoding.

[0057] In an embodiment of the present application, the training device of the speech recognition system further includes:

[0058] The corpus construction module is used to construct a corpus based on text corpus, which includes Wikipedia and news corpus.

[0059] In the embodiments of the present application, the above corpus building module is specifically used for:

[0060] Acquire first text information corresponding to the audio sample data of the training sample set, and mark first phoneme information corresponding to the first text information;

[0061] Obtaining second text information corresponding to the Wikipedia and news corpus, and marking second phoneme information corresponding to the second text information;

[0062] The first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information are used as a corpus.

[0063] In the embodiments of the present application, the above-mentioned language model training module is specifically used for:

[0064] The first text information and the second text information are used as language labels, the first phoneme information, the second phoneme information and the language label are used as inputs of the language model, the language model is trained using the Self-attention algorithm, and the parameters of the language model are iteratively updated based on the cross entropy loss function until the language model converges, thereby obtaining a trained language model;

[0065] Among them, the cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

[0066] In a fourth aspect, the present application provides a speech recognition device, the device comprising:

[0067] An acquisition module is used to acquire audio data to be recognized from a target speaker;

[0068] A recognition module, used for performing speech recognition on the audio data to be recognized based on the speech recognition system trained by the method shown in any embodiment of the first aspect to obtain a recognition result;

[0069] The identification module is specifically used for:

[0070] Input the audio data to be recognized into the trained acoustic model to obtain the phoneme sequence corresponding to the audio data to be recognized;

[0071] The phoneme sequence is input into the trained language model to obtain the text recognition result of the audio data to be recognized.

[0072] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method as shown in any embodiment of the first and second aspects of the present application.

[0073] In a sixth aspect, the present application provides an electronic device, the electronic device comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface communicate with each other via the communication bus;

[0074] The memory is used to store at least one executable instruction, which enables the processor to execute operations corresponding to the method shown in any embodiment of the first aspect and the second aspect of the present application.

[0075] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0076] The present application first extracts the acoustic features of the target speaker's audio data and the voiceprint features of the training sample set, and then combines the acoustic features and the voiceprint features as the input of the convolutional neural network to train the acoustic model, thereby realizing the training of a speech recognition system with adaptive voiceprint features. Compared with the prior art, which uses a single acoustic feature to train the acoustic model, the fused voiceprint feature combines the speaker's personalized voiceprint characteristics, expands the feature vector, enriches the feature dimension, and involves more data in the training, thereby improving the recognition accuracy of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0078] Figure 1 A flowchart of a method for training a speech recognition system provided in an embodiment of the present application;

[0079] Figure 2 A schematic diagram of the structure of an acoustic model in a training method for a speech recognition system provided in an embodiment of the present application;

[0080] Figure 3 An example of the content of a data sample in a training sample set in a training method for a speech recognition system provided in an embodiment of the present application;

[0081] Figure 4 A schematic diagram of the structure of a convolutional neural network module in a training method for a speech recognition system provided in an embodiment of the present application;

[0082] Figure 5 A schematic diagram of the structure of a speech recognition model in a speech recognition method provided in an embodiment of the present application;

[0083] Figure 6 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0084] Figure 7 A schematic diagram of the structure of a training device for a speech recognition system provided in an embodiment of the present application.

[0085] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0087] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0088] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0089] A complete statistical speech recognition system can be roughly divided into three parts: speech signal preprocessing and feature extraction, acoustic model and pattern matching, language model and language processing. Among them, the purpose of feature extraction is to analyze and process the speech signal, remove redundant information irrelevant to speech recognition, obtain important information that affects speech recognition, and compress the speech signal. At the same time, acoustic model training relies on a large amount of audio corpus, and the corpus needs to cover as many people as possible and as much audio material as possible to improve the recognition effect. Usually, the acquired speech features are used to train the acoustic model using a training algorithm.

[0090] In the prior art, only the MFCC or FBANK features of the audio corpus are obtained to train the acoustic model. However, due to the differences in speakers in the training sample set, the corresponding features are also very different. The above differences have a great interference on the training process and affect the accuracy of speech recognition.

[0091] In view of this, an embodiment of the present application proposes a training method for a speech recognition system. This method adds a voiceprint feature extraction process to the existing training method based on a deep learning speech model, and combines the voiceprint features with the acoustic features as the input of the convolutional network, so that the trained acoustic model and language model can realize voiceprint feature-adaptive speech recognition, thereby improving the accuracy of speech recognition.

[0092] In order to better understand the training method of the speech recognition system provided in the embodiment of the present application, Figure 1-8 The method is described in detail.

[0093] like Figure 1 As shown, an embodiment of the present application provides a flow chart of a method for training a speech recognition system, and the method may include the following steps:

[0094] S1, extract key features of the audio data of the target speaker to obtain the acoustic features of the audio data.

[0095] S2, extracting the voiceprint features of the training sample set according to the acoustic features and the preset training sample set.

[0096] S3, combining the acoustic features and the voiceprint features to obtain a fused voiceprint feature.

[0097] S4, a convolutional neural network module for training an acoustic model based on a training sample set and fused voiceprint features to obtain a trained acoustic model.

[0098] S5, training the constructed language model according to the training sample set to obtain a trained language model.

[0099] In practical applications, before step S1, the method may further include reading the audio data of the target speaker to perform voice activity detection (Voice Activity Detection, VAD), which may specifically include:

[0100] Adopt Google's open source WebRTC VAD algorithm and implement voice activity detection for audio data based on GMM. The algorithm works based on the fluctuation of the audio's time domain signal and calculates the energy migration change of the audio in the time domain, so as to identify the noise and silence parts in the audio data. First, set different energy thresholds, and then define the unit frame length (10 milliseconds, 20 milliseconds, or 30 milliseconds) according to specific needs, and then judge frame by frame. Frames with energy of 0 or exceeding the normal energy threshold range are abnormal frames.

[0101] According to the speaking habits of ordinary people, the pause time can be set to 0.5 seconds. To further improve the fault tolerance rate, audio with a duration of 300 milliseconds will be selected to determine whether it is a silent part, thereby achieving the effect of segmenting the audio and identifying the pause symbol. Specifically, the rules can be combined arbitrarily according to the unit frame length. This application uses 30 milliseconds as a frame to detect whether the energy of 10 consecutive frames (i.e. 300 milliseconds of audio) is all 0, thereby identifying the silent part in the audio and segmenting the entire audio data to be identified.

[0102] The purpose of voice activity detection is to identify noise and silence in audio. First, remove noise and silence from the audio, and reduce the amount of feature data and improve training efficiency when extracting acoustic features later. Second, audio data can be segmented by long periods of silence. For example, the pause time between commas is 0.5-1 seconds, the pause time between commas is 1-2 seconds, and the pause time between paragraphs is 2-3 seconds. Based on the length of continuous silence, the audio can be segmented well, which helps to align audio and text in the subsequent training process and reduce the time complexity of CTC (Connectionist Temporal Classification). Audio data VAD preprocessing can improve model training results and improve audio recognition accuracy.

[0103] In the embodiment of the present application, step S1 may specifically be to extract MFCC (Mel-Frequency Cepstral Coefficients) features from the audio data of the target speaker. The basic process is as follows:

[0104] 1. Pre-emphasis: Pre-emphasize the audio signal after VAD pre-processing, that is, multiply it by a pre-emphasis coefficient in the signal frequency domain. The pre-emphasis coefficient can be 0.95. This coefficient is positively correlated with the frequency to increase the amplitude of the high frequency in the signal. The role of pre-emphasis is to eliminate the effects caused by the vocal cords and lips during the phonation process to compensate for the high-frequency part of the speech signal suppressed by the pronunciation system, and to highlight the high-frequency resonance peak and reduce noise interference.

[0105] 2. Framing and windowing: Set the duration to 0.25ms as one frame and 0.1ms as the overlapping window. To avoid tailing during the conversion between the audio time domain and the frequency domain, add a Hamming window to each frame. The Hamming window length is 400 and the Hamming window parameter α is 0.46.

[0106] 3. Fast Fourier Transform: The number of sampling points of the Fourier transform is 512, and the spectrum of each frame is obtained so that the time domain signal can be converted into the frequency domain for subsequent frequency analysis.

[0107] 4. Mel-scale filter bank filtering: The number of filters in the filter bank is 26; the spectrum signal passes through a set of Mel-scale triangular filter banks, and the logarithmic energy of each filter bank output is calculated; the logarithmic energy is brought into discrete cosine transform to obtain the Mel spectrum.

[0108] 5. Cepstrum analysis: First, take the logarithm of the Mel spectrum, then perform discrete cosine transform to obtain the low-frequency information of the frequency spectrum, and take the coefficients after discrete cosine transform as MFCC coefficients. Among them, the MFCC coefficient order is 22, the cepstrum number is 13, plus the first-order difference and the second-order difference, and finally an audio data generates an n×39 feature matrix, where n depends on the length of the audio data. The longer the audio data, the larger the n value.

[0109] The present application first extracts the acoustic features of the target speaker's audio data and the voiceprint features of the training sample set, and then combines the acoustic features and the voiceprint features as the input of the convolutional neural network to train the acoustic model, thereby realizing the training of a speech recognition system with adaptive voiceprint features. Compared with the prior art, which uses a single acoustic feature to train the acoustic model, the fused voiceprint feature combines the speaker's personalized voiceprint characteristics, expands the feature vector, enriches the feature dimension, and involves more data in the training, thereby improving the recognition accuracy of the speech recognition system.

[0110] A possible implementation method is provided in an embodiment of the present application, and step S2 may specifically include:

[0111] S21, constructing a GMM model according to the audio sample data of each speaker in the training sample set, and training the GMM model to obtain an initialized GMM-UBM model, where the GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set.

[0112] S22, according to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker.

[0113] S23, obtaining a first feature vector according to the initialized GMM-UBM model, and obtaining a second feature vector according to the GMM-UBM model of the target speaker.

[0114] S24, obtaining the voiceprint features of the training sample set according to the first feature vector and the second feature vector.

[0115] like Figure 2 The figure shows the structure of the acoustic model. The voiceprint extraction module extracts the voiceprint features of the audio data based on GMM-UBM (Gaussian Mixture Model-Universal Background Model):

[0116] GMM is a model composed of multiple single Gaussian distributions. In theory, GMM can fit any type of distribution and is used to describe the distribution of data points / feature points in speech recognition. UBM is equivalent to a large mixed Gaussian distribution model. It is used to solve the problem of too little training data for target users and train a large GMM that fits general features with a large amount of non-target user data. The GMM-UBM model first uses a large amount of non-target user data to train the UBM, and then uses the MAP adaptive algorithm and the target speaker data to update the local parameters to obtain the corresponding GMM model. The specific process of extracting voiceprint features based on the GMM-UBM architecture is as follows:

[0117] First, according to the speaker ID in the training sample set, for each speaker, a GMM model is constructed, and the EM (Expectation Maximum) algorithm is used based on the MFCC features to train the GMM model to obtain an initialized GMM-UBM model. Because most of the speaker information is contained in the mean of the GMM, the mean vector of each Gaussian mixture component, that is, the first eigenvector mentioned above, is taken here as the audio feature of each speaker in the training sample set. In the actual application process of this application, the Gaussian distribution parameters are set to 512. Based on the obtained MFCC features, each speaker finally obtains a 512×39 feature matrix, which is expanded into a one-dimensional vector for the convenience of subsequent processing.

[0118] Second, the MAP (Maximum a posteriori) algorithm is used to combine the GMM-UBM model with the target speaker information to achieve the adaptation of the GMM-UBM model and obtain the GMM-UBM model of the target speaker. Based on the factor analysis method, the second feature vector is obtained according to the GMM-UBM of the target speaker; the second feature vector is combined with the audio features of each speaker in the training sample set, and the first N values ​​are extracted according to the weight to obtain the voiceprint features of the training sample set. To ensure that the voiceprint features are consistent with the MFCC feature dimensions, the custom parameter N is set to 39, and finally the voiceprint feature corresponding to each audio data is a 39-dimensional vector.

[0119] The most important advantage of the GMM-UBM model is that it estimates the model parameters through the MAP algorithm, avoiding overfitting. At the same time, we do not need to adjust all the parameters (weight, mean, variance) of the target user's GMM. We only need to estimate the mean parameters of each Gaussian component to achieve the best recognition performance. This can greatly reduce the parameters to be estimated. Fewer parameters also means faster convergence, and we do not need so much target user data to achieve good training of the model.

[0120] like Figure 2 The structural diagram of the acoustic model shown in FIG. 1 is a schematic diagram of the acoustic model, wherein the Concat module is used to merge feature vectors, and the specific calculation process of step S3 is as follows:

[0121] First, convert the voiceprint feature vector V into the basis vector V e , and then multiply it with each row vector in the MFCC feature to form a feature matrix (39×39). Finally, concatenate n (n depends on the number of rows in the original matrix) matrices into a high-dimensional matrix, expand the eigenvalues, and obtain the fused voiceprint feature. The calculation process is as follows:

[0122] Voiceprint basis vector: V e=[v1 v2 v3 … v m ] T ;

[0123] MFCC feature matrix:

[0124] Voiceprint vector weighting:

[0125]

[0126] …

[0127]

[0128] The weighted results are concatenated to obtain the fused voiceprint features:

[0129]

[0130] Another possible implementation method is provided in the embodiment of the present application. Step S4 may specifically include:

[0131] S41, according to the text content corresponding to the audio sample data in the training sample set, extract the phonemes of the training sample set as acoustic labels; taking Mandarin Chinese as an example, extract the Chinese pinyin corresponding to the text, such as Figure 3 It is an example of the content of the data samples in the training sample set, and each audio data sample corresponds to a pinyin sequence.

[0132] S42, constructing a dictionary table according to the relationship between the text content and the phonemes.

[0133] S43, numerically encode the acoustic labels according to the dictionary table to obtain label codes; specifically, one-hot encoding can be used, that is, converting the text labels (i.e., pinyin sequences) corresponding to the audio data in the sample set into binary vector codes to facilitate subsequent data processing.

[0134] S44, inputting the label code and the fused voiceprint feature into a convolutional neural network module to obtain a first phoneme corresponding to the fused voiceprint feature;

[0135] S45, iteratively updating the parameters of the convolutional neural network module using a CTC loss function until the acoustic model converges to obtain a trained acoustic model; wherein the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label encoding.

[0136] In practical applications, the structural diagram of the convolutional neural network module is as follows Figure 4As shown in the figure, the label encoding and fused voiceprint features are used as the input of the convolutional neural network module. The input data first passes through three two-channel convolution kernels and maximum pooling layers, and two two-channel 128*128 convolution kernel layers to learn the speech features and the semantic information between the features. Then, the output after the convolution kernel layer structure is integrated through the Reshape layer and the fully connected layer to accelerate the convergence of the model during subsequent training. Figure 2 As shown in the structural diagram of the acoustic model, the output phoneme of the convolutional neural network module passes through a fully connected layer to output the probability distribution of the phoneme at each step; then it passes through the Softmax+CTC layer, and the Softmax normalization function is used to calculate the posterior probability of the phoneme. The CTC (Connectionist temporal classification) loss function is used to calculate the difference between the output phoneme of each step of the neural convolutional network module and the actual result, that is, the label. The parameters of the neural network model are iteratively optimized according to the calculation results of the loss function until the model converges.

[0137] The specific steps for training the acoustic model are as follows: first, initialize the pre-built acoustic model of the above structure, then initialize an Adam optimizer, set the step size to 100 steps, the initial learning rate to 0.001, and perform model iteration.

[0138] Compared with traditional acoustic model training, acoustic model training using CTC as the loss function is a completely end-to-end acoustic model training that does not require pre-alignment of data and can be trained with only one input sequence and one output sequence. In this way, there is no need to align and label data one by one, and CTC directly outputs the probability of sequence prediction without the need for external post-processing.

[0139] Another possible implementation method is provided in the embodiment of the present application, which may further include before step S5:

[0140] S6, construct a corpus based on text corpus, which includes Wikipedia and news corpus.

[0141] Another possible implementation method is provided in the embodiment of the present application. Step S6 may specifically include:

[0142] S61, obtaining first text information corresponding to the audio sample data of the training sample set, and marking first phoneme information corresponding to the first text information.

[0143] S62, obtaining second text information corresponding to the Wikipedia and the news corpus, and marking second phoneme information corresponding to the second text information.

[0144] S63, taking the first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information as a corpus. In the present application, by adding the above-mentioned massive text corpus as a corpus to train the language model, the accuracy of the language model is improved; at the same time, the language model adopts a Transformer encoder structure based on a self-attention mechanism, takes the output of the acoustic model (i.e., the Chinese pinyin sequence) as the input of the model, and outputs a Chinese character sequence of the same length as the Chinese pinyin sequence through a multi-head attention network.

[0145] Another possible implementation method is provided in the embodiment of the present application, and step S5 specifically includes:

[0146] The first text information and the second text information are used as language labels, the first phoneme information, the second phoneme information and the language label are used as inputs of the language model, the language model is trained using the Self-attention algorithm, and the parameters of the language model are iteratively updated based on the cross entropy loss function until the language model converges, thereby obtaining a trained language model;

[0147] Among them, the cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

[0148] First, before training the language model, the Chinese characters and pinyin in the corpus are aligned, and the numerical mapping relationship between Chinese characters and pinyin is constructed according to the bag-of-words model to construct the pinyin table and the Chinese character table, and generate the numerical vector corresponding to the text and the pinyin; then, the Self-attention algorithm is used, with the total number of Chinese characters as the input length and the total number of pinyin as the length of the label. Then, a multi-head attention network is constructed according to the specified block, and the cross entropy (softmax_cross_entropy) function is used to calculate the loss, and the network parameters are iteratively optimized to train the language model. Among them, the main training parameter settings are: block is 5, multi head is 10, the length of the pinyin table is 1764, the length of the Chinese character table is 3593, the maximum length of the embedding size is 100, the number of hidden layer units is 512, the learning rate is 0.0001, and the step size is set to 25 steps.

[0149] Since the self-attention mechanism has a strong contextual learning ability and a relatively fast computing power, it can efficiently learn the contextual information of the text, making the model have a relatively strong inference ability and a relatively fast convergence speed. Since the Transformer self-attention model does not require dependencies between time steps, parallel computing at different time steps can be achieved through matrix multiplication, which greatly reduces the training time of the language model.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a speech recognition method, which includes:

[0151] S101, obtaining audio data to be recognized of a target speaker;

[0152] S102, performing speech recognition on the audio data to be recognized based on the speech recognition system trained by the training method of the speech recognition system shown in any of the above embodiments to obtain a recognition result;

[0153] Among them, S102 specifically includes:

[0154] Input the audio data to be recognized into the trained acoustic model to obtain the phoneme sequence corresponding to the audio data to be recognized;

[0155] The phoneme sequence is input into the trained language model to obtain the text recognition result of the audio data to be recognized.

[0156] The following is combined with Figure 5 and 6 A speech recognition method provided in an embodiment of the present application is described in detail, wherein: Figure 5 FIG. 1 is a schematic diagram showing the structure of the speech recognition model in this embodiment. Figure 6 The speech recognition method shown includes:

[0157] S110, reading audio data of a target speaker to perform voice activity detection on the audio data.

[0158] S120, extract key features from the audio data of the target speaker to obtain acoustic features.

[0159] S130, extracting voiceprint features of the training sample set according to the acoustic features and the preset training sample set;

[0160] In this embodiment, S130 is performed by Figure 5 The voiceprint extraction module is implemented based on the GMM-UBM model to extract the voiceprint features of the audio data.

[0161] Specifically, in this embodiment, S130 may specifically include:

[0162] S131, constructing a GMM model based on the audio sample data of each speaker in the training sample set, and training the GMM model using the EM algorithm to obtain an initialized GMM-UBM model. The GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set.

[0163] S132, according to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker.

[0164] S133, obtaining a first feature vector according to the initialized GMM-UBM model, and obtaining a second feature vector according to the GMM-UBM model of the target speaker;

[0165] In this embodiment, the MAP algorithm is used to combine the GMM-UBM model with the target speaker information to achieve the adaptation of the GMM-UBM model.

[0166] S134, obtaining the voiceprint features of the training sample set according to the first feature vector and the second feature vector.

[0167] S140, combining the acoustic feature and the voiceprint feature to obtain a fused voiceprint feature.

[0168] S150, a convolutional neural network module for training an acoustic model based on a training sample set and fused voiceprint features to obtain a trained acoustic model.

[0169] Specifically, in this embodiment, S150 may specifically include:

[0170] S151, extracting phonemes of the training sample set as acoustic labels according to text content corresponding to the audio sample data in the training sample set;

[0171] In this embodiment, taking Mandarin Chinese as an example, the extracted phonemes are the Chinese pinyin corresponding to the text.

[0172] S152, constructing a dictionary table according to the relationship between the text content and the phonemes.

[0173] S153, numerically encode the acoustic label according to the dictionary table to obtain the label code.

[0174] S154, input the label code and the fused voiceprint feature into a convolutional neural network module to obtain a first phoneme corresponding to the fused voiceprint feature.

[0175] S155, iteratively update the parameters of the convolutional neural network module using a CTC loss function until the acoustic model converges to obtain a trained acoustic model; wherein the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label encoding.

[0176] S160, constructing a corpus according to the text corpus, where the text corpus includes Wikipedia and news corpus.

[0177] Specifically, in this embodiment, S160 may specifically include:

[0178] S161, obtaining first text information corresponding to the audio sample data of the training sample set, and marking first phoneme information corresponding to the first text information.

[0179] S162, obtaining second text information corresponding to the Wikipedia and the news corpus, and marking second phoneme information corresponding to the second text information.

[0180] S163: taking the first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information as a corpus.

[0181] S170, training the constructed language model according to the training sample set to obtain a trained language model.

[0182] Specifically, in this embodiment, S170 may specifically include:

[0183] Using the first text information and the second text information as language labels, using the first phoneme information, the second phoneme information and the language label as inputs of the language model, using a self-attention algorithm to train the language model, and iteratively updating the parameters of the language model based on a cross entropy loss function until the language model converges, thereby obtaining a trained language model;

[0184] The cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

[0185] In this embodiment, the language model is adopted as follows Figure 5 The Transformer encoder structure shown in Figure 2 is based on the self-attention mechanism.

[0186] S180, obtaining audio data to be recognized of the target speaker; inputting the audio data to be recognized into the trained acoustic model to obtain a phoneme sequence corresponding to the audio data to be recognized.

[0187] S190, inputting the phoneme sequence into the trained language model to obtain a text recognition result of the audio data to be recognized.

[0188] The present application first extracts the acoustic features and voiceprint features of the audio data to be recognized, and then combines the acoustic features and voiceprint features as the input of the convolutional neural network to train the acoustic model, thereby realizing the training of a speech recognition system with adaptive voiceprint features. Compared with the prior art, in which a single acoustic feature is used to train the acoustic model, the fused voiceprint feature combines the personalized voiceprint characteristics of the speaker, expands the feature vector, enriches the feature dimension, and involves more data in the training, thereby improving the accuracy of the speech recognition system.

[0189] In the training of the voiceprint extraction module, the most important advantage of using the GMM-UBM model is that the model parameters are estimated through the MAP algorithm, which avoids overfitting. At the same time, we do not need to adjust all the parameters of the target user's GMM. We only need to estimate the mean parameters of each Gaussian component to achieve the best recognition performance. This can greatly reduce the parameters to be estimated. Fewer parameters also means faster convergence, and a good model training can be achieved without so much target user data.

[0190] In the training of the language model, the accuracy of the language model is improved by adding the above-mentioned massive text corpus as the corpus set for training the language model; at the same time, since the self-attention mechanism has a relatively strong contextual learning ability and relatively fast computing power, it can efficiently learn the contextual information of the text, making the model have a relatively strong inference ability and a faster convergence speed. Since the Transformer's self-attention model does not require dependencies between time steps, parallel computing at different time steps can be achieved through matrix multiplication, which greatly reduces the training time of the language model.

[0191] Based on the same inventive concept, the present application also provides a training device for a speech recognition system. Figure 7 As shown, the training device 30 of the speech recognition system may include: an acoustic feature extraction module 301, a voiceprint feature extraction module 302, a feature merging module 303, an acoustic model training module 304, and a language model training module 305, wherein:

[0192] An acoustic feature extraction module 301 is used to extract key features from the audio data of the target speaker to obtain acoustic features of the audio data;

[0193] A voiceprint feature extraction module 302 is used to extract voiceprint features of a training sample set according to acoustic features and a preset training sample set;

[0194] A feature merging module 303 is used to merge the acoustic feature and the voiceprint feature to obtain a fused voiceprint feature;

[0195] An acoustic model training module 304 is used to train a convolutional neural network module of an acoustic model based on a training sample set and fused voiceprint features to obtain a trained acoustic model;

[0196] The language model training module 305 is used to train the constructed language model according to the training sample set to obtain a trained language model.

[0197] Another possible implementation is provided in the embodiment of the present application. The voiceprint feature extraction module 302 is specifically used for:

[0198] According to the audio sample data of each speaker in the training sample set, a GMM model is constructed, and the GMM model is trained using the EM algorithm to obtain an initialized GMM-UBM model. The GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set;

[0199] According to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker.

[0200] A first feature vector is obtained according to the initialized GMM-UBM model, and a second feature vector is obtained according to the GMM-UBM model of the target speaker;

[0201] The voiceprint features of the training sample set are obtained according to the first eigenvector and the second eigenvector.

[0202] Another possible implementation is provided in the embodiment of the present application, where the acoustic model training module 304 is specifically used for:

[0203] According to the text content corresponding to the audio sample data in the training sample set, the phonemes of the training sample set are extracted as acoustic labels;

[0204] Construct a dictionary table based on the relationship between the text content and the phonemes;

[0205] Numerically encode the acoustic labels according to the dictionary table to obtain label codes;

[0206] Input the label code and the fused voiceprint feature into the convolutional neural network module to obtain the first phoneme corresponding to the fused voiceprint feature;

[0207] The CTC loss function is used to iteratively update the parameters of the convolutional neural network module until the acoustic model converges to obtain the trained acoustic model;

[0208] Among them, the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label encoding.

[0209] Another possible implementation is provided in the embodiment of the present application. The training device 30 of the speech recognition system may further include:

[0210] The corpus construction module is used to construct a corpus based on text corpus, which includes Wikipedia and news corpus.

[0211] Another possible implementation method is provided in the embodiment of the present application, and the above-mentioned corpus construction module is specifically used for:

[0212] Acquire first text information corresponding to the audio sample data of the training sample set, and mark first phoneme information corresponding to the first text information;

[0213] Obtaining second text information corresponding to the Wikipedia and news corpus, and marking second phoneme information corresponding to the second text information;

[0214] The first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information are used as a corpus.

[0215] Another possible implementation is provided in the embodiment of the present application. The above-mentioned language model training module 305 can be specifically used for:

[0216] The first text information and the second text information are used as language labels, the first phoneme information, the second phoneme information and the language label are used as inputs of the language model, the language model is trained using the Self-attention algorithm, and the parameters of the language model are iteratively updated based on the cross entropy loss function until the language model converges, thereby obtaining a trained language model;

[0217] Among them, the cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

[0218] For contents not described in detail in the training device for the speech recognition system provided in the embodiment of the present application, reference can be made to the training method for the speech recognition system provided in the above embodiment. The beneficial effects that can be achieved by the training device for the speech recognition system provided in the embodiment of the present application are the same as those of the training method for the speech recognition system provided in the above embodiment, and will not be repeated here.

[0219] The application of the embodiments of the present application has at least the following beneficial effects:

[0220] The present application first extracts the acoustic features and voiceprint features of the audio data to be recognized, and then combines the acoustic features and voiceprint features as the input of the convolutional neural network to train the acoustic model, thereby realizing the training of a speech recognition system with adaptive voiceprint features. Compared with the prior art, which uses a single acoustic feature to train the acoustic model, the fused voiceprint feature combines the personalized voiceprint characteristics of the speaker, expands the feature vector, enriches the feature dimension, and involves more data in the training, thereby improving the accuracy of the speech recognition system.

[0221] Based on the same inventive concept, the embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when it is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0223] Based on the same inventive concept, the embodiment of the present application further provides an electronic device 40, such as Figure 8 As shown, the electronic device includes a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the program, the steps of the method in the embodiment are implemented.

[0224] Figure 8 is a schematic diagram of the hardware structure of an electronic device for executing any of the methods in the above embodiments provided by an embodiment of the present invention, such as Figure 8 As shown, the electronic device includes one or more processors 401 and a memory 402. Figure 8 A processor 401 is taken as an example.

[0225] The electronic device executing any of the methods in the above embodiments may further include: an input device 403 and an output device 404 .

[0226] The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 8 The connection via bus 405 is taken as an example.

[0227] The processor 401 may be a central processing unit (CPU). The processor 401 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0228] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for training a speech recognition system, wherein the speech recognition system comprises an acoustic model and a language model, characterized in that: include: Extracting key features from the audio data of the target speaker to obtain acoustic features of the audio data; Extracting voiceprint features of the training sample set according to the acoustic features and a preset training sample set; the training sample set includes audio sample data of multiple speakers; Combining the acoustic feature with the voiceprint feature to obtain a fused voiceprint feature; Training the convolutional neural network module of the acoustic model based on the training sample set and the fused voiceprint features to obtain the acoustic model of the target speaker after training; The constructed language model is trained according to the training sample set to obtain a language model of the target speaker after training; and the voiceprint features of the training sample set are obtained according to the acoustic features and the preset training sample set, including: According to the audio sample data of each speaker in the training sample set, a GMM model is constructed, and the GMM model is trained to obtain an initialized GMM-UBM model, wherein the GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set; According to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker; Obtaining a first feature vector according to the initialized GMM-UBM model, and obtaining a second feature vector according to the GMM-UBM model of the target speaker; The voiceprint features of the training sample set are obtained according to the first feature vector and the second feature vector.

2. The method for training a speech recognition system according to claim 1, characterized in that: The convolutional neural network module of the acoustic model is trained based on the training sample set and the fused voiceprint feature to obtain the trained acoustic model, including: Extracting phonemes of the training sample set as acoustic labels according to text content corresponding to the audio sample data in the training sample set; Constructing a dictionary table according to the relationship between the text content and the phonemes; Numerically encoding the acoustic label according to the dictionary table to obtain a label code; Inputting the label code and the fused voiceprint feature into the convolutional neural network module to obtain a first phoneme corresponding to the fused voiceprint feature; Iteratively updating the parameters of the convolutional neural network module using a CTC loss function until the acoustic model converges to obtain a trained acoustic model; Among them, the CTC loss function represents the difference between the first phoneme output by the convolutional neural network module and the phoneme corresponding to the label code.

3. The method for training a speech recognition system according to any one of claims 1 to 2, characterized in that: Before training the constructed language model according to the training sample set to obtain the trained language model, the method further includes: A corpus set is constructed according to text corpus, wherein the text corpus includes Wikipedia and news corpus.

4. The method for training a speech recognition system according to claim 3, characterized in that: The corpus set is constructed based on text corpus, wherein the text corpus includes Wikipedia and news corpus, including: Acquire first text information corresponding to the audio sample data of the training sample set, and mark first phoneme information corresponding to the first text information; Acquire second text information corresponding to the Wikipedia and news corpus, and annotate second phoneme information corresponding to the second text information; The first text information annotated with the first phoneme information and the second text information annotated with the second phoneme information are used as the corpus.

5. The method for training a speech recognition system according to claim 4, characterized in that: The step of training the constructed language model according to the training sample set to obtain a trained language model includes: Using the first text information and the second text information as language labels, using the first phoneme information, the second phoneme information and the language label as inputs of the language model, using a self-attention algorithm to train the language model, and iteratively updating the parameters of the language model based on a cross entropy loss function until the language model converges, thereby obtaining a trained language model; The cross entropy loss function represents the difference between the output text recognition result of the language model and the language label.

6. A speech recognition method, characterized in that: include: Obtain audio data to be recognized from the target speaker; Based on the speech recognition system trained by the method according to any one of claims 1 to 5, speech recognition is performed on the audio data to be recognized to obtain a recognition result; The process of performing speech recognition on the audio data to be recognized to obtain a recognition result includes: Inputting the audio data to be recognized into the trained acoustic model to obtain a phoneme sequence corresponding to the audio data to be recognized; The phoneme sequence is input into the trained language model to obtain a text recognition result of the audio data to be recognized.

7. A training device for a speech recognition system, characterized in that: include: An acoustic feature extraction module, used to extract key features from the audio data of the target speaker to obtain the acoustic features of the audio data; A voiceprint feature extraction module, used to extract the voiceprint features of the training sample set according to the acoustic features and a preset training sample set; A feature merging module, used for merging the acoustic feature with the voiceprint feature to obtain a fused voiceprint feature; An acoustic model training module, used for training a convolutional neural network module of the acoustic model based on the training sample set and the fused voiceprint feature to obtain a trained acoustic model; A language model training module is used to train the constructed language model according to the training sample set to obtain a trained language model; When the voiceprint feature extraction module is used to extract the voiceprint features of the training sample set according to the acoustic features and the preset training sample set, it is specifically used to: According to the audio sample data of each speaker in the training sample set, a GMM model is constructed, and the GMM model is trained to obtain an initialized GMM-UBM model, wherein the GMM model is used to characterize the probability distribution of the audio features of each audio sample data in the training sample set; According to the acoustic features, an adaptive algorithm is used to update the parameters of the initialized GMM-UBM model to obtain a GMM-UBM model for the target speaker; Obtaining a first feature vector according to the initialized GMM-UBM model, and obtaining a second feature vector according to the GMM-UBM model of the target speaker; The voiceprint features of the training sample set are obtained according to the first feature vector and the second feature vector.

8. A language recognition device, characterized in that: include: An acquisition module is used to acquire audio data to be recognized from a target speaker; A recognition module, used for performing speech recognition on the audio data to be recognized based on a speech recognition system trained by the method according to any one of claims 1 to 5 to obtain a recognition result; Wherein, the identification module is also used for: Inputting the audio data to be recognized into the trained acoustic model to obtain a phoneme sequence corresponding to the audio data to be recognized; The phoneme sequence is input into the trained language model to obtain a text recognition result of the audio data to be recognized.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

10. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 6.

Images