Speech recognition method and device for non-standard speech, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a voice recognition method and device for non-standard voice, equipment and a medium. According to the method and the device, feature extraction is carried out on the standard voice by using the self-supervised model which learns the voice features of the non-native language person, so that the extracted voice features carry some features of the pronunciation of the non-native language person, and therefore, when voice reconstruction is carried out according to the voice features, non-standard voice with the pronunciation features of the non-native language person can be obtained; according to the method, the sample data of the non-standard voice is increased, the pre-trained voice recognition model is finely adjusted according to the non-standard voice, the recognition capability of the voice recognition model on the non-standard voice is enhanced, and therefore the recognition precision of the voice recognition model on the non-standard voice is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, device, equipment and medium for non-standard speech. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has been widely used in various fields such as education, entertainment, medical care, and transportation. At present, speech recognition models are obtained by collecting a large amount of corpus data in common scenarios, and speech recognition is performed based on the speech recognition model. Speech recognition is the conversion of speech into corresponding text. However, when using a speech recognition model to perform speech recognition on non-standard speech spoken by non-native speakers, due to the small amount of corpus data, the recognition accuracy of the speech recognition model is low and the corresponding text cannot be correctly recognized. Therefore, in the process of speech recognition of non-standard speech of non-native speakers, how to improve the accuracy of speech recognition has become an urgent problem to be solved. Summary of the invention

[0003] Based on this, it is necessary to provide a speech recognition method, device, equipment and medium for non-standard speech in order to solve the problem of low speech recognition accuracy in response to the above technical problems.

[0004] A first aspect of an embodiment of the present application provides a speech recognition method for non-standard speech, the speech recognition method comprising: Acquire a non-standard speech of a non-native speaker to be recognized, and a standard speech of the same language type as the non-standard speech, extract features of the standard speech using a self-supervised model trained based on a native language training sample of the non-native speaker to obtain speech features, and perform speech reconstruction processing on the speech features to obtain reconstructed speech; Using a pre-trained speech recognition model, respectively, the reconstructed speech and the standard speech are subjected to speech recognition to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech; According to the first recognition text and the second recognition text, fine-tuning the parameters of the pre-trained speech recognition model to obtain a fine-tuned speech recognition model; The fine-tuned speech recognition model is used to perform speech recognition on the non-standard speech to obtain a target recognition text.

[0005] A second aspect of an embodiment of the present application provides a speech recognition device for non-standard speech, the speech recognition device comprising: An acquisition module is used to acquire a non-standard speech of a non-native speaker to be recognized, and a standard speech of the same language type as the non-standard speech, extract features of the standard speech using a self-supervised model trained based on a native language training sample of the non-native speaker to obtain speech features, and perform speech reconstruction processing on the speech features to obtain reconstructed speech; A recognition module, used for performing speech recognition on the reconstructed speech and the standard speech respectively using a pre-trained speech recognition model to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech; A fine-tuning module, configured to perform parameter fine-tuning processing on the pre-trained speech recognition model according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model; A module is obtained, which is used to use the fine-tuned speech recognition model to perform speech recognition on the non-standard speech to obtain a target recognition text.

[0006] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech recognition method for non-standard speech as described in the first aspect when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech recognition method for non-standard speech as described in the first aspect is implemented.

[0008] Compared with the prior art, the present invention has the following beneficial effects: The non-standard speech of a non-native speaker to be recognized and the standard speech of the same language type as the non-standard speech are obtained, and a self-supervised model trained based on the native language training samples of the non-native speaker is used to extract features of the standard speech to obtain speech features, and the speech features are subjected to speech reconstruction processing to obtain reconstructed speech, and a pre-trained speech recognition model is used to perform speech recognition on the reconstructed speech and the standard speech respectively to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech, and the pre-trained speech recognition model is subjected to parameter fine-tuning processing according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model, and the fine-tuned speech recognition model is used to perform speech recognition on the non-standard speech to obtain a target recognition text. In the present application, feature extraction of standard speech is performed by using a self-supervised model that has learned the speech features of non-native speakers, so that the extracted speech features have some features of the pronunciation of non-native speakers. Therefore, when speech reconstruction is performed based on the speech features, non-standard speech with the pronunciation features of non-native speakers can be obtained, sample data of non-standard speech is increased, and a pre-trained speech recognition model is fine-tuned based on the non-standard speech, thereby enhancing the recognition ability of the speech recognition model for non-standard speech, thereby improving the recognition accuracy of the speech recognition model for non-standard speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0010] Figure 1 It is a schematic diagram of an application environment of a speech recognition method for non-standard speech provided by an embodiment of the present invention; Figure 2 It is a flowchart of a speech recognition method for non-standard speech provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a speech recognition device for non-standard speech provided by an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0013] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0014] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0015] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0016] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0017] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0018] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0019] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0020] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0021] An embodiment of the present invention provides a method for recognizing non-standard speech, which can be applied in the following aspects: Figure 1 In an application environment, a client communicates with a server. The client includes, but is not limited to, a PDA, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and other computer devices. The server can be implemented with an independent server or a server cluster generated by multiple servers.

[0022] See also Figure 2 , is a flow chart of a method for speech recognition of non-standard speech provided by an embodiment of the present invention. The above-mentioned method for speech recognition of non-standard speech can be applied to Figure 1 The server in the above connects to the corresponding client, such as Figure 2 As shown, the speech recognition method for non-standard speech may include the following steps.

[0023] S201: Obtain non-standard speech of a non-native speaker to be recognized, and standard speech of the same language type as the non-standard speech, extract features of the standard speech using a self-supervised model trained based on native language training samples of the non-native speaker to obtain speech features, perform speech reconstruction processing on the speech features to obtain reconstructed speech.

[0024] In step S201, non-standard speech of a non-native speaker to be identified is obtained, wherein the non-standard speech is speech spoken by the non-native speaker, and the standard speech of the same language type as the non-standard speech is speech spoken by a native speaker corresponding to the language type of the non-standard speech. A self-supervised model trained based on native language training samples of the non-native speaker is used to extract features of the standard speech to obtain speech features, and speech features are reconstructed to obtain reconstructed speech, wherein the reconstructed speech has features of the non-native speaker's native language.

[0025] In this embodiment, the non-standard speech of the non-native speaker to be recognized is obtained. For example, the speech of the native language spoken by a foreigner in the process of seeking medical treatment in his or her native country can be obtained. For example, the non-standard speech can be the Chinese speech spoken by a foreigner who speaks English. The standard speech of the same language type as the non-standard speech is the corresponding standard Chinese. It should be noted that the Chinese marked here refers to the correct grammar and pronunciation.

[0026] The self-supervised model trained based on the native language training samples of the non-native speaker is used to extract features from the standard speech to obtain speech features. For example, if the non-native speaker is a foreigner who speaks English, the native language training samples of the non-native speaker are standard English speech. The self-supervised model is trained using the native language training samples of the non-native speaker so that the self-supervised model can learn the pronunciation features of the non-native speaker. After the self-supervised model learns the pronunciation features of the non-native speaker, when the speech features of the standard speech of the same language type as the non-standard speech are extracted, the extracted speech features are made to carry the pronunciation features of the non-native speaker, so that the speech reconstructed according to the corresponding speech features can be used as the non-standard speech uttered by the non-target speaker. The self-supervised model can quickly generate non-standard speech, which is convenient for using the generated non-standard speech to train the speech recognition model.

[0027] Optionally, a self-supervised model trained based on native language training samples of non-native speakers is used to extract features from the standard speech to obtain speech features, including: Use the convolution layer in the self-supervised model to perform convolution operations on the standard speech to obtain the convolution features of the standard speech; Use the activation layer in the self-supervised model to activate the convolutional features to obtain the activation features of the convolutional features; The pooling operation is performed on the activated features using the pooling layer in the self-supervised model to obtain the pooled features of the activated features, and the pooled features are determined as the speech features of the standard speech.

[0028] In this embodiment, before using a self-supervised model trained based on native language training samples of non-native speakers to extract features of standard speech, a self-supervised model is first constructed. In this embodiment, a convolutional encoder and a convolutional decoder are constructed on the basis of an automatic encoder, so that the convolutional encoder extracts convolutional features of the standard speech, and then the convolutional layer in the self-supervised model is used to perform a convolution operation on the standard speech to obtain the convolutional features of the standard speech, and the activation layer in the self-supervised model is used to perform an activation operation on the convolutional features to obtain activation features of the convolutional features, and the pooling layer in the self-supervised model is used to perform a pooling operation on the activation features to obtain pooled features of the activation features, and the pooled features are determined as speech features of the standard speech, and the speech features are reconstructed according to the convolutional decoder to obtain reconstructed speech.

[0029] Before extracting the convolution feature, the standard speech can be subjected to dimensionality reduction processing to improve the extraction efficiency. Specifically, the dimensionality reduction processing can be implemented through a 1*1 convolution method, that is, a preset number of 1*1 convolution kernels are set, and a preset number of 1*1 convolution kernels are used to perform convolution processing on the input vector corresponding to the standard speech to obtain a reduced dimensionality convolution feature with unchanged size and reduced dimensionality. For example, in this embodiment, the input vector size is 32*32*192, wherein 32*32 corresponds to the size parameter of the input vector, and 192 corresponds to the number of dimensions of the input vector. Accordingly, the size of the 1*1 convolution kernel is specifically 1*1*192, and the preset number is set to 16. Then, the size of the obtained reduced dimensionality feature is 32*32*16. It can be seen that before and after the dimensionality reduction processing, the size of the vector has not changed, but the number of dimensions has been reduced.

[0030] The purpose of the dimensionality reduction layer processing may be to reduce the amount of calculation of the filter encoder. Obviously, fewer parameters need to be calculated for subsequent processing of the dimensionality reduction features, thus effectively improving the calculation efficiency.

[0031] Optionally, performing speech reconstruction processing on the speech features to obtain reconstructed speech includes: Use the fully connected layer in the preset decoder to transform the speech features to obtain fully connected features; Call the gated recurrent neural network in the decoder to extract the time series features from the fully connected features to obtain the time series features; Input the time series features into the output layer of the decoder to obtain the final speech features; The final speech feature is reconstructed using a preset vocoder to obtain the reconstructed speech.

[0032] In this embodiment, speech reconstruction is performed based on the decoder, and the speech features are transformed using the fully connected layer in the preset decoder to obtain fully connected features. The gated recurrent neural network in the decoder is called to extract time series features from the fully connected features to obtain time series features, wherein the transformation features may refer to features used to characterize the connection results. Specifically, the gated recurrent network may be used to retain some historical prediction information during the cyclic prediction process, thereby improving the accuracy of the prediction.

[0033] This embodiment uses a gated recurrent network to achieve prediction of the stimulus value, thereby retaining effective feature information of the generation process, thereby avoiding jumps in the prediction results and improving the accuracy of the stimulus value prediction.

[0034] The time series features are input into the output layer of the decoder to obtain the final speech features. The output layer of the decoder can be a fully connected layer. The time series features are input into the fully connected layer for feature mapping to obtain the final speech features. The final speech features are reconstructed using a preset vocoder to obtain the reconstructed speech, wherein the preset vocoder is a trained vocoder, and the trained vocoder can be a neural network vocoder, and can also be but not limited to a neural network vocoder based on autoregression such as a waveRNN vocoder, a neural network vocoder based on a Gan network such as a melGan vocoder, and a hifiGan vocoder.

[0035] It should be noted that the reconstructed speech is a non-standard local speech with the pronunciation characteristics of a non-native speaker. For example, if the non-native speaker is a foreigner who speaks English, and the language type of the corresponding non-standard speech is Chinese, then the reconstructed speech is non-standard Chinese. Using this method, a large number of non-standard speech can be obtained for training the speech recognition model, so that the trained speech recognition model can more accurately recognize the non-standard speech, which is conducive to obtaining accurate recognition text.

[0036] S202: Use a pre-trained speech recognition model to perform speech recognition on the reconstructed speech and the standard speech respectively, to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech.

[0037] In step S202, the reconstructed speech is a non-standard local speech with pronunciation characteristics of a non-native speaker, and the reconstructed speech and the standard speech are recognized using a pre-trained speech recognition model to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech, so as to calculate the loss between the recognition of the non-standard speech and the recognition of the standard speech, thereby adjusting the parameters of the speech recognition model so that the speech recognition model can better recognize the non-standard speech.

[0038] In this embodiment, a pre-trained speech recognition model is used to perform speech recognition on the reconstructed speech and the standard speech, wherein the pre-trained speech recognition model can be a deep learning model, and the pre-trained speech recognition model includes a classification recognition network and an attention recognition network. Among them, the classification recognition network is used to convert the reconstructed speech and the standard speech into text, and the classification recognition network includes a classification encoding module and a classification decoding module. Further, the classification encoding module can be constructed using neural networks such as CNN / RNN / BiRNN / GRU / LSTM, and the classification decoding module can be constructed using a CTC algorithm. The attention recognition network is also used to convert the reconstructed speech and the standard speech into text. The attention recognition network includes an attention encoding module, an attention processing module, and an attention decoding module. Further, the attention encoding module is the same as the classification encoding module, and can also be constructed using neural networks such as CNN / RNN / BiRNN / GRU / LSTM. The attention processing module is constructed based on the attention mechanism, and the attention decoding module can also be constructed using neural networks such as CNN / RNN / BiRNN / GRU / LSTM.

[0039] Furthermore, in addition to the above-described speech recognition model including a classification recognition network and an attention recognition network, the classification recognition network can also be directly used as a speech recognition model, or the attention recognition network can be directly used as a speech recognition model.

[0040] It should be noted that the pre-trained speech recognition model is used to perform speech recognition on the reconstructed speech and the standard speech respectively, to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech, wherein the reconstructed speech and the standard speech are semantically identical.

[0041] Optionally, before using the pre-trained speech recognition model to perform speech recognition on the reconstructed speech and the standard speech respectively, the method further includes: Obtaining standard speech training samples of the same language type as the non-standard speech and recognition labels of the standard speech training samples, as well as an initial speech recognition model; According to the standard speech training samples and recognition labels, the initial speech recognition model is supervised trained to obtain a trained speech recognition model, which is a pre-trained speech recognition model.

[0042] In this embodiment, an initial speech recognition model is obtained. The initial speech recognition model includes a classification recognition network and an attention recognition network. The initial speech recognition model is supervised and trained based on standard speech training samples and recognition labels. During training, a greedy loss function can be used to constrain the model to obtain a trained speech recognition model. The trained speech recognition model is a pre-trained speech recognition model.

[0043] S203: Fine-tune the parameters of the pre-trained speech recognition model according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model.

[0044] In step S203, the loss between the first recognized text and the second recognized text can be calculated based on the first recognized text and the second recognized text, and the parameters of the pre-trained speech recognition model can be fine-tuned according to the corresponding loss to obtain a fine-tuned speech recognition model, wherein the fine-tuned speech recognition model can perform speech recognition on unlabeled speech, thereby improving the recognition accuracy of non-standard speech.

[0045] In this embodiment, the pre-trained speech recognition model is fine-tuned, wherein the fine-tuning is to adjust the parameters in the pre-trained model so that the fine-tuned speech recognition model can reduce the loss between the first recognized text and the second recognized text.

[0046] Optionally, fine-tuning parameters of a pre-trained speech recognition model is performed according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model, including: Calculating the loss between the first recognized text and the second recognized text to obtain a loss value; The parameters of the pre-trained speech recognition model are fine-tuned according to the loss value to obtain a fine-tuned speech recognition model.

[0047] In this embodiment, the pre-trained speech recognition model is a model trained using standard speech. When the pre-trained speech recognition model is used to recognize the standard speech, the correct text recognition result corresponding to the standard speech can be obtained, that is, the second recognized text is the semantically correct text corresponding to the standard speech. Therefore, the recognition result of the standard speech, that is, the second recognized text, can be used as the corresponding label, and the loss between the first recognized text and the second recognized text can be calculated as the loss between the first recognized text and the label text. Therefore, the parameters of the pre-trained speech recognition model can be fine-tuned according to the corresponding loss to obtain a fine-tuned speech recognition model.

[0048] Optionally, calculating the loss between the first recognized text and the second recognized text to obtain a loss value includes: Extracting features from the first recognition text and the second recognition text respectively to obtain a first feature of the first recognition text and a second feature of the second recognition text; Calculate the difference between the first feature and the second feature to get the loss value.

[0049] In this embodiment, a preset encoder can be used to extract features from the first recognized text and the second recognized text to obtain corresponding text features, and a preset encoder can be used to feature encode the text to obtain text encoding features, wherein the preset encoder is composed of a character embedding layer, a position encoding layer and a FFTBlock module (the module is composed of multiple convolutional layers and a self-attention mechanism layer), the text input sequence is processed by the character embedding layer, encoded into a sequence vector of fixed dimension, a position vector is constructed through the position encoding layer, the encoded text sequence vector and the corresponding position of the position vector are added, and the result of the addition is sent to the multi-layer FFTBlock module and converted into a text encoding feature of fixed length to obtain the first feature of the first recognized text and the second feature of the second recognized text.

[0050] It should be noted that when encoding the first recognition text and the second recognition text, the first recognition text and the second recognition text need to be converted into corresponding phoneme sequences first, and the phoneme sequence is a sequence of prosodic features and phoneme features used to characterize the information of the first recognition text and the second recognition text. The phoneme sequence includes a prosodic identifier located between adjacent phonemes and a plurality of phonemes corresponding to the text information. Among them, the phoneme can be a combination of one or more phonetic units divided according to the natural attributes of the voice, and the phonetic unit can be a pinyin, initial consonant or final vowel corresponding to a Chinese character, or an English word, English phonetic symbol or English letter. The prosodic identifier is an identifier used to characterize the prosodic features corresponding to each phoneme in the text information, and the prosodic features include but are not limited to: the tone, syllable, prosodic word, prosodic phrase, intonation phrase, silence and pause corresponding to the phoneme.

[0051] It should be noted that the fine-grainedness of the prosodic identifier used to characterize pauses is higher than the fine-grainedness of the prosodic identifier used to characterize intonation phrases, the fine-grainedness used to characterize intonation phrases is higher than the fine-grainedness used to characterize prosodic phrases, the fine-grainedness used to characterize prosodic phrases is higher than the fine-grainedness used to characterize prosodic words, and the fine-grainedness used to characterize prosodic words is higher than the fine-grainedness used to characterize syllables. When encoding text using a preset encoder, the preset encoder splits the text into individual characters and obtains the sequential relationship between the individual characters.

[0052] In this embodiment, each first recognition text and the second recognition text are segmented to obtain a corresponding segmentation sequence. In this embodiment, the word vector of the segmentation sequence is calculated by a preset word vector calculation tool, and word2vec is used as a word vector calculation tool. Word2vec is a natural language processing tool, and its function is to convert words in natural language into word vectors that can be understood by computers. Traditional word vectors are easily troubled by the dimensionality disaster, and any two words are isolated and cannot reflect the relationship between words. Therefore, this embodiment uses word2vec to obtain word vectors, which can reflect the similarity between words by calculating the distance between vectors. Word2Vec mainly uses Skip-Gram and CBOW models. This embodiment uses Skip-Gram to realize the conversion of word vectors. The Skip-Gram model mainly predicts the words in the context through the center word.

[0053] The difference between the first feature and the second feature is calculated to obtain the loss value. When calculating the difference between the first feature and the second feature, the vector distance between the corresponding first feature and the second feature can be used. The larger the distance, the larger the loss value, and the smaller the distance, the smaller the loss value.

[0054] Optionally, fine-tuning parameters of the pre-trained speech recognition model is performed according to the loss value to obtain a fine-tuned speech recognition model, including: Using the loss value to perform initial parameter fine-tuning on the pre-trained speech recognition model, optimizing the parameters in the pre-trained speech recognition model, and obtaining the speech recognition model after initial fine-tuning; If the loss value does not meet the preset conditions, the initially fine-tuned speech recognition model is determined as the pre-trained speech recognition model, and the steps of performing speech recognition on the reconstructed speech and the standard speech are performed respectively until the loss value meets the preset conditions, thereby obtaining the fine-tuned speech recognition model.

[0055] In this embodiment, when fine-tuning the pre-trained speech recognition model, training fine-tuning is performed until the fine-tuned speech recognition model meets the conditions and the fine-tuning is stopped, wherein the loss value is used to perform initial parameter fine-tuning on the pre-trained speech recognition model, and the parameters in the pre-trained speech recognition model are optimized to obtain the initial fine-tuned speech recognition model. If the loss value does not meet the preset conditions, the initial fine-tuned speech recognition model is determined as the pre-trained speech recognition model, and the steps of performing speech recognition on the reconstructed speech and the standard speech are executed respectively until the loss value meets the preset conditions to obtain the fine-tuned speech recognition model.

[0056] It should be noted that a preset self-supervised model can be used to generate a large number of non-standard speech, and the generated non-standard speech can be used to fine-tune the pre-trained speech recognition model, which solves the problem of small number of non-standard speech samples. A large number of non-standard speech can be used for fine-tuning, so that the fine-tuned speech recognition model can improve the recognition accuracy of non-standard speech.

[0057] S204: Use the fine-tuned speech recognition model to perform speech recognition on the non-standard speech to obtain a target recognition text.

[0058] In this embodiment, the fine-tuned speech recognition model is used to perform speech recognition on non-standard speech to obtain target recognition text, so that the obtained target recognition text can be used for text classification, etc.

[0059] The non-standard speech of a non-native speaker to be recognized and the standard speech of the same language type as the non-standard speech are obtained, and a self-supervised model trained based on the native language training samples of the non-native speaker is used to extract features of the standard speech to obtain speech features, and the speech features are subjected to speech reconstruction processing to obtain reconstructed speech, and a pre-trained speech recognition model is used to perform speech recognition on the reconstructed speech and the standard speech respectively to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech, and the pre-trained speech recognition model is subjected to parameter fine-tuning processing according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model, and the fine-tuned speech recognition model is used to perform speech recognition on the non-standard speech to obtain a target recognition text. In the present application, feature extraction of standard speech is performed by using a self-supervised model that has learned the speech features of non-native speakers, so that the extracted speech features have some features of the pronunciation of non-native speakers. Therefore, when speech reconstruction is performed based on the speech features, non-standard speech with the pronunciation features of non-native speakers can be obtained, sample data of non-standard speech is increased, and a pre-trained speech recognition model is fine-tuned based on the non-standard speech, thereby enhancing the recognition ability of the speech recognition model for non-standard speech, thereby improving the recognition accuracy of the speech recognition model for non-standard speech.

[0060] See also Figure 3 , Figure 3 The structure block diagram of the speech recognition device for non-standard speech provided by the embodiment of the present application is shown. The above-mentioned sound conversion device is applied to the above-mentioned server. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Figure 3 The speech recognition device 30 includes: an acquisition module 31, a recognition module 32, a fine-tuning module 33, and a obtaining module 34.

[0061] The acquisition module 31 is used to acquire the non-standard speech of the non-native speaker to be recognized and the standard speech of the same language type as the non-standard speech, extract features of the standard speech using the self-supervised model trained based on the native language training samples of the non-native speaker to obtain speech features, and perform speech reconstruction processing on the speech features to obtain reconstructed speech; A recognition module 32, configured to perform speech recognition on the reconstructed speech and the standard speech respectively using a pre-trained speech recognition model to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech; A fine-tuning module 33, configured to perform parameter fine-tuning processing on the pre-trained speech recognition model according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model; The module 34 is used to perform speech recognition on the non-standard speech using the fine-tuned speech recognition model to obtain the target recognition text.

[0062] Optionally, the acquisition module 31 includes: A convolution unit, used to perform a convolution operation on the standard speech using the convolution layer in the self-supervised model to obtain the convolution features of the standard speech; An activation unit is used to activate the convolutional features using the activation layer in the self-supervised model to obtain the activation features of the convolutional features; The pooling unit is used to perform a pooling operation on the activated features using the pooling layer in the self-supervised model to obtain the pooled features of the activated features, and determine the pooled features as the speech features of the standard speech.

[0063] Optionally, the acquisition module 31 includes: A transformation unit, used to transform the speech features using a fully connected layer in a preset decoder to obtain a fully connected feature; An extraction unit, used to call the gated recurrent neural network in the decoder to extract time series features from the fully connected features to obtain time series features; The output unit is used to input the time series features into the output layer of the decoder to obtain the final speech features; The reconstruction unit is used to use a preset vocoder to reconstruct the waveform of the final speech feature to obtain the reconstructed speech.

[0064] Optionally, the speech recognition device 30 further includes: An acquisition unit, used to acquire a standard speech training sample of the same language type as the non-standard speech, an identification label of the standard speech training sample, and an initial speech recognition model; The supervised training unit is used to perform supervised training on the initial speech recognition model according to standard speech training samples and recognition labels to obtain a trained speech recognition model. The trained speech recognition model is a pre-trained speech recognition model.

[0065] Optionally, the fine-tuning module 33 includes: A calculation unit, used for calculating the loss between the first recognition text and the second recognition text to obtain a loss value; The obtaining unit is used to perform parameter fine-tuning processing on the pre-trained speech recognition model according to the loss value to obtain the fine-tuned speech recognition model.

[0066] Optionally, the calculation unit includes: A feature extraction subunit, used to extract features from the first recognition text and the second recognition text respectively, to obtain a first feature of the first recognition text and a second feature of the second recognition text; The difference calculation subunit is used to calculate the difference between the first feature and the second feature to obtain a loss value.

[0067] Optionally, the obtaining unit includes: An optimization unit, used to perform initial parameter fine-tuning processing on the pre-trained speech recognition model using the loss value, optimize the parameters in the pre-trained speech recognition model, and obtain the speech recognition model after initial fine-tuning; The iterative unit is used to determine the initially fine-tuned speech recognition model as the pre-trained speech recognition model if the loss value does not meet the preset conditions, and execute the steps of performing speech recognition on the reconstructed speech and the standard speech respectively until the loss value meets the preset conditions, thereby obtaining the fine-tuned speech recognition model.

[0068] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0069] Figure 4 Schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, any of the above-mentioned speech translation method steps are implemented.

[0070] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0071] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0072] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0073] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0074] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0075] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0077] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A speech recognition method for non-standard speech, characterized in that: The speech recognition method comprises: Acquire a non-standard speech of a non-native speaker to be recognized, and a standard speech of the same language type as the non-standard speech, extract features of the standard speech using a self-supervised model trained based on a native language training sample of the non-native speaker to obtain speech features, and perform speech reconstruction processing on the speech features to obtain reconstructed speech; Using a pre-trained speech recognition model, respectively, the reconstructed speech and the standard speech are subjected to speech recognition to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech; According to the first recognition text and the second recognition text, fine-tuning the parameters of the pre-trained speech recognition model to obtain a fine-tuned speech recognition model; The fine-tuned speech recognition model is used to perform speech recognition on the non-standard speech to obtain a target recognition text.

2. The speech recognition method according to claim 1, characterized in that: The method of extracting features from the standard speech using a self-supervised model trained based on the native language training samples of the non-native speaker to obtain speech features includes: Using the convolution layer in the self-supervisory model to perform a convolution operation on the standard speech to obtain convolution features of the standard speech; Using the activation layer in the self-supervisory model to perform an activation operation on the convolutional feature to obtain an activation feature of the convolutional feature; The activation feature is pooled using the pooling layer in the self-supervisory model to obtain a pooled feature of the activation feature, and the pooled feature is determined as the speech feature of the standard speech.

3. The speech recognition method according to claim 1, wherein: The performing speech reconstruction processing on the speech feature to obtain the reconstructed speech includes: Using a fully connected layer in a preset decoder to transform the speech feature to obtain a fully connected feature; Calling the gated recurrent neural network in the decoder to extract time series features from the fully connected features to obtain time series features; Inputting the time series features into the output layer of the decoder to obtain the final speech features; The final speech feature is reconstructed into a waveform using a preset vocoder to obtain a reconstructed speech.

4. The speech recognition method according to claim 1, characterized in that: Before using the pre-trained speech recognition model to perform speech recognition on the reconstructed speech and the standard speech respectively, the method further includes: Obtaining a standard speech training sample of the same language type as the non-standard speech and an identification label of the standard speech training sample, and an initial speech recognition model; The initial speech recognition model is supervisedly trained according to the standard speech training samples and the recognition labels to obtain a trained speech recognition model, which is a pre-trained speech recognition model.

5. The speech recognition method according to claim 1, wherein: The step of fine-tuning parameters of the pre-trained speech recognition model according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model includes: Calculating the loss between the first recognized text and the second recognized text to obtain a loss value; The pre-trained speech recognition model is fine-tuned according to the loss value to obtain a fine-tuned speech recognition model.

6. The speech recognition method according to claim 5, characterized in that: The calculating the loss between the first recognized text and the second recognized text to obtain a loss value includes: Extracting features from the first recognized text and the second recognized text respectively to obtain a first feature of the first recognized text and a second feature of the second recognized text; The difference between the first feature and the second feature is calculated to obtain the loss value.

7. The speech recognition method according to claim 5, characterized in that: The fine-tuning of parameters of the pre-trained speech recognition model according to the loss value to obtain a fine-tuned speech recognition model includes: Using the loss value to perform initial parameter fine-tuning processing on the pre-trained speech recognition model, optimizing the parameters in the pre-trained speech recognition model, and obtaining the speech recognition model after initial fine-tuning; If the loss value does not meet the preset conditions, the initially fine-tuned speech recognition model is determined as a pre-trained speech recognition model, and the steps of performing speech recognition on the reconstructed speech and the standard speech are executed respectively until the loss value meets the preset conditions, thereby obtaining a fine-tuned speech recognition model.

8. A speech recognition device for non-standard speech, characterized in that: The speech recognition device comprises: An acquisition module is used to acquire a non-standard speech of a non-native speaker to be recognized, and a standard speech of the same language type as the non-standard speech, extract features of the standard speech using a self-supervised model trained based on a native language training sample of the non-native speaker to obtain speech features, and perform speech reconstruction processing on the speech features to obtain reconstructed speech; A recognition module, used for performing speech recognition on the reconstructed speech and the standard speech respectively using a pre-trained speech recognition model to obtain a first recognition text of the reconstructed speech and a second recognition text of the standard speech; A fine-tuning module, configured to perform parameter fine-tuning processing on the pre-trained speech recognition model according to the first recognition text and the second recognition text to obtain a fine-tuned speech recognition model; A module is obtained, which is used to use the fine-tuned speech recognition model to perform speech recognition on the non-standard speech to obtain a target recognition text.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

Images