Speech recognition method and device, storage medium and equipment

Abstract

The invention discloses a speech recognition method and device, a storage medium and equipment. The method comprises the following steps: firstly, obtaining a to-be-recognized target speech and extracting a target acoustic feature of the to-be-recognized target speech; inputting the target acoustic feature into a pre-constructed speech recognition model, and recognizing to obtain a speech hot word recognition result of the target speech; wherein the speech recognition model comprises an audio encoder, a hot word encoder and a decoder; the decoder includes a hotword hybrid expert adapter and an attention mechanism network. The end-to-end speech recognition large model is trained to generate the speech recognition model based on the adapter by using the sample speech data and the corresponding text labeling result and the sample hot word text, so that the model training efficiency is improved, the training cost is reduced, and the recognition accuracy of the model is also effectively improved, and therefore, the recognition efficiency of the speech recognition model is improved. When the voice recognition model is used for performing voice hot word recognition on the target voice, the efficiency and the accuracy of a recognition result can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech recognition method, apparatus, storage medium and device. Background Art

[0002] With the continuous breakthroughs in artificial intelligence technology and the increasing popularity of various intelligent terminal devices, human-computer interaction appears more and more frequently in people's daily work and life. Voice interaction, as the next generation of human-computer interaction, can bring great convenience to people's lives. Among them, accurate voice recognition is more important. At present, the more commonly used voice recognition method is the voice recognition method based on end-to-end network. However, the end-to-end network lacks customization capabilities and can only realize voice recognition in general scenarios. The recognition effect of hot words in specific business scenarios such as certain special nouns or proper nouns in a certain field is poor.

[0003] At present, there are usually two methods to improve the recognition effect of speech hot words: one is hot word incentive, that is, scoring incentives for potential hot word candidates during the decoding process, but this method is commonly used in traditional speech recognition and has limited effect in end-to-end speech recognition. The other is to modify the end-to-end network model structure to improve the probability score of hot words from the model level, such as the CLAS (Contextual Listen, Attend and Spell, CLAS) framework, but this method requires full parameter updates. For large speech models, model training takes a long time and consumes a lot of resources. It may also damage the performance of existing baseline models and reduce the recognition effect of the model. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method, apparatus, storage medium and equipment, which can effectively improve the training efficiency of the model and the recognition effect of speech hot words.

[0005] The present application provides a speech recognition method, including:

[0006] Obtaining the target speech to be recognized;

[0007] Extracting target acoustic features of the target speech;

[0008] Inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0009] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word mixed expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0010] In a possible implementation, inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech includes:

[0011] Inputting the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain a coded feature vector of the target speech;

[0012] Using the hot word encoder of the speech recognition model to independently encode each preset hot word, to obtain a set of feature vectors of the hot words;

[0013] Fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0014] The target fusion feature vector is input into the decoder of the speech recognition model to be decoded through a hot word mixture expert adapter and an attention mechanism network to obtain a speech hot word recognition result of the target speech.

[0015] In a possible implementation, fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector includes:

[0016] The encoded feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained spliced ​​vector, a zero vector is added between the encoded feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0017] In a possible implementation, the speech recognition model is constructed as follows:

[0018] Acquire sample speech; and determine the text annotation result and sample hot word text corresponding to the sample speech;

[0019] Extracting sample acoustic features of the sample speech;

[0020] The sample acoustic features and text annotation results are used in combination with the sample hot word text to train the initial speech recognition model to obtain the speech recognition model.

[0021] In a possible implementation, the initial speech recognition model includes an audio encoder, an initial hot word encoder and an initial decoder; the decoder includes an initial hot word hybrid expert adapter and an attention mechanism network; the initial speech recognition model is trained by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model, including:

[0022] Inputting the sample acoustic features into an audio encoder of an initial speech recognition model to obtain a coded feature vector of the sample speech;

[0023] Fusing the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector;

[0024] Inputting the sample fusion feature vector into the initial decoder of the initial speech recognition model to decode it through the initial hot word mixed expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the sample speech;

[0025] According to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder and the initial hot word mixed expert adapter are updated until the preset conditions are met, then the updating of the model parameters is stopped and the speech recognition model is trained to obtain the speech recognition model.

[0026] In a possible implementation, the method further includes:

[0027] Obtaining a verification voice; and determining a text annotation result corresponding to the verification voice;

[0028] Extracting the verification acoustic features of the verification speech; and inputting the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech;

[0029] Fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector;

[0030] Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech;

[0031] When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0032] In a possible implementation, the target acoustic feature of the target speech is at least one of a Mel-cephalometric coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0033] The present application also provides a speech recognition device, the device comprising:

[0034] A first acquisition unit, used to acquire a target speech to be recognized;

[0035] A first extraction unit, configured to extract target acoustic features of the target speech;

[0036] A recognition unit, used for inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0037] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word mixed expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0038] In a possible implementation, the identification unit includes:

[0039] A first input subunit, configured to input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain a coded feature vector of the target speech;

[0040] An encoding subunit, used for independently encoding each preset hot word using a hot word encoder of the speech recognition model to obtain a set of feature vectors of the hot words;

[0041] A first fusion subunit is used to fuse the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0042] The second input subunit is used to input the target fusion feature vector into the decoder of the speech recognition model to decode it through the hot word mixed expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0043] In a possible implementation, the first fusion subunit is specifically used for:

[0044] The encoded feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained spliced ​​vector, a zero vector is added between the encoded feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0045] In a possible implementation, the device further includes:

[0046] The second acquisition unit is used to acquire a sample speech; and determine a text annotation result and a sample hot word text corresponding to the sample speech;

[0047] A second extraction unit, used to extract sample acoustic features of the sample speech;

[0048] The training unit is used to train the initial speech recognition model by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model.

[0049] In a possible implementation, the initial speech recognition model includes an audio encoder, an initial hot word encoder and an initial decoder; the decoder includes an initial hot word hybrid expert adapter and an attention mechanism network; the training unit includes:

[0050] A third input subunit, used for inputting the sample acoustic feature into the audio encoder of the initial speech recognition model to obtain a coding feature vector of the sample speech;

[0051] A second fusion subunit is used to fuse the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fusion feature vector;

[0052] A fourth input subunit is used to input the sample fusion feature vector into the initial decoder of the initial speech recognition model, so as to decode it through the initial hot word hybrid expert adapter and the attention mechanism network, and identify the speech hot word recognition result of the sample speech;

[0053] The training subunit is used to update the parameters of the initial hot word encoder and the initial hot word mixed expert adapter according to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, until the preset conditions are met, then stop updating the model parameters and train to obtain the speech recognition model.

[0054] In a possible implementation manner, the device further includes:

[0055] A third acquisition unit is used to acquire a verification voice; and determine a text annotation result corresponding to the verification voice;

[0056] A third extraction unit is used to extract the verification acoustic features of the verification speech; and input the verification acoustic features into the audio encoder of the speech recognition model to obtain the encoding feature vector of the verification speech;

[0057] A fusion unit, used for fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fusion feature vector;

[0058] An input unit, used to input the verification fusion feature vector into a decoder of a speech recognition model to obtain a speech hot word prediction result corresponding to the verification speech;

[0059] The updating unit is used to update the speech recognition model by reusing the verification speech as the sample speech when the speech hot word prediction result corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech.

[0060] In a possible implementation, the target acoustic feature of the target speech is at least one of a Mel-cephalometric coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0061] The embodiment of the present application also provides a speech recognition device, including: a processor, a memory, and a system bus;

[0062] The processor and the memory are connected via the system bus;

[0063] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned speech recognition method.

[0064] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech recognition method.

[0065] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech recognition method.

[0066] A speech recognition method, apparatus, storage medium and device provided in an embodiment of the present application first obtain a target speech to be recognized, and then extract the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; wherein the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0067] It can be seen that since the present application is first based on an adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model to generate a speech recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when the speech recognition model is used to recognize speech hot words for the target speech, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the speech recognition effect for the target speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 A schematic diagram of the Whisper speech recognition framework provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of the CLAS model framework provided in an embodiment of the present application;

[0071] Figure 3 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0072] Figure 4 An example diagram of the training process of the speech recognition model provided in the embodiment of the present application;

[0073] Figure 5 A schematic diagram of the composition of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] Speech recognition is a common processing content in the field of artificial intelligence. For example, in scenarios such as human-computer interaction and speech-to-text, speech recognition is an indispensable processing content. At present, the more commonly used speech recognition method is the speech recognition method based on the end-to-end network, which can solve the speech recognition needs in most general scenarios. However, for different fields and different users, different recognition results may be obtained for the same pronunciation based on the personalization of the field and the user, but the existing end-to-end network lacks customization capabilities, and its recognition results for certain special nouns or hot words such as proper nouns in a certain field are not accurate enough. Therefore, how to improve the recognition accuracy of hot words in specific fields of the end-to-end speech recognition model is a technical problem that needs to be solved urgently.

[0075] Moreover, the most popular deep learning sequence modeling is the end-to-end speech recognition large model based on the attention mechanism. This end-to-end modeling idea has become a research hotspot in sequence modeling tasks such as machine translation, handwriting recognition, and speech recognition. However, the end-to-end modeling method has been upgraded from the small parameter model of previous years to a large model with more than one billion parameters. Its model capacity is larger, the amount of data used is larger, and it supports more languages. Taking the Whisper speech recognition framework as an example, Figure 1 shown.

[0076] Most of the existing open-source speech models are trained based on hundreds of thousands or millions of hours of data. Traditional end-to-end speech recognition hot word solutions require full parameter updates, which is very costly for large speech models.

[0077] At present, there are usually two methods to improve the recognition effect of speech hot words: one is hot word incentive, that is, scoring incentives for potential hot word candidates during the decoding process, but this method is commonly used in traditional speech recognition and has limited effect in end-to-end speech recognition. The other is to modify the end-to-end network model structure to improve the probability score of hot words from the model level, such as Figure 2 The CLAS model framework 200 shown, etc.

[0078] Among them, compared with Figure 1 The baseline model 100 in Figure 2 The CLAS model framework shown in Figure 1 adds an additional bias encoder, as shown in Figure 2. Figure 2 As shown in the lower middle part of the figure. The CLAS model first uses the hot word encoder (BiasEncoder) to represent each candidate hot word as a fixed-dimensional vector, and then uses the state information of the decoder (Decoder) to select the hot words that match the input speech through the attention mechanism as the output, and then sends it to the decoder for decoding together with the output of the audio encoder (Audio Encoder) to obtain a relatively accurate recognition result. However, when using the CLAS model to update the model training, all parameters need to be updated, that is, the parameters of the Audio Encoder, Bias Encoder, and Decoder need to be updated at the same time so that the three modules can adapt to each other, and then the model can reach the optimal state. For large speech models, first of all, the number of parameters has increased from tens of megabytes (M) and hundreds of megabytes (M) to billions of parameters. The number of parameters is very large, and the full parameter update may destroy the performance of the existing baseline model and reduce the model recognition effect. Secondly, the amount of speech data used by the large speech model has increased from tens of thousands of hours to tens or millions of hours of data. The full parameter update cycle will also be very long, and the resource consumption is very large.

[0079] To solve the above-mentioned defects, the present application provides a speech recognition method, which first obtains the target speech to be recognized, and then extracts the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain the speech hot word recognition results of the target speech; wherein the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0080] It can be seen that since the present application is first based on an adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model to generate a speech recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when the speech recognition model is used to recognize speech hot words for the target speech, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the speech recognition effect for the target speech.

[0081] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] First embodiment

[0083] See also Figure 3 , is a flow chart of a speech recognition method provided in this embodiment, the method comprising the following steps:

[0084] S301: Acquire target speech to be recognized.

[0085] In this embodiment, any speech recognized by this embodiment is defined as a target speech to be recognized. It should be noted that this embodiment does not limit the language type of the target speech, for example, the target speech can be a speech composed of Chinese or English, etc.; at the same time, this embodiment does not limit the length of the target speech, for example, the target speech can be a sentence or a paragraph, etc.

[0086] It can be understood that the target voice can be obtained through recording or the like according to actual needs. For example, the telephone conversation voice or meeting recording in people's daily life can be used as the target voice. When the target voice is obtained, the target voice can be processed using the solution provided in this embodiment to identify the recognition results containing accurate hot words.

[0087] S302: Extracting target acoustic features of the target speech.

[0088] In this embodiment, after the target speech to be recognized is obtained through step S301, in order to accurately identify the recognition result containing accurate hot words corresponding to the target speech, the acoustic features of the target speech can be extracted by using existing or future feature extraction methods, and defined as target acoustic features as the basis for recognition, so as to realize accurate recognition of the target speech through the subsequent step S303.

[0089] Specifically, when extracting the target acoustic features of the target speech, it is first necessary to perform frame processing on the target speech to obtain a corresponding speech frame sequence, and then pre-emphasize the speech frame sequence after frame division; and then extract the acoustic features of each speech frame in turn, wherein the acoustic feature refers to the feature data used to characterize the acoustic information of the corresponding speech frame, such as, but not limited to, Mel-scale Frequency Cepstral Coefficients (MFCC) features, Perceptual Linear Predictive (PLP) features or filter bank (FilterBank) features, etc., which constitute the target acoustic features of the target speech.

[0090] It should be noted that the embodiment of the present application does not limit the method for extracting the target acoustic features of the target speech, nor does it limit the specific extraction process. An appropriate extraction method can be selected according to the actual situation, and corresponding feature extraction operations can be performed. For ease of understanding, this embodiment will be described later by taking the target acoustic features of the target speech as filter bank features as an example.

[0091] S303: Input the target acoustic features of the target speech into a pre-built speech recognition model to obtain the speech hot word recognition result of the target speech; wherein the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0092] In this embodiment, after obtaining the target acoustic features of the target speech through step S302, in order to effectively improve the accuracy of the recognition results of the target speech, the target acoustic features of the target speech can be further input into a pre-built speech recognition model to obtain the recognition results containing accurate hot words corresponding to the target speech.

[0093] It should be noted that in order to improve the accuracy of speech recognition results, this application pre-constructs a speech recognition model, such as Figure 4 As shown, the model includes an audio encoder (such as Figure 4 Audio Encoder in ), hotword encoder (such as Figure 4 Bias Encoder) and Decoder (such as Figure 4 The decoder includes a hot word mixture expert (mixture of expect) adapter (such as Figure 4 MOE Adaper in ) and attention mechanism network (such as Figure 4 Attention in the speech recognition model). The speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train the end-to-end speech recognition model. In the training process, only the parameters of the hot word encoder (Bias Encoder) and MOE Adaper need to be updated, and there is no need to update all the model parameters, thereby improving the training efficiency of the speech recognition model, reducing the investment in training resources, and reducing training costs. In this way, by inputting the acoustic features of the target speech into the speech recognition model for recognition, both efficiency and the recognition accuracy of the target speech can be achieved, thereby improving the speech recognition effect of the target speech.

[0094] Specifically, an optional implementation method is that the specific implementation process of "inputting the target acoustic features of the target speech into the pre-built speech recognition model to identify the speech hot word recognition result of the target speech" in step S303 may include the following steps S3031-S3034:

[0095] S3031: Input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain the encoded feature vector of the target speech.

[0096] In this implementation, after extracting the acoustic features (such as FilterBank features) of the target speech in step S302, Figure 4 As shown, the FilterBank features of the target speech can be further input into the audio encoder (Audio Encoder) of the speech recognition model to obtain the encoded feature vector of the target speech, which is expressed as [a1 ,a 2 ,……,a k ,……a K ] is used to execute the subsequent step S3033, which can be understood as the feature vector output by the last neural network layer of the audio encoder (Audio Encoder), and K is a positive integer greater than 0.

[0097] S3032: Use the hot word encoder of the speech recognition model to independently encode each preset hot word to obtain a set of feature vectors of the hot words.

[0098] In this implementation, in order to improve the accuracy of speech recognition results, a bias encoder (Bias Encoder) of the speech recognition model can be used to independently encode each preset hot word (the specific content is not limited and can be set by field according to actual conditions and experience values) to obtain a set of hot word feature vectors, which are expressed as [b 1 ,b 2 ,……,b m ,……b M ], to execute the subsequent step S3033, wherein b m Represents the feature vector of the mth hot word after being processed by the hot word encoder (BiasEncoder), where M is a positive integer greater than 0.

[0099] S3033: Fuse the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector.

[0100] In this implementation, the encoding feature vector [a 1 ,a 2 ,……,a k ,……a K ], and obtain a set of hot word feature vectors [b 1 ,b 2 ,……,b m ,……b M ], the coded feature vector [a 1 ,a 2 ,……,a k ,……a K ] and the feature vector of the hot word [b 1 ,b 2 ,……,b m ,……b M ] are fused to obtain the target fused feature vector for executing the subsequent step S3034.

[0101] Specifically, the encoded feature vector [a 1 ,a 2 ,……,a k ,……a K ] and the feature vector of the hot word [b 1 ,b 2 ,……,b m ,……b M ] is spliced ​​according to the time dimension to obtain the splicing vector of K+M frames, and in the splicing vector, the coded feature vector [a 1 ,a 2 ,……,a k ,……a K ] and the feature vector of the hot word [b 1 ,b 2 ,……,b m ,……b M ] to physically isolate the two in the data dimension by adding a zero vector [z] between them, and obtain the target fusion feature vector, which is defined as C = [a 1 ,a 2 ,……,a k ,……a K ,z,b 1 ,b 2 ,……,b m ,……b M ] to execute subsequent step S3034.

[0102] S3034: Input the target fusion feature vector into the decoder of the speech recognition model to decode it through the hot word mixed expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0103] In this implementation, after the target fusion feature vector is obtained in step S3033, Figure 4 As shown, the target fusion feature vector can be further input into the decoder of the speech recognition model to be used by the hot word mixing expert adapter (such as Figure 4 MOE Adaper in ) and attention mechanism network (such as Figure 4 The Attention in the target speech is used for coordinated decoding, and the text recognition result containing more accurate hot words corresponding to the target speech is output, thereby improving the recognition effect of speech hot words.

[0104] Next, this embodiment will introduce the process of constructing a speech recognition model. In one optional implementation, the process of constructing a speech recognition model may specifically include the following steps A1-A3:

[0105] Step A1: Obtain sample speech; and determine the text annotation result and sample hot word text corresponding to the sample speech.

[0106] In this embodiment, in order to build a speech recognition model, a lot of preparatory work needs to be done in advance. First, a large amount of speech data in various fields and scenarios needs to be collected. For example, the sound can be picked up by a microphone array. The sound pickup device can be a tablet computer or an intelligent hardware device, such as an intelligent speaker, a TV, and an air conditioner. Usually, 100 hours of speech data are collected in each field (such as education, medical care, games, etc.). The source of these voices can be unrequired. These voice data can cover a variety of application scenarios (such as shopping, home, etc.), and the collected voice data are used as sample voices. At the same time, it is also necessary to manually or machine-annotate the text recognition results corresponding to these sample voices as the text annotation results corresponding to the sample voices, and it is also possible to randomly select fragments from these text annotation results as sample hot word texts, or it is also possible to select some proper nouns from the fields to which these text annotation results belong as sample hot word texts, etc.

[0107] Step A2: Extracting sample acoustic features of the sample speech.

[0108] In this embodiment, in order to build a speech recognition model, after obtaining a large number of sample speech through step A1, a method similar to the method of extracting the target acoustic features of the target speech in step S302 can be further adopted to replace the target speech pair with the sample speech, and the sample acoustic features (such as FilterBank features) of each sample speech can be extracted. For relevant details, please refer to the introduction of step S302, which will not be repeated here to execute the subsequent step A3.

[0109] For example, for a certain sample speech, the text annotation result corresponding to the sample speech can be expressed as Y=[y 0 ,y 1 ,……,y t ,……,y T ], the extracted acoustic features of the sample speech can be expressed as X = [x 1 ,x 2 ,……,x k ,……,x K ].

[0110] Among them, x k represents the k-th frame speech feature vector in the sample acoustic feature X. This application takes the 40-dimensional FankBank feature as an example, and K represents the total number of speech frames; y t represents the tth character in the text annotation result Y, T+1 is the total number of characters in the total text annotation result, where y 0Refers to the sentence starter, which can be expressed as " <start>”,y 1 Refers to the language identifier, which can be represented as " <language>”,y 2 Refers to the identification task identifier, which can be expressed as " <transribe>”,y 3 Refers to the identification subtask identifier, which can be expressed as "<no timestamps> ”,y T Refers to the sentence terminator, which can be represented as " <eot>". Assuming that the text annotation result corresponding to the sample speech is "Welcome to iFLYTEK", the text annotation result is Y = [ <start> , <chinese> , <transribe>,<no timestamps> Welcome to Kexunfei <eot>].

[0111] Step A3: Use the sample acoustic features and text annotation results, combined with the sample hot word text, to train the initial speech recognition model to obtain a speech recognition model.

[0112] It should be noted that, in the first stage of training, the initial speech recognition model used in this application may include but is not limited to an audio encoder, an initial bias encoder, and an initial decoder. Figure 4 As shown in the figure, the initial decoder includes the initial hot word mixture expert adapter (MOEAdapter) and the attention mechanism network to improve the decoding accuracy.

[0113] Specifically, in this embodiment, the sample acoustic features (X=[x 1 ,x 2 ,……,x k ,……,x K ]), further, first, the sample acoustic features can be input into the audio encoder (Audio Encoder) of the initial speech recognition model for surface, to obtain the encoded feature vector of the sample speech, and still represent it as [a 1 ,a 2 ,……,a k ,……a K ] are for illustrative purposes only and are not intended to be limiting.

[0114] Then, the encoded feature vector of the sample speech (such as [a 1 ,a 2 ,……,a k ,……a K ]) and the feature vector of the hot word output by the initial hot word encoder (the feature vector obtained by encoding the sample hot word text, or the feature vector obtained by encoding other preset hot words, the specific content is not limited, and can be set according to the actual situation and experience value) are fused to obtain the sample fusion feature vector. In order to facilitate the explanation of the model training process, the feature vector of the hot word output by the initial hot word encoder can still be expressed as [b 1 ,b 2 ,……,b m ,……b M ], and the sample fusion feature vector is represented as C = [a 1 ,a 2 ,……,a k ,……a K ,z,b 1 ,b 2 ,……,b m ,……b M ] (hereinafter referred to as C) is only an example and is not limiting.

[0115] Next, the sample fusion feature vector can be input into the initial decoder (Decoder) of the initial speech recognition model to be decoded through the initial hot word mixture expert adapter (MOE Adapter) and attention mechanism network contained therein to obtain the speech recognition result containing the hot words of the sample speech.

[0116] Specifically, if Figure 4 As shown, after the sample fusion feature vector (C) is input into the initial decoder (Decoder) of the initial speech recognition model, since the initial decoder (Decoder) of the initial speech recognition model includes a standard attention mechanism decoding network, it requires three sets of vectors Q, K, and V to perform attention mechanism calculations, wherein Q, K, and V are used to project the input sample fusion feature vector (C) through projection transformation matrices Wq, Wk, and Wv, Q = Wq*C, K = Wk*C, and V = Wv*C. In addition, this embodiment uses an initial hot word mixed expert adapter (MOE Adapter) to fine-tune Q, K, and V, whose input is still the sample fusion feature vector (C), and the output is the transformed fine-tuning vector Adapter (C). Accordingly, Q, K, and V can be adjusted to be expressed as: Q = Wq*C+Adapter(C), K = Wk*C+Adapter(C), and V = Wv*C+Adapter(C). Each MOE Adapter used in this embodiment includes multiple adapters. The network structure of the multiple adapters included in each MOE Adapter is as follows: Figure 4 As shown in the upper right figure, each Adapter consists of a down-project layer, a nonlinear layer, and an up-project layer. Its input is the sample fusion feature vector (C), and its output is the fine-tuning vector Adapter (C). After obtaining Adapter1 (C)...AdapterN (C), the MOE Adapter sends it to the gated network (Gated route). The gated network finally outputs only one Adapter (C) to merge with the main network and output the speech hot word recognition result corresponding to the sample speech.

[0117] It can be seen that the present application achieves the distinction between the encoded feature vector of the sample speech and the feature vector of the hot word by inserting a zero vector z identifier without changing the main network structure and decoding mechanism of the model, and uses the additionally introduced MOE Adapter to control the output of each expert network and the final information fusion, and then merges the final information with the main network, so as to obtain a more accurate speech hot word recognition result.

[0118] Finally, according to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder (Bias Encoder) and the initial hot word mixed expert adapter (MOE Adapter) are updated until the preset conditions are met (the specific content is not limited and can be set according to actual conditions and experience values, such as reaching the preset maximum number of iterations (such as 100 times)), then the update of the model parameters is stopped and the speech recognition model is trained.

[0119] On this basis, after the speech recognition model is generated according to the sample speech training, the generated speech recognition can be further verified by using the verification speech. The specific verification process may include the following steps B1-B5:

[0120] Step B1: Acquire verification speech; and determine the text annotation result corresponding to the verification speech.

[0121] In this embodiment, in order to verify the speech recognition model, it is first necessary to obtain verification voice data. For example, with the user's permission, 1,000 voice data from different users can be collected as verification voices, where verification voice refers to voice information that can be used to verify the speech recognition model. After manually marking the text recognition results corresponding to each verification voice, the subsequent step B2 can be continued.

[0122] Step B2: extracting the verification acoustic features of the verification speech; and inputting the verification acoustic features into the audio encoder of the speech recognition model to obtain the encoded feature vector of the verification speech.

[0123] After obtaining the verification speech through step B1, the verification acoustic features (such as FilterBank features) of the verification speech can be further extracted, and the verification acoustic features can be input into the audio encoder of the speech recognition model to obtain the encoding feature vector corresponding to the verification speech to execute the subsequent step B3.

[0124] Step B3: Fuse the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector.

[0125] After obtaining the encoded feature vector of the verification speech through step B2, the verification fusion feature vector and the feature vector of the hot word output by the hot word encoder can be further fused, such as splicing the two, and adding a zero vector between the two in the obtained spliced ​​vector to fuse them, to obtain a verification fusion feature vector for executing the subsequent step B4.

[0126] Step B4: Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech.

[0127] After obtaining the verification fusion feature vector through step B3, the verification fusion feature vector can be further input into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech, so as to execute the subsequent step B5.

[0128] Step B5: When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0129] After obtaining the speech hot word prediction result corresponding to the verification voice through step B4, if the speech hot word prediction result corresponding to the verification voice is inconsistent with the text annotation result corresponding to the verification voice (i.e., the manually annotated text recognition result), the verification voice can be used as a sample voice again, and the parameters of the speech recognition model (i.e., the parameters of Bias Encoder and MOEAdapter) are updated. In this way, by timely adjusting and updating the speech recognition model, it is helpful to improve the recognition precision and accuracy of the speech recognition model.

[0130] In summary, the present embodiment provides a speech recognition method, which first obtains a target speech to be recognized, and then extracts the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain the speech hot word recognition result of the target speech; wherein the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0131] It can be seen that since the present application is first based on an adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model to generate a speech recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when the speech recognition model is used to recognize speech hot words for the target speech, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the speech recognition effect for the target speech.

[0132] Second embodiment

[0133] This embodiment will introduce a speech recognition device. For related content, please refer to the above method embodiment.

[0134] See also Figure 5 , is a schematic diagram of the composition of a speech recognition device provided in this embodiment, the device 500 includes:

[0135] A first acquisition unit 501 is used to acquire a target speech to be recognized;

[0136] A first extraction unit 502, configured to extract target acoustic features of the target speech;

[0137] The recognition unit 503 is used to input the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0138] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word mixed expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0139] In an implementation of this embodiment, the identification unit 503 includes:

[0140] A first input subunit, configured to input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain a coded feature vector of the target speech;

[0141] An encoding subunit, used for independently encoding each preset hot word using a hot word encoder of the speech recognition model to obtain a set of feature vectors of the hot words;

[0142] A first fusion subunit is used to fuse the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0143] The second input subunit is used to input the target fusion feature vector into the decoder of the speech recognition model to decode it through the hot word mixed expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0144] In an implementation of this embodiment, the first fusion subunit is specifically used for:

[0145] The encoded feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained spliced ​​vector, a zero vector is added between the encoded feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0146] In an implementation of this embodiment, the device further includes:

[0147] The second acquisition unit is used to acquire a sample speech; and determine a text annotation result and a sample hot word text corresponding to the sample speech;

[0148] A second extraction unit, used to extract sample acoustic features of the sample speech;

[0149] The training unit is used to train the initial speech recognition model by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model.

[0150] In one implementation of this embodiment, the initial speech recognition model includes an audio encoder, an initial hot word encoder and an initial decoder; the decoder includes an initial hot word hybrid expert adapter and an attention mechanism network; the training unit includes:

[0151] A third input subunit, used for inputting the sample acoustic feature into the audio encoder of the initial speech recognition model to obtain a coding feature vector of the sample speech;

[0152] A second fusion subunit is used to fuse the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fusion feature vector;

[0153] A fourth input subunit is used to input the sample fusion feature vector into the initial decoder of the initial speech recognition model, so as to decode it through the initial hot word hybrid expert adapter and the attention mechanism network, and identify the speech hot word recognition result of the sample speech;

[0154] The training subunit is used to update the parameters of the initial hot word encoder and the initial hot word mixed expert adapter according to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, until the preset conditions are met, then stop updating the model parameters and train to obtain the speech recognition model.

[0155] In an implementation of this embodiment, the device further includes:

[0156] A third acquisition unit is used to acquire a verification voice; and determine a text annotation result corresponding to the verification voice;

[0157] A third extraction unit is used to extract the verification acoustic features of the verification speech; and input the verification acoustic features into the audio encoder of the speech recognition model to obtain the encoding feature vector of the verification speech;

[0158] A fusion unit, used for fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fusion feature vector;

[0159] An input unit, used to input the verification fusion feature vector into a decoder of a speech recognition model to obtain a speech hot word prediction result corresponding to the verification speech;

[0160] The updating unit is used to update the speech recognition model by reusing the verification speech as the sample speech when the speech hot word prediction result corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech.

[0161] In an implementation of this embodiment, the target acoustic feature of the target speech is at least one of a Mel-cephalometric coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0162] Furthermore, an embodiment of the present application also provides a speech recognition device, including: a processor, a memory, and a system bus;

[0163] The processor and the memory are connected via the system bus;

[0164] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned speech recognition method.

[0165] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0166] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0167] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0168] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0169] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / eot> < / transribe> < / chinese> < / start> < / eot> < / transribe> < / language> < / start>

Claims

1. A speech recognition method, characterized in that: include: Obtaining the target speech to be recognized; Extracting target acoustic features of the target speech; Inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; The speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; The speech recognition model is obtained by training an end-to-end speech recognition large model based on an adapter using sample speech data and its corresponding text annotation results and sample hot word texts.

2. The method according to claim 1, characterized in that The step of inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech includes: Inputting the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain a coded feature vector of the target speech; Using the hot word encoder of the speech recognition model to independently encode each preset hot word, to obtain a set of feature vectors of the hot words; Fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector; The target fusion feature vector is input into the decoder of the speech recognition model to be decoded through a hot word mixture expert adapter and an attention mechanism network to obtain a speech hot word recognition result of the target speech.

3. The method according to claim 2, characterized in that The step of fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector includes: The encoded feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained spliced ​​vector, a zero vector is added between the encoded feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

4. The method according to claim 1, characterized in that: The speech recognition model is constructed as follows: Acquire sample speech; and determine the text annotation result and sample hot word text corresponding to the sample speech; Extracting sample acoustic features of the sample speech; The sample acoustic features and text annotation results are used in combination with the sample hot word text to train the initial speech recognition model to obtain the speech recognition model.

5. The method according to claim 4, characterized in that The initial speech recognition model includes an audio encoder, an initial hot word encoder and an initial decoder; the decoder includes an initial hot word mixed expert adapter and an attention mechanism network; The method of using the sample acoustic features and text annotation results in combination with the sample hot word text to train the initial speech recognition model to obtain the speech recognition model includes: Inputting the sample acoustic features into an audio encoder of an initial speech recognition model to obtain a coded feature vector of the sample speech; Fusing the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector; Inputting the sample fusion feature vector into the initial decoder of the initial speech recognition model to decode it through the initial hot word mixed expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the sample speech; According to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder and the initial hot word mixed expert adapter are updated until the preset conditions are met, then the updating of the model parameters is stopped and the speech recognition model is trained to obtain the speech recognition model.

6. The method according to claim 4, characterized in that The method further comprises: Obtaining a verification voice; and determining a text annotation result corresponding to the verification voice; Extracting the verification acoustic features of the verification speech; and inputting the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech; Fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector; Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech; When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

7. The method according to any one of claims 1 to 6, characterized in that: The target acoustic feature of the target speech is at least one of a Mel-cephalometric coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

8. A speech recognition device, characterized in that: include: A first acquisition unit, used to acquire a target speech to be recognized; A first extraction unit, configured to extract target acoustic features of the target speech; A recognition unit, used for inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; The speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; The speech recognition model is obtained by training an end-to-end speech recognition large model based on an adapter using sample speech data and its corresponding text annotation results and sample hot word texts.

9. A speech recognition device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images