A speech recognition method, device, storage medium and equipment

Abstract

The present application discloses a speech recognition method, apparatus, storage medium and device, which comprises: first obtaining a target speech to be recognized and extracting its target acoustic features; then inputting the target acoustic features into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; wherein the speech recognition model comprises an audio encoder, a hot word encoder and a decoder; and the decoder comprises a hot word hybrid expert adapter and an attention mechanism network. Since the present application first trains an end-to-end speech recognition large model based on an adapter, using sample speech data and its corresponding text annotation results and sample hot word text to generate a speech recognition model, it not only improves the model training efficiency and reduces the training cost, but also effectively improves the recognition accuracy of the model. In this way, when the speech recognition model is used to perform speech hot word recognition on the target speech, the efficiency and accuracy of the recognition results can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech recognition method, apparatus, storage medium, and device. Background Art

[0002] With continuous breakthroughs in artificial intelligence (AI) technology and the increasing popularity of various smart devices, human-computer interaction (HCI) is becoming increasingly common in our daily lives. Voice interaction, as the next generation of HCI, can greatly facilitate our lives. Accurate speech recognition is crucial. Currently, the most commonly used speech recognition method is based on end-to-end networks. However, these networks lack customization capabilities and can only support general speech recognition. They are less effective at recognizing hot words in specific business scenarios, such as specific nouns or domain-specific terms.

[0003] Currently, there are two common methods for improving speech hotword recognition: one is hotword incentives, which incentivizes potential hotword candidates by increasing their scores during the decoding process. However, this method, commonly used in traditional speech recognition, has limited effectiveness in end-to-end speech recognition. The other is to modify the end-to-end network model structure to improve the probability scores of hotwords at the model level, such as the CLAS (Contextual Listen, Attend and Spell) framework. However, this method requires full parameter updates, which is time-consuming and resource-intensive for large speech models. It may also undermine the performance of existing baseline models and reduce model recognition effectiveness. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method, device, storage medium and equipment that can effectively improve the training efficiency of the model and the recognition effect of speech hot words.

[0005] The present invention provides a speech recognition method, including:

[0006] Obtain the target speech to be recognized;

[0007] extracting target acoustic features of the target speech;

[0008] Inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0009] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0010] In one possible implementation, inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hotword recognition result of the target speech includes:

[0011] Inputting the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain an encoded feature vector of the target speech;

[0012] Using the hot word encoder of the speech recognition model to independently encode each preset hot word to obtain a set of feature vectors of the hot words;

[0013] Fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0014] The target fusion feature vector is input into the decoder of the speech recognition model to be decoded through the hot word mixture expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0015] In a possible implementation, fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector includes:

[0016] The encoding feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained splicing vector, a zero vector is added between the encoding feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0017] In one possible implementation, the speech recognition model is constructed as follows:

[0018] Obtaining a sample speech; and determining a text annotation result and a sample hot word text corresponding to the sample speech;

[0019] Extracting sample acoustic features of the sample speech;

[0020] The sample acoustic features and text annotation results are used in combination with the sample hot word text to train the initial speech recognition model to obtain the speech recognition model.

[0021] In one possible implementation, the initial speech recognition model includes an audio encoder, an initial hotword encoder, and an initial decoder; the decoder includes an initial hotword hybrid expert adapter and an attention mechanism network; the initial speech recognition model is trained using the sample acoustic features and text annotation results in combination with the sample hotword text to obtain the speech recognition model, including:

[0022] Inputting the sample acoustic features into the audio encoder of the initial speech recognition model to obtain a coding feature vector of the sample speech;

[0023] Fusing the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector;

[0024] Inputting the sample fusion feature vector into the initial decoder of the initial speech recognition model for decoding through the initial hot word hybrid expert adapter and the attention mechanism network to obtain a speech hot word recognition result of the sample speech;

[0025] According to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder and the initial hot word mixed expert adapter are updated until the preset conditions are met, then the updating of the model parameters is stopped and the speech recognition model is trained to obtain the speech recognition model.

[0026] In a possible implementation, the method further includes:

[0027] Obtaining a verification voice; and determining a text annotation result corresponding to the verification voice;

[0028] Extracting verification acoustic features of the verification speech; and inputting the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech;

[0029] Fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector;

[0030] Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech;

[0031] When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0032] In a possible implementation, the target acoustic feature of the target speech is at least one of a Mel-cepstral coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0033] The present application also provides a speech recognition device, comprising:

[0034] A first acquisition unit, configured to acquire a target speech to be recognized;

[0035] a first extraction unit, configured to extract target acoustic features of the target speech;

[0036] A recognition unit, configured to input the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0037] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0038] In a possible implementation, the identification unit includes:

[0039] A first input subunit, configured to input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain an encoded feature vector of the target speech;

[0040] An encoding subunit, configured to independently encode each preset hot word using a hot word encoder of the speech recognition model to obtain a set of feature vectors of the hot words;

[0041] A first fusion subunit is configured to fuse the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0042] The second input subunit is used to input the target fusion feature vector into the decoder of the speech recognition model for decoding through the hot word mixture expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0043] In a possible implementation, the first fusion subunit is specifically configured to:

[0044] The encoding feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained splicing vector, a zero vector is added between the encoding feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0045] In a possible implementation, the apparatus further includes:

[0046] The second acquisition unit is used to acquire a sample speech; and determine the text annotation result and the sample hot word text corresponding to the sample speech;

[0047] a second extraction unit, configured to extract sample acoustic features of the sample speech;

[0048] The training unit is used to train the initial speech recognition model by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model.

[0049] In one possible implementation, the initial speech recognition model includes an audio encoder, an initial hotword encoder, and an initial decoder; the decoder includes an initial hotword hybrid expert adapter and an attention mechanism network; and the training unit includes:

[0050] A third input subunit is configured to input the sample acoustic features into an audio encoder of an initial speech recognition model to obtain a coding feature vector of the sample speech;

[0051] A second fusion subunit is configured to fuse the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector;

[0052] a fourth input subunit, configured to input the sample fusion feature vector into the initial decoder of the initial speech recognition model, so as to decode the sample speech through the initial hotword hybrid expert adapter and the attention mechanism network, and obtain a speech hotword recognition result of the sample speech;

[0053] The training subunit is used to update the parameters of the initial hot word encoder and the initial hot word mixed expert adapter according to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, until the preset conditions are met, then stop updating the model parameters and train to obtain the speech recognition model.

[0054] In a possible implementation, the apparatus further includes:

[0055] A third acquisition unit is configured to acquire a verification voice and determine a text annotation result corresponding to the verification voice;

[0056] a third extraction unit, configured to extract verification acoustic features of the verification speech; and input the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech;

[0057] A fusion unit, configured to fuse the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector;

[0058] An input unit, configured to input the verification fusion feature vector into a decoder of a speech recognition model to obtain a speech hot word prediction result corresponding to the verification speech;

[0059] An updating unit is used to update the speech recognition model by reusing the verification speech as the sample speech when the speech hot word prediction result corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech.

[0060] In a possible implementation, the target acoustic feature of the target speech is at least one of a Mel-cepstral coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0061] The embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0062] The processor and the memory are connected via the system bus;

[0063] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one implementation of the above-mentioned speech recognition method.

[0064] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0065] An embodiment of the present application further provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementations of the above-mentioned speech recognition method.

[0066] The embodiments of the present application provide a speech recognition method, apparatus, storage medium and device, which first obtain a target speech to be recognized and then extract the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain the speech hot word recognition results of the target speech; wherein, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, and uses sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0067] It can be seen that since this application is first based on the adapter, using sample voice data and its corresponding text annotation results and sample hot word text to train the end-to-end voice recognition large model to generate a voice recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when using the voice recognition model to perform voice hot word recognition on the target voice, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the voice recognition effect for the target voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 A schematic diagram of the Whisper speech recognition framework provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of the CLAS model framework provided in an embodiment of the present application;

[0071] Figure 3 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0072] Figure 4 An example diagram of the training process of the speech recognition model provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of the composition of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] Speech recognition is a common processing task in the field of artificial intelligence. For example, in scenarios such as human-computer interaction and speech-to-text, speech recognition is an indispensable processing task. Currently, the more commonly used speech recognition method is the speech recognition method based on end-to-end networks, which can meet the speech recognition needs in most common scenarios. However, for different fields and different users, different recognition results may be obtained for the same pronunciation based on the field and user personalization. However, the existing end-to-end network lacks customization capabilities, and its recognition results for certain hot words such as specific nouns or proper nouns in a certain field are not accurate enough. Therefore, how to improve the recognition accuracy of end-to-end speech recognition models for hot words in specific fields is a technical problem that needs to be solved urgently.

[0075] Moreover, the most popular deep learning sequence modeling is the end-to-end speech recognition large model based on the attention mechanism. This end-to-end modeling approach has become a research hotspot in sequence modeling tasks such as machine translation, handwriting recognition, and speech recognition. However, the end-to-end modeling method has been upgraded from the small parameter model of previous years to a large model with more than one billion parameters. Its model capacity is larger, the amount of data used is larger, and it supports more languages. Taking the Whisper speech recognition framework as an example, Figure 1 shown.

[0076] Most existing open-source speech models are trained on hundreds of thousands or even millions of hours of data. Traditional end-to-end speech recognition hotword solutions require full parameter updates, which is very costly for large speech models.

[0077] Currently, there are two common methods to improve speech hot word recognition: one is hot word incentive, which is to score incentives for potential hot word candidates during the decoding process. However, this method is commonly used in traditional speech recognition and has limited effect in end-to-end speech recognition. The other is to modify the end-to-end network model structure to improve the probability score of hot words at the model level, such as Figure 2 The CLAS model framework 200 shown, etc.

[0078] Among them, compared with Figure 1 The baseline model 100 in Figure 2 The CLAS model framework shown in Figure 1 adds a bias encoder, as shown in Figure 2. Figure 2 As shown in the lower middle part of the figure, the CLAS model first uses the BiasEncoder to represent each candidate hotword as a fixed-dimensional vector. It then uses the decoder's state information through an attention mechanism to select hotwords that match the input speech. These are output, and then, along with the output from the audio encoder, are fed into the decoder for decoding, resulting in relatively accurate recognition results. However, when training and updating the CLAS model, a full parameter update is required. This means that the parameters of the audio encoder, bias encoder, and decoder must be updated simultaneously to ensure that the three modules adapt to each other and achieve optimal model performance. For large speech models, the number of parameters increases from tens of megabytes (M) to hundreds of megabytes (M). This is a significant number, and a full parameter update can potentially undermine the performance of the existing baseline model and reduce model recognition effectiveness. Furthermore, the amount of speech data used by large speech models increases from tens of thousands of hours to tens or millions of hours, which increases the full parameter update cycle and consumes a lot of resources.

[0079] To address the above-mentioned defects, the present application provides a speech recognition method, which first obtains the target speech to be recognized, and then extracts the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain the speech hot word recognition results of the target speech; wherein, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, and uses sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0080] It can be seen that since this application is first based on the adapter, using sample voice data and its corresponding text annotation results and sample hot word text to train the end-to-end voice recognition large model to generate a voice recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when using the voice recognition model to perform voice hot word recognition on the target voice, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the voice recognition effect for the target voice.

[0081] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] First embodiment

[0083] See also Figure 3 , is a flow chart of a speech recognition method provided in this embodiment, the method comprising the following steps:

[0084] S301: Acquire target speech to be recognized.

[0085] In this embodiment, any speech recognized by this embodiment is defined as a target speech to be recognized. It should be noted that this embodiment does not limit the language type of the target speech. For example, the target speech can be a speech composed of Chinese or English. At the same time, this embodiment does not limit the length of the target speech. For example, the target speech can be a sentence or a paragraph.

[0086] It can be understood that the target voice can be obtained through recording or other means according to actual needs. For example, the telephone conversation voice in people's daily life or the conference recording can be used as the target voice. When the target voice is obtained, the target voice can be processed using the solution provided in this embodiment to identify the recognition results containing accurate hot words.

[0087] S302: Extracting target acoustic features of the target speech.

[0088] In this embodiment, after obtaining the target speech to be recognized through step S301, in order to accurately identify the recognition result containing accurate hot words corresponding to the target speech, the acoustic features of the target speech can be further extracted using existing or future feature extraction methods, and defined as target acoustic features as the basis for recognition, so as to achieve accurate recognition of the target speech through the subsequent step S303.

[0089] Specifically, when extracting the target acoustic features of the target speech, the target speech needs to be framed first to obtain the corresponding speech frame sequence, and then the framed speech frame sequence is pre-emphasized; and then the acoustic features of each speech frame are extracted in turn, where the acoustic features refer to feature data used to characterize the acoustic information of the corresponding speech frame, such as but not limited to Mel-scale Frequency Cepstral Coefficients (MFCC) features, Perceptual Linear Predictive (PLP) features or filter bank (FilterBank) features, etc., which constitute the target acoustic features of the target speech.

[0090] It should be noted that the embodiments of the present application do not limit the method for extracting the target acoustic features of the target speech, nor do they limit the specific extraction process. An appropriate extraction method can be selected according to the actual situation, and corresponding feature extraction operations can be performed. For ease of understanding, this embodiment will be described later using the target acoustic features of the target speech as filter bank features as an example.

[0091] S303: Input the target acoustic features of the target speech into a pre-built speech recognition model to obtain the speech hot word recognition result of the target speech; wherein, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train the end-to-end speech recognition large model.

[0092] In this embodiment, after obtaining the target acoustic features of the target speech through step S302, in order to effectively improve the accuracy of the recognition results of the target speech, the target acoustic features of the target speech can be further input into a pre-built speech recognition model to obtain the recognition results containing accurate hot words corresponding to the target speech.

[0093] It should be noted that in order to improve the accuracy of speech recognition results, this application pre-constructs a speech recognition model, such as Figure 4 As shown, the model includes an audio encoder (such as Figure 4 Audio Encoder in ), hot word encoder (such as Figure 4 Bias Encoder in) and decoder (such as Figure 4 The decoder includes a hot word mixture expert (mixture of expect) adapter (such as Figure 4 MOE Adaper in ) and attention mechanism network (such as Figure 4 The speech recognition model is based on an adapter, using sample speech data, its corresponding text annotation results, and sample hot word text to train an end-to-end speech recognition large model. During the training process, only the parameters of the hot word encoder (Bias Encoder) and the MOE Adaper need to be updated, without updating all model parameters. This can improve the training efficiency of the speech recognition model, reduce the investment in training resources, and lower training costs. In this way, by inputting the acoustic features of the target speech into the speech recognition model for recognition, it can achieve both efficiency and improve the recognition accuracy of the target speech, thereby improving the speech recognition effect of the target speech.

[0094] Specifically, an optional implementation method is that the specific implementation process of "inputting the target acoustic features of the target speech into the pre-built speech recognition model to obtain the speech hot word recognition result of the target speech" in step S303 may include the following steps S3031-S3034:

[0095] S3031: Input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain the encoded feature vector of the target speech.

[0096] In this implementation, after extracting the acoustic features of the target speech (such as FilterBank features) in step S302, Figure 4As shown, the FilterBank features of the target speech can be further input into the audio encoder of the speech recognition model to obtain the encoded feature vector of the target speech, which is expressed as [a1, a2, ..., a k ,……a K ] is used to execute the subsequent step S3033, which can be understood as the feature vector output by the last neural network layer of the audio encoder (Audio Encoder), and K is a positive integer greater than 0.

[0097] S3032: Use the hot word encoder of the speech recognition model to independently encode each preset hot word to obtain a set of feature vectors of the hot words.

[0098] In this implementation, in order to improve the accuracy of the speech recognition results, a bias encoder (Bias Encoder) of the speech recognition model can be used to independently encode each preset hot word (the specific content is not limited and can be set by field according to actual conditions and experience values) to obtain a set of hot word feature vectors, which are expressed as [b1, b2, ..., b m ,……b M ], to execute the subsequent step S3033, wherein b m Represents the feature vector of the mth hot word after being processed by the hot word encoder (BiasEncoder), where M is a positive integer greater than 0.

[0099] S3033: Fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector.

[0100] In this implementation, the encoding feature vector [a1, a2, ..., a k ,……a K ], and obtain a set of hot word feature vectors [b1, b2, ..., b m ,……b M ], we can further use the existing or future feature fusion methods to combine the target speech encoding feature vector [a1, a2, ..., a k ,……a K ] and the feature vectors of hot words [b1,b2,…,b m ,……b M ] are fused to obtain the target fusion feature vector for executing the subsequent step S3034.

[0101] Specifically, the encoding feature vector of the target speech [a1, a2, ..., a k ,……a K] and the feature vectors of hot words [b1,b2,…,b m ,……b M ] is spliced ​​according to the time dimension to obtain the splicing vector of K+M frames, and in the splicing vector, the coded feature vector [a1, a2, ..., a k ,……a K ] and the feature vectors of hot words [b1,b2,…,b m ,……b M ] to physically isolate the two in the data dimension by using the zero vector [z], and obtain the target fusion feature vector, which is defined as C = [a1, a2, ..., a k ,……a K ,z,b1,b2,……,b m ,……b M ] to execute the subsequent step S3034.

[0102] S3034: Input the target fusion feature vector into the decoder of the speech recognition model for decoding through the hot word mixture expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0103] In this implementation, after obtaining the target fusion feature vector in step S3033, Figure 4 As shown, the target fusion feature vector can be further input into the decoder of the speech recognition model to be used by the hot word mixture expert adapter (such as Figure 4 MOE Adaper in ) and attention mechanism network (such as Figure 4 The Attention in the target speech is used for coordinated decoding, and the text recognition result containing more accurate hot words corresponding to the target speech is output, thereby improving the recognition effect of speech hot words.

[0104] Next, this embodiment will introduce the process of constructing a speech recognition model. In one optional implementation, the process of constructing a speech recognition model may specifically include the following steps A1-A3:

[0105] Step A1: Obtain sample speech; and determine the text annotation results and sample hot word text corresponding to the sample speech.

[0106] In this embodiment, in order to build a speech recognition model, a lot of preparatory work needs to be done in advance. First, a large amount of speech data in various fields and scenarios needs to be collected. For example, the sound can be picked up by a microphone array. The sound pickup device can be a tablet computer or an intelligent hardware device such as an intelligent speaker, a TV and an air conditioner. Usually, 100 hours of speech data need to be collected in each field (such as education, medical care, games, etc.). The source of these voices can be free of charge. These voice data can cover a variety of application scenarios (such as shopping, home, etc.), and each piece of collected voice data is used as a sample voice. At the same time, the text recognition results corresponding to these sample voices need to be annotated manually or by machine as the text annotation results corresponding to the sample voices, and fragments can be randomly selected from these text annotation results as sample hot word texts, or some proper nouns can be selected from the fields to which these text annotation results belong as sample hot word texts, etc.

[0107] Step A2: Extracting sample acoustic features of the sample speech.

[0108] In this embodiment, in order to build a speech recognition model, after obtaining a large number of sample speech through step A1, a method similar to the method of extracting the target acoustic features of the target speech in step S302 can be further adopted to replace the target speech pair with the sample speech, and the sample acoustic features (such as FilterBank features) of each sample speech can be extracted. For relevant details, please refer to the introduction of step S302, which will not be repeated here to execute the subsequent step A3.

[0109] For example, for a certain sample speech, the text annotation result corresponding to the sample speech can be expressed as Y=[y0,y1,……,y t ,……,y T ], the extracted acoustic features of the sample speech can be expressed as X = [x1, x2, ..., x k ,……,x K ].

[0110] Among them, x k represents the k-th frame speech feature vector in the sample acoustic feature X. This application takes the 40-dimensional FankBank feature as an example, and K represents the total number of speech frames; t represents the tth character in the text annotation result Y, T+1 is the total number of characters in the total text annotation result, where y0 refers to the sentence start symbol, which can be expressed as " <start>", y1 refers to the language identifier, which can be expressed as" <language>", y2 refers to the identification task identifier, which can be expressed as" <transribe>", y3 refers to the identification subtask indicator, which can be expressed as"<no timestamps> ”,y T Refers to the sentence terminator, which can be represented as " <eot>". Assume that the text annotation result corresponding to the sample speech is "Welcome to iFLYTEK", then the text annotation result is Y = [ <start> , <chinese> , <transribe>,<no timestamps> Welcome to Kexunfei. <eot>].

[0111] Step A3: Use the sample acoustic features and text annotation results, combined with the sample hot word text, to train the initial speech recognition model to obtain a speech recognition model.

[0112] It should be noted that, in the first stage of training, the initial speech recognition model used in this application may include but is not limited to an audio encoder, an initial hot word encoder (Bias Encoder) and an initial decoder (Decoder), such as Figure 4 The initial decoder includes the initial hot word mixture expert adapter (MOEAdapter) and the attention mechanism network to improve decoding accuracy.

[0113] Specifically, in this embodiment, the sample acoustic features of the sample speech (X=[x1, x2, ……, x k ,……,x K ]), further, first, the sample acoustic features can be input into the audio encoder (Audio Encoder) of the initial speech recognition model for surface, to obtain the encoded feature vector of the sample speech, and still express it as [a1, a2, ..., a k ,……a K ] is for illustrative purposes only and is not intended to be limiting.

[0114] Then, the encoded feature vector of the sample speech (such as [a1, a2, ..., a k ,……a K ]) and the feature vector of the hot word output by the initial hot word encoder (the feature vector obtained by encoding the sample hot word text, or the feature vector obtained by encoding other preset hot words, the specific content is not limited and can be set according to the actual situation and experience value) are fused to obtain the sample fusion feature vector. In order to facilitate the explanation of the model training process, the feature vector of the hot word output by the initial hot word encoder can still be expressed as [b1, b2, ..., b m ,……b M ], and the sample fusion feature vector is represented as C = [a1, a2, ..., a k ,……a K ,z,b1,b2,……,b m ,……b M ] (hereinafter referred to as C) is only an example and is not limiting.

[0115] Next, the sample fusion feature vector can be input into the initial decoder of the initial speech recognition model to be decoded through the initial hot word mixture of experts adapter (MOE Adapter) and attention mechanism network contained therein to obtain the speech recognition result containing the hot words of the sample speech.

[0116] Specifically, if Figure 4 As shown, after the sample fusion feature vector (C) is input into the initial decoder (Decoder) of the initial speech recognition model, since the initial decoder (Decoder) of the initial speech recognition model includes a standard attention mechanism decoding network, it requires three sets of vectors Q, K, and V to perform attention mechanism calculations, where Q, K, and V are projected on the input sample fusion feature vector (C) through projection transformation matrices Wq, Wk, and Wv, Q = Wq*C, K = Wk*C, V = Wv*C, and this embodiment uses an initial hot word mixture expert adapter (MOE Adapter) to fine-tune Q, K, and V. Its input is still the sample fusion feature vector (C), and its output is the transformed fine-tuning vector Adapter(C). Accordingly, Q, K, and V can be adjusted to be expressed as: Q = Wq*C+Adapter(C), K = Wk*C+Adapter(C), and V = Wv*C+Adapter(C). Each MOE Adapter used in this embodiment includes multiple adapters. The network structure of the multiple adapters included in each MOE Adapter is as follows: Figure 4 As shown in the upper right figure, each Adapter consists of a down-project layer, a nonlinear layer, and an up-project layer. Its input is the sample fusion feature vector (C), and its output is the fine-tuning vector Adapter (C). After obtaining Adapter1 (C)...AdapterN (C), the MOE Adapter sends them to the gated network (Gated route). The gated network ultimately outputs only one Adapter (C), which is merged with the main network to output the speech hot word recognition result corresponding to the sample speech.

[0117] It can be seen that the present application achieves the distinction between the encoded feature vector of the sample speech and the feature vector of the hot word by inserting a zero vector z identifier without changing the main network structure and decoding mechanism of the model, and uses the additionally introduced MOE Adapter to control the output of each expert network and the final information fusion, and then merges the final information with the main network, so as to obtain more accurate speech hot word recognition results.

[0118] Finally, based on the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder (Bias Encoder) and the initial hot word mixed expert adapter (MOE Adapter) are updated until the preset conditions are met (the specific content is not limited and can be set according to actual conditions and experience values, such as reaching the preset maximum number of iterations (such as 100 times)). Then, the update of the model parameters is stopped and the speech recognition model is trained.

[0119] On this basis, after generating a speech recognition model based on sample speech training, the generated speech recognition can be further verified using verification speech. The specific verification process may include the following steps B1-B5:

[0120] Step B1: Obtain verification speech; and determine the text annotation result corresponding to the verification speech.

[0121] In this embodiment, in order to verify the speech recognition model, it is first necessary to obtain verification voice data. For example, with the user's permission, 1,000 voice data sent by different users can be collected as verification voices. The verification voice refers to voice information that can be used to verify the speech recognition model. After manually marking the text recognition results corresponding to each verification voice, the subsequent step B2 can be continued.

[0122] Step B2: extracting verification acoustic features of the verification speech; and inputting the verification acoustic features into the audio encoder of the speech recognition model to obtain a coding feature vector of the verification speech.

[0123] After obtaining the verification speech through step B1, the verification acoustic features of the verification speech (such as FilterBank features) can be further extracted, and the verification acoustic features can be input into the audio encoder of the speech recognition model to obtain the encoding feature vector corresponding to the verification speech for executing the subsequent step B3.

[0124] Step B3: Fuse the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector.

[0125] After obtaining the encoded feature vector of the verification speech through step B2, the verification fusion feature vector and the feature vector of the hot word output by the hot word encoder can be further fused, such as splicing the two and adding a zero vector between the two in the obtained spliced ​​vector to obtain a verification fusion feature vector for executing the subsequent step B4.

[0126] Step B4: Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech.

[0127] After obtaining the verification fusion feature vector through step B3, the verification fusion feature vector can be further input into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech, so as to execute the subsequent step B5.

[0128] Step B5: When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

[0129] After obtaining the speech hot word prediction result corresponding to the verification voice through step B4, if the speech hot word prediction result corresponding to the verification voice is inconsistent with the text annotation result corresponding to the verification voice (i.e., the manually annotated text recognition result), the verification voice can be used as a sample voice again to update the parameters of the speech recognition model (i.e., the parameters of Bias Encoder and MOEAdapter). In this way, by timely adjusting and updating the speech recognition model, it is helpful to improve the recognition precision and accuracy of the speech recognition model.

[0130] In summary, the present embodiment provides a speech recognition method, which first obtains the target speech to be recognized, and then extracts the target acoustic features of the target speech; then, the target acoustic features of the target speech can be input into a pre-built speech recognition model to obtain the speech hot word recognition results of the target speech; wherein, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, and uses sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0131] It can be seen that since this application is first based on the adapter, using sample voice data and its corresponding text annotation results and sample hot word text to train the end-to-end voice recognition large model to generate a voice recognition model, it not only improves the model training efficiency, reduces the investment in training resources, and reduces training costs, but also effectively improves the recognition accuracy of the model. In this way, when using the voice recognition model to perform voice hot word recognition on the target voice, the efficiency and accuracy of the recognition results can be effectively improved, thereby improving the voice recognition effect for the target voice.

[0132] Second embodiment

[0133] This embodiment will introduce a speech recognition device. For related content, please refer to the above method embodiment.

[0134] See also Figure 5 , is a schematic diagram of the composition of a speech recognition device provided in this embodiment, the device 500 includes:

[0135] A first acquisition unit 501 is used to acquire a target speech to be recognized;

[0136] A first extraction unit 502 is configured to extract target acoustic features of the target speech;

[0137] The recognition unit 503 is used to input the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech;

[0138] Among them, the speech recognition model includes an audio encoder, a hot word encoder and a decoder; the decoder includes a hot word hybrid expert adapter and an attention mechanism network; the speech recognition model is based on the adapter, using sample speech data and its corresponding text annotation results and sample hot word text to train an end-to-end speech recognition large model.

[0139] In one implementation of this embodiment, the identification unit 503 includes:

[0140] A first input subunit, configured to input the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain an encoded feature vector of the target speech;

[0141] An encoding subunit, configured to independently encode each preset hot word using a hot word encoder of the speech recognition model to obtain a set of feature vectors of the hot words;

[0142] A first fusion subunit is configured to fuse the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector;

[0143] The second input subunit is used to input the target fusion feature vector into the decoder of the speech recognition model for decoding through the hot word mixture expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

[0144] In one implementation of this embodiment, the first fusion subunit is specifically configured to:

[0145] The encoding feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained splicing vector, a zero vector is added between the encoding feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

[0146] In one implementation of this embodiment, the apparatus further includes:

[0147] The second acquisition unit is used to acquire a sample speech; and determine the text annotation result and the sample hot word text corresponding to the sample speech;

[0148] a second extraction unit, configured to extract sample acoustic features of the sample speech;

[0149] The training unit is used to train the initial speech recognition model by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model.

[0150] In one implementation of this embodiment, the initial speech recognition model includes an audio encoder, an initial hotword encoder, and an initial decoder; the decoder includes an initial hotword hybrid expert adapter and an attention mechanism network; and the training unit includes:

[0151] A third input subunit is configured to input the sample acoustic features into an audio encoder of an initial speech recognition model to obtain a coding feature vector of the sample speech;

[0152] A second fusion subunit is configured to fuse the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector;

[0153] a fourth input subunit, configured to input the sample fusion feature vector into the initial decoder of the initial speech recognition model, so as to decode the sample speech through the initial hotword hybrid expert adapter and the attention mechanism network, and obtain a speech hotword recognition result of the sample speech;

[0154] The training subunit is used to update the parameters of the initial hot word encoder and the initial hot word mixed expert adapter according to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, until the preset conditions are met, then stop updating the model parameters and train to obtain the speech recognition model.

[0155] In one implementation of this embodiment, the apparatus further includes:

[0156] A third acquisition unit is configured to acquire a verification voice and determine a text annotation result corresponding to the verification voice;

[0157] a third extraction unit, configured to extract verification acoustic features of the verification speech; and input the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech;

[0158] A fusion unit, configured to fuse the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector;

[0159] An input unit, configured to input the verification fusion feature vector into a decoder of a speech recognition model to obtain a speech hot word prediction result corresponding to the verification speech;

[0160] An updating unit is used to update the speech recognition model by reusing the verification speech as the sample speech when the speech hot word prediction result corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech.

[0161] In an implementation of this embodiment, the target acoustic feature of the target speech is at least one of a Mel-cepstral coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

[0162] Furthermore, an embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0163] The processor and the memory are connected via the system bus;

[0164] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one of the implementation methods of the above-mentioned speech recognition method.

[0165] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0166] Furthermore, an embodiment of the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0167] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0168] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0169] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0170] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.< / eot> < / transribe> < / chinese> < / start> < / eot> < / transribe> < / language> < / start>

Claims

1. A speech recognition method, characterized in that: include: Obtain the target speech to be recognized; extracting target acoustic features of the target speech; Inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; The speech recognition model includes an audio encoder, a hotword encoder, and a decoder; the decoder includes a hotword hybrid expert adapter and an attention mechanism network; the speech recognition model is trained on an end-to-end speech recognition large model based on the adapter, using sample speech data and its corresponding text annotation results and sample hotword text; The attention mechanism network uses three groups of vectors Q, K, and V to perform attention mechanism calculations; and the hot word mixing expert adapter is used to fine-tune the three groups of vectors Q, K, and V, with its input being the target fusion feature vector and its output being the transformed fine-tuning vector; wherein, each hot word mixing expert adapter contains multiple adapters, and the network structure of each adapter is composed of a dimensionality reduction projection layer, a nonlinear layer, and a dimensionality increase projection layer; the hot word mixing expert adapter is used to send the transformed fine-tuning vector output by each adapter to the gated network after obtaining it, so that the gated network finally outputs only one fine-tuning vector to be merged with the main network, and outputs the speech hot word recognition result corresponding to the target speech; wherein, the target fusion feature vector is obtained by fusing the encoded feature vector of the target speech and the feature vector of the hot word; the feature vector of the hot word is determined by independently encoding each preset hot word using the hot word encoder.

2. The method according to claim 1, characterized in that The step of inputting the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hotword recognition result of the target speech includes: Inputting the target acoustic features of the target speech into the audio encoder of the speech recognition model to obtain an encoded feature vector of the target speech; Using the hot word encoder of the speech recognition model to independently encode each preset hot word to obtain a set of feature vectors of the hot words; Fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector; The target fusion feature vector is input into the decoder of the speech recognition model to be decoded through the hot word mixture expert adapter and the attention mechanism network to obtain the speech hot word recognition result of the target speech.

3. The method according to claim 2, characterized in that The step of fusing the encoded feature vector of the target speech and the feature vector of the hot word to obtain a target fused feature vector includes: The encoding feature vector of the target speech and the feature vector of the hot word are spliced ​​according to the time dimension, and in the obtained splicing vector, a zero vector is added between the encoding feature vector of the target speech and the feature vector of the hot word for fusion to obtain a target fused feature vector.

4. The method according to claim 1, wherein The speech recognition model is constructed as follows: Obtaining a sample speech; and determining a text annotation result and a sample hot word text corresponding to the sample speech; Extracting sample acoustic features of the sample speech; The sample acoustic features and text annotation results are used in combination with the sample hot word text to train the initial speech recognition model to obtain the speech recognition model.

5. The method according to claim 4, characterized in that The initial speech recognition model includes an audio encoder, an initial hotword encoder, and an initial decoder; the decoder includes an initial hotword hybrid expert adapter and an attention mechanism network; The method of training the initial speech recognition model by using the sample acoustic features and text annotation results in combination with the sample hot word text to obtain the speech recognition model includes: Inputting the sample acoustic features into the audio encoder of the initial speech recognition model to obtain a coding feature vector of the sample speech; Fusing the encoded feature vector of the sample speech with the feature vector of the hot word output by the initial hot word encoder to obtain a sample fused feature vector; Inputting the sample fusion feature vector into the initial decoder of the initial speech recognition model for decoding through the initial hot word hybrid expert adapter and the attention mechanism network to obtain a speech hot word recognition result of the sample speech; According to the difference between the speech hot word recognition result of the sample speech and the text annotation result and the sample hot word text, the parameters of the initial hot word encoder and the initial hot word mixed expert adapter are updated until the preset conditions are met, then the updating of the model parameters is stopped and the speech recognition model is trained to obtain the speech recognition model.

6. The method according to claim 4, characterized in that The method further comprises: Obtaining a verification voice; and determining a text annotation result corresponding to the verification voice; Extracting verification acoustic features of the verification speech; and inputting the verification acoustic features into an audio encoder of a speech recognition model to obtain a coding feature vector of the verification speech; Fusing the encoded feature vector of the verification speech with the feature vector of the hot word output by the hot word encoder to obtain a verification fused feature vector; Input the verification fusion feature vector into the decoder of the speech recognition model to obtain the speech hot word prediction result corresponding to the verification speech; When the prediction result of the speech hot words corresponding to the verification speech is inconsistent with the text annotation result corresponding to the verification speech, the verification speech is used as the sample speech again to update the speech recognition model.

7. The method according to any one of claims 1 to 6, characterized in that The target acoustic feature of the target speech is at least one of a Mel-cepstral coefficient feature, a perceptual linear prediction feature, and a filter bank feature.

8. A speech recognition device, characterized in that: include: A first acquisition unit, configured to acquire a target speech to be recognized; a first extraction unit, configured to extract target acoustic features of the target speech; A recognition unit, configured to input the target acoustic features of the target speech into a pre-built speech recognition model to obtain a speech hot word recognition result of the target speech; The speech recognition model includes an audio encoder, a hotword encoder, and a decoder; the decoder includes a hotword hybrid expert adapter and an attention mechanism network; the speech recognition model is trained on an end-to-end speech recognition large model based on the adapter, using sample speech data and its corresponding text annotation results and sample hotword text; The attention mechanism network uses three groups of vectors Q, K, and V to perform attention mechanism calculations; and the hot word mixing expert adapter is used to fine-tune the three groups of vectors Q, K, and V, with its input being the target fusion feature vector and its output being the transformed fine-tuning vector; wherein, each hot word mixing expert adapter contains multiple adapters, and the network structure of each adapter is composed of a dimensionality reduction projection layer, a nonlinear layer, and a dimensionality increase projection layer; the hot word mixing expert adapter is used to send the transformed fine-tuning vector output by each adapter to the gated network after obtaining it, so that the gated network finally outputs only one fine-tuning vector to be merged with the main network, and outputs the speech hot word recognition result corresponding to the target speech; wherein, the target fusion feature vector is obtained by fusing the encoded feature vector of the target speech and the feature vector of the hot word; the feature vector of the hot word is determined by independently encoding each preset hot word using the hot word encoder.

9. A speech recognition device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is configured to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images