Speech recognition model training method, speech recognition method and equipment

Abstract

The invention relates to a speech recognition model training method, a speech recognition method and equipment, and the method comprises the steps: carrying out the processing of a first speech sample based on an initial speech recognition model, obtaining a first speech feature, carrying out the feature extraction of a keyword text, obtaining a first text feature, carrying out the aggregation of the first speech feature and the first text feature, and obtaining a speech recognition model; the method comprises the steps of obtaining a first aggregation feature, performing keyword prediction and voice text recognition based on the first aggregation feature to obtain a predicted keyword text and a predicted voice text of a first voice sample, determining model loss based on the predicted keyword text and the predicted voice text, and training an initial voice recognition model based on the model loss. And obtaining a target speech recognition model. By adopting the method, the recall capability of the keyword in the voice can be improved, and the accuracy of voice recognition is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a training method for a speech recognition model, a speech recognition method and a device. Background Art

[0002] With the development of artificial intelligence, speech recognition technology has been widely used in people's lives. Although the recognition effect of current speech recognition systems has been significantly improved, the recognition effect is still poor for some words, especially for some words that appear frequently in the required speech recognition scenarios but rarely appear in the training data. Summary of the invention

[0003] The present disclosure provides a speech recognition model training method, speech recognition method, device, medium, electronic device and program product to improve the ability to recall key words in speech, thereby improving the accuracy of speech recognition.

[0004] In a first aspect, the present disclosure provides a method for training a speech recognition model, comprising: Processing the first speech sample based on the initial speech recognition model to obtain a first speech feature; Extracting features from the keyword text to obtain a first text feature, wherein the keyword text is obtained by extracting keywords from a text label of the first voice sample and a text label of a second voice sample, wherein the first voice sample and the second voice sample are voice samples from different training batches in a voice sample set; Aggregating the first speech feature and the first text feature to obtain a first aggregated feature; Perform keyword prediction and speech-to-text recognition based on the first aggregated features to obtain predicted keyword text and predicted speech text of the first speech sample respectively; A model loss is determined based on the predicted keyword text and the predicted voice text, and the initial voice recognition model is trained based on the model loss to obtain a target voice recognition model.

[0005] In a second aspect, the present disclosure provides a speech recognition method, comprising: Determining a second speech feature of the speech to be recognized through a target speech recognition model; Determine a second text feature of the preset keyword text; Aggregating the second speech feature and the second text feature to obtain a second aggregated feature; Based on the second aggregated features, the predicted speech text of the speech to be recognized and / or the predicted keyword text of the speech to be recognized are obtained; wherein the target speech recognition model is trained according to the method described in the first aspect.

[0006] In a third aspect, the present disclosure provides a training device for a speech recognition model, comprising: A first processing module is configured to process the first speech sample based on the initial speech recognition model to obtain a first speech feature; A second processing module is configured to perform feature extraction on the keyword text to obtain a first text feature, wherein the keyword text is obtained by performing keyword extraction on the text label of the first voice sample and the text label of the second voice sample, and the first voice sample and the second voice sample are voice samples in different training batches in the voice sample set; an aggregation module, configured to aggregate the first speech feature and the first text feature to obtain a first aggregate feature; A prediction module is configured to perform keyword prediction based on the first aggregate feature to obtain a predicted keyword text of the first voice sample, and perform voice text recognition based on the first aggregate feature to obtain a predicted voice text of the first voice sample; The training module is configured to determine the model loss based on the predicted keyword text and the predicted voice text, and train the initial voice recognition model based on the model loss to obtain a target voice recognition model.

[0007] In a fourth aspect, the present disclosure provides a speech recognition device, comprising: A third processing module is configured to determine a second speech feature of the speech to be recognized through a target speech recognition model; A fourth processing module is configured to determine a second text feature of a preset keyword text; a fifth processing module, configured to aggregate the second speech feature and the second text feature to obtain a second aggregate feature; The sixth processing module is configured to obtain the predicted speech text of the speech to be recognized and / or the predicted keyword text of the speech to be recognized based on the second aggregated feature.

[0008] Wherein, the target speech recognition model is obtained according to the training method of the speech recognition model described in the first aspect.

[0009] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect or the second aspect.

[0010] In a sixth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method in the first aspect or the second aspect.

[0011] In a seventh aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect or the second aspect when executed by a processor.

[0012] Through the above technical solution, when the initial speech recognition model is trained, the model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss, so that the obtained target speech recognition model can establish a correspondence between the speech text and the keyword text, and improve the ability to focus on keywords in the speech. In addition, since it is based on the first speech feature obtained by the first speech sample, and the first text feature is obtained by extracting the keyword text from the text label of the first speech sample and the text label of the second speech sample, then the first speech feature and the first text feature are aggregated to obtain the first aggregated feature, and the predicted keyword text and the predicted speech text are obtained based on the first aggregated feature, and since the first speech sample and the first text feature are obtained by extracting the keyword text from the text label of the first speech sample and the text label of the second speech sample, the first speech feature and the first text feature are aggregated to obtain the first aggregated feature, and the predicted keyword text and the predicted speech text are obtained by processing based on the first aggregated feature, and since the first speech sample and the first The two voice samples are voice samples from different training batches in the voice sample set. In this way, when the initial voice recognition model is trained using the first voice sample, in addition to effectively paying attention to the keywords in the text labels of the sample data (the first voice sample) given in the parameter update process of this training batch, it is also possible to pay attention to the keywords in the text labels of the second voice sample other than the first voice sample. That is, it is possible to pay attention to the keywords in the text labels of the sample data (i.e., the sample data of other training batches) not given in the parameter update process of this training batch, thereby enriching the keywords and improving the target voice recognition model's ability to capture unknown keywords. Furthermore, the target voice recognition model trained by the above technical solution can improve the ability to recall keywords in the voice, thereby improving the accuracy of voice recognition.

[0013] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1is a schematic diagram of an application scenario of a speech recognition model according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram of a training method for a speech recognition model and an application environment of the speech recognition method according to an exemplary embodiment of the present disclosure; Figure 3 is a flowchart of a method for training a speech recognition model according to an exemplary embodiment of the present disclosure; Figure 4 is a structural schematic diagram of a speech recognition model according to an exemplary embodiment of the present disclosure; Figure 5 is a structural diagram of a basic model of speech recognition according to an exemplary embodiment of the present disclosure; Figure 6 is a flowchart of a method for training a speech recognition model according to an exemplary embodiment of the present disclosure; Figure 7 is a flowchart of a speech recognition method according to an exemplary embodiment of the present disclosure; Figure 8 is a structural block diagram of a training device for a speech recognition model according to an exemplary embodiment of the present disclosure; Fig. 9 is a structural block diagram of a speech recognition device according to an exemplary embodiment of the present disclosure; Fig.10 It is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0017] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] In the end-to-end speech recognition of related technologies, deep networks often have stronger generalization capabilities. However, the recognition target of end-to-end speech recognition is often based on characters. Unlike the hybrid model that uses phonemes as modeling units, the model that uses characters as modeling units is more dependent on training data, and the semantic information it carries is more likely to be inclined to the training set. When the application scenario cannot obtain speech training data, it is impossible to accurately recognize certain specific words in these scenarios, such as the new words in the e-commerce field "10 billion subsidies" and "one yuan purchase", etc. As a result, these words may be recognized as other words in the field to which the training data belongs, which leads to speech recognition errors.

[0023] In the embodiment of the present application, words that frequently appear in the required speech recognition scenarios but rarely appear in the training data and often appear in specific scenarios can be understood as keywords. In some implementations, keywords can also be called hot words.

[0024] In view of this, the embodiments of the present disclosure provide a training method for a speech recognition model, a speech recognition method, an apparatus, a medium, an electronic device and a program product. By establishing a correspondence between speech and keyword text during the training process, the sampled keyword information is obtained as much as possible, and the ability to focus on keyword information in a piece of speech data is improved, thereby achieving the effect of improving the keyword capture ability of the speech recognition model in unknown scenarios, avoiding keyword recognition errors, and further improving the accuracy of speech recognition.

[0025] The application scenarios of the solutions provided in the embodiments of the present application are described below.

[0026] The training method of the speech recognition model and the speech recognition method provided in the embodiments of the present disclosure can be applied to scenarios where speech recognition is required, and the method can be applied to products in these scenarios, for example, in anti-fraud outbound call systems, call quality inspection systems, etc.

[0027] The anti-fraud outbound call system is used to conduct anti-fraud inquiries and reminders to users based on their voice responses. For example, Figure 1 As shown, an anti-fraud outbound call system is provided, including a voice acquisition module 101, a voice recognition module 102, an intention understanding module 103, a text generation module 104 and a voice synthesis module 105. The voice acquisition module 101 acquires the user's real-time voice stream signal, inputs the voice stream signal into the voice recognition module 102, and after fast decoding, inputs the output result into the intention understanding module 103, and through the intention understanding judgment, the judgment logic is matched to the reply to be answered, and the text to be synthesized is generated by the text generation module 104, and then answered through the voice synthesis module 105 to complete a round of reply, and this is repeated.

[0028] The call quality inspection system is used to recognize the agent's voice and obtain the corresponding speech text. Then, the speech text can be classified, for example, into responsible text and non-responsible text, so as to evaluate the call quality and regulate the agent's behavior.

[0029] The following describes an application environment applicable to the embodiments of the present application.

[0030] See also Figure 2 , Figure 2 The following is a schematic diagram showing an application environment of a speech recognition model training method and a speech recognition method provided in an embodiment of the present application. As an implementation mode, the speech recognition model training method and the speech recognition method provided in an embodiment of the present application can be applied to the same electronic device. The electronic device can be, for example, Figure 2The server 210 shown in the figure can be connected to the terminal device 220 through a network. The network is used to provide a medium for a communication link between the server 210 and the terminal device 220. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which is not limited in the embodiment of the present application.

[0031] It should be understood that Figure 2 The server 210, network and terminal device 220 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 210 may be a physical server or a server cluster composed of multiple servers, and the terminal device 220 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 220 to access the server 210 at the same time.

[0032] As another implementation, the speech recognition model training method and speech recognition method provided in the embodiment of the present application can be applied to different electronic devices. For example, the speech recognition model training method is applied to a server, and the speech recognition method is applied to a terminal device. For another example, the speech recognition model training method is applied to a terminal device, and the speech recognition method is applied to a server. The embodiment of the present application does not limit the electronic devices to which the speech recognition model training method and the speech recognition method are applied.

[0033] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.

[0034] Figure 3 is a flowchart of a method for training a speech recognition model according to an exemplary embodiment of the present disclosure. The method can be applied to an electronic device, which can be a terminal device or a server. Figure 3 , the training method of the speech recognition model may include the following steps: S301: Processing a first speech sample based on an initial speech recognition model to obtain a first speech feature.

[0035] The initial language recognition model can be understood as a language recognition model to be trained. The feature corresponding to the first speech sample can be understood as a first speech feature.

[0036] In the embodiment of the present application, after the first voice sample is input into the initial voice recognition model, the initial voice recognition model can perform feature extraction on the voice data in the first voice sample to obtain the first voice feature of the first voice sample.

[0037] The speech data in the speech sample can be in the form of Mel-frequency cepstral coefficients (MFCCs) or filterbank features.

[0038] In the embodiment of the present application, when training the initial speech recognition model, a different first speech sample can be input into the initial speech recognition model each time, and the initial speech recognition model performs the same iterative training process for each first speech sample input, and after multiple iterations, the final trained target speech recognition model can be obtained. In the subsequent embodiments of the present application, the first speech sample of a certain time is mainly used as an example for explanation.

[0039] S302: Extract features from the keyword text to obtain a first text feature. The keyword text is obtained by extracting keywords from a text label of a first voice sample and a text label of a second voice sample. The first voice sample and the second voice sample are voice samples from different training batches in a voice sample set.

[0040] In the embodiment of the present application, when the initial speech recognition model is trained, it is also necessary to prepare a keyword text corresponding to the first speech sample. Similarly, the keyword text corresponding to the first speech sample can also be input into the initial speech recognition model, and the keyword text corresponding to the first speech sample can be extracted by the initial speech recognition model to obtain the corresponding feature, i.e., the first text feature.

[0041] In some implementations, the first voice sample may be a plurality or a plurality of groups of voice data. For example, the first voice sample may be a batch in a voice sample set.

[0042] Among them, batch, which can also be understood as batch or training batch, refers to the set of samples sent to the model for training in one iteration. It is the unit of data processed by the model simultaneously.

[0043] In some implementations, the second voice sample may be a plurality of or a plurality of groups of voice data. For example, the second voice sample may be a plurality of batches in a voice sample set.

[0044] In an embodiment of the present application, the training batch to which the second voice sample belongs is different from the training batch to which the first voice sample belongs. Therefore, when the initial speech recognition model is trained using the first voice sample, in addition to being able to effectively focus on the keywords in the text labels of the given sample data (first voice sample) in the parameter update process of this training batch, it is also possible to focus on the keywords in the text labels of the second voice sample other than the first voice sample, thereby enriching the keywords.

[0045] In some implementations, the speech sample set may include multiple batches of speech samples, any batch of speech samples may be regarded as the first speech samples, and the batches of speech samples in the speech sample set excluding the first speech samples may be regarded as the second speech samples.

[0046] In combination with the foregoing content, it can be known that in some embodiments, the first voice sample can be a batch. In this case, the initial voice recognition model can perform feature extraction on the multiple voice data included in the batch to obtain features corresponding to the multiple voice data included in the batch. Thus, the features corresponding to the multiple voice data constitute the first voice feature of the first voice sample.

[0047] In some embodiments, the keyword text corresponding to the first voice sample may include one or more keyword texts extracted from the text label of the first voice sample, and the one or more keyword texts can be understood as keyword positive samples, and may also include one or more keyword texts extracted from the text label of the second voice sample, and the one or more keyword texts can be understood as keyword negative samples.

[0048] In some embodiments, the keyword text may include M keywords. The initial speech recognition model may perform feature extraction on each keyword in the keyword text to obtain features corresponding to each keyword. By concatenating the features corresponding to each keyword, the first text feature of the keyword text may be obtained. That is, the feature corresponding to each keyword may be understood as a sub-feature of the first text feature.

[0049] In the embodiment of the present application, each voice data in the first voice sample corresponds to the same first text feature.

[0050] S303: Aggregate the first speech feature and the first text feature to obtain a first aggregated feature.

[0051] In the embodiment of the present application, after obtaining the first voice feature and the first text feature, the first voice feature and the first text feature may be aggregated to obtain a first aggregated feature.

[0052] By aggregating the first voice feature and the first text feature, a correspondence between the voice text and the keyword text can be established, thereby improving the ability to focus on keywords in the voice.

[0053] S304: Perform keyword prediction based on the first aggregated feature to obtain predicted keyword text of the first voice sample, and perform voice-to-text recognition based on the first aggregated feature to obtain predicted voice text of the first voice sample.

[0054] In an embodiment of the present application, after obtaining the first aggregate feature, the initial speech recognition model can perform keyword prediction based on the first aggregate feature to obtain predicted keyword text in the first speech sample. In addition, speech text recognition can also be performed based on the first aggregate feature to obtain predicted speech text.

[0055] The predicted speech text of the first speech sample refers to the text corresponding to the speech recognition result of the first speech sample, and the keyword text of the first speech sample refers to the text corresponding to the keyword in the predicted speech text of the first speech sample.

[0056] In some embodiments, in order to improve the accuracy of keyword prediction processing, keyword prediction based on the first aggregate feature can be based on the first aggregate feature and the first voice feature to perform keyword prediction to obtain predicted keyword text of the first voice sample, and voice-to-text recognition based on the first aggregate feature can be based on the first aggregate feature and the first voice feature to perform voice-to-text recognition to obtain predicted voice text of the first voice sample.

[0057] S305: Determine the model loss based on the predicted keyword text and the predicted voice text, and train the initial voice recognition model based on the model loss to obtain a target voice recognition model.

[0058] In an embodiment of the present application, after the predicted keyword text and the predicted voice text are determined, the model loss can be determined based on the predicted keyword text and the predicted voice text, and the initial speech recognition model can be trained based on the model loss to obtain the target speech recognition model.

[0059] In some implementations, speech samples from different training batches in the speech sample set may be used as first speech samples respectively, and the same training process as above may be performed to finally obtain a target speech recognition model.

[0060] Through the above technical solution, when the initial speech recognition model is trained, the model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss, so that the obtained target speech recognition model can establish a correspondence between the speech text and the keyword text, and improve the ability to focus on keywords in the speech. In addition, since it is based on the first speech feature obtained by the first speech sample, and the first text feature is obtained by extracting the keyword text from the text label of the first speech sample and the text label of the second speech sample, then the first speech feature and the first text feature are aggregated to obtain the first aggregated feature, and the predicted keyword text and the predicted speech text are obtained based on the first aggregated feature, and since the first speech sample and the first text feature are obtained by extracting the keyword text from the text label of the first speech sample and the text label of the second speech sample, the first speech feature and the first text feature are aggregated to obtain the first aggregated feature, and the predicted keyword text and the predicted speech text are obtained by processing based on the first aggregated feature, and since the first speech sample and the first The two voice samples are voice samples from different training batches in the voice sample set. In this way, when the initial voice recognition model is trained using the first voice sample, in addition to effectively paying attention to the keywords in the text labels of the sample data (the first voice sample) given in the parameter update process of this training batch, it is also possible to pay attention to the keywords in the text labels of the second voice sample other than the first voice sample. That is, it is possible to pay attention to the keywords in the text labels of the sample data (i.e., the sample data of other training batches) not given in the parameter update process of this training batch, thereby enriching the keywords and improving the target voice recognition model's ability to capture unknown keywords. Furthermore, the target voice recognition model trained by the above technical solution can improve the ability to recall keywords in the voice, thereby improving the accuracy of voice recognition.

[0061] In some implementations, in step S301, processing the first speech sample to obtain the first speech feature may include the following steps: Encoding the first speech sample to obtain a first intermediate feature, and encoding the first intermediate feature to obtain a second intermediate feature; Perform speech frame prediction based on the first intermediate feature to obtain a first speech frame in the first speech sample; Based on the sequence relationship of the first speech frame in the first speech sample, the context relationship of the second intermediate feature corresponding to the first speech frame is reconstructed to obtain the first speech feature of the first speech sample.

[0062] The first speech frame refers to a frame with speech activity in the first speech sample, and speech activity may refer to a speech frame containing information such as speech, human voice, speech text, etc. The speech data of the first speech sample may include speech frames with speech activity and non-speech frames without speech activity.

[0063] Exemplarily, assuming that the speech data of the first speech sample includes 10 frames, among which the frames with sound are the 2nd, 3rd, 4th, 6th, 7th, 8th, and 9th frames, then the first speech frames in the first speech sample are the 2nd, 3rd, 4th, 6th, 7th, 8th, and 9th frames, and the non-speech frames are the 1st, 5th, and 10th frames.

[0064] In some embodiments, Figure 4 As shown, the initial speech recognition model 400 may include a speech coding network 401, a speech frame prediction network 402, and a contextual relationship reconstruction network 407. In this case, the first speech sample can be encoded by the speech coding network 401 to obtain a first intermediate feature, and the first intermediate feature can be encoded by the speech coding network 401 to obtain a second intermediate feature. The speech frame prediction network 402 can also perform speech frame prediction based on the first intermediate feature to obtain a first speech frame in the first speech sample. The contextual relationship of the second intermediate feature of the first speech frame can also be reconstructed by the contextual relationship reconstruction network 407 based on the time sequence relationship of the first speech frame in the first speech sample to obtain a reconstructed feature. The reconstructed feature is the first speech feature of the first speech sample.

[0065] Among them, contextual relationship reconstruction refers to using the features of the previous and next frames to enhance the feature representation of the current frame. Using the above example, assuming that the first speech frame in the first speech sample is the 2nd, 3rd, 4th, 6th, 7th, 8th, and 9th frames, the second intermediate feature corresponding to the 3rd frame can be enhanced by the second intermediate feature corresponding to the 2nd frame and the second intermediate feature corresponding to the 4th frame.

[0066] In some embodiments, the speech coding network may include multiple stacked speech encoders. In this case, the features output by the speech encoder of the middle layer of the speech coding network can be used as the first intermediate features. At the same time, the features output by the speech encoder of the last layer of the speech coding network can also be used as the second intermediate features.

[0067] For example, continue as Figure 4 As shown, the speech coding network 401 may include 12 stacked speech encoders. Optionally, the features output by the fourth stacked speech encoder may be used as the first intermediate features, and the features output by the twelfth stacked speech encoder may be used as the second intermediate features.

[0068] In some implementations, the speech encoder may be a conformer (convolution-attention mechanism) module. Using the conformer module as a speech encoder enables the speech coding network to better model long-term global interaction information and local features, thereby improving the accuracy of feature expression.

[0069] In some implementations, with respect to the first intermediate feature, a speech frame prediction network may be used to perform speech frame prediction processing on the first intermediate feature of each frame in the first speech sample to determine the first speech frame in the first speech sample.

[0070] In some implementations, after determining the first speech frame in the first speech sample, the feature values ​​of the non-speech frames therein may be set to 0, thereby obtaining a reduced vector, that is, obtaining a second intermediate feature corresponding to the first speech frame.

[0071] In some embodiments, the speech frame prediction network may include a linear layer and an activation function layer. The linear layer is used to perform a linear transformation on the first intermediate feature, and the activation function layer includes an activation function, which is used to convert the linear transformation result into a probability distribution, thereby obtaining the probability that each frame in the first speech sample belongs to a speech frame, and then the first speech frame in the first speech sample can be obtained according to the probability.

[0072] Since the underlying pronunciation information of phonemes and the like is clearer in the underlying information of the speech coding network, the first speech frame in the first speech sample can be predicted by using the first intermediate features of each frame in the first speech sample output by the intermediate layer of the speech coding network, thereby improving the accuracy of selecting the first speech frame.

[0073] By setting up a speech frame prediction network, the output results of the last layer of the speech encoding network can be guided to tell the speech recognition model where the frames containing speech activities are, thereby making the speech recognition model's learning goals for keywords clearer and further improving the accuracy of subsequent speech recognition.

[0074] Although only frames with voice activity have actual voice information, if the second intermediate feature corresponding to the first voice frame is directly selected as the first voice feature, the contextual association relationship of the original voice data will be destroyed. Therefore, in an embodiment of the present application, the contextual relationship of the second intermediate feature corresponding to the first voice frame can be reconstructed based on the sequential relationship of the first voice frame in the first voice sample to obtain the first voice feature of the first voice sample.

[0075] In some implementations, a contextual relationship reconstruction network may be used to perform contextual relationship reconstruction on the second intermediate feature of the first speech frame to obtain the first speech feature, thereby improving the accuracy of the first speech feature.

[0076] Exemplarily, the second intermediate features corresponding to multiple first speech frames with a time sequence relationship can be input into a contextual relationship reconstruction network, and the contextual relationship reconstruction network outputs reconstructed features corresponding to each first speech frame. The reconstructed features can then be sequentially spliced ​​to obtain the first speech feature.

[0077] In combination with the foregoing content, it can be known that the first text feature may include text sub-features corresponding to each keyword in the keyword text. Therefore, in some embodiments, the first text feature may include M text sub-features, each text sub-feature corresponding to at least one keyword; in this case, the first voice feature and the first text feature are aggregated to obtain a first aggregated feature, including: Determining a correlation degree between the first speech feature and the M text sub-features; Assign weights to the M text sub-features according to the degree of association; Based on the weights of the M text sub-features, the M text sub-features are weighted aggregated to obtain a first aggregated feature.

[0078] In the embodiment of the present application, in order to extract the corresponding relationship between the voice data and the keywords, it is possible to consider using M text sub-features to represent the first voice feature and perform subsequent text recognition processing. In addition, based on the degree of association between the first voice feature and the M text sub-features, a weight is assigned to each text sub-feature, and based on the weights of the M text sub-features, the M text sub-features are weighted aggregated to obtain a first aggregated feature, which can improve the accuracy of the first aggregated feature.

[0079] In combination with the foregoing content, it can be known that in some embodiments, the first speech feature is a feature obtained after the context relationship is reconstructed. In this case, the first speech feature and the first text feature are aggregated to obtain a first aggregated feature. The degree of association between the first speech feature obtained after the context relationship is reconstructed and the M text sub-features can be first determined, and then weights are assigned to the M text sub-features according to the degree of association. Then, based on the weights of the M text sub-features, the M text sub-features are weightedly aggregated to obtain the first aggregated feature.

[0080] In some embodiments, an attention mechanism may be used to perform the above-mentioned step of aggregating the first speech feature and the first text feature to obtain a first aggregate feature. In this case, the first speech feature may be input as a query (Query, Q), and the first text feature may be input as a key (Key, K) input and a value (Value, V). Based on the attention mechanism, the step of aggregating the first speech feature and the first text feature to obtain a first aggregate feature is performed.

[0081] That is, the attention mechanism can be used to determine the degree of correlation between the first speech feature and the M text sub-features, assign weights to the M text sub-features based on the degree of correlation, and perform weighted aggregation on the M text sub-features based on the weights of the M text sub-features to obtain the first aggregated feature.

[0082] In some embodiments, continuing as Figure 4As shown, the initial speech recognition model may also include a keyword core network 405, and the keyword core network 405 may include an attention mechanism module, and the query input of the attention mechanism module is a first speech feature, and the key input and the value input are both first text features. Thus, the keyword core network 405 can perform the step of aggregating the first speech feature and the first text feature based on the attention mechanism to obtain a first aggregated feature.

[0083] In some embodiments, continuing as Figure 4 As shown, the initial speech recognition model 400 may further include a speech decoding network 403. In this case, the speech decoding network 403 may perform speech text recognition based on the first aggregated features to obtain predicted speech text.

[0084] In some implementations, the speech decoding network may include multiple stacked speech decoders.

[0085] For example, continue as Figure 4 As shown, the speech decoding network 403 may include six stacked speech decoders.

[0086] In some implementations, the speech decoder may be a transformer decoder.

[0087] Continue as Figure 4 As shown, in some implementations, the first aggregate feature and the feature before context reconstruction may be weightedly summed, that is, the second intermediate feature of the first speech frame and the first aggregate feature may be weightedly summed to obtain the input feature of the speech decoder.

[0088] In some embodiments, continuing as Figure 4 As shown, the initial speech recognition model 400 may further include a keyword decoding network 406. In this case, the keyword decoding network 406 may perform keyword prediction based on the first aggregated feature to obtain a predicted keyword text in the first speech sample.

[0089] In some implementations, the keyword decoding network may include a plurality of stacked decoder modules. In addition, the keyword decoding network may also include a linear layer and an activation function layer, etc.

[0090] For example, continue as Figure 4 As shown, the keyword decoding network 406 may include two stacked decoder modules.

[0091] In some implementations, the decoder module included in the keyword decoding network may be a transformer decoder.

[0092] In some implementations, the speech recognition model may further include a keyword encoding network. In this case, in step S302, feature extraction is performed on the keyword text to obtain the first text feature, which may be performed by extracting the feature of the keyword text through the keyword encoding network.

[0093] Considering that there is a sequential position relationship between the multiple characters included in each keyword, a sequence processing network can be used for processing. Figure 4 As shown, the keyword encoding network 404 may include two stacked GRU networks, which perform vector encoding of the keyword text through the two stacked GRU (gated recurrent unit) networks to obtain the features corresponding to each keyword in the keyword text. For example, each keyword in the keyword text is encoded through the keyword encoding network to obtain the first text feature.

[0094] In some implementations, in step S305, determining the model loss based on the predicted keyword text and the predicted voice text, and training the initial voice recognition model based on the model loss to obtain the target voice recognition model may include the following steps: Determine a first loss value according to the predicted keyword text and the extracted keyword text; Determine a second loss value according to the predicted speech text and the text label of the first speech sample; The initial speech recognition model is trained according to the first loss value and the second loss value to obtain a target speech recognition model.

[0095] In the embodiment of the present application, for the same first voice sample, the corresponding loss value, i.e., the first loss value, can be calculated based on the difference between the corresponding predicted keyword text and the corresponding extracted keyword text through a loss function. In addition, the corresponding loss value, i.e., the second loss value, can also be calculated based on the difference between the corresponding predicted voice text and the corresponding text label through a loss function. Subsequently, back propagation can be performed based on the first loss value and the second loss value corresponding to the first voice sample to adjust the model parameters of the initial voice recognition model to complete the process of training the initial voice recognition model using the first voice sample.

[0096] Considering that the input of the keyword prediction process is a vector of indefinite length and the output is a keyword, in order to align the dimensions of the input and output, in some embodiments, the loss function used to determine the first loss value based on the predicted keyword text and the extracted keyword text can be a CTC (Connectionist Temporal Classification) loss function.

[0097] In some implementations, the loss function used to determine the second loss value based on the predicted speech text and the text label of the first speech sample may be a cross entropy loss function.

[0098] In some implementations, in step S305, training the initial speech recognition model based on the model loss to obtain the target speech recognition model may include the following steps: Keeping the first and second parameters pre-trained in the initial speech recognition model unchanged, and updating the third and fourth parameters in the initial speech recognition model based on the model loss; Among them, the first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

[0099] In an embodiment of the present application, the initial speech recognition model may include multiple parameters to be trained, for example, a first parameter for determining a first speech feature, a second parameter for determining a predicted speech text, a third parameter for determining a first text feature, and a fourth parameter for determining a predicted keyword text.

[0100] In the embodiment of the present application, taking into account that text recognition is the main task and keyword recall is the auxiliary task, therefore, in order to prevent the extracted keyword text from affecting the first parameter and the second parameter in the text recognition task to ensure the overall effect of text recognition, when the initial speech recognition model is trained based on the model loss, only the third parameter and the fourth parameter in the keyword recall task can be adjusted, that is, the first parameter and the second parameter pre-trained in the initial speech recognition model can be kept unchanged, and the third parameter and the fourth parameter in the initial speech recognition model can be updated based on the model loss. In this way, the ability to recall keywords in the speech can be improved while ensuring the overall speech recognition effect.

[0101] In combination with the foregoing, it can be known that in some embodiments, the model loss may include a first loss value and a second loss value. In this case, the first and second parameters pre-trained in the initial speech recognition model are kept unchanged, and the third and fourth parameters in the initial speech recognition model are updated based on the model loss. The first and second parameters pre-trained in the initial speech recognition model may be kept unchanged, and the third and fourth parameters in the initial speech recognition model are updated based on the first loss value and the second loss value.

[0102] Continue as Figure 4As shown, in some embodiments, when the initial speech recognition model 400 includes a speech encoding network 401, a speech frame prediction network 402, a speech decoding network 403, a keyword encoding network 404, a keyword core network 405, a keyword decoding network 406, and a contextual relationship reconstruction network 407, a first loss value can be determined based on the predicted keyword text and the extracted keyword text, and a second loss value can be determined based on the predicted speech text and the text label of the first speech sample. When the speech encoding network 401, the speech frame prediction network 402, and the speech decoding network 403 pre-trained in the initial speech recognition model are frozen, the keyword encoding network 404, the keyword core network 405, the keyword decoding network 406, and the contextual relationship reconstruction network 407 are jointly trained based on the first loss and the second loss.

[0103] Through the above embodiment, the third parameter and the fourth parameter in the initial speech recognition model are updated through the first speech sample and the corresponding keyword text, that is, the update process of the third parameter and the fourth parameter in the initial speech recognition model is completed. Figure 4 The keyword encoding network, the keyword core network, the keyword decoding network and the contextual relationship reconstruction network in the training process are as follows. In combination with the specific embodiment, the first parameter and the second parameter in the initial speech recognition model are updated, that is, Figure 4 The process of training the speech encoding network and speech decoding network in is explained.

[0104] In some embodiments, Figure 5 As shown, a basic speech recognition model consisting of a speech coding network, a speech frame prediction network and a speech decoding network can be constructed. When the parameters of the speech coding network, the speech frame prediction network and the speech decoding network are jointly trained, an additional speech sample set can be prepared. The additional speech sample set includes multiple third speech samples, each third speech sample includes speech data and a corresponding text label. The additional speech sample set is input into the basic speech recognition model, and the basic speech recognition model executes the same process for each third speech sample in the additional speech sample set.

[0105] In the embodiment of the present application, the process of the speech recognition basic model processing each third speech sample is the same. For each third speech sample, after it is input into the speech coding network, the third speech sample is processed by the speech coding network to obtain the third intermediate features of each frame in the third speech sample output by the middle layer of the speech coding network and the fourth intermediate features of each frame in the third speech sample output by the last layer of the speech coding network.

[0106] The third intermediate feature of each frame in the third speech sample can be processed by the speech frame prediction network to obtain the second speech frame in the third speech sample, and the fourth intermediate feature of each frame in the third speech sample can be processed by the speech decoding network to obtain the predicted speech text corresponding to the third speech sample. Subsequently, the third loss value can be determined according to the second speech frame in the third speech sample and the corresponding text label, the fourth loss value can be determined according to the predicted speech text corresponding to the third speech sample and the corresponding text label, the fifth loss value can be determined according to the fourth intermediate feature of each frame in the third speech sample and the text label corresponding to the third speech sample, and the parameters of the speech encoding network, the speech frame prediction network and the speech decoding network can be further adjusted according to the third loss value, the fourth loss value and the fifth loss value.

[0107] Optionally, the parameters of the speech encoding network, the speech frame prediction network and the speech decoding network are adjusted according to the third loss value, the fourth loss value and the fifth loss value. The parameters can be adjusted by back propagation based on the third loss value, the fourth loss value and the fifth loss value corresponding to each third speech sample in the same batch, and after multiple parameter adjustments, the trained speech recognition basic model is obtained.

[0108] In some implementations, the third loss value may be calculated based on a CTC loss function, and the fifth loss value may be calculated based on a CTC loss function.

[0109] In some embodiments, the initial speech recognition model may not include a speech frame prediction network. In this case, the parameter calculation process associated with the speech frame prediction network can be removed, that is, the parameters of the speech encoding network and the speech decoding network are adjusted directly according to the fourth loss value and the fifth loss value, thereby completing the update process of the first parameter and the second parameter in the initial speech recognition model.

[0110] Through the above embodiments, the structure and training process of the initial speech recognition model have been described. Next, in combination with specific embodiments, the process of extracting keywords from the text labels of the first speech sample and the second speech sample to obtain keyword text is explained.

[0111] In some implementations, the method of the embodiment of the present disclosure may further include the following steps: Extract X adjacent characters from the text label of the first voice sample to obtain a first keyword text; Extracting Y adjacent characters from the text label of the second voice sample to obtain a second keyword text; The keyword text is obtained based on a first preset number of first keyword texts and a second preset number of second keyword texts, wherein the sum of the first preset number and the second preset number is equal to the number of the first voice samples.

[0112] In the embodiment of the present application, in order to obtain the keyword text corresponding to the first voice sample, the keyword text may be extracted from the text label of the first voice sample, and the keyword text may be extracted from the text label of the second voice sample. The keyword text extracted from the text label of the first voice sample may be referred to as the first keyword text, and the keyword text extracted from the text label of the second voice sample may be referred to as the second keyword text.

[0113] In some embodiments, in order to improve the randomness of the keyword text, the starting position of keyword text extraction is not fixed. For example, the second word in the text label can be used as the extraction starting position for extraction, or the third word in the text label can be used as the extraction starting position for extraction.

[0114] In addition, in some embodiments, the length of the extracted text, that is, X and Y, may also be random. For example, three adjacent characters may be extracted from the text label of the first voice sample to obtain the first keyword text, or five adjacent characters may be extracted from the text label of the first voice sample to obtain the first keyword text.

[0115] In the embodiment of the present application, although the sum of the first preset number and the second preset number is equal to the number of the first voice samples, the first preset number may be random. For example, when the first voice sample includes 32 voice samples, 3 first keyword texts and 29 second keyword texts may be extracted to form the keyword text corresponding to the first voice sample, or 5 first keyword texts and 27 second keyword texts may be extracted to form the keyword text corresponding to the first voice sample.

[0116] It should be noted that the above process of extracting keyword texts can be performed before the first voice sample is input into the initial voice recognition model each time, or can be performed after the first voice sample is determined from the voice sample set.

[0117] Figure 6 is a flowchart of a method for training a speech recognition model according to an exemplary embodiment of the present disclosure. The method can be applied to an electronic device, which can be a terminal device or a server. Figure 6 , the training method of the speech recognition model may include the following steps: S601: Process any third speech sample in the speech sample set based on the speech coding network to obtain the third intermediate features of each frame in the third speech sample output by the middle layer of the speech coding network and the fourth intermediate features of each frame in the third speech sample output by the last layer of the speech coding network.

[0118] S602: Process the third intermediate feature based on the speech frame prediction network to obtain a second speech frame in the third speech sample.

[0119] S603: Process the fourth intermediate feature based on the speech decoding network to obtain a predicted speech text corresponding to the third speech sample.

[0120] S604: Determine a third loss value, a fourth loss value, and a fifth loss value, and adjust parameters of a speech encoding network, a speech frame prediction network, and a speech decoding network according to the third loss value, the fourth loss value, and the fifth loss value.

[0121] Among them, the third loss value is determined based on the second speech frame in the third speech sample and the corresponding text label, the fourth loss value is determined based on the predicted speech text corresponding to the third speech sample and the corresponding text label, and the fifth loss value is determined based on the fourth intermediate feature of each frame in the third speech sample and the corresponding text label of the third speech sample.

[0122] After the above steps S601-S604, the training of the speech coding network, speech frame prediction network and speech decoding network can be completed. Next, the speech coding network, speech frame prediction network and speech decoding network are loaded, and the initial speech recognition model is formed with the keyword coding network, keyword core network, keyword decoding network and context relationship reconstruction network. Subsequently, the parameters of the speech coding network, speech frame prediction network and speech decoding network are frozen, and the keyword coding network, keyword core network, keyword decoding network and context relationship reconstruction network are trained through steps S605-S612.

[0123] S605: Process the first speech sample based on the speech coding network to obtain a first intermediate feature output by an intermediate layer of the speech coding network and a second intermediate feature output by a final layer of the speech coding network.

[0124] S606: Perform speech frame prediction on the first intermediate feature through a speech frame prediction network to obtain a first speech frame in the first speech sample.

[0125] S607: Based on the sequence relationship of the first speech frame in the first speech sample, reconstruct the context relationship of the second intermediate feature corresponding to the first speech frame to obtain the first speech feature of the first speech sample.

[0126] S608: Determine a first text feature of the keyword text corresponding to the first voice sample through a keyword encoding network.

[0127] The keyword text corresponding to the first voice sample includes a first keyword text extracted from the text label of the first voice sample and a second keyword text extracted from the text label of the second voice sample.

[0128] S609: Using the keyword core network, taking the first voice feature as the query input and the first text feature as the key input and the value input, based on the attention mechanism, aggregate the first voice feature and the first text feature to obtain a first aggregated feature.

[0129] S610: Determine a predicted keyword text in the first voice sample according to the first aggregated feature through a keyword decoding network.

[0130] S611: Determine the predicted speech text according to the second intermediate feature and the first aggregate feature corresponding to the first speech frame through the speech decoding network.

[0131] S612: Determine a first loss value and a second loss value, and adjust parameters of a keyword encoding network, a keyword core network, a keyword decoding network, and a context relationship reconstruction network according to the first loss value and the second loss value.

[0132] Among them, the first loss value is determined based on the predicted keyword text and the extracted keyword text, and the second loss value is determined based on the predicted voice text and the text label of the first voice sample. Through the above steps S605-S612, the training of the keyword encoding network, the keyword core network, the keyword decoding network and the context relationship reconstruction network can be completed.

[0133] It should be noted that the embodiment of the present application does not limit the execution order of the above steps S601-S612. For example, step S608 can be executed simultaneously with any step in steps S605-S607, or before any step in steps S605-S607.

[0134] Among them, the steps not described in detail in the above steps S601-S612 can refer to the previous embodiments and will not be repeated here.

[0135] Next, combine Figure 4 as well as Figure 5 The model structure diagram shown in the figure uses a complete embodiment to explain in detail the training method of the speech recognition model: Data processing stage: The pure speech recognition results of 50,000 hours of general fields are indexed to obtain the text index corresponding to each speech. In addition, each speech is pre-emphasized, framed, windowed, discrete Fourier transformed, and Mel-filtered to obtain the 80-dimensional fbank feature corresponding to each speech. Among them, the fbank feature is a frequency domain feature.

[0136] For any common speech, the 80-dimensional fbank feature corresponding to the common speech is used as the speech data included in the speech sample, and the text index corresponding to the common speech is used as the text label included in the speech sample to construct a speech sample.

[0137] First training phase: The multiple speech samples obtained by the above process are used to form a speech sample set to Figure 5 The speech recognition basic model 500 shown is trained. Figure 5 In the present invention, a conformer is used as a speech encoding network (encoder) 401 of a speech recognition basic model 500, and a transformer is used as a speech decoding network (decoder) 403 of the speech recognition basic model 500. The features outputted from the intermediate layer of the speech encoding network 401 are connected to a speech frame prediction network 402 including a linear layer and an activation function layer.

[0138] In this training phase, the speech sample (the third speech sample) in the speech sample set is processed by the speech coding network 401, wherein the acoustic feature vector (the fourth intermediate feature of each frame) output by the last layer of the speech coding network 401 can be mapped to high-level semantic information. Then, the high-level semantic information output by the speech coding network 401 is input to the speech decoding network 403. After being processed by the speech decoding network 403, the predicted speech text can be obtained. In addition, the intermediate layers of the speech coding network 401 (for example Figure 4 The acoustic feature vector (the third intermediate feature of each frame) output by the fourth layer in the speech frame prediction network 402 passes through the linear layer and the activation function layer, and then the second speech frame can be predicted.

[0139] Subsequently, the cross entropy loss (fourth loss value) can be calculated based on the predicted speech text and the text label carried by the third speech sample, the CTC loss (fifth loss value) can be calculated based on the acoustic features output by the last layer of the speech coding network 401 (the fourth intermediate features of each frame) and the text label carried by the speech sample, and the CTC loss (third loss value) can be calculated based on the second speech frame output by the speech frame prediction network 402 and the text label carried by the speech sample. Based on the above three losses, the speech coding network 401, the speech decoding network 403 and the speech frame prediction network 402 are jointly trained.

[0140] Second training phase: The model architecture of the second training stage is as follows Figure 4 As shown, in addition to the speech recognition basic model 500 trained in the first training phase, a keyword encoding network 404, a keyword core network 405, a keyword decoding network 406, and a contextual relationship reconstruction network 407 may be added. In the second training phase, the network parameters of the speech encoding network 401, the speech decoding network 403, and the speech frame prediction network 402 are frozen, and the multiple speech samples obtained by the above processing are used to form the first speech samples in batches, and the keyword texts corresponding to each batch of the first speech samples are prepared to be extracted to train the speech recognition basic model 500. Figure 5 The keyword encoding network 404, keyword core network 405, keyword decoding network 406 and contextual relationship reconstruction network 407 shown are trained.

[0141] Figure 5 In the present invention, two stacked GRU networks are used as the keyword encoding network 404, the attention mechanism is used as the keyword core network 405, two layers of transformer, linear layer and activation function layer are used as the keyword decoding network 406 to perform keyword prediction, and TDNN (Time Delay Neural Network) is used as the contextual relationship reconstruction network 407.

[0142] In this training phase, on the one hand, any batch of first speech samples is processed through the speech coding network 401, wherein the acoustic feature vector (the second intermediate feature of each frame) output by the last layer of the speech coding network 401 can be mapped to high-level semantic information. In addition, the acoustic feature vector (the first intermediate feature of each frame) output by the middle layer of the speech coding network 401 can be predicted to obtain the first speech frame after passing through the linear layer and activation function layer of the speech frame prediction network 402. Then, the second intermediate feature corresponding to the first speech frame is selected from the second intermediate feature of each frame, and the contextual relationship reconstruction network 407 is used to reconstruct the contextual relationship of the selected second intermediate feature corresponding to the first speech frame to obtain the first speech feature of each speech sample in the first speech sample. On the other hand, the keyword text corresponding to the batch of first speech samples is processed through the keyword coding network 404 to obtain the first text feature.

[0143] The first speech feature is used as the query Q of the keyword core network 405, and the first text feature is used as the key K and value V of the keyword core network 405. The keyword core network 405 processes the first aggregated feature.

[0144] After obtaining the first aggregated feature, on the one hand, the second intermediate feature of the first speech frame and the first aggregated feature are weighted and summed, and the result is input into the speech decoding network 403, and speech text recognition is performed through the speech decoding network 403 to obtain the predicted speech text. Among them, the weight value can be set to 1.0 during training and set to an adjustable value during inference to indicate the degree of enhancement of the corresponding feature of the keyword. On the other hand, the first aggregated feature is input into the keyword decoding network 406, and the keyword is predicted by the keyword decoding network 406 to obtain the keyword, that is, the predicted keyword text.

[0145] Subsequently, the cross entropy loss (second loss value) can be calculated based on the predicted speech text and the text label carried by the first speech sample, and the CTC loss (first loss value) can be calculated based on the predicted keyword text and the extracted keyword text. Based on the above two losses, the keyword encoding network 404, the keyword core network 405, the keyword decoding network 406 and the context relationship reconstruction network 407 are jointly trained.

[0146] Finally, the trained speech encoding network 401, speech decoding network 403, speech frame prediction network 402, keyword encoding network 404, keyword core network 405, keyword decoding network 406 and contextual relationship reconstruction network 407 can be loaded to obtain a speech recognition network for speech recognition.

[0147] Based on the same technical concept, the embodiment of the present application also provides a speech recognition method. Figure 7 is a flowchart of a speech recognition method according to an exemplary embodiment of the present disclosure, the method can be applied to an electronic device, the electronic device can be a terminal device or a server, a trained speech recognition model is run in the electronic device, the trained speech recognition model is trained by the training method of the speech recognition model of the above embodiment, and reference is made to Figure 7 , the speech recognition method may include the following steps: S701: Determine a second speech feature of the speech to be recognized through a target speech recognition model.

[0148] The target speech recognition model is obtained by training using the training method of the aforementioned embodiment.

[0149] The speech to be recognized may be speech with speech recognition requirements obtained after feature extraction, for example, speech in a specific field or scenario as described in the above embodiments may be obtained after feature extraction.

[0150] The speech to be recognized can be an 80-dimensional fbank feature.

[0151] S702: Determine a second text feature of a preset keyword text.

[0152] The preset keyword text may include a plurality of pre-prepared keywords belonging to the same field as the speech to be recognized.

[0153] In some implementations, by inputting a preset keyword text into a target speech recognition model, the target speech recognition model may determine a second text feature of the preset keyword text.

[0154] That is, in the embodiment of the present application, when applying the trained target speech recognition model, the input of the model includes two parts, namely the speech to be recognized and the preset keyword text.

[0155] In some embodiments, the trained target speech recognition model determines the second text feature of the preset keyword text by first processing the preset keyword text when the model is loaded and started, obtaining and saving the second text feature of the preset keyword text, and subsequently, when processing different speech to be recognized, the second text feature of the preset keyword text can be determined by obtaining the saved second text feature.

[0156] S703: Aggregate the second speech feature and the second text feature to obtain a second aggregated feature.

[0157] S704: Based on the second aggregated feature, obtain predicted speech text of the speech to be recognized and / or predicted keyword text of the speech to be recognized.

[0158] In some embodiments, after obtaining the second speech feature and the second text feature, the second speech feature and the second text feature can be further aggregated through the target speech recognition model to obtain a second aggregated feature, and then based on the second aggregated feature, the predicted speech text of the speech to be recognized and / or the predicted keyword text of the speech to be recognized can be obtained.

[0159] It should be noted that in the embodiment of the present application, when applying the trained target speech recognition model, at least one of the network for predicting speech text or the network for predicting keyword text can be selected and loaded according to actual needs. When the network for predicting speech text and the network for predicting keyword text are loaded at the same time, the target speech recognition model can output the predicted speech text of the speech to be recognized as well as the predicted keyword text of the speech to be recognized.

[0160] It should be noted that during the application phase of the target speech recognition model, the execution process of each network in the target speech recognition model can refer to the data execution process during the training phase, which will not be repeated here.

[0161] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0162] Based on the same concept, the embodiment of the present disclosure also provides a training device for a speech recognition model, which can be part or all of an electronic device through software, hardware, or a combination of software and hardware. Figure 8 As shown, the training device 800 of the speech recognition model may include: A first processing module 801 is configured to process a first speech sample based on an initial speech recognition model to obtain a first speech feature; The second processing module 802 is configured to perform feature extraction on the keyword text to obtain a first text feature, wherein the keyword text is obtained by performing keyword extraction on the text label of the first voice sample and the text label of the second voice sample, wherein the first voice sample and the second voice sample are voice samples in different training batches in the voice sample set; An aggregation module 803 is configured to aggregate the first speech feature and the first text feature to obtain a first aggregate feature; A prediction module 804 is configured to perform keyword prediction based on the first aggregated feature to obtain a predicted keyword text of the first voice sample, and perform voice-to-text recognition based on the first aggregated feature to obtain a predicted voice text of the first voice sample; The training module 805 is configured to determine the model loss based on the predicted keyword text and the predicted voice text, and train the initial voice recognition model based on the model loss to obtain a target voice recognition model.

[0163] In a possible implementation, the first processing module 801 includes: an encoding submodule, configured to encode the first speech sample to obtain a first intermediate feature, and to encode the first intermediate feature to obtain a second intermediate feature; a speech frame prediction submodule, configured to perform speech frame prediction based on the first intermediate feature to obtain a first speech frame in the first speech sample; The reconstruction submodule is configured to reconstruct the context relationship of the second intermediate feature corresponding to the first speech frame based on the sequence relationship of the first speech frame in the first speech sample to obtain the first speech feature of the first speech sample.

[0164] In a possible implementation, the first text feature includes M text sub-features, each text sub-feature corresponds to at least one keyword; and the aggregation module includes: A correlation degree determination submodule, configured to determine a correlation degree between the first speech feature and the M text sub-features; A weight assignment submodule, configured to assign weights to the M text sub-features according to the association degree; The weighted aggregation submodule is configured to perform weighted aggregation on the M text sub-features based on the weights of the M text sub-features to obtain the first aggregated feature.

[0165] In a possible implementation, the training module 805 includes: The first training submodule is configured to determine a first loss value based on the predicted keyword text and the extracted keyword text; determine a second loss value based on the predicted speech text and the text label of the first speech sample; and train the initial speech recognition model based on the first loss value and the second loss value to obtain a target speech recognition model.

[0166] In a possible implementation, the training module 805 includes: A second training submodule is configured to keep the first and second parameters pre-trained in the initial speech recognition model unchanged, and update the third and fourth parameters in the initial speech recognition model based on the model loss; Among them, the first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

[0167] In a possible implementation, the speech recognition model training device 800 further includes: A first extraction module is configured to extract X adjacent characters from the text label of the first voice sample to obtain a first keyword text; A second extraction module is configured to extract Y adjacent characters from the text label of the second voice sample to obtain a second keyword text; The keyword text acquisition module is configured to obtain the keyword text based on a first preset number of the first keyword texts and a second preset number of the second keyword texts, wherein the sum of the first preset number and the second preset number is equal to the number of the first voice samples.

[0168] Regarding the speech recognition model training device 800 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0169] Based on the same concept, the embodiment of the present disclosure also provides a speech recognition device, which can be part or all of the electronic device through software, hardware, or a combination of software and hardware. Fig. 9 As shown, the speech recognition device 900 may include: The third processing module 901 is configured to determine the second speech feature of the speech to be recognized through the target speech recognition model; The fourth processing module 902 is configured to determine a second text feature of a preset keyword text; A fifth processing module 903 is configured to aggregate the second speech feature and the second text feature to obtain a second aggregate feature; The sixth processing module is configured to obtain the predicted speech text of the speech to be recognized and / or the predicted keyword text of the speech to be recognized based on the second aggregated feature.

[0170] Wherein, the trained target speech recognition model is obtained according to the aforementioned speech recognition model training method.

[0171] Regarding the speech recognition device 900 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0172] Based on the same concept, the embodiment of the present disclosure further provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the training method of any of the above-mentioned speech recognition models, or to implement the steps of the above-mentioned speech recognition method.

[0173] Fig.10 An electronic device (eg, Figure 2 A schematic diagram of the structure of a terminal device or server in FIG. 1000.

[0174] The electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0175] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0176] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of any of the above-mentioned speech recognition models are implemented, or the steps of the above-mentioned speech recognition method are implemented.

[0177] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the training method of any of the above-mentioned speech recognition models, or implements the steps of the above-mentioned speech recognition method.

[0178] It should be understood that the above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) to form a technical solution.

[0179] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0180] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A method for training a speech recognition model, characterized in that: include: Processing the first speech sample based on the initial speech recognition model to obtain a first speech feature; Extracting features from the keyword text to obtain a first text feature, wherein the keyword text is obtained by extracting keywords from a text label of the first voice sample and a text label of a second voice sample, wherein the first voice sample and the second voice sample are voice samples from different training batches in a voice sample set; Aggregating the first speech feature and the first text feature to obtain a first aggregated feature; Performing keyword prediction based on the first aggregated feature to obtain predicted keyword text of the first voice sample, and performing voice-to-text recognition based on the first aggregated feature to obtain predicted voice text of the first voice sample; A model loss is determined based on the predicted keyword text and the predicted voice text, and the initial voice recognition model is trained based on the model loss to obtain a target voice recognition model.

2. The method according to claim 1, characterized in that: The processing of the first speech sample to obtain the first speech feature includes: encoding the first speech sample to obtain a first intermediate feature, and encoding the first intermediate feature to obtain a second intermediate feature; Perform speech frame prediction based on the first intermediate feature to obtain a first speech frame in the first speech sample; Based on the sequence relationship of the first speech frame in the first speech sample, the context relationship of the second intermediate feature corresponding to the first speech frame is reconstructed to obtain the first speech feature of the first speech sample.

3. The method according to claim 1 or 2, characterized in that: The first text feature includes M text sub-features, each text sub-feature corresponds to at least one keyword; the first speech feature and the first text feature are aggregated to obtain a first aggregate feature, including: Determining a correlation degree between the first speech feature and the M text sub-features; According to the degree of association, weights are assigned to the M text sub-features; Based on the weights of the M text sub-features, the M text sub-features are weighted aggregated to obtain the first aggregated feature.

4. The method according to claim 1, characterized in that: The method of determining a model loss based on the predicted keyword text and the predicted voice text, and training the initial voice recognition model based on the model loss to obtain a target voice recognition model includes: Determine a first loss value according to the predicted keyword text and the extracted keyword text; Determine a second loss value according to the predicted speech text and the text label of the first speech sample; The initial speech recognition model is trained according to the first loss value and the second loss value to obtain a target speech recognition model.

5. The method according to claim 1 or 4, characterized in that: The step of training the initial speech recognition model based on the model loss to obtain a target speech recognition model includes: Keeping the first and second parameters pre-trained in the initial speech recognition model unchanged, and updating the third and fourth parameters in the initial speech recognition model based on the model loss; Among them, the first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first keyword text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

6. The method according to claim 1, characterized in that The method further comprises: Extracting X adjacent characters from the text label of the first voice sample to obtain a first keyword text; Extracting Y adjacent characters from the text label of the second voice sample to obtain a second keyword text; The keyword text is obtained based on a first preset number of the first keyword texts and a second preset number of the second keyword texts, wherein the sum of the first preset number and the second preset number is equal to the number of the first voice samples.

7. A speech recognition method, characterized in that: include: Determining a second speech feature of the speech to be recognized through a target speech recognition model; Determine a second text feature of the preset keyword text; Aggregating the second speech feature and the second text feature to obtain a second aggregated feature; Based on the second aggregated features, the predicted speech text of the speech to be recognized and / or the predicted keyword text of the speech to be recognized are obtained; wherein the target speech recognition model is trained according to the method described in any one of claims 1-6.

8. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images