Speech recognition model training method, speech recognition method, and device

Abstract

The present disclosure relates to a speech recognition model training method, a speech recognition method and equipment. The method comprises: processing a first speech sample based on an initial speech recognition model to obtain first speech features, performing feature extraction on keyword text to obtain first text features, aggregating the first speech features and the first text features to obtain first aggregated features, performing keyword prediction and speech text recognition based on the first aggregated features to obtain predicted keyword text and predicted speech text of the first speech sample, determining a model loss based on the predicted keyword text and the predicted speech text, and training the initial speech recognition model based on the model loss to obtain a target speech recognition model. The method can improve the recall ability of keywords in speech and improve the accuracy of speech recognition.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to a training method for a speech recognition model, a speech recognition method, and a device. Background Technology

[0002] With the development of artificial intelligence, speech recognition technology has been widely applied in people's lives. Although the recognition effect of current speech recognition systems has been significantly improved, the recognition effect is still not good for some words. This is especially true for words that frequently appear in the required speech recognition scenarios but are less common in the training data. Summary of the Invention

[0003] This disclosure provides a training method for a speech recognition model, a speech recognition method, an apparatus, a medium, an electronic device, and a program product to improve the recall capability of keywords in speech, thereby improving the accuracy of speech recognition.

[0004] Firstly, this disclosure provides a method for training a speech recognition model, including:

[0005] The first speech sample is processed based on the initial speech recognition model to obtain the first speech features;

[0006] Feature extraction is performed on the keyword text to obtain the first text feature. The keyword text is obtained by extracting keywords from the text labels of the first speech sample and the text labels of the second speech sample. The first speech sample and the second speech sample are speech samples from different training batches in the speech sample set.

[0007] The first speech feature and the first text feature are aggregated to obtain the first aggregated feature;

[0008] Based on the first aggregated features, keyword prediction and speech-text recognition are performed to obtain the predicted keyword text and predicted speech text of the first speech sample, respectively.

[0009] The model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss to obtain the target speech recognition model.

[0010] Secondly, this disclosure provides a speech recognition method, including:

[0011] The second speech features of the speech to be recognized are determined using the target speech recognition model.

[0012] Determine the second text features of the preset keyword text;

[0013] The second speech feature and the second text feature are aggregated to obtain the second aggregated feature;

[0014] Based on the second aggregation feature, the predicted speech text of the speech to be identified and / or the predicted keyword text of the speech to be identified are obtained; wherein, the target speech recognition model is trained according to the method described in the first aspect.

[0015] Thirdly, this disclosure provides a training apparatus for a speech recognition model, comprising:

[0016] The first processing module is configured to process the first speech sample based on the initial speech recognition model to obtain the first speech features;

[0017] The second processing module is configured to extract features from the keyword text to obtain the first text features. The keyword text is obtained by extracting keywords from the text labels of the first speech sample and the text labels of the second speech sample. The first speech sample and the second speech sample are speech samples from different training batches in the speech sample set.

[0018] The aggregation module is configured to aggregate the first speech feature and the first text feature to obtain a first aggregated feature;

[0019] The prediction module is configured to perform keyword prediction based on the first aggregation feature to obtain the predicted keyword text of the first speech sample, and to perform speech text recognition based on the first aggregation feature to obtain the predicted speech text of the first speech sample.

[0020] The training module is configured to determine the model loss based on the predicted keyword text and the predicted speech text, and to train the initial speech recognition model based on the model loss to obtain the target speech recognition model.

[0021] Fourthly, this disclosure provides a voice recognition device, comprising:

[0022] The third processing module is configured to determine the second speech features of the speech to be recognized through the target speech recognition model;

[0023] The fourth processing module is configured to determine the second text features of the preset keyword text;

[0024] The fifth processing module is configured to aggregate the second speech feature and the second text feature to obtain the second aggregated feature;

[0025] The sixth processing module is configured to obtain the predicted speech text and / or the predicted keyword text of the speech to be identified based on the second aggregation feature.

[0026] The target speech recognition model is obtained according to the training method of the speech recognition model described in the first aspect.

[0027] Fifthly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first or second aspect.

[0028] Sixthly, this disclosure provides an electronic device, comprising:

[0029] A storage device on which computer programs are stored;

[0030] A processing device for executing the computer program in the storage device to implement the steps of the method in the first or second aspect.

[0031] In a seventh aspect, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0032] Through the above technical solution, when training the initial speech recognition model, since the model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss, the resulting target speech recognition model can establish the correspondence between speech text and keyword text, improving the ability to focus on keywords in speech. Furthermore, since the first speech features are obtained from the first speech sample, and the first text features are obtained from the keyword text extracted from the text labels of the first speech sample and the text labels of the second speech sample, the first speech features and the first text features are then aggregated to obtain the first aggregated features. The predicted keyword text and predicted speech text are then processed based on the first aggregated features. And since the first speech sample and the first text features are... The second speech sample is a speech sample from different training batches in the speech sample set. This allows the initial speech recognition model to be trained using the first speech sample. In addition to effectively focusing on the keywords in the text labels of the sample data (the first speech sample) given in the parameter update process of this training batch, it can also focus on the keywords in the text labels of the second speech sample other than the first speech sample. That is, it can focus on the keywords in the text labels of the sample data not given in the parameter update process of this training batch (i.e., the sample data of other training batches), enriching the keywords and thus improving the target speech recognition model's ability to capture unknown keywords. As a result, the target speech recognition model trained through the above technical solution can improve its recall ability of keywords in speech, thereby improving the accuracy of speech recognition.

[0033] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0034] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0035] Figure 1 This is a schematic diagram illustrating an application scenario of a speech recognition model according to an exemplary embodiment of the present disclosure;

[0036] Figure 2 This is a schematic diagram illustrating a training method for a speech recognition model and an application environment of the speech recognition method according to an exemplary embodiment of this disclosure;

[0037] Figure 3 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment of the present disclosure;

[0038] Figure 4 This is a schematic diagram illustrating the structure of a speech recognition model according to an exemplary embodiment of the present disclosure;

[0039] Figure 5 This is a schematic diagram illustrating the structure of a basic speech recognition model according to an exemplary embodiment of the present disclosure;

[0040] Figure 6 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment of the present disclosure;

[0041] Figure 7 This is a flowchart illustrating a speech recognition method according to an exemplary embodiment of the present disclosure;

[0042] Figure 8 This is a structural block diagram of a speech recognition model training device according to an exemplary embodiment of the present disclosure;

[0043] Figure 9 This is a structural block diagram of a speech recognition device according to an exemplary embodiment of the present disclosure;

[0044] Figure 10 This is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0045] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0046] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0047] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0048] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0049] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0050] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0051] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0052] In end-to-end speech recognition technologies, deeper networks often possess stronger generalization capabilities. However, end-to-end speech recognition targets are typically at the character level, unlike hybrid models which use phonemes as modeling units. Character-based models rely more heavily on training data, and the semantic information they carry is more likely to be biased towards the training set. When training data is unavailable for the application scenario, it becomes impossible to accurately identify certain specific words within those scenarios. For example, new e-commerce terms like "billions in subsidies" or "one-yuan purchase" might be misidentified as words from other domains within the training data, leading to speech recognition errors.

[0053] In this embodiment, words that frequently appear in the required speech recognition scenarios but less frequently in the training data, often appearing only in specific contexts, can be understood as keywords. In some implementations, keywords may also be referred to as hot words.

[0054] In view of this, the present disclosure provides a training method for a speech recognition model, a speech recognition method, a device, a medium, an electronic device, and a program product. By establishing a correspondence between speech and keyword text during the training process, the sampled keyword information is obtained as much as possible, improving the ability to focus on keyword information in a speech data, thereby achieving the effect of improving the keyword capture capability of the speech recognition model in unknown scenarios, avoiding keyword recognition errors, and thus improving the accuracy of speech recognition.

[0055] The application scenarios of the solutions provided in the embodiments of this application will be described below.

[0056] The speech recognition model training method and speech recognition method provided in this disclosure can be applied to scenarios that require speech recognition, and thus the method can be applied to products in these scenarios, such as anti-fraud outbound call systems and call quality inspection systems.

[0057] Anti-fraud outbound calling systems are used to conduct anti-fraud inquiries and reminders to users based on their voice responses. For example, ... Figure 1 As shown, an anti-fraud outbound calling system is provided, including a voice acquisition module 101, a voice recognition module 102, an intent understanding module 103, a text generation module 104, and a voice synthesis module 105. The voice acquisition module 101 acquires the user's real-time voice stream signal, inputs the voice stream signal to the voice recognition module 102, and after fast decoding, inputs the output result to the intent understanding module 103. Through intent understanding judgment, the judgment logic is mapped to the response to be answered. The text generation module 104 generates the text to be synthesized, and then the voice synthesis module 105 responds to complete one round of response, and so on.

[0058] The call quality inspection system is used to recognize the voices of agents and obtain the corresponding script texts. Then, the script texts can be classified, for example, into responsible texts and non-responsible texts, in order to evaluate the call quality and thus standardize agent behavior.

[0059] The application environment applicable to the embodiments of this application is described below.

[0060] Please see Figure 2 , Figure 2 This illustration shows a training method for a speech recognition model and a schematic diagram of an application environment for a speech recognition method according to an embodiment of this application. As one implementation, the training method for the speech recognition model and the speech recognition method provided in this embodiment can be applied to the same electronic device. This electronic device can be, for example,... Figure 2 The server 210 shown can be connected to the terminal device 220 via a network. The network serves as a medium for providing a communication link between the server 210 and the terminal device 220. The network can include various connection types, such as wired communication links, wireless communication links, etc., and this embodiment does not limit this.

[0061] It should be understood that Figure 2 The server 210, network, and terminal device 220 shown are merely illustrative. Depending on the implementation requirements, any number of servers, networks, and terminal devices can be included. For example, server 210 can be a physical server or a server cluster consisting of multiple servers, and terminal device 220 can be a mobile phone, tablet, desktop computer, laptop computer, etc. It is understood that embodiments of this application can also allow multiple terminal devices 220 to access server 210 simultaneously.

[0062] As another implementation, the speech recognition model training method and speech recognition method provided in this application can be applied to different electronic devices. For example, the speech recognition model training method can be applied to a server, and the speech recognition method can be applied to a terminal device; or, for example, the speech recognition model training method can be applied to a terminal device, and the speech recognition method can be applied to a server. This application does not limit the electronic devices to which the speech recognition model training method and speech recognition method are applied.

[0063] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0064] Figure 3 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment of the present disclosure. This method can be applied to electronic devices, which may be terminal devices or servers. (Refer to...) Figure 3The training method for this speech recognition model may include the following steps:

[0065] S301: Process the first speech sample based on the initial speech recognition model to obtain the first speech features.

[0066] The initial language recognition model can be understood as the language recognition model to be trained. The features corresponding to the first speech sample can be understood as the first speech features.

[0067] In this embodiment of the application, after the first speech sample is input into the initial speech recognition model, the initial speech recognition model can extract features from the speech data in the first speech sample to obtain the first speech feature of the first speech sample.

[0068] Speech data in speech samples can be in the form of Mel-frequency cepstral coefficients (MFCCs) or filterbank features.

[0069] In this embodiment, when training the initial speech recognition model, different first speech samples can be input into the initial speech recognition model each time. The initial speech recognition model performs the same iterative training process for each input first speech sample. After multiple iterations, the final trained target speech recognition model can be obtained. In subsequent embodiments of this application, a specific first speech sample will be used as an example for illustration.

[0070] S302: Extract features from the keyword text to obtain the first text features. The keyword text is obtained by extracting keywords from the text labels of the first speech sample and the text labels of the second speech sample. The first speech sample and the second speech sample are speech samples from different training batches in the speech sample set.

[0071] In this embodiment of the application, when training the initial speech recognition model, it is also necessary to prepare the keyword text corresponding to the first speech sample. Similarly, the keyword text corresponding to the first speech sample can also be input into the initial speech recognition model, and the initial speech recognition model can extract features from the keyword text corresponding to the first speech sample to obtain the corresponding features, i.e., the first text features.

[0072] In some implementations, the first speech sample can be multiple or multiple sets of speech data. For example, the first speech sample can be a batch in a speech sample set.

[0073] In this context, "batch" can also be understood as a batch or training batch, referring to the set of samples fed into the model for training during one iteration. It is the data unit that the model processes simultaneously.

[0074] In some implementations, the second speech sample can be multiple or multiple sets of speech data. For example, the second speech sample can be multiple batches from a speech sample set.

[0075] In this embodiment, the training batch to which the second speech sample belongs is different from the training batch to which the first speech sample belongs. This allows the first speech sample to be used to train the initial speech recognition model. In addition to effectively focusing on the keywords in the text labels of the given sample data (first speech sample) during the parameter update process of this training batch, the keywords in the text labels of the second speech sample (other than the first speech sample) can also be focused on, thus enriching the keywords.

[0076] In some implementations, the speech sample set may include multiple batches of speech samples, and any one batch of speech samples may be regarded as the first speech sample, and the batches of speech samples in the speech sample set other than the first speech sample may be regarded as the second speech sample.

[0077] As can be seen from the foregoing, in some implementations, the first speech sample can be a batch. In this case, the initial speech recognition model can extract features from the multiple speech data included in the batch to obtain the features corresponding to the multiple speech data included in the batch. Thus, the features corresponding to the multiple speech data constitute the first speech feature of the first speech sample.

[0078] In some implementations, the keyword text corresponding to the first speech sample may include one or more keyword texts extracted from the text tags of the first speech sample, which can be understood as positive keyword samples. It may also include one or more keyword texts extracted from the text tags of the second speech sample, which can be understood as negative keyword samples.

[0079] In some implementations, the keyword text may include M keywords. The initial speech recognition model can extract features from each keyword in the keyword text to obtain the features corresponding to each keyword. By concatenating the features corresponding to each keyword, the first text feature of the keyword text can be obtained. That is, the feature corresponding to each keyword can be understood as a sub-feature of the first text feature.

[0080] In this embodiment of the application, each voice data in the first voice sample corresponds to the same first text feature.

[0081] S303: Aggregate the first speech feature and the first text feature to obtain the first aggregated feature.

[0082] In this embodiment of the application, after obtaining the first speech feature and the first text feature, the first speech feature and the first text feature can be aggregated to obtain the first aggregated feature.

[0083] By aggregating the first speech features and the first text features, a correspondence between speech text and keyword text can be established, thereby improving the ability to focus on keywords in speech.

[0084] S304: Based on the first aggregation feature, perform keyword prediction to obtain the predicted keyword text of the first speech sample, and based on the first aggregation feature, perform speech text recognition to obtain the predicted speech text of the first speech sample.

[0085] In this embodiment of the application, after obtaining the first aggregated feature, the initial speech recognition model can perform keyword prediction based on the first aggregated feature to obtain the predicted keyword text in the first speech sample. In addition, it can also perform speech text recognition based on the first aggregated feature to obtain the predicted speech text.

[0086] Wherein, the predicted speech text of the first speech sample refers to the text corresponding to the speech recognition result of the first speech sample, and the keyword text of the first speech sample refers to the text corresponding to the keywords in the predicted speech text of the first speech sample.

[0087] In some implementations, in order to improve the accuracy of keyword prediction processing, keyword prediction based on the first aggregation feature can be based on the first aggregation feature and the first speech feature to obtain the predicted keyword text of the first speech sample. Speech text recognition based on the first aggregation feature can be based on the first aggregation feature and the first speech feature to obtain the predicted speech text of the first speech sample.

[0088] S305: Determine the model loss based on the predicted keyword text and the predicted speech text, and train the initial speech recognition model based on the model loss to obtain the target speech recognition model.

[0089] In this embodiment of the application, after determining the predicted keyword text and the predicted speech text, the model loss can be determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model can be trained based on the model loss to obtain the target speech recognition model.

[0090] In some implementations, the target speech recognition model can be finally obtained by taking speech samples from different training batches in the speech sample set as the first speech samples and performing the same training process as described above.

[0091] Through the above technical solution, when training the initial speech recognition model, since the model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss, the resulting target speech recognition model can establish the correspondence between speech text and keyword text, improving the ability to focus on keywords in speech. Furthermore, since the first speech features are obtained from the first speech sample, and the first text features are obtained from the keyword text extracted from the text labels of the first speech sample and the text labels of the second speech sample, the first speech features and the first text features are then aggregated to obtain the first aggregated features. The predicted keyword text and predicted speech text are then processed based on the first aggregated features. And since the first speech sample and the first text features are... The second speech sample is a speech sample from different training batches in the speech sample set. This allows the initial speech recognition model to be trained using the first speech sample. In addition to effectively focusing on the keywords in the text labels of the sample data (the first speech sample) given in the parameter update process of this training batch, it can also focus on the keywords in the text labels of the second speech sample other than the first speech sample. That is, it can focus on the keywords in the text labels of the sample data not given in the parameter update process of this training batch (i.e., the sample data of other training batches), enriching the keywords and thus improving the target speech recognition model's ability to capture unknown keywords. As a result, the target speech recognition model trained through the above technical solution can improve its recall ability of keywords in speech, thereby improving the accuracy of speech recognition.

[0092] In some implementations, step S301, processing the first speech sample to obtain first speech features, may include the following steps:

[0093] The first speech sample is encoded to obtain a first intermediate feature, and the first intermediate feature is encoded to obtain a second intermediate feature;

[0094] Based on the first intermediate feature, speech frame prediction is performed to obtain the first speech frame in the first speech sample.

[0095] Based on the sequential relationship of the first speech frame in the first speech sample, the context relationship of the second intermediate feature corresponding to the first speech frame is reconstructed to obtain the first speech feature of the first speech sample.

[0096] The first speech frame refers to a frame in the first speech sample that contains speech activity. Speech activity can mean that the speech frame contains speaking sounds, human voices, or text-to-speech information. The speech data of the first speech sample may include speech frames with speech activity and non-speech frames without speech activity.

[0097] For example, suppose the speech data of the first speech sample includes 10 frames, of which the frames with sound are the 2nd, 3rd, 4th, 6th, 7th, 8th and 9th frames. Then the first speech frames in the first speech sample are the 2nd, 3rd, 4th, 6th, 7th, 8th and 9th frames, and the non-speech frames are the 1st, 5th and 10th frames.

[0098] In some implementations, such as Figure 4 As shown, the initial speech recognition model 400 may include a speech coding network 401, a speech frame prediction network 402, and a context relationship reconstruction network 407. In this case, the first speech sample can be encoded by the speech coding network 401 to obtain a first intermediate feature, and the first intermediate feature can be encoded by the speech coding network 401 to obtain a second intermediate feature. The speech frame prediction network 402 can also predict the first speech frame in the first speech sample based on the first intermediate feature. The context relationship reconstruction network 407 can also reconstruct the context relationship of the second intermediate feature of the first speech frame based on the temporal order relationship of the first speech frame in the first speech sample to obtain the reconstructed feature. The reconstructed feature is the first speech feature of the first speech sample.

[0099] Contextual relationship reconstruction refers to using features from preceding and following frames to enhance the feature representation of the current frame. Continuing with the previous example, assuming the first speech frames in the first speech sample are frames 2, 3, 4, 6, 7, 8, and 9, the second intermediate feature corresponding to frame 3 can be enhanced using the second intermediate feature corresponding to frame 2 and the second intermediate feature corresponding to frame 4.

[0100] In some implementations, a speech coding network may include multiple stacked speech encoders. In this case, the features output by the speech encoders in the middle layers of the speech coding network may be used as the first intermediate features, and the features output by the speech encoders in the last layer of the speech coding network may be used as the second intermediate features.

[0101] For example, continue as follows Figure 4 As shown, the speech coding network 401 may include 12 stacked speech encoders. Optionally, the features output by the fourth stacked speech encoder may be used as the first intermediate feature, and the features output by the twelfth stacked speech encoder may be used as the second intermediate feature.

[0102] In some implementations, the speech encoder can be a conformer (convolutional attention mechanism) module. Using a conformer module as a speech encoder allows the speech coding network to model long-term global interaction information and local features better, thereby improving the accuracy of feature representation.

[0103] In some implementations, for the first intermediate feature, a speech frame prediction network can be used to perform speech frame prediction processing on the first intermediate feature of each frame in the first speech sample to determine the first speech frame in the first speech sample.

[0104] In some implementations, after determining the first speech frame in the first speech sample, the feature values ​​of the non-speech frames therein can be set to 0, thereby obtaining a reduced vector, that is, obtaining the second intermediate feature corresponding to the first speech frame.

[0105] In some implementations, the speech frame prediction network may include a linear layer and an activation function layer. The linear layer performs a linear transformation on the first intermediate features, and the activation function layer includes an activation function to convert the linear transformation result into a probability distribution, thereby obtaining the probability that each frame in the first speech sample belongs to a speech frame. Then, the first speech frame in the first speech sample can be obtained based on the probability.

[0106] Since the underlying pronunciation information of phonemes is more explicit in the underlying information of the speech coding network, the accuracy of the first speech frame selection can be improved by using the first intermediate features of each frame in the first speech sample output by the intermediate layer of the speech coding network to predict the first speech frame in the first speech sample.

[0107] By setting up a speech frame prediction network, the output of the last layer of the speech coding network can be guided, telling the speech recognition model where the frames containing speech activity are. This makes the speech recognition model more specific in its learning objectives for keywords, thereby further improving the accuracy of subsequent speech recognition.

[0108] Although only frames with voice activity have actual voice information, directly selecting the second intermediate feature corresponding to the first voice frame as the first voice feature would destroy the original contextual relationship of the voice data. Therefore, in this embodiment, the contextual relationship of the second intermediate feature corresponding to the first voice frame can be reconstructed based on the order relationship of the first voice frame in the first voice sample to obtain the first voice feature of the first voice sample.

[0109] In some implementations, a context relation reconstruction network can be used to reconstruct the context relation of the second intermediate features of the first speech frame to obtain the first speech features, thereby improving the accuracy of the first speech features.

[0110] For example, the second intermediate features corresponding to multiple first speech frames with temporal relationships can be input into the context relationship reconstruction network to obtain the reconstructed features corresponding to each first speech frame output by the context relationship reconstruction network. The reconstructed features can then be sequentially concatenated to obtain the first speech features.

[0111] Based on the foregoing, the first text feature can include text sub-features corresponding to each keyword in the keyword text. Therefore, in some embodiments, the first text feature can include M text sub-features, each corresponding to at least one keyword. In this case, the first speech feature and the first text feature are aggregated to obtain the first aggregated feature, including:

[0112] Determine the degree of association between the first speech feature and the M text sub-features;

[0113] Based on the degree of correlation, assign weights to the M text sub-features;

[0114] Based on the weights of the M text sub-features, the M text sub-features are weighted and aggregated to obtain the first aggregated feature.

[0115] In this embodiment, to extract the correspondence between speech data and keywords, M text sub-features can be used to represent the first speech feature, and subsequent text recognition processing can be performed. Furthermore, based on the correlation between the first speech feature and the M text sub-features, a weight is assigned to each text sub-feature, and based on the weights of the M text sub-features, the M text sub-features are weighted and aggregated to obtain a first aggregated feature, which can improve the accuracy of the first aggregated feature.

[0116] As can be seen from the foregoing, in some implementations, the first speech feature is a feature obtained after contextual relationship reconstruction. In this case, the first speech feature and the first text feature are aggregated to obtain the first aggregated feature. This can be achieved by first determining the degree of association between the first speech feature obtained after contextual relationship reconstruction and M text sub-features, then assigning weights to the M text sub-features according to the degree of association, and then performing weighted aggregation on the M text sub-features based on their weights to obtain the first aggregated feature.

[0117] In some implementations, an attention mechanism can be used to perform the step of aggregating the first speech feature and the first text feature to obtain the first aggregated feature. In this case, the first speech feature can be used as a query (Q) input, and the first text feature can be used as a key (K) input and a value (V) input. Based on the attention mechanism, the step of aggregating the first speech feature and the first text feature to obtain the first aggregated feature is performed.

[0118] That is, the degree of association between the first speech feature and the M text sub-features can be determined through the attention mechanism, and weights can be assigned to the M text sub-features according to the degree of association. Based on the weights of the M text sub-features, the M text sub-features can be weighted and aggregated to obtain the first aggregated feature.

[0119] In some implementations, continue as Figure 4 As shown, the initial speech recognition model may further include a keyword core network 405. The keyword core network 405 may include an attention mechanism module, and the query input of the attention mechanism module is a first speech feature, while the key input and value input are both first text features. Thus, the keyword core network 405 can perform the step of aggregating the first speech feature and the first text feature based on the attention mechanism to obtain the first aggregated feature.

[0120] In some implementations, continue as Figure 4 As shown, the initial speech recognition model 400 may also include a speech decoding network 403. In this case, the speech decoding network 403 can perform speech-text recognition based on the first aggregated features to obtain the predicted speech-text.

[0121] In some implementations, the speech decoding network may include multiple stacked speech decoders.

[0122] For example, continue as follows Figure 4 As shown, the speech decoding network 403 may include six stacked speech decoders.

[0123] In some implementations, the speech decoder may be a transformer decoder.

[0124] Continue as Figure 4 As shown, in some implementations, the first aggregated feature can also be weighted and summed with the features before context reconstruction. That is, the second intermediate feature of the first speech frame can be weighted and summed with the first aggregated feature to obtain the input features of the speech decoder.

[0125] In some implementations, continue as Figure 4 As shown, the initial speech recognition model 400 may also include a keyword decoding network 406. In this case, the keyword decoding network 406 can perform keyword prediction based on the first aggregated features to obtain the predicted keyword text in the first speech sample.

[0126] In some implementations, the keyword decoding network may include multiple stacked decoder modules. In addition, the keyword decoding network may also include linear layers and activation function layers, etc.

[0127] For example, continue as follows Figure 4 As shown, the keyword decoding network 406 may include two stacked decoder modules.

[0128] In some implementations, the keyword decoding network may include a transformer decoder module.

[0129] In some implementations, the speech recognition model may also include a keyword encoding network. In this case, in step S302, feature extraction is performed on the keyword text to obtain first text features, which may specifically be feature extraction of the keyword text through a keyword encoding network.

[0130] Considering the sequential relationship between the multiple characters included in each keyword, a sequence processing network can be used for processing. Optionally, in some implementations, the process continues as follows: Figure 4 As shown, the keyword encoding network 404 may include two stacked GRU networks. The keyword text is vector-encoded by the two stacked GRU (gated recurrent unit) networks to obtain the features corresponding to each keyword in the keyword text. For example, the first text feature is obtained by encoding each keyword in the keyword text through the keyword encoding network.

[0131] In some implementations, step S305, determining the model loss based on the predicted keyword text and the predicted speech text, and training the initial speech recognition model based on the model loss to obtain the target speech recognition model, may include the following steps:

[0132] The first loss value is determined based on the predicted keyword text and the extracted keyword text;

[0133] The second loss value is determined based on the predicted speech text and the text labels of the first speech sample;

[0134] The initial speech recognition model is trained based on the first loss value and the second loss value to obtain the target speech recognition model.

[0135] In this embodiment, for the same first speech sample, a first loss value can be calculated using a loss function based on the difference between its corresponding predicted keyword text and the corresponding extracted keyword text. Furthermore, a second loss value can be calculated using a loss function based on the difference between its corresponding predicted speech text and the corresponding text label. Subsequently, backpropagation can be performed based on the first and second loss values ​​corresponding to the first speech sample to adjust the model parameters of the initial speech recognition model, thus completing the process of training the initial speech recognition model using the first speech sample.

[0136] Given that the input to keyword prediction processing is a vector of variable length and the output is keywords, in order to align the dimensions of the input and output, in some implementations, the loss function used to determine the first loss value based on the predicted keyword text and the extracted keyword text can be the CTC (Connectionist Temporal Classification) loss function.

[0137] In some implementations, the loss function used to determine the second loss value based on the predicted speech text and the text labels of the first speech sample can be the cross-entropy loss function.

[0138] In some implementations, step S305, training the initial speech recognition model based on the model loss to obtain the target speech recognition model, may include the following steps:

[0139] Keeping the first and second parameters of the initial speech recognition model unchanged, the third and fourth parameters of the initial speech recognition model are updated based on the model loss;

[0140] The first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

[0141] In this embodiment of the application, the initial speech recognition model may include multiple parameters to be trained, such as a first parameter for determining a first speech feature, a second parameter for determining the predicted speech text, a third parameter for determining a first text feature, and a fourth parameter for determining the predicted keyword text.

[0142] In this embodiment, considering that text recognition is the primary task and keyword recall is an auxiliary task, in order to prevent the extracted keyword text from affecting the first and second parameters in the text recognition task and to ensure the overall effect of text recognition, when training the initial speech recognition model based on the model loss, only the third and fourth parameters in the keyword recall task can be adjusted. That is, the first and second parameters in the pre-trained initial speech recognition model can be kept unchanged, and the third and fourth parameters in the initial speech recognition model can be updated based on the model loss. In this way, the recall ability of keywords in speech can be improved while ensuring the overall speech recognition effect.

[0143] As can be seen from the foregoing, in some implementations, the model loss may include a first loss value and a second loss value. In this case, keeping the first and second pre-trained parameters in the initial speech recognition model unchanged, and updating the third and fourth parameters in the initial speech recognition model based on the model loss, may involve keeping the first and second pre-trained parameters in the initial speech recognition model unchanged, and updating the third and fourth parameters in the initial speech recognition model based on the first loss value and the second loss value.

[0144] Continue as Figure 4 As shown, in some embodiments, when the initial speech recognition model 400 includes a speech coding network 401, a speech frame prediction network 402, a speech decoding network 403, a keyword coding network 404, a keyword core network 405, a keyword decoding network 406, and a context relationship reconstruction network 407, a first loss value can be determined based on the predicted keyword text and the extracted keyword text, and a second loss value can be determined based on the predicted speech text and the text labels of the first speech sample. When the pre-trained speech coding network 401, speech frame prediction network 402, and speech decoding network 403 in the initial speech recognition model are frozen, the keyword coding network 404, keyword core network 405, keyword decoding network 406, and context relationship reconstruction network 407 are jointly trained based on the first loss and the second loss.

[0145] Through the above embodiments, using the first speech sample and the corresponding keyword text, the update process of the third and fourth parameters in the initial speech recognition model was completed, that is, the update process of the initial speech recognition model was completed. Figure 4 The process of training the keyword encoding network, keyword core network, keyword decoding network, and context relationship reconstruction network in the initial speech recognition model is described below. Next, using a specific embodiment, we will explain the update process of the first and second parameters in the initial speech recognition model, that is, the process of updating the first and second parameters. Figure 4 The process of training the speech coding network and speech decoding network in this paper is explained.

[0146] In some implementations, such as Figure 5 As shown, a basic speech recognition model can be constructed, consisting of a speech coding network, a speech frame prediction network, and a speech decoding network. When jointly training the parameters of the speech coding network, the speech frame prediction network, and the speech decoding network, an additional speech sample set can be prepared. The additional speech sample set includes multiple third speech samples, each of which includes speech data and corresponding text labels. The additional speech sample set is input into the basic speech recognition model, and the basic speech recognition model performs the same process on each third speech sample in the additional speech sample set.

[0147] In this embodiment of the application, the basic speech recognition model processes each third speech sample in the same way. For each third speech sample, after it is input into the speech coding network, the speech coding network processes the third speech sample to obtain the third intermediate feature of each frame of the third speech sample output by the intermediate layer of the speech coding network, and the fourth intermediate feature of each frame of the third speech sample output by the final layer of the speech coding network.

[0148] For the third intermediate feature of each frame in the third speech sample, a speech frame prediction network can be used to process it to obtain the second speech frame in the third speech sample. For the fourth intermediate feature of each frame in the third speech sample, a speech decoding network can be used to process it to obtain the predicted speech text corresponding to the third speech sample. Subsequently, a third loss value can be determined based on the second speech frame and its corresponding text label in the third speech sample, a fourth loss value can be determined based on the predicted speech text and its corresponding text label in the third speech sample, and a fifth loss value can be determined based on the fourth intermediate feature of each frame in the third speech sample and its corresponding text label. Furthermore, the parameters of the speech coding network, the speech frame prediction network, and the speech decoding network can be adjusted based on the third, fourth, and fifth loss values.

[0149] Optionally, the parameters of the speech coding network, speech frame prediction network, and speech decoding network can be adjusted based on the third, fourth, and fifth loss values. This can be achieved by backpropagating based on the third, fourth, and fifth loss values ​​corresponding to each third speech sample in the same batch, and after multiple parameter adjustments, the trained speech recognition basic model can be obtained.

[0150] In some implementations, the third loss value can be calculated based on the CTC loss function, and the fifth loss value can be calculated based on the CTC loss function.

[0151] In some implementations, the initial speech recognition model may not include a speech frame prediction network. In this case, the parameter calculation process associated with the speech frame prediction network can be removed. That is, the parameters of the speech coding network and the speech decoding network can be adjusted directly based on the fourth and fifth loss values, thus completing the update process of the first and second parameters in the initial speech recognition model.

[0152] The structure and training process of the initial speech recognition model have been described through the above embodiments. Next, in conjunction with specific embodiments, the process of extracting keyword text from the text labels of the first and second speech samples will be explained.

[0153] In some implementations, the method of this disclosure may further include the following steps:

[0154] From the text labels of the first speech sample, extract X adjacent characters to obtain the first keyword text;

[0155] From the text labels of the second speech sample, extract Y adjacent characters to obtain the second keyword text;

[0156] Based on a first preset number of first keyword texts and a second preset number of second keyword texts, keyword texts are obtained, and the sum of the first preset number and the second preset number is equal to the number of first speech samples.

[0157] In this embodiment of the application, in order to obtain the keyword text corresponding to the first speech sample, keyword text can be extracted from the text tags of the first speech sample, and keyword text can be extracted from the text tags of the second speech sample. The keyword text extracted from the text tags of the first speech sample can be referred to as the first keyword text, and the keyword text extracted from the text tags of the second speech sample can be referred to as the second keyword text.

[0158] In some implementations, to improve the randomness of keyword text extraction, the starting position of keyword text extraction is not fixed. For example, extraction can start from the second character in the text tag, or it can start from the third character in the text tag.

[0159] Furthermore, in some implementations, the lengths of the extracted text, i.e., X and Y, can also be random. For example, the first keyword text can be obtained by extracting three adjacent characters from the text tags of the first speech sample, or by extracting five adjacent characters from the text tags of the first speech sample.

[0160] In this embodiment of the application, although the sum of the first preset quantity and the second preset quantity is equal to the number of first speech samples, the first preset quantity can be random. For example, when the first speech sample includes 32 speech samples, 3 first keyword texts and 29 second keyword texts can be extracted to form the keyword text corresponding to the first speech sample, or 5 first keyword texts and 27 second keyword texts can be extracted to form the keyword text corresponding to the first speech sample.

[0161] It should be noted that the process of extracting keyword text described above can be performed before each input of the first speech sample into the initial speech recognition model, or it can be performed after the first speech sample has been determined from the speech sample set.

[0162] Figure 6This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment of the present disclosure. This method can be applied to electronic devices, which may be terminal devices or servers. (Refer to...) Figure 6 The training method for this speech recognition model may include the following steps:

[0163] S601: Based on the speech coding network, process any third speech sample in the speech sample set to obtain the third intermediate feature of each frame of the third speech sample output by the intermediate layer of the speech coding network, and the fourth intermediate feature of each frame of the third speech sample output by the last layer of the speech coding network.

[0164] S602: The third intermediate feature is processed based on the speech frame prediction network to obtain the second speech frame in the third speech sample.

[0165] S603: The fourth intermediate feature is processed based on the speech decoding network to obtain the predicted speech text corresponding to the third speech sample.

[0166] S604: Determine the third, fourth, and fifth loss values, and adjust the parameters of the speech coding network, speech frame prediction network, and speech decoding network based on the third, fourth, and fifth loss values.

[0167] The third loss value is determined based on the second speech frame and the corresponding text label in the third speech sample; the fourth loss value is determined based on the predicted speech text and the corresponding text label of the third speech sample; and the fifth loss value is determined based on the fourth intermediate feature of each frame in the third speech sample and the corresponding text label of the third speech sample.

[0168] After steps S601-S604, the training of the speech coding network, speech frame prediction network, and speech decoding network is completed. Next, the speech coding network, speech frame prediction network, and speech decoding network are loaded and combined with the keyword coding network, keyword core network, keyword decoding network, and context relationship reconstruction network to form an initial speech recognition model. Subsequently, the parameters of the speech coding network, speech frame prediction network, and speech decoding network are frozen, and the keyword coding network, keyword core network, keyword decoding network, and context relationship reconstruction network are trained through steps S605-S612.

[0169] S605: The first speech sample is processed based on the speech coding network to obtain the first intermediate feature output by the intermediate layer of the speech coding network and the second intermediate feature output by the last layer of the speech coding network.

[0170] S606: The first intermediate feature is predicted by the speech frame prediction network to obtain the first speech frame in the first speech sample.

[0171] S607: Based on the sequential relationship of the first speech frame in the first speech sample, the context relationship of the second intermediate feature corresponding to the first speech frame is reconstructed to obtain the first speech feature of the first speech sample.

[0172] S608: Determine the first text features of the keyword text corresponding to the first speech sample through a keyword encoding network.

[0173] The keyword text corresponding to the first speech sample includes the first keyword text extracted from the text labels of the first speech sample, and the second keyword text extracted from the text labels of the second speech sample.

[0174] S609: Using the keyword core network, the first speech feature is used as the query input, and the first text feature is used as the key and value input. Based on the attention mechanism, the first speech feature and the first text feature are aggregated to obtain the first aggregated feature.

[0175] S610: Determine the predicted keyword text in the first speech sample based on the first aggregation feature using the keyword decoding network.

[0176] S611: Determine the predicted speech text based on the second intermediate feature and the first aggregate feature corresponding to the first speech frame through the speech decoding network.

[0177] S612: Determine the first loss value and the second loss value, and adjust the parameters of the keyword encoding network, the keyword core network, the keyword decoding network, and the context relationship reconstruction network based on the first loss value and the second loss value.

[0178] The first loss value is determined based on the predicted keyword text and the extracted keyword text, while the second loss value is determined based on the predicted speech text and the text labels of the first speech sample.

[0179] After the above steps S605-S612, the training of the keyword encoding network, keyword core network, keyword decoding network, and context relationship reconstruction network can be completed.

[0180] It should be noted that the execution order of steps S601-S612 in this embodiment is not limited. For example, step S608 can be executed synchronously with any of steps S605-S607, or it can be executed before any of steps S605-S607.

[0181] For steps not described in detail in steps S601-S612 above, please refer to the foregoing embodiments, and they will not be repeated here.

[0182] Below, in conjunction with Figure 4 as well as Figure 5 The schematic diagram of the model structure shown illustrates the training method of the speech recognition model in detail using a complete embodiment:

[0183] Data processing stage:

[0184] The pure speech recognition results from 50,000 hours of general domain data were indexed to obtain the text index corresponding to each speech. Furthermore, each speech was pre-emphasized, framed, windowed, subjected to Discrete Fourier Transform, and Mel-filtered processes to obtain 80-dimensional fbank features for each speech. The fbank features are frequency domain features.

[0185] Specifically, for any given speech, the 80-dimensional fbank features corresponding to that speech are used as the speech data included in the speech sample, and the text index corresponding to that speech is used as the text label included in the speech sample, thus constructing a speech sample.

[0186] First training phase:

[0187] The multiple speech samples obtained through the above processing are used to form a speech sample set, which is then used to analyze... Figure 5 The speech recognition basic model 500 shown is used for training. Figure 5 In the speech recognition model 500, a speech encoding network 401 using a transformer is used as the speech recognition base model 500, and a speech decoding network 403 using a transformer is used as the speech recognition base model 500. The features output from the intermediate layer of the speech encoding network 401 are connected to a speech frame prediction network 402 that includes a linear layer and an activation function layer.

[0188] In this training phase, the speech samples (third speech samples) in the speech sample set are processed by the speech coding network 401. The acoustic feature vector output by the final layer of the speech coding network 401 (the fourth intermediate feature of each frame) can be mapped to high-level semantic information. Then, the high-level semantic information output by the speech coding network 401 is input into the speech decoding network 403. After processing by the speech decoding network 403, the predicted speech text can be obtained. Furthermore, the intermediate layers of the speech coding network 401 (e.g., Figure 4 The acoustic feature vector (the third intermediate feature of each frame) output by the fourth layer of the speech frame prediction network 402 is passed through the linear layer and activation function layer to predict the second speech frame.

[0189] Subsequently, cross-entropy loss (fourth loss value) can be calculated based on the predicted speech text and the text labels carried by the third speech sample. CTC loss (fifth loss value) can be calculated based on the acoustic features output by the last layer of the speech coding network 401 (the fourth intermediate feature of each frame) and the text labels carried by the speech samples. CTC loss (third loss value) can be calculated based on the second speech frame output by the speech frame prediction network 402 and the text labels carried by the speech samples. The speech coding network 401, the speech decoding network 403, and the speech frame prediction network 402 are jointly trained based on the above three losses.

[0190] Second training phase:

[0191] The model architecture for the second training phase is as follows: Figure 4 As shown, in addition to the speech recognition base model 500 trained in the first training phase, a keyword encoding network 404, a keyword core network 405, a keyword decoding network 406, and a context relationship reconstruction network 407 can be added. In the second training phase, the network parameters of the speech encoding network 401, the speech decoding network 403, and the speech frame prediction network 402 are frozen. Multiple speech samples obtained through the above processing are used to form the first speech sample in batches. Simultaneously, the extracted keyword text corresponding to each batch of the first speech sample is prepared for analysis. Figure 5 The keyword encoding network 404, keyword core network 405, keyword decoding network 406, and context relationship reconstruction network 407 shown are trained.

[0192] Figure 5 In this process, two stacked GRU networks are used as the keyword encoding network 404, an attention mechanism is used as the keyword core network 405, two transformer layers, a linear layer and an activation function layer are used as the keyword decoding network 406 for keyword prediction, and TDNN (Time Delay Neural Network) is used as the context relationship reconstruction network 407.

[0193] In this training phase, on the one hand, the speech coding network 401 processes any batch of first speech samples. The acoustic feature vector output from the final layer of the speech coding network 401 (the second intermediate feature of each frame) can be mapped to high-level semantic information. Furthermore, the acoustic feature vector output from the intermediate layers of the speech coding network 401 (the first intermediate feature of each frame) is passed through the linear layer and activation function layer of the speech frame prediction network 402 to predict the first speech frame. Next, the second intermediate features corresponding to the first speech frame are selected from the second intermediate features of each frame, and the context relationship reconstruction network 407 is used to reconstruct the context relationship of the selected second intermediate features corresponding to the first speech frame, obtaining the first speech feature of each speech sample in the first speech sample. On the other hand, the keyword encoding network 404 processes the keyword text corresponding to the batch of first speech samples to obtain the first text feature.

[0194] The first speech feature is used as the query Q of the keyword core network 405, and the first text feature is used as the key K and value V of the keyword core network 405. The first aggregated feature is output by processing through the keyword core network 405.

[0195] After obtaining the first aggregated feature, on the one hand, the second intermediate feature of the first speech frame and the first aggregated feature are weighted and summed, and the result is input into the speech decoding network 403. The speech decoding network 403 performs speech-to-text recognition to obtain the predicted speech-to-text. The weight value can be set to 1.0 during training and is set to an adjustable value during inference to represent the degree of enhancement to the features corresponding to the keywords. On the other hand, the first aggregated feature is input into the keyword decoding network 406, which predicts the keywords, i.e., the predicted keyword text.

[0196] Subsequently, cross-entropy loss (second loss value) can be calculated based on the predicted speech text and the text labels carried by the first speech sample, and CTC loss (first loss value) can be calculated based on the predicted keyword text and the extracted keyword text. Based on the above two losses, the keyword encoding network 404, the keyword core network 405, the keyword decoding network 406, and the context relationship reconstruction network 407 can be jointly trained.

[0197] Finally, the trained speech coding network 401, speech decoding network 403, speech frame prediction network 402, keyword encoding network 404, keyword core network 405, keyword decoding network 406, and context relationship reconstruction network 407 can be loaded to obtain a speech recognition network for speech recognition.

[0198] Based on the same technical concept, this application also provides a speech recognition method. Figure 7This is a flowchart illustrating a speech recognition method according to an exemplary embodiment of the present disclosure. The method can be applied to an electronic device, which may be a terminal device or a server. The electronic device runs a trained speech recognition model, which is trained using the speech recognition model training method of the above embodiment. (Refer to...) Figure 7 The speech recognition method may include the following steps:

[0199] S701: Determine the second speech features of the speech to be recognized through the target speech recognition model.

[0200] The target speech recognition model is trained using the training method described in the aforementioned embodiments.

[0201] The speech to be recognized can be obtained by extracting features from speech that requires speech recognition. For example, it can be obtained by extracting features from speech in a specific domain or scenario as described in the foregoing embodiments.

[0202] The speech to be recognized can be an 80-dimensional fbank feature.

[0203] S702: Determine the second text features of the preset keyword text.

[0204] The preset keyword text can include multiple pre-prepared keywords belonging to the same domain as the speech to be recognized.

[0205] In some implementations, by inputting preset keyword text into a target speech recognition model, the target speech recognition model can determine the second text features of the preset keyword text.

[0206] That is, in the embodiments of this application, when applying the trained target speech recognition model, the input of the model includes two parts, namely the speech to be recognized and the preset keyword text.

[0207] In some implementations, the trained target speech recognition model can determine the second text features of the preset keyword text by first processing the preset keyword text when the model is loaded and started, obtaining the second text features of the preset keyword text and saving them. Subsequently, when processing different speech to be recognized, the second text features of the preset keyword text can be determined by obtaining the saved second text features.

[0208] S703: Aggregate the second speech feature and the second text feature to obtain the second aggregated feature.

[0209] S704: Based on the second aggregation feature, obtain the predicted speech text of the speech to be identified and / or the predicted keyword text of the speech to be identified.

[0210] In some implementations, after obtaining the second speech features and the second text features, the second speech features and the second text features can be further aggregated by the target speech recognition model to obtain the second aggregated features. Then, based on the second aggregated features, the predicted speech text and / or the predicted keyword text of the speech to be recognized can be obtained.

[0211] It should be noted that, in the embodiments of this application, when applying the trained target speech recognition model, at least one of the network for predicting speech text or the network for predicting keyword text can be selected and loaded according to actual needs. When both the network for predicting speech text and the network for predicting keyword text are loaded simultaneously, the target speech recognition model can output not only the predicted speech text of the speech to be recognized, but also the predicted keyword text of the speech to be recognized.

[0212] It should be noted that during the application phase of the target speech recognition model, the execution process of each network in the target speech recognition model can be referred to the data execution process during the training phase, and will not be repeated here.

[0213] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0214] Based on the same concept, embodiments of this disclosure also provide a training device for a speech recognition model, which can be part or all of an electronic device through software, hardware, or a combination of both. For example... Figure 8 As shown, the training device 800 for the speech recognition model may include:

[0215] The first processing module 801 is configured to process the first speech sample based on the initial speech recognition model to obtain the first speech features;

[0216] The second processing module 802 is configured to extract features from the keyword text to obtain the first text features. The keyword text is obtained by extracting keywords from the text labels of the first speech sample and the text labels of the second speech sample. The first speech sample and the second speech sample are speech samples from different training batches in the speech sample set.

[0217] The aggregation module 803 is configured to aggregate the first speech feature and the first text feature to obtain a first aggregated feature;

[0218] The prediction module 804 is configured to perform keyword prediction based on the first aggregated features to obtain the predicted keyword text of the first speech sample, and to perform speech text recognition based on the first aggregated features to obtain the predicted speech text of the first speech sample.

[0219] The training module 805 is configured to determine the model loss based on the predicted keyword text and the predicted speech text, and to train the initial speech recognition model based on the model loss to obtain the target speech recognition model.

[0220] In a possible implementation, the first processing module 801 includes:

[0221] The encoding submodule is configured to encode the first speech sample to obtain a first intermediate feature, and to encode the first intermediate feature to obtain a second intermediate feature;

[0222] The speech frame prediction submodule is configured to predict speech frames based on the first intermediate feature to obtain the first speech frame in the first speech sample.

[0223] The reconstruction submodule is configured to reconstruct the context relationship of the second intermediate feature corresponding to the first speech frame based on the sequential relationship of the first speech frame in the first speech sample, so as to obtain the first speech feature of the first speech sample.

[0224] In a possible implementation, the first text feature includes M text sub-features, each text sub-feature corresponding to at least one keyword; the aggregation module includes:

[0225] The correlation degree determination submodule is configured to determine the correlation degree between the first speech feature and the M text sub-features;

[0226] The weight allocation submodule is configured to assign weights to the M text sub-features based on the degree of correlation.

[0227] The weighted aggregation submodule is configured to perform weighted aggregation on the M text sub-features based on their weights to obtain the first aggregated feature.

[0228] In a possible implementation, training module 805 includes:

[0229] The first training submodule is configured to determine a first loss value based on the predicted keyword text and the extracted keyword text; determine a second loss value based on the predicted speech text and the text label of the first speech sample; and train the initial speech recognition model based on the first loss value and the second loss value to obtain the target speech recognition model.

[0230] In a possible implementation, training module 805 includes:

[0231] The second training submodule is configured to keep the first and second pre-trained parameters in the initial speech recognition model unchanged, and update the third and fourth parameters in the initial speech recognition model based on the model loss.

[0232] Wherein, the first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

[0233] In a possible implementation, the training device 800 for the speech recognition model further includes:

[0234] The first extraction module is configured to extract X adjacent characters from the text tags of the first speech sample to obtain the first keyword text.

[0235] The second extraction module is configured to extract Y adjacent characters from the text labels of the second speech sample to obtain the second keyword text.

[0236] The keyword text acquisition module is configured to obtain the keyword text based on a first preset number of the first keyword text and a second preset number of the second keyword text, wherein the sum of the first preset number and the second preset number is equal to the number of the first speech samples.

[0237] Regarding the training device 800 for the speech recognition model in the above embodiments, the specific methods by which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0238] Based on the same concept, embodiments of this disclosure also provide a voice recognition device, which can be part or all of an electronic device through software, hardware, or a combination of software and hardware. For example... Figure 9 As shown, the voice recognition device 900 may include:

[0239] The third processing module 901 is configured to determine the second speech features of the speech to be recognized through the target speech recognition model;

[0240] The fourth processing module 902 is configured to determine the second text features of the preset keyword text;

[0241] The fifth processing module 903 is configured to aggregate the second speech feature and the second text feature to obtain a second aggregated feature;

[0242] The sixth processing module is configured to obtain the predicted speech text and / or the predicted keyword text of the speech to be identified based on the second aggregation feature.

[0243] The trained target speech recognition model is obtained according to the aforementioned speech recognition model training method.

[0244] Regarding the speech recognition device 900 in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0245] Based on the same concept, embodiments of this disclosure also provide an electronic device, including:

[0246] A storage device on which computer programs are stored;

[0247] A processing device is configured to execute the computer program in the storage device to implement the steps of the training method for any of the above-described speech recognition models, or to implement the steps of the above-described speech recognition method.

[0248] Figure 10 This illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 2 A structural diagram of terminal equipment or server 1000.

[0249] Electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0250] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0251] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for any of the above-described speech recognition models, or implements the steps of the above-described speech recognition method.

[0252] Based on the same concept, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the training method for any of the above-described speech recognition models, or the steps of the above-described speech recognition method.

[0253] It should be understood that the above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0254] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0255] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for training a speech recognition model, characterized in that, include: The first speech sample is processed based on the initial speech recognition model to obtain the first speech features; Feature extraction is performed on the keyword text to obtain the first text feature. The keyword text is obtained by extracting keywords from the text labels of the first speech sample and the text labels of the second speech sample. The first speech sample and the second speech sample are speech samples from different training batches in the speech sample set. The keyword text includes positive keyword samples extracted from the text labels of the first speech sample and negative keyword samples extracted from the text labels of the second speech sample. The first speech feature and the first text feature are aggregated to obtain the first aggregated feature; Based on the first aggregation feature, keyword prediction is performed to obtain the predicted keyword text of the first speech sample, and based on the first aggregation feature, speech text recognition is performed to obtain the predicted speech text of the first speech sample. The model loss is determined based on the predicted keyword text and the predicted speech text, and the initial speech recognition model is trained based on the model loss to obtain the target speech recognition model.

2. The method according to claim 1, characterized in that, The process of processing the first speech sample to obtain the first speech features includes: The first speech sample is encoded to obtain a first intermediate feature, and the first intermediate feature is encoded to obtain a second intermediate feature; Based on the first intermediate feature, speech frame prediction is performed to obtain the first speech frame in the first speech sample. Based on the sequential relationship of the first speech frame in the first speech sample, the context relationship of the second intermediate feature corresponding to the first speech frame is reconstructed to obtain the first speech feature of the first speech sample.

3. The method according to claim 1 or 2, characterized in that, The first text feature includes M text sub-features, each text sub-feature corresponding to at least one keyword; the aggregation of the first speech feature and the first text feature to obtain the first aggregated feature includes: Determine the degree of association between the first speech feature and the M text sub-features; Based on the degree of correlation, weights are assigned to the M text sub-features; Based on the weights of the M text sub-features, the M text sub-features are weighted and aggregated to obtain the first aggregated feature.

4. The method according to claim 1, characterized in that, The step of determining the model loss based on the predicted keyword text and the predicted speech text, and training the initial speech recognition model based on the model loss to obtain the target speech recognition model includes: A first loss value is determined based on the predicted keyword text and the extracted keyword text; A second loss value is determined based on the predicted speech text and the text labels of the first speech sample; The initial speech recognition model is trained based on the first loss value and the second loss value to obtain the target speech recognition model.

5. The method according to claim 1 or 4, characterized in that, The step of training the initial speech recognition model based on the model loss to obtain the target speech recognition model includes: Keeping the first and second parameters of the pre-trained initial speech recognition model unchanged, the third and fourth parameters of the initial speech recognition model are updated based on the model loss; Wherein, the first parameter is the parameter used by the initial speech recognition model to determine the first speech feature, the second parameter is the parameter used by the initial speech recognition model to determine the predicted speech text, the third parameter is the parameter used by the initial speech recognition model to determine the first keyword text feature, and the fourth parameter is the parameter used by the initial speech recognition model to determine the predicted keyword text.

6. The method according to claim 1, characterized in that, The method further includes: From the text tags of the first speech sample, extract X adjacent characters to obtain the first keyword text; From the text labels of the second speech sample, extract Y adjacent characters to obtain the second keyword text; The keyword text is obtained based on a first preset number of the first keyword text and a second preset number of the second keyword text, wherein the sum of the first preset number and the second preset number is equal to the number of the first speech samples.

7. A speech recognition method, characterized in that, include: The second speech features of the speech to be recognized are determined using the target speech recognition model. Determine the second text features of the preset keyword text; The second speech feature and the second text feature are aggregated to obtain the second aggregated feature; Based on the second aggregation feature, the predicted speech text of the speech to be identified and / or the predicted keyword text of the speech to be identified are obtained; wherein, the target speech recognition model is trained by the method according to any one of claims 1-6.

8. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images