A speech recognition method, apparatus and device

Abstract

The speech recognition method provided in this application includes: extracting features from the speech to be recognized to obtain multiple frames of acoustic features corresponding to the speech; inputting the multiple frames of acoustic features into a pre-trained first speech recognition model and a pre-trained second speech recognition model, respectively; the first speech recognition model outputs a recognition result corresponding to each frame of acoustic features based on the input acoustic features, and the second speech recognition model outputs a posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features; determining a search space composed of a weighted finite-state transcriber (WFST) based on the recognition results; and, for each frame of acoustic features, using the posterior probability matrix corresponding to that frame of acoustic features as input, performing a decoding operation in the search space under the constraints of the recognition results to obtain an alignment result corresponding to that frame of acoustic features. The speech recognition method, apparatus, and device provided in this application can simultaneously achieve both content recognition accuracy and boundary alignment accuracy.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus and device. Background Technology

[0002] With the development of human-computer interaction and intelligent assistant technologies, speech recognition, as an important key technology, is receiving increasing attention and application. Speech recognition technology can convert human speech signals into understandable text or commands, greatly improving the convenience and efficiency of human-computer interaction.

[0003] Currently, speech recognition is often performed using speech recognition models, but these models cannot simultaneously achieve both content recognition accuracy and boundary alignment accuracy. Summary of the Invention

[0004] In view of this, this application provides a speech recognition method, apparatus, and device to solve the problem that existing speech recognition technologies cannot simultaneously achieve both content recognition accuracy and boundary alignment accuracy.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] The first aspect of this application provides a speech recognition method, the method comprising:

[0007] Feature extraction is performed on the speech to be recognized to obtain multi-frame acoustic features corresponding to the speech;

[0008] The multi-frame acoustic features are respectively input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features.

[0009] Based on the identification results, a search space consisting of a weighted finite-state transcription machine (WFST) is determined.

[0010] For each frame of acoustic features, the posterior probability matrix corresponding to the acoustic features of that frame is used as input. Under the constraint of the recognition result, a decoding operation is performed in the search space to obtain the alignment result corresponding to the acoustic features of that frame.

[0011] A second aspect of this application provides a speech recognition device, the device comprising an extraction module, a processing module, a determination module, and a decoding module, wherein...

[0012] The extraction module is used to extract features from the speech to be recognized, and obtain multi-frame acoustic features corresponding to the speech.

[0013] The processing module is used to input the multi-frame acoustic features into a pre-trained first speech recognition model and a pre-trained second speech recognition model, respectively. The first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features.

[0014] The determining module is used to determine the search space composed of the weighted finite-state transcription machine (WFST) based on the identification result;

[0015] The decoding module is used to take the posterior probability matrix corresponding to the acoustic features of each frame as input, and perform a decoding operation in the search space under the constraint of the recognition result to obtain the alignment result corresponding to the acoustic features of that frame.

[0016] A third aspect of this application provides a speech recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0017] A fourth aspect of this application provides a storage medium having a program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0018] The speech recognition method, apparatus, and device provided in this application extract features from the speech to be recognized to obtain multiple frames of acoustic features corresponding to the speech. These multiple frames of acoustic features are then input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs a recognition result corresponding to each frame of acoustic features based on the input acoustic features, and the second speech recognition model outputs a posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features. A search space composed of a weighted finite-state transcriber (WFST) is determined based on the recognition results. For each frame of acoustic features, the posterior probability matrix corresponding to that frame is used as input, and under the constraint of the recognition results, a decoding operation is performed in the search space to obtain an alignment result corresponding to that frame of acoustic features. In this way, during the speech recognition process, the extracted acoustic features from multiple frames are input into the first speech recognition model and the second speech recognition model respectively. Under the combined action of the first speech recognition model and the second speech recognition model, the final alignment result is obtained. Since two complementary speech recognition models are used, and decoding is performed based on the constraints of the recognition results when decoding in the search space, both content recognition accuracy and boundary alignment accuracy can be taken into account at the same time. Attached Figure Description

[0019] Figure 1 A flowchart of Embodiment 1 of the speech recognition method provided in this application;

[0020] Figure 2 This is a schematic diagram illustrating the structure of a first speech recognition model as an exemplary embodiment of this application;

[0021] Figure 3 A flowchart illustrating a second embodiment of the speech recognition method provided in this application.

[0022] Figure 4 This is a schematic diagram of the structure of a conformer unit shown in an exemplary embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating the structure of a second speech recognition model as an exemplary embodiment of this application;

[0024] Figure 6 A flowchart of Embodiment 3 of the speech recognition method provided in this application;

[0025] Figure 7 This is a schematic diagram of the structure of a TDNNF module shown in an exemplary embodiment of this application;

[0026] Figure 8 This is a hardware structure diagram of the speech recognition device in which the speech recognition device of this application is located;

[0027] Figure 9 This is a schematic diagram of the structure of the speech recognition device according to Embodiment 1 of the present application;

[0028] Figure 10 This is a schematic diagram of the structure of Embodiment 2 of the speech recognition device provided in this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0030] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0032] This application provides a speech recognition method, apparatus, and device to address the problem that existing speech recognition technologies cannot simultaneously achieve both content recognition accuracy and boundary alignment accuracy.

[0033] The speech recognition method, apparatus, and device provided in this application extract features from the speech to be recognized to obtain multiple frames of acoustic features corresponding to the speech. These multiple frames of acoustic features are then input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs a recognition result corresponding to each frame of acoustic features based on the input acoustic features, and the second speech recognition model outputs a posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features. A search space composed of a weighted finite-state transcriber (WFST) is determined based on the recognition results. For each frame of acoustic features, the posterior probability matrix corresponding to that frame is used as input, and under the constraint of the recognition results, a decoding operation is performed in the search space to obtain an alignment result corresponding to that frame of acoustic features. In this way, during the speech recognition process, the extracted acoustic features from multiple frames are input into the first speech recognition model and the second speech recognition model respectively. Under the combined action of the first speech recognition model and the second speech recognition model, the final alignment result is obtained. Since two complementary speech recognition models are used, and decoding is performed based on the constraints of the recognition results when decoding in the search space, both content recognition accuracy and boundary alignment accuracy can be taken into account at the same time.

[0034] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0035] Figure 1 This is a flowchart of an embodiment of the speech recognition method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0036] S101. Extract features from the speech to be recognized to obtain multi-frame acoustic features corresponding to the speech.

[0037] Specifically, feature extraction can be performed using traditional feature extraction methods or deep learning-based models; this embodiment does not limit the specific methods used. For example, when using traditional feature extraction methods, a predefined algorithm can be used to extract speech features. As another example, when using deep learning-based models for feature extraction, a model for feature extraction can be pre-trained, and then the speech can be input into the model to extract its features.

[0038] Furthermore, at least one of the following features can be extracted from the speech: Filter Bank (FBANK) features, fundamental frequency (PITCH) features, and Mel Frequency Cepstral Coefficients (MFCC) features. The following explanation uses the extraction of FBANK and PITCH features as examples.

[0039] For example, in one embodiment, the feature extraction step may include:

[0040] (1) Perform frame-by-frame processing on the speech to obtain each frame of speech.

[0041] Specifically, the frame length during frame processing is set according to actual needs, and in this embodiment, its specific value is not limited. For example, in one embodiment, the frame length is 10ms.

[0042] (2) For each frame of speech, extract the filter bank FBANK feature and the fundamental frequency PITCH feature of the speech frame.

[0043] Specifically, the process of extracting FBANK features may include:

[0044] Windowing, Fourier transform, wave filter group design, wave filter group applications, logarithmic operations, discrete cosine transform (DCT), feature normalization.

[0045] In practice, a windowing function is applied to each frame of speech, followed by a Fourier transform to calculate the energy of the filter bank. A logarithmic transform is then performed on the energy output of each filter, and a discrete cosine transform is applied to the logarithmic energy coefficients to obtain the FBANK features. Finally, the obtained FBANK features are normalized. For example, with 10ms as one frame, in this embodiment, the extracted FBANK features have a dimension of 40.

[0046] Furthermore, the process of extracting PITCH features may include:

[0047] The process includes windowing, autocorrelation function calculation, autocorrelation peak detection, fundamental frequency estimation, fundamental frequency post-processing, and fundamental frequency feature extraction. It should be noted that in this embodiment, the extracted PITCH features are 1-dimensional.

[0048] (3) Calculate the derivative of the PITCH feature and the derivative of the derivative, and use the FBANK feature, the PITCH feature, the derivative of the PITCH feature and the derivative of the derivative as the acoustic features of the speech in this frame.

[0049] Referring to the preceding description, the extracted FBANK feature has a dimension of 40, and the extracted PITCH feature has a dimension of 1. Correspondingly, the derivative of the PITCH feature has a dimension of 1, and the derivative of the derivative of the PITCH feature has a dimension of 1. Therefore, in this embodiment, the extracted acoustic features of a single frame of speech have a dimension of 43.

[0050] S102. The multi-frame acoustic features are respectively input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features.

[0051] Specifically, the input to the first speech recognition model is multiple frames of acoustic features, and the output is the recognition result corresponding to each frame of acoustic features. It should be noted that the above recognition results are character-level recognition results.

[0052] Furthermore, the second speech recognition model takes multiple frames of acoustic features as input and outputs a posterior probability matrix corresponding to each frame of acoustic features. Specifically, the posterior probability matrix is ​​a posterior probability matrix at the phoneme or word level.

[0053] The internal structure and working principle of the first and second speech recognition models will be described in detail in the following embodiments, and will not be repeated here.

[0054] S103. Determine the search space consisting of the weighted finite state transcription machine (WFST) based on the identification results.

[0055] In practical implementation, a search space can be generated based on the speech model. This search space can include a weighted finite-state transducer (WFST). It should be noted that the search space refers to the set of possible text sequences during the decoding process.

[0056] S104. For each frame of acoustic features, the posterior probability matrix corresponding to the acoustic features of that frame is used as input. Under the constraint of the recognition result, a decoding operation is performed in the search space to obtain the alignment result corresponding to the acoustic features of that frame.

[0057] In specific implementation, performing decoding operations in the search space under the constraints of the recognition results can be understood as follows: when searching for decoding results in the decoding space using a search algorithm, decoding should be performed according to the target decoding path. In this step, the target decoding path can be found based on the recognition results, and the text sequence corresponding to each node on the target decoding path contains the aforementioned recognition results.

[0058] For example, if the recognition result is "sparrows outside the window", then during decoding, the decoding should be performed according to the decoding path containing the text sequence "sparrows outside the window" to obtain the final alignment result.

[0059] The speech recognition method provided in this embodiment extracts features from the speech to be recognized to obtain multiple frames of acoustic features corresponding to the speech. These multiple frames of acoustic features are then input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs the recognition result corresponding to each frame of acoustic features based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features. A search space composed of a weighted finite-state transcriber (WFST) is determined based on the recognition results. For each frame of acoustic features, the posterior probability matrix corresponding to that frame is used as input, and under the constraint of the recognition results, a decoding operation is performed in the search space to obtain an alignment result corresponding to that frame of acoustic features. In this way, during the speech recognition process, the extracted acoustic features from multiple frames are input into the first speech recognition model and the second speech recognition model respectively. Under the combined action of the first speech recognition model and the second speech recognition model, the final alignment result is obtained. Since two complementary speech recognition models are used, and decoding is performed based on the constraints of the recognition results when decoding in the search space, both content recognition accuracy and boundary alignment accuracy can be taken into account at the same time.

[0060] Figure 2 This is a schematic diagram illustrating the structure of a first speech recognition model as an exemplary embodiment of this application. Please refer to... Figure 2 The first speech recognition model includes an encoder, a first decoder connected to the output of the encoder, and a second decoder connected to both the output of the first decoder and the output of the encoder.

[0061] Figure 3The flowchart is for Embodiment 2 of the speech recognition method provided in this application. Please refer to... Figure 3 The method provided in this embodiment, based on the above embodiments, combines... Figure 2 The process by which the first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features may include:

[0062] S301. The encoder encodes the acoustic features to obtain coded features.

[0063] Specifically, before encoding the acoustic features, the acoustic features can be subjected to specified processing; wherein, the specified processing includes at least one of the following: enhancement processing, regularization processing, and linear transformation processing.

[0064] Specifically, augmentation processing can include spatial augmentation. It should be noted that spatial augmentation expands the dataset by transforming, distorting, or occluding acoustic features, thereby improving the model's robustness and generalization ability. For example, in one embodiment, a certain width of features is randomly sampled in both the time and feature dimensions and zeroed out, introducing different temporal and frequency occlusions for data augmentation, dimensionality reduction, or local feature extraction.

[0065] Specifically, during regularization, Dropout regularization can be used; correspondingly, linear transformation processing can include linear predictive analysis.

[0066] Specifically, in this embodiment, the encoder includes multiple conformer units (i.e., conformer blocks). Figure 4 This is a schematic diagram illustrating the structure of a conformer unit as shown in an exemplary embodiment of this application. Please refer to... Figure 4 Each conformer unit comprises, in series, a first feedforward module, a multi-head self-attention module, a convolutional module, a second feedforward module, and a layer normalization module. Furthermore, the input of the multi-head self-attention module is connected to the input of the conformer unit; the input of the convolutional module is connected to the output of the first feedforward module; the input of the second feedforward module is connected to the output of the multi-head self-attention module; and the input of the layer normalization module is connected to the output of the convolutional module. This allows for the introduction of residuals.

[0067] Furthermore, the encoder is mainly used to encode acoustic features to obtain encoded features. In this embodiment, the obtained encoded features are a probability matrix.

[0068] S302. The first decoder decodes the encoded features to obtain multiple candidate recognition results corresponding to the acoustic features of each frame and the confidence level of each candidate recognition result, and outputs the first N candidate recognition results in descending order of confidence level.

[0069] Specifically, the first decoder is the component responsible for decoding the encoded features, primarily used to map the encoded features to text. In this embodiment, the first decoder is implemented using a Connectionist Temporal Classification (CTC) model. Furthermore, the decoding process of the first decoder can involve generating a series of candidate recognition results based on the acoustic information corresponding to the encoded features. It should be noted that in this embodiment, the candidate recognition results are text sequences or phoneme sequences.

[0070] Furthermore, for each candidate recognition result, the first decoder calculates the confidence level of that candidate recognition result. It should be noted that the confidence level of each candidate recognition result characterizes the degree of credibility of that candidate recognition result. In other words, the higher the confidence level of a candidate recognition result, the higher the credibility of that candidate recognition result.

[0071] Finally, as described above, candidate recognition results with higher confidence are considered more reliable. Therefore, the first decoder ultimately uses the top N candidate recognition results with the highest confidence as the final output.

[0072] S303. The second decoder decodes the encoded features, calculates the re-scoring confidence of the N candidate recognition results corresponding to the acoustic features in each frame, and outputs the target candidate recognition result with the highest re-scoring confidence as the recognition result corresponding to the acoustic features in that frame.

[0073] Specifically, the second decoder is another component that decodes the coded features. The main task of the second decoder is to calculate the re-scoring confidence of each frame's acoustic features corresponding to the N candidate recognition results output by the first decoder in the previous step, and select the target candidate recognition result with the highest re-scoring confidence as the final recognition result output for that frame's acoustic features.

[0074] Specifically, the internal workflow of the second decoder may include:

[0075] (1) The second decoder receives the encoded features and the N candidate recognition results output by the first decoder as input.

[0076] (2) Align the encoded features of each frame with the corresponding candidate recognition results to ensure that each candidate result is consistent with its corresponding encoded features.

[0077] (3) For each frame of encoded features and corresponding candidate recognition results, calculate the re-scoring confidence of the candidate recognition results.

[0078] (4) For each frame of acoustic features, select the candidate recognition result with the highest re-scoring confidence as the final recognition result output for that frame.

[0079] It should be noted that the method provided in this embodiment, by using a second decoder, can more accurately evaluate multiple candidate recognition results generated by the first decoder and select the target recognition result with the highest re-scoring confidence as the final recognition result of the acoustic features for each frame. This helps to improve the accuracy and stability of speech recognition.

[0080] The speech recognition method provided in this embodiment includes a first speech recognition model comprising an encoder, a first decoder connected to the output of the encoder, and a second decoder connected to both the output of the first decoder and the output of the encoder. In the speech recognition process, the first speech recognition model first encodes the acoustic features of each frame based on the encoder to obtain encoded features. Then, the first decoder decodes the encoded features based on the encoded features, outputting the top N candidate recognition results with the highest confidence. The second decoder then calculates the re-scoring confidence of each frame's acoustic features corresponding to the N candidate recognition results output by the first decoder, and selects the target candidate recognition result with the highest re-scoring confidence as the recognition result for that frame's acoustic features. Thus, by using the second decoder, the multiple candidate recognition results generated by the first decoder can be evaluated more accurately, contributing to improved accuracy and stability of speech recognition.

[0081] Figure 5 This is a schematic diagram illustrating the structure of a second speech recognition model as an exemplary embodiment of this application. Please refer to... Figure 5The second speech recognition model includes a first semi-orthogonal constrained time-delay neural network (TDNNF) and a second TDNNF network. The first TDNNF network includes multiple parallel first TDNNF modules, each with a different time window size. The second TDNNF network includes multiple serial second TDNNF modules, each with the same time window size. The output of the first TDNNF network is connected to the input of the second TDNNF network, and the output of a specific first TDNNF module in the first TDNNF network is connected to the input of the last second TDNNF module in the second TDNNF network.

[0082] Figure 6 The flowchart is for Embodiment 3 of the speech recognition method provided in this application. Please refer to... Figure 6 The method provided in this embodiment, based on the above embodiments, combines... Figure 5 The process by which the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features may include:

[0083] S601. Each of the first TDNNF modules in the first TDNNF network performs feature extraction on the acoustic features to obtain multiple context features corresponding to the acoustic features.

[0084] It should be noted that before feature extraction of acoustic features, the acoustic features can be subjected to specified processing; wherein, the specified processing includes at least one of the following processing: enhancement processing, regularization processing, and linear transformation processing.

[0085] Referring to the preceding description, the first TDNNF network includes multiple parallel first TDNNF modules with time windows configured. It should be noted that the first TDNNF modules with time windows refer to time-dependent TDNNF modules. Furthermore, the number of first TDNNF modules is set according to actual needs; in this embodiment, the number of first TDNNF modules is not limited. For example, in one embodiment, the number of first TDNNF modules is 3, 4, or 5. Further details can be found in [link to documentation]. Figure 5 ,exist Figure 5 In the example shown, the number of the first TDNNF modules is 2.

[0086] It should be noted that each first TDNNF module is a TDNNF module with a time window, and the size of the time window is different for each first TDNNF module. For example, in Figure 5In the example shown, the time windows for the two first TDNNF modules are 80ms and 30ms, respectively.

[0087] It should be noted that the time window refers to the temporal context considered when processing the input acoustic features. In this embodiment, the time windows of the multiple first TDNNF modules are of different sizes, which allows the second speech recognition model to capture different features at different time scales.

[0088] Figure 7 This is a schematic diagram illustrating the structure of a TDNNF module as shown in an exemplary embodiment of this application. Please refer to... Figure 7 In this embodiment, the TDNNF module (the first TDNNF module and / or the second TDNNF module) mainly includes four key units: a batch normalization unit, a ReLU activation function, an affine transformation unit, and a linear fully connected layer. The specific working principles of each unit can be found in descriptions in related technologies, and will not be repeated here.

[0089] S602. The second TDNNF network identifies the multiple contextual features and outputs the posterior probability matrix corresponding to the acoustic features.

[0090] It should be noted that the second TDNNF network comprises multiple cascaded second TDNNF modules. The time windows of these multiple cascaded second TDNNF modules are of the same size. Furthermore, the number of second TDNNF modules included in the second TDNNF network is set according to actual needs; this embodiment does not impose a limitation on this. For example, in... Figure 5 In the example shown, the second TDNNF network contains three second TDNNF modules.

[0091] It should be noted that, see Figure 5 The first TDNNF module can be the one with the smaller time window among multiple first TDNNF modules. For example, in Figure 5 In the example shown, the first TDNNF module is specified as a TDNNF module with a time window size of 30ms.

[0092] It should be noted that the posterior probability matrix is ​​a phoneme or character-level posterior probability matrix.

[0093] The speech recognition method provided in this embodiment includes a second speech recognition model comprising a first TDNNF network and a second TDNNF network. The first TDNNF network comprises multiple parallel first TDNNF modules, each with a different time window size. The second TDNNF network comprises multiple cascaded second TDNNF modules, each with the same time window size. The output of the first TDNNF network is connected to the input of the second TDNNF network, and the output of a specific first TDNNF module in the first TDNNF network is connected to the input of the last second TDNNF module in the second TDNNF network. In this way, by acquiring multiple contextual features through the first TDNNF network and then using the second TDNNF network to recognize these features and output a posterior probability matrix, the modeling ability and classification accuracy of acoustic features can be improved, thereby enhancing the performance of speech recognition.

[0094] Corresponding to the aforementioned embodiment of a speech recognition method, this application also provides an embodiment of a speech recognition device.

[0095] An embodiment of a speech recognition device provided in this application can be applied to a speech recognition device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the speech recognition device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of a speech recognition device in which the speech recognition device of this application is located. (Except for...) Figure 8 In addition to the processor, memory, network interface, and non-volatile memory shown, the speech recognition device in the embodiment may also include other hardware depending on the actual function of the speech recognition device, which will not be described in detail here.

[0096] Figure 9 This is a schematic diagram of the structure of a speech recognition device according to Embodiment 1 of this application. Please refer to... Figure 9 The apparatus provided in this embodiment may include an extraction module 910, a processing module 920, a determination module 930, and a decoding module 940, wherein...

[0097] The extraction module 910 is used to extract features from the speech to be recognized, and obtain multi-frame acoustic features corresponding to the speech.

[0098] The processing module 920 is used to input the multi-frame acoustic features into a pre-trained first speech recognition model and a pre-trained second speech recognition model, respectively. The first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features.

[0099] The determining module 930 is used to determine the search space composed of the weighted finite-state transcription machine (WFST) based on the identification result;

[0100] The decoding module 940 is used to take the posterior probability matrix corresponding to the acoustic features of each frame as input, and perform a decoding operation in the search space under the constraint of the recognition result to obtain the alignment result corresponding to the acoustic features of that frame.

[0101] The apparatus provided in this embodiment can be used to perform... Figure 1 The steps, implementation principles, and implementation processes of the method embodiments shown can be found in the descriptions of the previous embodiments, and will not be repeated here.

[0102] Optionally, the first speech recognition model includes an encoder, a first decoder connected to the output of the encoder, and a second decoder connected to both the output of the first decoder and the output of the encoder; wherein, the process by which the first speech recognition model outputs the recognition result corresponding to each frame of acoustic features based on the input acoustic features includes:

[0103] The encoder encodes the acoustic features to obtain coded features;

[0104] The first decoder decodes the encoded features to obtain multiple candidate recognition results corresponding to the acoustic features of each frame and the confidence level of each candidate recognition result, and outputs the first N candidate recognition results in descending order of confidence level;

[0105] The second decoder decodes the encoded features, calculates the re-scoring confidence of the N candidate recognition results corresponding to the acoustic features in each frame, and outputs the target candidate recognition result with the highest re-scoring confidence as the recognition result corresponding to the acoustic features in that frame.

[0106] Optionally, the second speech recognition model includes a first semi-orthogonal constrained time-delay neural network (TDNNF) and a second TDNNF network, wherein the first TDNNF network includes multiple parallel first TDNNF modules; wherein the time window size of each first TDNNF module is different; the second TDNNF network includes multiple cascaded second TDNNF modules; wherein the time window size of each second TDNNF module is the same; the output of the first TDNNF network is connected to the input of the second TDNNF network, and the output of a specified first TDNNF module in the first TDNNF network is connected to the input of the last second TDNNF module in the second TDNNF network; the process of the second speech recognition model outputting the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features includes:

[0107] Each of the first TDNNF modules in the first TDNNF network performs feature extraction on the acoustic features to obtain multiple context features corresponding to the acoustic features;

[0108] The second TDNNF network identifies the multiple contextual features and outputs the posterior probability matrix corresponding to the acoustic features.

[0109] Optionally, the processing module 920 is further configured to perform specified processing on the acoustic features before inputting the multi-frame acoustic features into the pre-trained first speech recognition model and the pre-trained second speech recognition model respectively; wherein the specified processing includes at least one of the following processes: enhancement processing, regularization processing and linear transformation processing.

[0110] Optional, Figure 10 This is a schematic diagram of the second embodiment of the speech recognition device provided in this application. Please refer to... Figure 10 Based on the above embodiments, the apparatus provided in this embodiment includes an extraction module 910 comprising a splitting unit 9101, an extraction unit 9102, and a combination unit 9103; wherein,

[0111] The splitting unit 9101 is used to perform frame-segmentation processing on the speech to obtain each frame of speech.

[0112] The extraction unit 9102 is used to extract the filter bank FBANK features and the fundamental frequency PITCH features of each frame of speech.

[0113] The extraction unit 9102 is also used to calculate the derivative of the PITCH feature and the derivative of the derivative;

[0114] The combination unit 9103 is used to use the FBANK feature, the PITCH feature, the derivative of the PITCH feature, and the derivative of the derivative as the acoustic features of the speech in the frame.

[0115] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0116] Please continue to refer to Figure 7 This application also provides a voice recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0117] This application also provides a storage medium having a program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in this application.

[0118] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0119] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A speech recognition method, characterized in that, The method includes: Feature extraction is performed on the speech to be recognized to obtain multi-frame acoustic features corresponding to the speech; The multi-frame acoustic features are respectively input into a pre-trained first speech recognition model and a pre-trained second speech recognition model. The first speech recognition model outputs the recognition result corresponding to the acoustic features of each frame based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features. Based on the identification results, a search space consisting of a weighted finite-state transcription machine (WFST) is determined. For each frame of acoustic features, the posterior probability matrix corresponding to the acoustic features of that frame is used as input. Under the constraint of the recognition result, a decoding operation is performed in the search space to obtain the alignment result corresponding to the acoustic features of that frame. The second speech recognition model includes a first TDNNF network and a second TDNNF network, wherein the first TDNNF network includes multiple parallel first TDNNF modules, and the time window size of each first TDNNF module is different; the second TDNNF network includes multiple cascaded second TDNNF modules, and the time window size of each second TDNNF module is the same; the output of the first TDNNF network is connected to the input of the second TDNNF network, and the output of a specified first TDNNF module in the first TDNNF network is connected to the input of the last second TDNNF module in the second TDNNF network; the process of the second speech recognition model outputting the posterior probability matrix corresponding to the acoustic features of each frame based on the input acoustic features includes: Each of the first TDNNF modules in the first TDNNF network performs feature extraction on the acoustic features to obtain multiple context features corresponding to the acoustic features; The second TDNNF network identifies the multiple contextual features and outputs the posterior probability matrix corresponding to the acoustic features.

2. The method according to claim 1, characterized in that, The first speech recognition model includes an encoder, a first decoder connected to the output of the encoder, and a second decoder connected to both the output of the first decoder and the output of the encoder; wherein, the process by which the first speech recognition model outputs the recognition result corresponding to each frame of acoustic features based on the input acoustic features includes: The encoder encodes the acoustic features to obtain coded features; The first decoder decodes the encoded features to obtain multiple candidate recognition results corresponding to the acoustic features of each frame and the confidence level of each candidate recognition result, and outputs the first N candidate recognition results in descending order of confidence level; The second decoder decodes the encoded features, calculates the re-scoring confidence of the N candidate recognition results corresponding to the acoustic features in each frame, and outputs the target candidate recognition result with the highest re-scoring confidence as the recognition result corresponding to the acoustic features in that frame.

3. The method according to claim 1, characterized in that, Before inputting the multi-frame acoustic features into the pre-trained first speech recognition model and the pre-trained second speech recognition model, the method further includes: The acoustic features are subjected to specified processing; wherein the specified processing includes at least one of the following: enhancement processing, regularization processing, and linear transformation processing.

4. The method according to claim 1, characterized in that, The process of extracting features from the speech to be recognized to obtain multi-frame acoustic features corresponding to the speech includes: The speech is segmented into frames to obtain each frame of speech; For each frame of speech, extract the filter bank FBANK features and the fundamental frequency PITCH features of that frame of speech. Calculate the derivative of the PITCH feature and the derivative of the derivative, and use the FBANK feature, the PITCH feature, the derivative of the PITCH feature, and the derivative of the derivative as the acoustic features of the speech in this frame.

5. A voice recognition device, characterized in that, The device includes an extraction module, a processing module, a determination module, and a decoding module, wherein, The extraction module is used to extract features from the speech to be recognized, and obtain multi-frame acoustic features corresponding to the speech. The processing module is used to input the multi-frame acoustic features into a pre-trained first speech recognition model and a pre-trained second speech recognition model, respectively. The first speech recognition model outputs the recognition result corresponding to each frame of acoustic features based on the input acoustic features, and the second speech recognition model outputs the posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features. The second speech recognition model includes a first TDNNF network and a second TDNNF network, wherein the first TDNNF network includes multiple parallel first TDNNF modules, and the time window size of each first TDNNF module is different; the second TDNNF network includes multiple cascaded second TDNNF modules, and the time window size of each second TDNNF module is the same; the output end of the first TDNNF network is connected to the input end of the second TDNNF network, and the output end of a specified first TDNNF module in the first TDNNF network is connected to the input end of the last second TDNNF module in the second TDNNF network; the process of the second speech recognition model outputting the posterior probability matrix corresponding to each frame of acoustic features based on the input acoustic features includes: Each of the first TDNNF modules in the first TDNNF network performs feature extraction on the acoustic features to obtain multiple context features corresponding to the acoustic features; The second TDNNF network identifies the multiple contextual features and outputs the posterior probability matrix corresponding to the acoustic features; The determining module is used to determine the search space composed of the weighted finite-state transcription machine (WFST) based on the identification result; The decoding module is used to take the posterior probability matrix corresponding to the acoustic features of each frame as input, and perform a decoding operation in the search space under the constraint of the recognition result to obtain the alignment result corresponding to the acoustic features of that frame.

6. The apparatus according to claim 5, characterized in that, The extraction module includes a splitting unit, an extraction unit, and a combination unit; wherein... The splitting unit is used to perform frame-by-frame processing on the speech to obtain each frame of speech; The extraction unit is used to extract the filter bank FBANK features and the fundamental frequency PITCH features of each frame of speech. The extraction unit is also used to calculate the derivative of the PITCH feature and the derivative of the derivative; The combining unit is used to take the FBANK feature, the PITCH feature, the derivative of the PITCH feature, and the derivative of the derivative as the acoustic features of the speech in the frame.

7. The apparatus according to claim 5, characterized in that, The processing module is further configured to perform specified processing on the acoustic features before inputting the multi-frame acoustic features into the pre-trained first speech recognition model and the pre-trained second speech recognition model, respectively; wherein the specified processing includes at least one of the following processes: enhancement processing, regularization processing, and linear transformation processing.

8. A voice recognition device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method according to any one of claims 1-4.

9. A storage medium having a program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.

Images