Speech recognition method, device, and computer-readable storage medium

Abstract

The present application discloses a speech recognition method, apparatus, and computer-readable storage medium. The speech recognition method includes: identifying at least two different features of a speech signal to be recognized to obtain at least two word graphs; determining a first evaluation score for each emitting arc in each word graph; determining a weight for each emitting arc in each word graph; determining a second evaluation score for each path based on the first evaluation scores and weights of all emitting arcs in each path; fusing all word graphs to obtain a fused word graph, and setting a third evaluation score for the emitting arc in each path in the fused word graph based on the second evaluation score of each path; and determining a recognition result of the speech signal to be recognized based on the fused word graph and the third evaluation scores of the emitting arcs in the fused word graph. The speech recognition method provided by the present application can improve the accuracy of speech recognition.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, and computer-readable storage medium. Background Art

[0002] Speech recognition technology is widely used in different fields such as human-computer interaction, smart home, office meeting scenarios, etc. With the increasing richness of usage scenarios, speech recognition technology faces challenges from multiple factors such as background interference, multilingualism, colloquialism, and long sentences. Therefore, the accuracy of speech recognition needs to be further improved. Summary of the Invention

[0003] The present application provides a speech recognition method, device, and computer-readable storage medium, which can improve the accuracy of speech recognition.

[0004] A first aspect of an embodiment of the present application provides a speech recognition method, the method comprising: identifying at least two different features of a speech signal to be recognized to obtain at least two word graphs; determining a first evaluation score for each emitting arc in each of the word graphs, respectively, wherein the size relationship between the first evaluation scores of any two emitting arcs in the same word graph is correlated with the size relationship between the probabilities of corresponding output labels; determining a weight for each emitting arc in each of the word graphs, respectively, wherein in all the word graphs, the size relationship between the weights of any two emitting arcs under the same path is correlated with the size relationship between the probabilities of corresponding output labels; determining a second evaluation score for each path based on the first evaluation scores and the weights of all the emitting arcs under each path; fusing all the word graphs to obtain a fused word graph, and setting a third evaluation score for the emitting arc under each path in the fused word graph based on the second evaluation score of each path; and determining a recognition result of the speech signal to be recognized based on the fused word graph and the third evaluation scores of the emitting arcs in the fused word graph.

[0005] A second aspect of an embodiment of the present application provides a speech recognition device, which includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit, respectively. Program data is stored in the memory. The processor implements the steps in the above method by executing the program data in the memory.

[0006] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the above method.

[0007] The beneficial effect is that the speech recognition method of the present application can increase the distinction between the correct path and the confused path in the fused word graph after fusion by setting weights for the emission arcs in the word graph, thereby facilitating the search for the correct path and improving the accuracy and robustness of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0009] Figure 1 This is a flow chart of an embodiment of the speech recognition method of the present application;

[0010] Figure 2 It is a partial structural diagram of the word graph;

[0011] Figure 3 yes Figure 1 Flow chart of step S130;

[0012] Figure 4 This is a structural diagram of an embodiment of the speech recognition device of the present application;

[0013] Figure 5 This is a structural diagram of another embodiment of the speech recognition device of the present application;

[0014] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] See Figure 1 , Figure 1 This is a flow chart of an embodiment of a speech recognition method of the present application, which includes:

[0017] S110: Identify at least two different features of the speech signal to be recognized to obtain at least two word graphs.

[0018] Here, at least two different feature extractions are performed on the speech signal to be recognized to obtain at least two different speech features, and then speech recognition is performed on each speech feature to obtain a word graph for each speech feature.

[0019] The speech features of the speech signal to be recognized include the sub-speech features of each frame signal in the speech signal to be recognized. The sub-speech features of each frame signal are concatenated to obtain the speech features of the speech signal to be recognized.

[0020] The types of speech features are different, and we can focus on information features of different aspects of the speech signal to be recognized.

[0021] In one application scenario, step S110 includes:

[0022] (a) performing at least two different feature extractions on the speech signal to be recognized to obtain speech features of at least two different dimensions.

[0023] (b) Recognize the speech features of each dimension separately to obtain at least two word graphs.

[0024] The different dimensions of the two speech features specifically refer to that the dimensions of the sub-speech features included in the two speech features are different, wherein the sub-speech features can be represented by vectors, that is, the lengths of the vectors are different.

[0025] For example, the at least two different speech features include 13-dimensional MFCC (Mel Frequency Cepstral Coefficents) features and 40-dimensional MFCC features.

[0026] In other application scenarios, the at least two different speech features may also be different specific acoustic features. For example, the at least two different speech features include MFCC features, FBANK (Mel filterbank) features, PLP (Perceptual Linear Predictive) features, and bottle-neck features (language features of audio).

[0027] In other application scenarios, speech features may also have different dimensions and specific acoustic features.

[0028] In one application scenario, to improve recognition accuracy and efficiency, different speech recognition systems are used to identify different speech features, thereby obtaining word graphs for each speech feature. When at least two different speech features include 13-dimensional MFCC features, a GMM-HMM (Gaussian Mixture-Hidden Markov Model) is used to identify these 13-dimensional MFCC features and obtain the corresponding word graphs. When 40-dimensional MFCC features are included, a DNN-HMM (Deep Neural Network-Hidden Markov Model) is used to identify these 40-dimensional MFCC features and obtain the corresponding word graphs.

[0029] S120: Determine the first evaluation score of each emission arc in each word graph respectively, wherein the magnitude relationship between the first evaluation scores of any two emission arcs in the same word graph is correlated with the magnitude relationship between the probabilities of the corresponding output labels.

[0030] Among them, the word graph is also called a lattice graph, which includes multiple state nodes, which are connected by transition arcs. Among the multiple transition arcs in the word graph, some transition arcs are emitting arcs (emitting arcs) and some transition arcs are non-emitting arcs (nonemitting arcs). The basic structure on the emitting arc includes: input label, output label, graph cost score and acoustic cost score. The input label represents the identifier corresponding to the decoded sub-speech feature (such as ID number), and the output label is the decoded word or character sequence. The graph cost score is composed of the transition probability between the language model and the HMM state, and the acoustic cost score is obtained by fitting different acoustic models.

[0031] For example, Figure 2 In the figure, the input label of the emission arc numbered 101 is "35", the output label is "I am true", the acoustic cost is "234.66", and the image cost is "-93.00"; the input label of the emission arc numbered 102 is "24", the output label is "good", the acoustic cost is "277.09", and the image cost is "-117.00"; the input label of the emission arc numbered 103 is "23", the output label is "good", the acoustic cost is "277.09", and the image cost is "-46.00".

[0032] Among them, the possibility of the output label of the emission arc represents the accuracy of the output label of the emission arc. Specifically, the greater the possibility of the output label of the emission arc, the greater the accuracy of the output label of the emission arc, and the graph cost score and acoustic cost score of the emission arc are both related to the possibility of the output label of the emission arc. Specifically, the greater the sum of the graph cost score and the acoustic cost score of the emission arc, the greater the possibility of the output label of the emission arc.

[0033] In this embodiment, in the same word graph, the greater the possibility of the emitted arc outputting a label, the greater the first evaluation score of the emitted arc.

[0034] In this embodiment, step S120 specifically includes: determining a first evaluation score for each emission arc in the corresponding word graph based on the graph cost scores and acoustic cost scores of all emission arcs in each word graph.

[0035] For any word graph, the first evaluation score of any emission arc included in the word graph is determined according to the graph cost scores and acoustic cost scores of all emission arcs included in the word graph.

[0036] In one application scenario, the first evaluation score is the minimum Bayesian risk assessment score, namely, the MRB score. The calculation process of the MRB score belongs to the existing technology and will not be described in detail here.

[0037] In other application scenarios, other formulas can also be used to determine the first evaluation score of each emitting arc in the word graph. For example, in any word graph, the sum of the graph cost score and acoustic cost score of each emitting arc is calculated, and then the sum values corresponding to all the emitting arcs in the word graph are added to obtain the sum value corresponding to the word graph. The ratio of the sum of the graph cost score and acoustic cost score of each emitting arc in the word graph to the sum value of the word graph is calculated respectively. This ratio is the first evaluation score corresponding to the emitting arc.

[0038] In other embodiments, the first evaluation score of the emission arc may also be the sum of the graph cost score and the acoustic cost score of the emission arc, or the graph cost score or the acoustic cost score of the emission arc.

[0039] S130: Determine the weight of each emission arc in each word graph respectively, wherein, in all word graphs, the size relationship between the weights of any two emission arcs in the same path is related to the size relationship between the probabilities of the corresponding output labels.

[0040] If two emitting arcs have the same input label and the same output label, then the paths of the two emitting arcs are the same. In other words, the input and output labels of the emitting arcs in the same path are the same. At the same time, for any emitting arc in each word graph, an emitting arc with the same path can be found in any other word graph.

[0041] In this embodiment, the greater the probability of the output label of the emitting arc in the same path, the greater the weight of the emitting arc. In other embodiments, when the probability of the output label of the emitting arc in the same word graph is smaller, the smaller the first evaluation score of the emitting arc, and the smaller the probability of the output label of the emitting arc in the same path, the weight of the emitting arc is smaller.

[0042] See Figure 3In this embodiment, step S130 includes:

[0043] S131: Add the first evaluation scores of the emission arcs of the same path in different word graphs to obtain the first sum value corresponding to each path.

[0044] S132: Determine the ratio of the first evaluation score of each emission arc to the first sum value of the corresponding path.

[0045] S133: Determine the weight of each emission arc according to the ratio corresponding to each emission arc.

[0046] To better understand steps S131 to S133, an example is provided below:

[0047] There are word graphs A and B. The first evaluation scores of the five emission arcs included in word graph A are [A1, A2, A3, A4, A5], and the first evaluation scores of the five emission arcs included in word graph B are [B1, B2, B3, B4, B5]. The emission arc with the first evaluation score of A1 has the same path as the emission arc with the first evaluation score of B1, both of which are path 1. The emission arc with the first evaluation score of A2 has the same path as the emission arc with the first evaluation score of B2, both of which are path 2. The emission arc with the first evaluation score of A3 has the same path as the emission arc with the first evaluation score of B2, both of which are path 2. The emission arc path is the same as the emission arc path with the first evaluation score of B3, which is path 3. The emission arc path with the first evaluation score of A4 is the same as the emission arc path with the first evaluation score of B4, which is path 4. The emission arc path with the first evaluation score of A5 is the same as the emission arc path with the first evaluation score of B5, which is path 5. Then determine the sum of A1 and B1 to obtain the first sum corresponding to path 1; determine the sum of A2 and B2 to obtain the first sum corresponding to path 2; determine the sum of A3 and B3 to obtain the first sum corresponding to path 3, and so on.

[0048] Then determine the ratio corresponding to each emission arc:

[0049] The ratio of the emission arc with the first evaluation score A1 is The ratio of the emission arc with the first evaluation score A2 is The ratio of the emission arc with the first evaluation score of A3 is The ratio of the emission arc with the first evaluation score of A4 is The ratio of the emission arc with the first evaluation score of A5 is The ratio of the emission arc with the first evaluation score B1 is The ratio of the emission arc with the first evaluation score of B2 is The ratio of the emission arc with the first evaluation score of B3 is The ratio of the emission arc with the first evaluation score of B4 is And the ratio of the emission arc with the first evaluation score of B5 is

[0050] After obtaining the ratio corresponding to each emission arc, the ratio can be directly used as the weight of each emission arc. That is, the weight of the emission arc with an evaluation value of A1 is The weight of the emission arc with evaluation value A2 is And so on.

[0051] That is to say, the sum of the weights of all emission arcs under the same path is 1.

[0052] In other embodiments, after obtaining the ratio corresponding to each emission arc, the ratio of each emission arc may be corrected to obtain a correction value corresponding to each emission arc, and then the weight of the emission arc may be set to the corresponding correction value.

[0053] The correction process may be to add the ratio of each emission arc to the corresponding preset value, or to multiply the ratio of each emission arc to the corresponding preset value, etc., which is not limited here.

[0054] Alternatively, in other embodiments, the process of determining the weight in step S130 may also include: adding the graph cost scores of all emission arcs under each path respectively to obtain the second sum value corresponding to each path; determining the ratio of the graph cost score of each emission arc to the second sum value of the corresponding path respectively; setting the weight of each emission arc to the ratio corresponding to the emission arc respectively, or, including: adding the acoustic cost scores of all emission arcs under each path respectively to obtain the third sum value corresponding to each path; determining the ratio of the acoustic cost score of each emission arc to the third sum value of the corresponding path respectively; setting the weight of each emission arc to the ratio corresponding to the emission arc respectively.

[0055] S140: Determine a second evaluation score for each path according to the first evaluation scores and weights of all emission arcs under each path.

[0056] After the above steps, each emission arc has a first evaluation score and a weight. Then, in step S140, the first evaluation scores of all emission arcs under each path are weighted and summed according to the corresponding weights to obtain the second evaluation score of each path.

[0057] Still using the above example to illustrate step S140:

[0058] calculate Get the second evaluation score of path 1; calculate Get the second evaluation score of path 2; calculate Get the second evaluation score of path 3; calculate Get the second evaluation score of path 4; calculate The second evaluation score of path 5 is obtained.

[0059] S150: Fuse all word graphs to obtain a fused word graph, and set the third evaluation score of the emission arc under each path in the fused word graph according to the second evaluation score of each path.

[0060] All word graphs are fused into one word graph. The process of word graph fusion belongs to the existing technology and will not be introduced in detail here.

[0061] In the fused word graph, each emission arc includes an input label and an output label, and for any emission arc in the fused word graph, an emission arc with the same path can be found in the word graph before fusion.

[0062] In this embodiment, step S150 includes: setting the third evaluation score of each emission arc in the fused word graph as the second evaluation score of the path corresponding to the emission arc.

[0063] Still using the above example to illustrate:

[0064] In the fused word graph, the third evaluation score of the emission arc whose path is path 1 is set to Set the third evaluation score of the emitting arc whose path is path 2 to Set the third evaluation score of the emitting arc whose path is path 3 to Set the third evaluation score of the launch arc whose path is path 4 to Set the third evaluation score of the launch arc whose path is path 5 to

[0065] In other embodiments, after obtaining the second evaluation score of each path, the second evaluation score of each path can be corrected to obtain the correction value corresponding to each path, and then the third evaluation score of the emitted arc in the fused word graph is set to the correction value of the corresponding path.

[0066] The correction process may be to add the second evaluation score of each path and the corresponding preset value, or to multiply the second evaluation score of each path and the corresponding preset value, etc., which is not limited here.

[0067] S160: Determine a recognition result of the speech signal to be recognized based on the fused word graph and the third evaluation score of the emission arc in the fused word graph.

[0068] After the above steps, the basic structure of the emission arc in the fused word graph includes the input label, the output label, and the third evaluation score. It can be understood that the basic structure of the emission arc in the word graph before fusion includes two scores: the graph cost score and the acoustic cost score, while the basic structure of the reflection arc in the fused word graph after fusion only includes the third evaluation score.

[0069] The Viterbi algorithm can be used to find the optimal path in the fused word graph, and the word sequence on the optimal path is obtained to obtain the recognition result of the speech signal to be recognized.

[0070] In this embodiment, by setting weights for the reflex arcs in each word graph, the distinction between the correct path and the confusing path in the fused word graph can be improved, thereby improving the accuracy and robustness of speech recognition.

[0071] See Figure 4 , Figure 4 2 is a schematic diagram of the structure of one embodiment of a speech recognition device of the present application. The speech recognition device 200 includes a processor 210, a memory 220, and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230, respectively. The memory 220 stores program data. The processor 210 executes the program data in the memory 220 to implement the steps of any of the above-mentioned embodiments. The detailed steps can be found in the above-mentioned embodiments and will not be repeated here.

[0072] The speech recognition device 200 can be any device with algorithm processing capabilities, such as a computer or a mobile phone, and is not limited here.

[0073] See Figure 5 , Figure 5 The speech recognition device 300 includes a feature recognition module 310 , a first determination module 320 , a second determination module 330 , a third determination module 340 , a word-graph fusion module 350 , and a speech recognition module 360 .

[0074] The feature recognition module 310 is used to recognize at least two different features of the speech signal to be recognized, and obtain at least two word graphs.

[0075] The first determination module 320 is connected to the feature recognition module 310 and is used to determine the first evaluation score of each emission arc in each word graph respectively, wherein the size relationship between the first evaluation scores of any two emission arcs in the same word graph is related to the size relationship between the probabilities of the corresponding output labels.

[0076] The second determination module 330 is connected to the feature recognition module 310 and is used to determine the weight of each emission arc in each word graph respectively, wherein in all word graphs, the size relationship between the weights of any two emission arcs under the same path is related to the size relationship between the probabilities of the corresponding output labels.

[0077] The third determination module 340 is connected to the first determination module 320 and the second determination module 330 , and is configured to determine the second evaluation score of each path according to the first evaluation scores and weights of all emission arcs under each path.

[0078] The word graph fusion module 350 is connected to the third determination module 340 and is used to fuse all word graphs to obtain a fused word graph, and set the third evaluation score of the emission arc under each path in the fused word graph according to the second evaluation score of each path.

[0079] The speech recognition module 360 is connected to the word graph fusion module 350 and is used to determine the recognition result of the speech signal to be recognized based on the fused word graph and the third evaluation score of the emission arc in the fused word graph.

[0080] In one embodiment, the second determination module 330 is specifically used to: add the first evaluation scores of the emission arcs of the same path in different word graphs to obtain the first sum value corresponding to each path; determine the ratio of the first evaluation score of each emission arc to the first sum value of the corresponding path; and determine the weight of each emission arc according to the ratio corresponding to each emission arc.

[0081] In one embodiment, the second determining module 330 is specifically configured to set the weight of each emission arc to a corresponding ratio.

[0082] In one embodiment, the first determining module 320 is configured to determine a first evaluation score for each emission arc in the corresponding word graph based on the graph cost scores and the acoustic cost scores of all emission arcs in each word graph.

[0083] In one embodiment, the first evaluation score is a minimum Bayesian risk assessment score.

[0084] In one embodiment, the third determining module 340 is specifically configured to set the third evaluation score of each emission arc in the fused word graph as the second evaluation score of the path corresponding to the emission arc.

[0085] In one embodiment, the third determining module 340 is specifically configured to perform weighted summation of the first evaluation scores of all emission arcs under each path according to corresponding weights to obtain the second evaluation score of each path.

[0086] In one embodiment, the feature recognition module 310 is specifically configured to: perform at least two different feature extractions on the speech signal to be recognized to obtain speech features of at least two different dimensions; and recognize the speech features of each dimension respectively to obtain at least two word graphs.

[0087] The speech recognition device 300 executes the steps of the speech recognition method in any of the above embodiments when working. The detailed steps can be found in the above embodiments and will not be repeated here.

[0088] The speech recognition device 300 can be any device with algorithm processing capabilities, such as a computer or a mobile phone, and is not limited here.

[0089] See Figure 6 , Figure 6 The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps of any of the above methods.

[0090] Among them, the computer-readable storage medium 400 can specifically be a device that can store the computer program 410, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the computer program 410. The server can send the stored computer program 410 to other devices for execution, or it can also run the stored computer program 410 itself.

[0091] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech recognition method, characterized in that: The method comprises: Recognizing at least two different features of the speech signal to be recognized to obtain at least two word graphs; Determine a first evaluation score for each emission arc in each word graph, respectively, wherein a magnitude relationship between the first evaluation scores of any two emission arcs in the same word graph is correlated with a magnitude relationship between probabilities of corresponding output labels; Determining the weight of each of the emission arcs in each of the word graphs, respectively, wherein, in all the word graphs, the magnitude relationship between the weights of any two emission arcs in the same path is correlated with the magnitude relationship between the probabilities of the corresponding output labels; Determine a second evaluation score for each path according to the first evaluation scores and the weights of all the emission arcs under each path; Fusing all of the word graphs to obtain a fused word graph, and setting a third evaluation score of the emission arc under each path in the fused word graph according to the second evaluation score of each path; A recognition result of the to-be-recognized speech signal is determined according to the fused word graph and the third evaluation score of the emission arc in the fused word graph.

2. The method according to claim 1, characterized in that The step of respectively determining the weight of each of the emission arcs in each of the word graphs comprises: Adding the first evaluation scores of the emission arcs of the same path in different word graphs to obtain a first sum value corresponding to each path; respectively determining a ratio of the first evaluation score of each of the emission arcs to the first sum value of the corresponding path; The weight of each of the emission arcs is determined according to the ratio corresponding to each of the emission arcs.

3. The method according to claim 2, characterized in that The step of determining the weight of each of the emission arcs according to the ratio corresponding to each of the emission arcs comprises: The weight of each of the emission arcs is set to the corresponding ratio.

4. The method according to claim 1, wherein The step of respectively determining a first evaluation score for each emission arc in each word graph comprises: The first evaluation score corresponding to each of the emission arcs in the word graph is determined based on the graph cost scores and the acoustic cost scores of all the emission arcs in each of the word graphs.

5. The method according to claim 4, characterized in that The first evaluation score is the minimum Bayesian risk assessment score.

6. The method according to claim 1, characterized in that The step of setting a third evaluation score of an emission arc under each path in the fused word graph according to the second evaluation score of each path includes: The third evaluation score of each of the emission arcs in the fused word graph is set as the second evaluation score of the path corresponding to the emission arc.

7. The method according to claim 1, characterized in that The step of determining the second evaluation score of each path according to the first evaluation scores and the weights of all the emission arcs under each path includes: The first evaluation scores of all the emission arcs under each path are weightedly summed according to the corresponding weights to obtain the second evaluation score of each path.

8. The method according to claim 1, characterized in that The step of identifying at least two different features of the speech signal to be recognized to obtain at least two word graphs includes: Performing at least two different feature extractions on the speech signal to be recognized to obtain speech features of at least two different dimensions; The speech features of each dimension are recognized respectively to obtain at least two word graphs.

9. A speech recognition device, characterized in that: The speech recognition device includes a processor, a memory and a communication circuit, the processor is coupled to the memory and the communication circuit respectively, the memory stores program data, and the processor implements the steps in the method according to any one of claims 1 to 8 by executing the program data in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images