A speech recognition system decoding method, system and storage medium

Abstract

The present invention provides a speech recognition system decoding method, system, and storage medium. The method includes preprocessing the speech data to be recognized to obtain a speech feature frame sequence; feeding the speech feature frame sequence into a trained deep neural network, randomly performing dropout on each layer of the deep neural network except the output layer using a dropout strategy, repeating this operation N times to obtain N different deep neural networks, thereby achieving sufficient sampling of the same deep neural network; performing a forward propagation process on the input speech feature frame sequence using the N different deep neural networks based on the dropout strategy to obtain forward calculation results of N different deep neural networks; feeding the forward calculation results into a decoder, and merging them using the synchronous merging unbiased decoding algorithm proposed by the present invention to obtain an unbiased decoded word graph. The present application can eliminate the bias of the decoding results of the speech recognition system, obtain high-quality unbiased decoding results, and improve the performance of the speech recognition system.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech recognition system decoding method, system and storage medium. Background Art

[0002] With the increasing application of deep learning technology, the performance of speech recognition systems has been significantly improved. In current speech recognition systems modeled using deep neural networks, whether using unidirectional recurrent neural networks, bidirectional recurrent neural networks, or convolutional neural networks, a single model is obtained after training. However, a single model can be viewed as a point estimate of the distribution formed by many possible candidate models. Therefore, a single model can introduce systematic biases, affecting the performance of the speech recognition system.

[0003] In speech recognition systems modeled using deep neural networks, using a dropout strategy, whether during model training or testing and inference, is equivalent to sampling the model set constructed by the deep neural network. The inventors of this application have discovered that using only one dropout strategy to obtain a single deep neural network model is equivalent to performing a biased point estimate on the model set, which will cause irreparable systematic errors during the decoding phase and degrade the performance of the speech recognition system. Summary of the Invention

[0004] In the existing speech recognition system that adopts deep neural network for modeling, only one Dropout strategy is adopted to obtain a single deep neural network model for sampling. The single model can be regarded as a point estimate of the distribution formed by many possible candidate models, which will cause irreparable system errors in the decoding stage. Therefore, the single model will bring about systematic deviations and degrade the performance of the speech recognition system. The present invention provides a speech recognition system decoding method.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A speech recognition system decoding method comprises the following steps:

[0007] S1. Preprocessing the speech data to be recognized to obtain a speech feature frame sequence;

[0008] S2. Feed the speech feature frame sequence into the trained deep neural network, use the Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks to achieve sufficient sampling of the same deep neural network;

[0009] S3, Monte Carlo sampling method based on Dropout strategy, N different deep neural networks perform forward propagation process on the input speech feature frame sequence, and obtain the forward calculation results of N different deep neural networks;

[0010] S4. The forward calculation results of the N different deep neural networks are sent to the decoder, and the forward calculation results of the N different deep neural networks are merged in the beam search step of the existing decoding using a synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation to obtain an unbiased decoding word graph. The basic idea of ​​the merging is: use the N forward calculation results to execute the N decoding processes according to the existing decoding algorithm respectively, the N decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rules based on the N decoding processes. The N+1th decoding process is the result of the fusion and merging of the previous N decoding processes, and the N+1th decoding process outputs the final decoding result.

[0011] Furthermore, in step S1, the speech data to be recognized is sequentially subjected to pre-emphasis, frame windowing, discrete Fourier transform, and Mel filter bank preprocessing operations to obtain a speech feature frame sequence.

[0012] Furthermore, the deep neural network trained in step S2 is selected from at least one of CNN, LSTM, RNN, TDNN, and Transformer neural network structures.

[0013] Furthermore, the first, second, ..., and Nth decoding processes of the N-path decoding process in step S4 are executed synchronously and independently, and all include the following identical search and pruning steps:

[0014] S41. At time t, the existing decoding program traverses the forward links of the tokens in the token list from time t-1 to time t, and calculates the difference between the corresponding value of the forward link arc corresponding to the token and the forward link list pruning threshold according to the set forward link list pruning threshold. Then, the difference is compared with the forward link list pruning threshold. If the difference is greater than the forward link list pruning threshold, the forward link of the token is pruned. If the difference is less than or equal to the forward link list pruning threshold, the forward link of the token is retained and the existing difference value of the token is updated with the difference value, so that the token records the lowest difference value, thereby completing the forward link pruning at time t-1.

[0015] S42. After completing the forward link pruning at time t-1, the existing decoding program traverses the token list at the current time, i.e., time t, and determines whether the total loss value of the optimal path in the token list to reach the token is greater than the token list pruning threshold according to a preset token list pruning threshold. If so, the token is pruned.

[0016] S43. At time t-1, a beam search is performed on the token list of the current frame. Based on the pruning results of multiple token lists at time t-1, the token lists of the current time step t are merged, and the tot_cost average value of the tokens of the current frame is calculated during the merging process. The tot_cost value is calculated using the following formula:

[0017] tot_cost=cur_cost+ac_cost+graph_cost

[0018] Among them, cur_cost represents the total loss value of the forward link from a token at the previous moment, i.e., time t-1, to the token at the current moment; ac_cost represents the acoustic score of the token at the current moment, i.e., time t; graph_cost represents the value corresponding to the forward link arc from a token at time t-1 to the current token at time t. The calculation method of graph_cost value is provided in the common decoding algorithm in the industry;

[0019] S44, N decoding processes are executed synchronously and independently, all processing the same frame of voice data synchronously, and the tokens in the token chain list are also one-to-one corresponding, and each token chain list will save the lowest tot_cost and the path information corresponding to the lowest tot_cost; according to the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation, based on the first N decoding processes, a virtual N+1th decoding process is created through the merging rule composed of the token merging rule and the forward link merging rule; wherein,

[0020] The token merging rules for the 1st to Nth decoding process are as follows:

[0021] ①, the sum of the tot_cost of each corresponding token in the token list of the current frame t-1 of the 1st to Nth decoding is divided by N to get the average value, and at the same time, the corresponding token is created for the N+1th decoding, and the average value is used as the tot_cost of the token created for the N+1th decoding;

[0022] ②, the sum of the pruning thresholds of the token list of the current frame t-1 from the 1st to the Nth decoding is divided by N to obtain the average pruning threshold of the N-way token list. This average is added to the lowest tot_cost value stored in the token list corresponding to the N+1th decoding, and is used as the token pruning threshold of the token list at the t-1 moment of the frame in the N+1th decoding.

[0023] The forward link merging rules in the token list of the 1st to Nth decoding process are as follows:

[0024] For each token in the token list at time t-1, the existing decoding program will traverse its forward links and calculate the difference link_extra_cost between each forward link and the tot_cost of the optimal path at that time. If the difference link_extra_cost is greater than lattice_beam, the forward link will be pruned. Otherwise, token->extra_cost is defined as the minimum link_extra_cost of all forward links of the token, and the forward link will be retained. If all forward links are pruned, token->extra_cost will be set to infinity. In the next step, token->extra_cost will be used to determine whether the token has a forward link.

[0025] S45. The tokens and their pruning thresholds corresponding to each frame data in the N+1th decoding process, as well as the forward links and their pruning thresholds between the tokens, are all obtained by merging the tokens and their pruning thresholds, the forward links and their pruning thresholds in the synchronously and independently executed 1st to Nth decoding processes, and the N+1th decoding process performs beam search and backtracking of the optimal path according to the existing decoding process. The decoding result corresponding to the obtained speech signal is the output result of the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation.

[0026] Furthermore, in the step S44 at time t-1, during the synchronous merging unbiased decoding process of the 1st to Nth paths, the merging of the forward links of the merged tokens has the following conditions:

[0027] ①, At the same time, token pruning starts from the first token in the token chain list of the 1st to the Nth path. In the decoding process, after step S42, the first {i1, i2, ..., i k} path is retained, and the corresponding tokens are pruned during the decoding process of the remaining paths; the token pruning process of the first token in the token chain list from the 1st path to the Nth path is as described above, and the process is repeated until the token pruning process of the last token in the token chain list from the 1st path to the Nth path is completed; where {i1, i2, ..., i k} is any subset of {i|i≥1 and i≤N, i is a natural number};

[0028] Copy the forward link of the retained token to the corresponding token in the N+1th decoding as the forward link of the token in the N+1th decoding, and take the average value of the weight of each forward link;

[0029] ② During the decoding process, the tokens corresponding to the 1st to Nth paths are retained.

[0030] The weights of each forward link of the retained tokens in the 1st to Nth decoding are summed and divided by N to obtain an average value, which is used as the forward link value of the corresponding token in the N+1th decoding; the forward link pruning thresholds of the tokens in the 1st to Nth decoding are summed and divided by N to obtain an average value, which is added to the lowest tot_cost value stored in the N+1th token chain list, and the resulting sum is used as the forward link pruning threshold of the N+1th token;

[0031] ③、The corresponding tokens from the 1st to the Nth path are all cut off

[0032] This token has no forward link in the N+1th decoding.

[0033] The present invention also provides a speech recognition system decoding system, comprising a feature extraction module, a multi-channel sampling module, a forward calculation result acquisition module and a synchronous merging unbiased decoding module; wherein,

[0034] The feature extraction module is configured to: perform a preprocessing operation on the speech data to be recognized to obtain a speech feature frame sequence;

[0035] The multi-channel sampling module is configured to: feed the speech feature frame sequence into a trained deep neural network, use a Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks, thereby achieving sufficient sampling of the same deep neural network;

[0036] The forward calculation result acquisition module is configured to: perform a forward propagation process on the input speech feature frame sequence using a Monte Carlo sampling method based on a Dropout strategy, using N different deep neural networks to obtain forward calculation results of N different deep neural networks;

[0037] The synchronous merging unbiased decoding module is configured to: send the forward calculation results of N different deep neural networks into the decoder, and use the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation to merge the forward calculation results of N different deep neural networks in the beam search step of the existing decoding to obtain an unbiased decoding word graph, wherein the basic idea of ​​merging is: respectively use the N-path forward calculation results to execute N-path decoding processes according to the existing decoding algorithm, the N-path decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rules based on the N-path decoding processes, the N+1th decoding process is the result of the fusion and merging of the previous N-path decoding processes, and the N+1th decoding process outputs the final decoding result.

[0038] The present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned speech recognition system decoding method is implemented.

[0039] Compared with the existing technology, the speech recognition system decoding method, system and storage medium provided by the present invention send the speech feature frame sequence into a trained deep neural network, use the Monte Carlo sampling method based on the Dropout strategy to achieve sufficient sampling of the same deep neural network by multi-channel Dropout, and send the forward calculation results of N different deep neural networks obtained into the decoder, and use the synchronous merging unbiased decoding algorithm based on multi-channel Dropout forward calculation to merge them. In this way, on the basis of eliminating the deviation of the decoding results of the speech recognition system, the decoding results can integrate the advantages of multiple candidate deep neural network models, thereby improving the performance of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of the decoding method of the speech recognition system provided by the present invention.

[0041] Figure 2 This is a schematic diagram of the multi-path Dropout forward calculation synchronous merging unbiased decoding technology route provided by the present invention.

[0042] Figure 3 This is a flow chart of a synchronous merging unbiased decoding algorithm based on multi-path dropout forward calculation provided by the present invention.

[0043] Figure 4 Schematic diagram of the synchronous merging Dropout decoding process provided by the present invention.

[0044] Figure 5 It is a basic structural diagram of the speech recognition system decoding system provided by the present invention.

[0045] Figure 6 This is the framework diagram of the existing DL-HMM hybrid acoustic model speech recognition system.

[0046] Figure 7 This is the framework diagram of the existing end-to-end speech recognition system. DETAILED DESCRIPTION

[0047] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below with reference to specific illustrations.

[0048] Please refer to Figure 1 and Figure 2 As shown, the present invention provides a speech recognition system decoding method, comprising the following steps:

[0049] S1. Preprocessing the speech data to be recognized to obtain a speech feature frame sequence;

[0050] S2. Feed the speech feature frame sequence into the trained deep neural network, use the Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks to achieve sufficient sampling of the same deep neural network;

[0051] S3, Monte Carlo sampling method based on Dropout strategy, N different deep neural networks perform forward propagation process on the input speech feature frame sequence, and obtain forward calculation results of multiple N different deep neural networks;

[0052] S4. The forward calculation results of the N different deep neural networks are sent to the decoder, and the forward calculation results of the N different deep neural networks are merged in the beam search step of the existing decoding using a synchronous merging decoding algorithm based on multi-path Dropout forward calculation to obtain an unbiased decoding word graph. The basic idea of ​​the merging is: use the N forward calculation results to execute the N decoding processes according to the existing decoding algorithm respectively, the N decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rules based on the N decoding processes. The N+1th decoding process is the result of the fusion and merging of the previous N decoding processes, and the N+1th decoding process outputs the final decoding result.

[0053] As a specific embodiment, in step S1, the speech data to be recognized is sequentially subjected to pre-emphasis, frame windowing, discrete Fourier transform, and Mel filter bank preprocessing operations to obtain a speech feature frame sequence. The specific processing operations of pre-emphasis, frame windowing, discrete Fourier transform, and Mel filter bank are well known to those skilled in the art and are not further described here.

[0054] As a specific embodiment, the deep neural network trained in step S2 belongs to the existing technology and can be various deep neural network structures available in the field of deep learning. Specifically, it can be selected from at least one of the existing CNN, LSTM, RNN, TDNN, and Transformer neural network structures, that is, it can be selected from a single neural network among CNN, LSTM, RNN, TDNN, and Transformer neural network structures, or it can be selected from a hybrid neural network among CNN, LSTM, RNN, TDNN, and Transformer neural network structures, such as a hybrid neural network such as TDNN-LSTM.

[0055] As a specific embodiment, the first, second, ..., and Nth decoding processes of the N-path decoding process in step S4 are executed synchronously and independently, and all include the following identical search and pruning steps:

[0056] S41. At time t, the existing decoding program traverses the forward link (forward link) of the token (token) in the token list from time t-1 to time t, and calculates the difference (link_extra_cost) between the corresponding value of the forward link arc corresponding to the token and the forward link list pruning threshold (lattice_beam) according to the set forward link list pruning threshold (lattice_beam). Then, the difference is compared with the forward link list pruning threshold. If the difference is greater than the forward link list pruning threshold, the forward link of the token is pruned. If the difference is less than or equal to the forward link list pruning threshold, the forward link of the token is retained, and the existing difference (link_extra_cost) of the token is updated with the difference, so that the token records the lowest difference (link_extra_cost), thereby completing the forward link pruning at time t-1;

[0057] S42. After completing the forward link pruning at time t-1, the existing decoding program traverses the token list at the current time, i.e., time t, and determines whether the total cost value (tot_cost) of the optimal path in the token list to reach the token is greater than the token list pruning threshold according to a preset token list pruning threshold. If so, the token is pruned.

[0058] S43. At time t-1, a beam search is performed on the token list of the current frame. Based on the pruning results of multiple token lists at time t-1, the token lists of the current time step t are merged, and the tot_cost average value of the tokens of the current frame is calculated during the merging process. The tot_cost value is calculated using the following formula:

[0059] tot_cost=cur_cost+ac_cost+graph_cost

[0060] Among them, cur_cost represents the total loss value of the forward link from a token at the previous moment, i.e., time t-1, to the token at the current moment; ac_cost represents the acoustic score of the token at the current moment, i.e., time t; graph_cost represents the value corresponding to the forward link arc from a token at time t-1 to the current token at time t. The calculation method of graph_cost value is provided in the common decoding algorithm in the industry;

[0061] S44, N decoding processes are executed synchronously and independently, all processing the same frame of voice data synchronously, and the tokens in the token chain list are also one-to-one corresponding, and each token chain list will save the lowest tot_cost and the path information corresponding to the lowest tot_cost; according to the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation, based on the first N decoding processes, a virtual N+1th decoding process is created through the merging rule composed of the token merging rule and the forward link merging rule; wherein,

[0062] The token merging rules for the 1st to Nth decoding process are as follows:

[0063] ①, the sum of the tot_cost of each corresponding token in the token list of the current frame t-1 of the 1st to Nth decoding is divided by N to get the average value, and at the same time, the corresponding token is created for the N+1th decoding, and the average value is used as the tot_cost of the token created for the N+1th decoding;

[0064] ②, the sum of the pruning thresholds of the token list of the current frame t-1 from the 1st to the Nth decoding is divided by N to obtain the average pruning threshold of the N-way token list. This average is added to the lowest tot_cost value stored in the token list corresponding to the N+1th decoding, and is used as the token pruning threshold of the token list at the t-1 moment of the frame in the N+1th decoding.

[0065] The forward link merging rules in the token list of the 1st to Nth decoding process are as follows:

[0066] For each token in the token list at time t-1, the existing decoding program will traverse its forward links and calculate the difference link_extra_cost between each forward link and the tot_cost of the optimal path at that time. If the difference link_extra_cost is greater than lattice_beam, the forward link will be pruned. Otherwise, token->extra_cost is defined as the minimum link_extra_cost of all forward links of the token, and the forward link will be retained. If all forward links are pruned, token->extra_cost will be set to infinity. In the next step, token->extra_cost will be used to determine whether the token has a forward link.

[0067] S45. The tokens and their pruning thresholds corresponding to each frame data in the N+1th decoding process, as well as the forward links and their pruning thresholds between the tokens, are all obtained by merging the tokens and their pruning thresholds, the forward links and their pruning thresholds in the synchronously and independently executed 1st to Nth decoding processes, and the N+1th decoding process performs beam search and backtracking of the optimal path according to the existing decoding process. The decoding result corresponding to the obtained speech signal is the output result of the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation.

[0068] As a specific embodiment, in step S44 at time t-1, during the synchronous merging unbiased decoding process from the 1st path to the Nth path, the merging of the forward links of the merged tokens has the following conditions:

[0069] ①, At the same time, token pruning starts from the first token in the token chain list of the 1st to the Nth path. In the decoding process, after step S42, the first {i1, i2, ..., i k} path is retained, and the corresponding tokens are pruned during the decoding process of the remaining paths; the token pruning process of the first token in the token chain list from the 1st path to the Nth path is as described above, and the process is repeated until the token pruning process of the last token in the token chain list from the 1st path to the Nth path is completed; where {i1, i2, ..., i k} is any subset of {i|i≥1 and i≤N, i is a natural number};

[0070] The forward link of the retained token is copied to the corresponding token in the N+1th decoding as the forward link of the token in the N+1th decoding, and the weight of each forward link is averaged.

[0071] ② During the decoding process, the tokens corresponding to the 1st to Nth paths are retained.

[0072] The weights of each forward link of the retained tokens in the 1st to Nth decoding are summed up and divided by N to obtain the average value, which is used as the value of the forward link of the corresponding token in the N+1th decoding; the forward link pruning thresholds of the tokens in the 1st to Nth decoding are summed up and divided by N to obtain the average value, which is added to the lowest tot_cost value stored in the N+1th token list, and the resulting sum is used as the forward link pruning threshold of the N+1th token.

[0073] ③、The corresponding tokens from the 1st to the Nth path are all cut off

[0074] This token has no forward link in the N+1th decoding.

[0075] The method for using the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation in a speech recognition system provided by the present invention is as follows: Figure 3 As shown, before using the voice data, the voice signal needs to be preprocessed to obtain the characteristics of the voice frame signal, which is the input of the deep neural network. After steps S2 and S3, the forward calculation result of the voice frame at time t is obtained, as shown in the red dotted box. In the constructed WFST decoding graph, the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation provided by the present invention involves ac_cost, cur_cost, and tot_cost, wherein the multi-path Dropout forward calculation results are synchronously merged according to tot_cost. In addition, the optimal decoding path can also be generated according to tot_cost to complete the decoding task.

[0076] In order to better understand the synchronous merging unbiased decoding process scheme based on multi-way Dropout forward calculation, the following takes the synchronous merging unbiased decoding process based on 2-way Dropout forward calculation as an example to explain in detail. The synchronous merging unbiased decoding algorithm based on 2-way Dropout forward calculation specifically includes:

[0077] Figure 4 The synchronous merging unbiased decoding algorithm based on two-way dropout forward computation shown in the figure includes two layers of pruning and merging. Time t determines whether the token at the previous time, time t-1, can continue to be retained. If the tot_cost value of a token in the token list at time t-1 is greater than the tot_cost value of the optimal token, and the loss difference is greater than the token pruning threshold cur_cutoff at the current time, then the token will not be passed on (this path is too costly). Time t determines whether the token retained at time t-1 can pass information to the current time along the path constraints in the decoding graph. This step requires estimating the optimal weight at the current time. If the token transferred along a forward link is lower than the estimated optimal weight, the token is not allowed to pass on. The threshold for the first layer of pruning is named cur_cutoff, and the threshold for the second layer of pruning is named next_cutoff.

[0078] For next_cutoff, it is initialized according to the industry's mature decoding algorithm. Then, during the second-level pruning of the token at time t-1, if the cost generated by the token transfer (the value is tot_cost + adaptive_beam) is greater than next_cutoff, the forward link will not be passed. Otherwise, a corresponding new token will be generated at time t, and the next_cutoff value will be updated at the same time.

[0079] like Figure 4As shown in the figure, at time t-1, a cluster search is performed on the token list of the current frame, where the yellow nodes in the figure represent the tokens retained after pruning, and the solid arrows represent the forward links retained after pruning; the gray-black grid nodes represent the tokens deleted after pruning, and the dotted arrows represent the forward links deleted after pruning. Figure 4 As shown in the figure, based on the pruning results of multiple token lists at time t-1, the token lists of the current time step t are merged, and the tot_cost average of the current frame token is calculated during the merging process. The tot_cost is calculated using the following formula:

[0080] tot_cost=cur_cost+ac_cost+graph_cost

[0081] Among them, cur_cost represents the total loss value from a certain token at the previous moment, i.e., time t-1, along the corresponding forward link to the token at the current moment; ac_cost represents the acoustic score of the token at the current moment, i.e., time t; graph_cost represents the value corresponding to the forward link arc from a certain token at time t-1 to the current token at time t. The calculation method of the graph_cost value is provided in the common decoding algorithm process in the industry.

[0082] The optimal path is generated during the decoding process based on tot_cost to complete the decoding task. Tokens pruned from the previous frame are no longer included in the forward link list merge process for the current frame. The tokens that remain after pruning preserve their link information through forward links. For example, there is a forward link between the third token in the token list at time t-1 and the first token in the token list at time t. This link stores the input and output labels of the transition arc between the corresponding state nodes of these two tokens, as well as the value (cost) corresponding to the transition arc and the acoustic score for the current frame at time t.

[0083] Figure 4 As described above, the second path process in is executed synchronously with the first path process. At the same time, during the first and second path synchronous execution processes, since they are the same frame, the tokens in their token lists are one-to-one corresponding, and each token list will save the lowest tot_cost and the path information corresponding to the lowest tot_cost; according to the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation, according to the first and second path decoding processes, a virtual third path decoding process is created through the merging rule composed of token merging rule and forward link merging rule. The third path decoding process is the result of the fusion and merging of the first and second path decoding processes; wherein,

[0084] The token merging rules for the first and second decoding processes are as follows:

[0085] ①. For the first route, add the tot_cost of each token in the token list at time t-1 of the current frame to the tot_cost of each token in the token list at time t-1 of the current frame in the second route, then divide by 2 to get the average value. At the same time, create the corresponding token for the third route decoding, and use this average value as the tot_cost of the token created by the third route decoding.

[0086] ②. At time t-1 of the current frame in the first path, the pruning threshold in the token list is added to the pruning threshold at time t-1 of the current frame in the second path, and then divided by 2 to get the average value. The average value is added to the lowest tot_cost value stored in the token list token corresponding to the third path decoding, and the value is used as the pruning threshold of the token list token at time t-1 of the frame in the third path decoding.

[0087] The forward link merging rules in the token lists of the 1st and 2nd decoding processes are as follows:

[0088] For each token in the token list at time t-1, the existing decoding process traverses its forward links and calculates the difference between each forward link and the tot_cost of the optimal path at that time, link_extra_cost. If the difference link_extra_cost is greater than lattice_beam, the forward link is pruned. Otherwise, token->extra_cost is defined as the minimum link_extra_cost of all forward links for the token, and the forward link is retained. If all forward links are pruned, token->extra_cost is set to infinity. In the next step, token->extra_cost is used to determine whether the token has a forward link.

[0089] At time t-1, during the unbiased decoding process of the first and second paths, the forward link of the merged tokens is merged as follows:

[0090] ① The token in the first path is retained, and the corresponding token in the second path is cut off

[0091] The forward link of the retained token in the first path is copied to the corresponding token in the merged third path decoding as the forward link of the token in the third path decoding.

[0092] ② The token in the second path is retained, and the corresponding token in the first path is cut off

[0093] The forward link of the retained token in the second path is copied to the corresponding token in the merged third path decoding as the forward link of the token in the third path decoding.

[0094] ③. The corresponding tokens in the 1st and 2nd routes are retained.

[0095] The weight of each forward link of the retained token in the first path is added to the corresponding forward link weight of the retained token in the second path, and then divided by 2 to obtain the average value, which is used as the corresponding forward link value of the token in the merged third path decoding; the forward link pruning threshold of the third path token is: the forward link pruning threshold of the first path token plus the forward link pruning threshold of the second path token, and then divided by 2 to obtain the average value, and the average value is added to the lowest tot_cost value stored in the third path token list, and the resulting sum is used as the forward link pruning threshold of the third path token.

[0096] ④. The corresponding tokens in the 1st and 2nd routes are cut off.

[0097] There is no forward link for this token in the 3rd decoding after merging.

[0098] After the two-way Dropout decoding is merged, independent token list pruning and forward link list pruning are also performed synchronously on the merged decoding results. The pruned tokens in the previous frame will no longer participate in the merging of the forward link list of the current frame.

[0099] Please refer to Figure 5 As shown, the present invention also provides a speech recognition system decoding system, including a feature extraction module, a multi-channel sampling module, a forward calculation result acquisition module and a synchronous merging unbiased decoding module; wherein,

[0100] The feature extraction module is configured to: perform a preprocessing operation on the speech data to be recognized to obtain a speech feature frame sequence;

[0101] The multi-channel sampling module is configured to: feed the speech feature frame sequence into a trained deep neural network, use a Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks, thereby achieving sufficient sampling of the same deep neural network;

[0102] The forward calculation result acquisition module is configured to: perform a forward propagation process on the input speech feature frame sequence using a Monte Carlo sampling method based on a Dropout strategy, using N different deep neural networks to obtain forward calculation results of N different deep neural networks;

[0103] The synchronous merging unbiased decoding module is configured to: send the forward calculation results of N different deep neural networks into the decoder, and use the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation to merge the forward calculation results of N different deep neural networks in the beam search step of the existing decoding to obtain an unbiased decoding word graph, wherein the basic idea of ​​merging is: respectively use the N-path forward calculation results to execute N-path decoding processes according to the existing decoding algorithm, the N-path decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rules based on the N-path decoding processes, the N+1th decoding process is the result of the fusion and merging of the previous N-path decoding processes, and the N+1th decoding process outputs the final decoding result.

[0104] As a specific embodiment, the speech recognition system can be as follows Figure 6 The existing DL-HMM hybrid acoustic model speech recognition system shown can also be as follows Figure 7 The existing end-to-end speech recognition system shown. Whether it is a DL-HMM hybrid acoustic model speech recognition system or an end-to-end speech recognition system, the speech recognition system decoding method of the present application can eliminate the bias of the speech recognition system decoding results, so that the fusion process of Dropout prediction posterior probability runs through the beam search step of decoding, obtaining high-quality unbiased decoding results, improving the performance of the speech recognition system, and can be applied in various speech recognition scenarios.

[0105] The present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned speech recognition system decoding method is implemented.

[0106] Compared with the existing technology, the speech recognition system decoding method, system and storage medium provided by the present invention send the speech feature frame sequence into a trained deep neural network, use the Monte Carlo sampling method based on the Dropout strategy to achieve sufficient sampling of the same deep neural network by multi-channel Dropout, and send the forward calculation results of N different deep neural networks obtained into the decoder, and use the synchronous merging unbiased decoding algorithm based on multi-channel Dropout forward calculation to merge them. In this way, on the basis of eliminating the deviation of the decoding results of the speech recognition system, the decoding results can integrate the advantages of multiple candidate deep neural network models, thereby improving the performance of the speech recognition system.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A decoding method for a speech recognition system, characterized in that: The following steps are involved: S1. Preprocessing the speech data to be recognized to obtain a speech feature frame sequence; S2. Feed the speech feature frame sequence into the trained deep neural network, use the Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks to achieve sufficient sampling of the same deep neural network; S3, Monte Carlo sampling method based on Dropout strategy, N different deep neural networks perform forward propagation process on the input speech feature frame sequence, and obtain the forward calculation results of N different deep neural networks; S4. The forward calculation results of the obtained N different deep neural networks are sent to the decoder, and the forward calculation results of the N different deep neural networks are merged in the beam search step of the existing decoding using a synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation to obtain an unbiased decoding word graph. The basic idea of ​​the merging is: respectively use the N forward calculation results to execute the N decoding processes according to the existing decoding algorithm, the N decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rules based on the N decoding processes. The N+1th decoding process is the result of the fusion and merging of the previous N decoding processes, and the N+1th decoding process outputs the final decoding result; wherein, The token merging rules for the 1st to Nth decoding process include: ①, the sum of the tot_cost of each corresponding token in the token list of the current frame t-1 of the 1st to Nth decoding is divided by N to get the average value, and the corresponding token is created for the N+1th decoding at the same time, and the average value is used as the tot_cost of the token created by the N+1th decoding. The tot_cost value is calculated using the following formula: tot_cost=cur_cost+ac_cost+graph_cost Among them, cur_cost represents the total loss value from a token at the previous moment, i.e., time t-1, along the corresponding forward link to the token at the current moment; ac_cost represents the acoustic score of the token at the current moment, i.e., time t; graph_cost represents the value corresponding to the forward link arc from a token at time t-1 to the current token at time t.

2. The speech recognition system decoding method according to claim 1, characterized in that: In step S1, the speech data to be recognized is sequentially subjected to pre-emphasis, frame windowing, discrete Fourier transform, and Mel filter bank preprocessing operations to obtain a speech feature frame sequence.

3. The speech recognition system decoding method according to claim 1, characterized in that: The deep neural network trained in step S2 is selected from at least one of CNN, LSTM, RNN, TDNN, and Transformer neural network structures.

4. The speech recognition system decoding method according to claim 1, characterized in that: In step S4, the first, second, ..., and Nth decoding processes of the N-path decoding process are executed synchronously and independently, and all include the following identical search and pruning steps: S41. At time t, the existing decoding program traverses the forward links of the tokens in the token list from time t-1 to time t, and calculates the difference between the corresponding value of the forward link arc corresponding to the token and the forward link list pruning threshold according to the set forward link list pruning threshold. Then, the difference is compared with the forward link list pruning threshold. If the difference is greater than the forward link list pruning threshold, the forward link of the token is pruned. If the difference is less than or equal to the forward link list pruning threshold, the forward link of the token is retained and the existing difference value of the token is updated with the difference value, so that the token records the lowest difference value, thereby completing the forward link pruning at time t-1. S42. After completing the forward link pruning at time t-1, the existing decoding program traverses the token list at the current time, i.e., time t, and determines whether the total loss value of the optimal path in the token list to reach the token is greater than the token list pruning threshold according to a preset token list pruning threshold. If so, the token is pruned. S43. At time t-1, a beam search is performed on the token list of the current frame. Based on the pruning results of multiple token lists at time t-1, the token lists of the current time step t are merged, and the average tot_cost of the tokens of the current frame is calculated during the merging process. S44, N decoding processes are executed synchronously and independently, all processing the same frame of voice data synchronously, and the tokens in the token chain list are also one-to-one corresponding, and each token chain list will save the lowest tot_cost and the path information corresponding to the lowest tot_cost; according to the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation, based on the first N decoding processes, a virtual N+1th decoding process is created through the merging rule composed of the token merging rule and the forward link merging rule; wherein, The token merging rules for the 1st to Nth decoding process also include: ②, the sum of the pruning thresholds of the token list of the current frame t-1 from the 1st to the Nth decoding is divided by N to obtain the average pruning threshold of the N-way token list. This average is added to the lowest tot_cost value stored in the token list corresponding to the N+1th decoding, and is used as the token pruning threshold of the token list at the t-1 moment of the frame in the N+1th decoding. The forward link merging rules in the token list of the 1st to Nth decoding process are as follows: For each token in the token list at time t-1, the existing decoding program will traverse its forward links and calculate the difference link_extra_cost between each forward link and the tot_cost of the optimal path at that time. If the difference link_extra_cost is greater than lattice_beam, the forward link will be pruned. Otherwise, token->extra_cost is defined as the minimum link_extra_cost of all forward links of the token, and the forward link will be retained. If all forward links are pruned, token->extra_cost will be set to infinity. In the next step, token->extra_cost will be used to determine whether the token has a forward link. S45. The tokens and their pruning thresholds corresponding to each frame data in the N+1th decoding process, as well as the forward links and their pruning thresholds between the tokens, are all obtained by merging the tokens and their pruning thresholds, the forward links and their pruning thresholds in the synchronously and independently executed 1st to Nth decoding processes, and the N+1th decoding process performs beam search and backtracking of the optimal path according to the existing decoding process. The decoding result corresponding to the obtained speech signal is the output result of the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation.

5. The speech recognition system decoding method according to claim 4, characterized in that: In step S44, at time t-1, during the synchronous merging unbiased decoding process from the 1st path to the Nth path, the merging of the forward links of the merged tokens has the following conditions: ①, At the same time, token pruning starts from the first token in the token chain list of the 1st to the Nth path. In the decoding process, after step S42, the first {i1, i2, ..., i k } path is retained, and the corresponding tokens are pruned during the decoding process of the remaining paths; the token pruning process of the first token in the token chain list from the 1st path to the Nth path is as described above, and the process is repeated until the token pruning process of the last token in the token chain list from the 1st path to the Nth path is completed; where {i1, i2, ..., i k } is any subset of {i|i≥1 and i≤N, i is a natural number}; Copy the forward link of the retained token to the corresponding token in the N+1th decoding as the forward link of the token in the N+1th decoding, and take the average value of the weight of each forward link; ② During the decoding process, the tokens corresponding to the 1st to Nth paths are retained. The weights of each forward link of the retained tokens in the 1st to Nth decoding are summed and divided by N to obtain an average value, which is used as the forward link value of the corresponding token in the N+1th decoding; the forward link pruning thresholds of the tokens in the 1st to Nth decoding are summed and divided by N to obtain an average value, which is added to the lowest tot_cost value stored in the N+1th token chain list, and the resulting sum is used as the forward link pruning threshold of the N+1th token; ③、The corresponding tokens from the 1st to the Nth path are all cut off This token has no forward link in the N+1th decoding.

6. A speech recognition system decoding system, characterized in that: It includes feature extraction module, multi-channel sampling module, forward calculation result acquisition module and synchronous merging unbiased decoding module; among them, The feature extraction module is configured to: perform a preprocessing operation on the speech data to be recognized to obtain a speech feature frame sequence; The multi-channel sampling module is configured to: feed the speech feature frame sequence into a trained deep neural network, use a Monte Carlo sampling method based on the Dropout strategy, randomly perform Dropout on each layer of the deep neural network except the output layer, repeat this operation N times, and obtain N different deep neural networks, thereby achieving sufficient sampling of the same deep neural network; The forward calculation result acquisition module is configured to: perform a forward propagation process on the input speech feature frame sequence using a Monte Carlo sampling method based on a Dropout strategy, using N different deep neural networks to obtain forward calculation results of N different deep neural networks; The synchronous merging unbiased decoding module is configured to: send the forward calculation results of N different deep neural networks obtained into the decoder, and use the synchronous merging unbiased decoding algorithm based on multi-path Dropout forward calculation to merge the forward calculation results of N different deep neural networks in the existing decoding beam search step to obtain an unbiased decoding word graph, wherein the basic idea of ​​merging is: respectively use the N-path forward calculation results to execute N-path decoding processes according to the existing decoding algorithm, the N-path decoding processes are executed synchronously and independently, and the N+1th virtual decoding process is created according to the merging rule based on the N-path decoding process, the N+1th decoding process is the result of the fusion and merging of the previous N-path decoding processes, and the N+1th decoding process outputs the final decoding result; wherein, The token merging rules for the 1st to Nth decoding process include: ①, the sum of the tot_cost of each corresponding token in the token list of the current frame t-1 of the 1st to Nth decoding is divided by N to get the average value, and the corresponding token is created for the N+1th decoding at the same time, and the average value is used as the tot_cost of the token created by the N+1th decoding. The tot_cost value is calculated using the following formula: tot_cost=cur_cost+ac_cost+graph_cost Among them, cur_cost represents the total loss value from a token at the previous moment, i.e., time t-1, along the corresponding forward link to the token at the current moment; ac_cost represents the acoustic score of the token at the current moment, i.e., time t; graph_cost represents the value corresponding to the forward link arc from a token at time t-1 to the current token at time t.

7. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the speech recognition system decoding method according to any one of claims 1 to 5 is implemented.

Images