Speech recognition model output post-processing method, speech recognition method and device

Abstract

The application provides a speech recognition model output post-processing method, a speech recognition method and device. A probability vector output by a speech recognition model is determined frame by frame to determine a keyword label meeting a requirement, the number of valid frames for each keyword label is counted, and an optimal label is determined from the keyword labels according to a preset counting threshold value and a recognition result is fed back. In multi-keyword recognition, the method can filter out incorrect prediction results mixed in the prediction results of continuous multiple speech frames, improve the continuity and accuracy of the prediction results of the speech recognition model, and the algorithm is simple.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a post-processing method of a speech recognition model, a speech recognition method and device. BACKGROUND

[0002] Keyword recognition is widely used in edge devices with relatively small computing power and memory space to realize relatively simple speech recognition functions such as wake-up word recognition and command word recognition. Due to the influence of computing resources, the size of the recognition model is limited, which leads to a certain degree of decline in the prediction accuracy of the model, and this influence is more obvious in streaming recognition. Therefore, an effective post-processing method is needed to improve the performance of keyword recognition algorithms.

[0003] Therefore, it is necessary to provide an improved technical solution to overcome the above technical problems in the prior art. SUMMARY

[0004] To solve the above technical problems, the present application provides a post-processing method of a speech recognition model, a speech recognition method and device, which can solve the problem of discontinuous prediction results of small computing power devices and the inability to select the optimal label in multi-keyword recognition.

[0005] According to the first aspect of the present application, a post-processing method of a speech recognition model is provided, comprising:

[0006] determining the maximum probability in a probability vector output by the speech recognition model and the classification label corresponding to the maximum probability frame by frame, wherein the probability vector represents the corresponding probability of the speech frame signal to be processed being recognized as a plurality of classification labels, and the plurality of classification labels include a plurality of keyword labels corresponding to a plurality of keywords;

[0007] counting the number of valid frames of each keyword label in the plurality of keyword labels as an optimal label, wherein each keyword label participates in the counting process when the respective probability is the maximum probability in the probability vector and is greater than a preset trigger threshold;

[0008] determining an optimal label from the plurality of keyword labels according to a preset counting threshold, and feeding back the determined optimal label as a recognition result.

[0009] In some preferred embodiments, the number of valid frames of each keyword label as an optimal label represents the difference between the number of frames in which the determined keyword label that can participate in counting is the keyword label and the number of frames in which the determined keyword label that can participate in counting is not the keyword label in the continuous multiple speech frames in which the optimal label is the keyword label.

[0010] The counting of the effective frame number of each keyword tag in the plurality of keyword tags as an optimal tag comprises:

[0011] When the first count value is in an initial state, updating the optimal tag to the newly determined keyword tag capable of participating in counting, and controlling the first count value to increase by one count unit, wherein the first count value represents the effective frame number of each keyword tag as an optimal tag.

[0012] When the first count value is in a non-initial state, increasing / decreasing the first count value according to the newly determined keyword tag capable of participating in counting.

[0013] In some preferred embodiments, the increasing / decreasing of the first count value according to the newly determined keyword tag capable of participating in counting comprises:

[0014] When the newly determined keyword tag capable of participating in counting is the same keyword tag as the current optimal tag, controlling the first count value to increase by one count unit; or

[0015] When the newly determined keyword tag capable of participating in counting is a different keyword tag from the current optimal tag, controlling the first count value to decrease by one count unit.

[0016] In some preferred embodiments, the determining of the optimal tag from the plurality of keyword tags according to a preset count threshold value comprises:

[0017] When the first count value reaches the preset count threshold value, determining the keyword tag currently as an optimal tag as the optimal tag.

[0018] In some preferred embodiments, the effective frame number of each keyword tag as an optimal tag represents the frame number of the determined keyword tag capable of participating in counting.

[0019] The counting of the effective frame number of each keyword tag in the plurality of keyword tags as an optimal tag comprises:

[0020] Counting the effective frame number of each keyword tag in the plurality of keyword tags as an optimal tag by using a plurality of second count values, wherein each second count value is configured to increase by one count unit when the keyword tag corresponding thereto is determined to be capable of participating in counting.

[0021] In some preferred embodiments, the determining of the optimal tag from the plurality of keyword tags according to a preset count threshold value comprises:

[0022] The keyword label corresponding to the second count value that first reaches the preset count threshold is determined as the optimal label.

[0023] In some preferred embodiments, the plurality of classification labels further comprises at least one background label.

[0024] When it is detected that the probability corresponding to each background label is the maximum probability in the probability vector and is greater than the preset trigger threshold, the output post-processing of the next speech frame is directly started.

[0025] In some preferred embodiments, after the optimal label is determined, the method further comprises:

[0026] Initializing the count result.

[0027] In some preferred embodiments, after the optimal label is determined, the method further comprises:

[0028] After K continuous speech frames, the number of valid frames of each keyword label in the plurality of keyword labels that is tentatively determined as the optimal label is counted again to determine the optimal label again, K being a positive integer.

[0029] According to a second aspect of the present application, a multi-keyword-based speech recognition method is provided, comprising:

[0030] Obtaining a speech signal to be processed, and calculating feature data available for recognition of the speech signal;

[0031] Inputting the feature data into a neural network model frame by frame, and outputting a probability vector;

[0032] The output of the speech recognition model is post-processed according to the output post-processing method according to any embodiment of the present application, and the feedback of the recognition result is realized.

[0033] According to a third aspect of the present application, an output post-processing device of a speech recognition model is provided, comprising a processor and a memory storing a computer program, wherein the processor can realize the output post-processing method of the speech recognition model according to any embodiment of the present application when executing the computer program.

[0034] According to a fourth aspect of the present application, a multi-keyword-based speech recognition device is provided, comprising a processor and a memory storing a computer program, wherein the processor can realize the multi-keyword-based speech recognition method according to any embodiment of the present application when executing the computer program.

[0035] According to a fifth aspect of the present application, a terminal device is provided, comprising:

[0036] An audio collector configured to collect a voice signal and output;

[0037] A voice recognition device configured to receive the voice signal, perform voice recognition according to the multi-keyword based voice recognition method as described in any embodiment of the present application, and output a feedback signal according to a recognition result.

[0038] According to a sixth aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is capable of implementing the output post-processing method of the voice recognition model as described in any embodiment of the present application, or implementing the multi-keyword based voice recognition method as described in any embodiment of the present application when executed by a processor.

[0039] The technical solutions in any embodiment of the present application can realize optimal label selection in multi-keyword recognition, improve the continuity and accuracy of the prediction result of the voice recognition model, and the algorithm is simple.

[0040] It should be noted that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flow chart of the output post-processing method of the voice recognition model provided by an embodiment of the present application is shown;

[0042] Figure 2 A flow chart of the output post-processing method of the voice recognition model provided by another embodiment of the present application is shown;

[0043] Figure 3 A flow chart of the output post-processing method of the voice recognition model provided by still another embodiment of the present application is shown;

[0044] Figure 4 A schematic diagram of the probability vector output by the voice recognition model provided by an embodiment of the present application is shown;

[0045] Figure 5 A flow chart of the multi-keyword based voice recognition method provided by an embodiment of the present application is shown;

[0046] Figure 6 A schematic diagram of the output post-processing device of the voice recognition model provided by an embodiment of the present application is shown;

[0047] Figure 7 A schematic diagram of the terminal device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] For the purpose of clarity, the present application will be described with reference to the accompanying drawings. The preferred embodiments of the present application are shown in the drawings. However, the present application can be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as part of the disclosure to convey the principles of the present application to those skilled in the art.

[0049] In the description of the present application, the word "exemplary" or "for example" is used to mean an example, an illustration, or another instance. Any embodiment described as "exemplary" or "for example" in the present application is not necessarily to be construed as preferred or advantageous over other embodiments. In the present application, "and / or" is a description of an associative relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists alone, A and B exist together, and B exists alone. "Multiple" means two or more than two. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, the same items or similar items with basically the same functions and roles are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0050] In describing representative, exemplary embodiments, the specification can have presented the method and / or process as a particular sequence of steps. However, to the extent that the method or process depends on the performance of certain steps, the method or process is not limited to the performance of the steps in the order described. Alternative sequencing is possible, as will be appreciated by those skilled in the art. Therefore, the specific order of the steps recited in the specification is not to be construed as a limitation on the claims. Further, the claims should not be limited to the steps of the methods and / or processes in the order written, as other sequences of steps can be readily apparent to those skilled in the art and still fall within the scope of the application.

[0051] For a speech recognition model (hereinafter referred to as "model" for short), a relatively stable probability distribution can be obtained by mean filtering the output of the model, and the performance of the model algorithm is improved. When the use scene is complex, such as when the command word length changes obviously, the background sound is complex, the command word similarity is high, etc., the prediction result of the speech recognition model will appear in most frames, the prediction category is correct, but there are several frames in the middle, the prediction category is wrong, that is, the prediction result is discontinuous. At this time, if a method similar to mean filtering is used to obtain a relatively stable recognition result, more frames need to be processed for averaging, which can easily lead to a large delay in the output prediction result.

[0052] To this end, an embodiment of the present application provides an output post-processing method of a speech recognition model, a multi-keyword-based speech recognition method and device, equipment, and a storage medium, wherein the execution subject of the output post-processing method of the speech recognition model (hereinafter referred to as "output post-processing method") can be an output post-processing device of the speech recognition model (hereinafter referred to as "output post-processing device") provided by an embodiment of the present application, or a terminal device integrated with the output post-processing device; and the execution subject of the multi-keyword-based speech recognition method (hereinafter referred to as "speech recognition method") can be a multi-keyword-based speech recognition device (hereinafter referred to as "speech recognition device") provided by an embodiment of the present application, or a terminal device integrated with the speech recognition device, wherein the output post-processing device and the speech recognition device can be realized in the form of hardware or software, and the terminal device can be a computing device such as a laptop computer, a computer monitor containing an embedded computer, a tablet computer, a cellular phone, a media player, or other handheld or portable terminal devices, smaller devices (such as wristwatch devices, pendant devices, earphone or handset devices, devices embedded in glasses or other devices worn on the head of a user, or other wearable or miniature devices), a television set, a computer display not containing an embedded computer, a game device, a navigation device, an embedded system (such as a system in which a terminal device with a display is installed in an information kiosk or a car), and the like. In an embodiment of the present application, the error results mixed in the recognition results of the model output are regarded as spike noise in the output signal, and the counting filter method which is only applicable to processing a single signal (i.e., only applicable to a two-classification task similar to single wake-up word / command word recognition) is extended to multi-signal filtering (i.e., a multi-keyword recognition task), for example, by introducing a counting reduction rule or a multi-counter filter to post-process the output of the model, only by setting a reasonable counting threshold, the optimal label corresponding to the target keyword in the multi-keyword recognition can be selected while ensuring the continuity of the prediction results of the model output, compared with the mean filter method, the output post-processing method provided in the embodiment of the present application needs to design a simple algorithm, has small requirements on the device, and can improve the adaptability of the post-processing to the model precision.

[0053] As shown in Figure 1 The output post-processing method provided by an embodiment of the present application comprises:

[0054] In step 110, the maximum probability in the probability vector output by the speech recognition model and the classification label corresponding to the maximum probability are determined frame by frame, wherein the probability vector represents the corresponding probability that the speech frame signal to be processed is recognized as a plurality of classification labels, and the plurality of classification labels include a plurality of keyword labels corresponding to a plurality of keywords.

[0055] The speech recognition model of this step is a recognition model that can recognize multiple keywords. The keywords can be wake-up words or command words. The speech recognition model processes the speech signal to be processed and outputs a probability vector corresponding to the speech signal frame by frame.

[0056] The speech signal processed by the speech recognition model can be a signal generated by converting the sound emitted by the user via an audio collector (such as a built-in microphone or an external microphone), or a signal generated by converting the sound emitted by other audio playback devices or video playback devices via an audio collector (such as a built-in microphone or an external microphone), or a signal output via a signal port of an electronic module. The present application does not make strict restrictions on this.

[0057] The speech recognition model performs frame segmentation, windowing, and other operations on the received speech signal to obtain a frame-by-frame speech signal (also referred to as "speech frame signal" herein). The speech recognition model inputs the identifiable data features in the speech frame signal to be processed (i.e., the speech signal corresponding to the current speech frame) into a neural network model in each speech frame, and obtains a probability vector in each speech frame through the neural network model. The neural network model in the speech recognition model is a model pre-trained using training samples including multiple keywords, and thus can be used to determine the corresponding probability of the speech frame signal to be processed in each speech frame being recognized as multiple classification labels, and output a probability vector corresponding to the frame.

[0058] In an exemplary embodiment, the multiple classification labels include multiple keyword labels corresponding to the multiple keywords. In another exemplary embodiment, the multiple classification labels include multiple keyword labels corresponding to the multiple keywords and at least one background label. By setting the background label, background noise interference can be more accurately filtered out, and the accuracy of speech recognition can be improved. Figure 4 In the example shown, the probability vector 410 output by the speech recognition model in each speech frame includes probabilities 411 corresponding to the keyword label 1, the keyword label 2, the keyword label 3, and the background label 1, respectively. When post-processing the output of the speech recognition model, the maximum probability and the classification label corresponding to the maximum probability need to be determined from the probability vector 410 of each speech frame frame by frame. Each keyword label can correspond to only one keyword or multiple keywords (e.g., multiple similar keywords implementing the same command). The present embodiment does not make strict restrictions on this.

[0059] Step 120: Counting the number of valid frames in which each keyword label in the multiple keyword labels is the optimal label. Each keyword label participates in the counting process when the respective probability is the maximum probability in the probability vector and is greater than a preset trigger threshold.

[0060] In this step, each keyword label only participates in the counting process when the respective probability is the maximum probability in the probability vector and is greater than the preset trigger threshold. When each probability in the probability vector corresponding to a speech frame is less than the preset trigger threshold, it does not participate in the counting process (i.e., does not affect the current counting result) and the output post-processing of the next speech frame can be directly started. When multiple classification labels include at least one background label and it is detected that the respective probability of each background label is the maximum probability in the probability vector and is greater than the preset trigger threshold, it does not participate in the counting process (i.e., does not affect the current counting result) and the output post-processing of the next speech frame can be directly started.

[0061] In step 130, the optimal label is determined from the multiple keyword labels according to a preset counting threshold, and the determined optimal label is fed back as a recognition result.

[0062] The optimal label is the classification label actually corresponding to the speech signal to be processed (i.e., the speech signal input into the neural network model of the speech recognition model), that is, the target keyword label. The speech recognition model performs frame recognition on the speech signal to be processed in the form of processing speech frame signals, and outputs a corresponding probability vector in each speech frame to represent the prediction result of each speech frame signal. In this way, the maximum value (hereinafter also referred to as the "maximum probability") can be selected from the probability vector corresponding to each speech frame, and when it is determined that the maximum probability is greater than the preset trigger threshold and the classification label corresponding to the maximum probability is a keyword label, the classification label is taken as the prediction result of the speech frame signal to be processed in the speech frame, indicating that the keyword corresponding to the keyword label is recognized in the speech frame. Otherwise, it is determined that the speech signal to be processed does not contain any keyword corresponding to any keyword label in the speech frame, which is an invalid speech signal. In this embodiment, each keyword label corresponding to each keyword that the speech signal to be processed may contain in the continuous multiple speech frames is taken as a candidate optimal label, and the number of valid frames when each keyword label is taken as a candidate optimal label (i.e., as an optimal label) is counted. By setting a reasonable counting threshold, the optimal label can be selected, and at the same time, the error results mixed in the prediction results of the continuous multiple speech frames can be effectively filtered out, so that the final output prediction result has better stability and accuracy.

[0063] In an exemplary embodiment, the number of valid frames of each keyword label taken as an optimal label represents the difference between the number of frames in which the optimal label is the keyword label and the number of frames in which the determined keyword label that can participate in counting is not the keyword label in the continuous multiple speech frames. For example, in the case of the keyword label "A", if the number of frames in which the keyword label "A" is determined to be able to participate in counting is 10 and the number of frames in which the keyword label "A" is not determined to be able to participate in counting is 5, the number of valid frames of the keyword label "A" taken as an optimal label is 5.Figure 4 In the example shown, it is assumed that the preset trigger threshold is 0.7, n is equal to 6, and the optimal label is keyword label 1 in the first to sixth speech frames. In this case, the following is obtained:

[0064] In the probability vector 410 corresponding to the first speech frame, it can be determined that the maximum probability is 0.75, and the keyword label 1 corresponds to the keyword label 1. Therefore, the keyword label that can participate in counting in the first speech frame is keyword label 1. Similarly, the keyword label that can participate in counting in the second speech frame is keyword label 1, the keyword label that can participate in counting in the third speech frame is keyword label 2, no keyword label that can participate in counting is identified in the fourth and fifth speech frames, and the keyword label that can participate in counting in the sixth speech frame is keyword label 1. In this case, among the six continuous speech frames, the number of frames in which the keyword label that can participate in counting is keyword label 1 is 3 (including the first, second, and sixth speech frames), the number of frames in which the keyword label that can participate in counting is not keyword label 1 is 1 (including the third speech frame), and the effective frame number of keyword label 1 as the optimal label is 3-1=2.

[0065] In this embodiment, the effective frame number of each keyword label as the optimal label is counted, including:

[0066] When the first count value is in an initial state, the optimal label is updated to the keyword label that can participate in counting that is newly determined, and the first count value is controlled to increase by one count unit, where the first count value represents the effective frame number of each keyword label as the optimal label.

[0067] When the first count value is in a non-initial state, the first count value is increased / decreased according to the keyword label that can participate in counting that is newly determined, specifically including: when the keyword label that can participate in counting that is newly determined and the current optimal label are the same keyword label, the first count value is controlled to increase by one count unit; or when the keyword label that can participate in counting that is newly determined and the current optimal label are different keyword labels, the first count value is controlled to decrease by one count unit.

[0068] In this embodiment, the optimal label is determined from the plurality of keyword labels according to a preset count threshold, including: when the first count value reaches the preset count threshold, the keyword label that is currently used as the optimal label is determined as the optimal label.

[0069] In Figure 4In the shown example, it is assumed that the first count value is an initial state (e.g., the first count value is equal to zero) in the first speech frame, and after determining that the keyword label capable of participating in counting is keyword label 1, the optimal label is updated to keyword label 1, and the first count value is controlled to increase by one count unit. From the second speech frame, since the first count value is in a non-initial state, counting of the effective frame number when keyword label 1 is assumed to be the optimal label is started according to the recognition result of the speech frame signal to be processed in each speech frame, until the first count value becomes the initial state again. In the second speech frame, the latest determined keyword label capable of participating in counting is keyword label 1, and the first count value is controlled to increase by one count unit. In the third speech frame, the latest determined keyword label capable of participating in counting is keyword label 2, and the first count value is controlled to decrease by one count unit. In the fourth speech frame and the fifth speech frame, no keyword label capable of participating in counting is recognized, and the first count value remains unchanged. In the nth speech frame (n is an integer greater than 1), the latest determined keyword label capable of participating in counting is keyword label 1, and the first count value is controlled to increase by one count unit. It is assumed that the first count value reaches the preset count threshold at this time, and keyword label 1 is determined to be the optimal label and is fed back as the final recognition result. It is assumed that the first count value does not reach the preset count threshold at this time, and in the (n+1)th speech frame, the latest determined keyword label capable of participating in counting is not keyword label 1, and after the first count value is controlled to decrease by one count unit, the first count value becomes the initial state, and counting of the effective frame number when keyword label 1 is assumed to be the optimal label is ended. In addition, when the initial state of the first count value is a number other than zero, the effective frame number when each keyword label is assumed to be the optimal label can be represented according to the change amount of the first count value relative to the initial state.

[0070] In the above-described embodiments of the present application, the count increase and count decrease rules are introduced at the same time, and the count filtering applicable to a single keyword is extended to multi-keyword filtering. By using only the first count value for cyclic counting, counting of the effective frame number when each keyword label is assumed to be the optimal label can be implemented, which not only enables optimal label selection in multi-keyword recognition, improves the continuity and accuracy of the prediction result of the speech recognition model, but also has a small demand for counting applications (e.g., a counter) and a low design cost.

[0071] In another exemplary embodiment, the effective frame number when each keyword label is assumed to be the optimal label is represented by the frame number in which the determined keyword label capable of participating in counting is the keyword label. For example, in the shown example, it is assumed that the preset trigger threshold is 0.7, and n is equal to 6, and then: Figure 4

[0072] ​In the probability vector 410 corresponding to the first speech frame, it can be determined that the maximum probability is 0.75, and the keyword label 1 is corresponded, and thus the keyword label capable of participating in counting determined in the first speech frame is the keyword label 1. Similarly, the keyword label capable of participating in counting determined in the second speech frame is the keyword label 1, the keyword label capable of participating in counting determined in the third speech frame is the keyword label 2, no keyword label capable of participating in counting is identified in the fourth speech frame and the fifth speech frame, and the keyword label capable of participating in counting determined in the sixth speech frame is the keyword label 1. At this time, in the continuous six speech frames, the effective frame number of the keyword label 1 as the optimal label is 3 (including the first speech frame, the second speech frame and the sixth speech frame), and the effective frame number of the keyword label 2 as the optimal label is 1 (including the third speech frame).

[0073] In the embodiment, the counting of the effective frame number of each keyword label as the optimal label includes:

[0074] The counting of the effective frame number of each keyword label as the optimal label includes:

[0075] In the embodiment, the optimal label is determined from the plurality of keyword labels according to a preset counting threshold, and the counting of the effective frame number of each keyword label as the optimal label includes:

[0076] The keyword label corresponding to the second counting value first reaching the preset counting threshold is determined as the optimal label.

[0077] In the embodiment, the counting of the effective frame number of each keyword label as the optimal label includes: Figure 4 In the example shown, in the first speech frame, the keyword label 1 is determined to be capable of participating in the counting process, and the second counting value corresponding to the keyword label 1 is controlled to increase by one counting unit; in the second speech frame, the keyword label 1 is determined to be capable of participating in the counting process, and the second counting value corresponding to the keyword label 1 is controlled to increase by one counting unit; in the third speech frame, the keyword label 2 is determined to be capable of participating in the counting process, and the second counting value corresponding to the keyword label 2 is controlled to increase by one counting unit; in the fourth speech frame and the fifth speech frame, no keyword label capable of participating in counting is identified, and the second counting value corresponding to each keyword label is kept unchanged; in the nth speech frame, the keyword label 1 is determined to be capable of participating in the counting process, and the second counting value corresponding to the keyword label 1 is controlled to increase by one counting unit.

[0078] Assuming that the second count value corresponding to the keyword label 2 controlled in the third speech frame increases by one count unit, and the second count value corresponding to the keyword label 2 reaches the preset count threshold first, the keyword label 2 is determined as the optimal label, and is fed back as the final recognition result. Or, assuming that the second count value corresponding to the keyword label 1 controlled in the n-th speech frame increases by one count unit, and the second count value corresponding to the keyword label 1 reaches the preset count threshold first, the keyword label 1 is determined as the optimal label, and is fed back as the final recognition result.

[0079] In the above embodiments of the present application, the second count values are used to count the effective frame numbers of each keyword label when the keyword label is tentatively selected as the optimal label. This not only realizes the selection of the optimal label in multi-keyword recognition, improves the continuity and accuracy of the prediction results of the speech recognition model, but also simplifies the algorithm rules and is easier to program.

[0080] In some preferred embodiments of the present application, after the optimal label is determined, the method further includes initializing the count result. Specifically, the first count value is restored to the initial state, or the second count value corresponding to each keyword label is restored to the initial state, so as to facilitate the output post-processing of the next speech signal.

[0081] In some preferred embodiments of the present application, after the optimal label is determined and fed back as the recognition result, the method further includes continuing to count the effective frame numbers of each keyword label in the plurality of keyword labels tentatively selected as the optimal label after the continuous K speech frames, so as to determine the optimal label again, that is, ignoring the recognition results in the continuous K speech frames after that, so as to prevent repeated recognition or false recognition, wherein K is a positive integer.

[0082] In an embodiment of the present application, in order to realize the selection of the optimal label in multi-keyword recognition and improve the continuity and accuracy of the prediction results of the speech recognition model, an output post-processing method of a speech recognition model is provided, as shown in Figure 2 The method includes the following steps.

[0083] In step 201, the first count value cnt_1 is initialized.

[0084] This step occurs before the first output post-processing of the speech recognition model, or occurs after the first recognition result is fed back and before the next output post-processing, including restoring the first count value cnt_1 to the initial state, for example, initializing the first count value cnt_1 to zero.

[0085] In some other examples, this also includes: initializing the optimal label cls_best, for example, initializing the optimal label cls_best as the background label, to avoid affecting the subsequent counting of the number of valid frames for each keyword label that is proposed as the optimal label.

[0086] Step 202: Obtain the probability vector P of the i-th speech frame. t_i , where i is an integer greater than or equal to 1.

[0087] Step 203, determine P t_i The maximum probability P in max_i The category label d corresponding to the highest probability max_i .

[0088] Step 204, determine P max_i Is the threshold Thresh_p greater than the trigger threshold? If yes, proceed to step 206; otherwise, proceed to step 205.

[0089] When every probability in the probability vector corresponding to a speech frame is less than a preset trigger threshold, that speech frame will not be included in the counting of valid frames when each keyword tag is considered the optimal tag, and post-processing of the next speech frame can begin directly. This is equivalent to filtering out the maximum probability P. max_i The prediction results of the speech recognition model in speech frames that have not reached the trigger threshold Thresh_p are used to reduce the amount of computation and ensure that the current speech frame does not affect the count of the number of effective frames when each keyword tag is to be the optimal tag.

[0090] Step 205, increment i by 1. After executing step 205, proceed to step 202.

[0091] After this step, post-processing operations will begin on the probability vector of the next speech frame output by the speech recognition model.

[0092] Step 206, determine d max_i Does it correspond to a background label? If yes, proceed to step 205; otherwise, proceed to step 207.

[0093] This step is performed based on the inclusion of background labels among multiple classification labels. When the background label is detected in the speech frame, the probability corresponding to it is the probability vector P. t_i The maximum probability P in max_iWhen the probability is greater than the preset trigger threshold Thresh p, the speech frame does not participate in counting of the effective frame number when each keyword label is taken as the optimal label, and the output post-processing of the next speech frame can be directly started. This is equivalent to filtering the influence of the false prediction that the prediction result is background noise on the prediction result of the speech recognition model, reduces the subsequent calculation amount, and ensures that the current speech frame does not affect the counting result of the effective frame number when each keyword label is taken as the optimal label.

[0094] Step 207: Determine whether cnt_1 is equal to 0. If yes, jump to step 208; otherwise, jump to step 209.

[0095] This step is executed on the basis that the initial value of the first counting value cnt_1 is defined as 0, which is equivalent to determining whether the first counting value cnt_1 is in the initial state.

[0096] Step 208: Let the optimal label cls_best be equal to d max_i , and cnt_1 is incremented by 1. After step 208 is executed, jump to step 212.

[0097] When the first counting value is in the initial state, update the optimal label cls_best to the classification label d max_i determined in the current frame, and control the first counting value cnt_1 to be increased by one counting unit to start counting the effective frame number when the classification label d max_i determined in the current frame is taken as the optimal label cls_best.

[0098] Step 209: Determine whether d max_i is equal to cls_best. If yes, jump to step 210; otherwise, jump to step 211.

[0099] When the first counting value is in the initial state, increase / decrease the first counting value according to the classification label d max_i determined in the current frame.

[0100] Step 210: cnt_1 is incremented by 1. After step 210 is executed, jump to step 212.

[0101] This step is executed when the classification label d max_i determined in the current frame is the same keyword label as the current optimal label cls_best, and the first counting value cnt_1 is controlled to be increased by one counting unit.

[0102] Step 211: cnt_1 is decremented by 1. After step 211 is executed, jump to step 212.

[0103] This step is executed when the classification label d max_iWhen the current optimal label cls_best is different from the keyword label, the first count value cnt_1 is decremented by one. The count decrement rule is applied in this step to filter out the incorrect prediction results in the prediction results of the continuous multiple speech frames in the multi-keyword recognition model, reduce the possibility that the discontinuous prediction results of the command word recognition model lead to repeated prediction or missed detection in the final recognition result, and is also the key to realize the loop counting, so that the model can have the ability to select the optimal label cls_best in multi-keyword recognition.

[0104] In step 212, it is determined whether the first count value cnt_1 is greater than or equal to the count threshold Thresh_N. If yes, the process jumps to step 213, otherwise, the process jumps to step 205.

[0105] This step is performed after the first count value cnt_1 changes each time. When the first count value cnt_1 is greater than or equal to the count threshold Thresh_N, it can be determined that the final optimal label has been selected, that is, the target keyword label, and then the recognition result can be fed back. When the first count value cnt_1 is less than the count threshold Thresh_N, the output post-processing of the next speech frame needs to be continued.

[0106] In step 213, the predicted optimal label cls_best is output.

[0107] After the optimal label cls_best is determined, the determined optimal label cls_best is fed back as the recognition result.

[0108] The algorithm flow shown in this embodiment has a small demand for counting (for example, a counter) and a low design cost, and can be used as one of the implementation schemes of the foregoing loop counting to select the optimal label by using the first count value and improve the continuity and accuracy of the prediction results of the speech recognition model.

[0109] In another embodiment of the present application, in order to realize the selection of the optimal label in multi-keyword recognition and improve the continuity and accuracy of the prediction results of the speech recognition model, another output post-processing method of the speech recognition model is provided, as shown in Figure 3 The method comprises the following steps.

[0110] In step 301, a plurality of second count values cnt_2_M are initialized, M is an integer greater than or equal to 1, and corresponds to the number of keyword labels in the classification labels.

[0111] This step occurs before the first post-processing of the speech recognition model's output, or after a recognition result has been returned and before the next post-processing of the output, and includes restoring each second count value cnt_2_M to its initial state, for example, initializing each second count value cnt_2_M to zero.

[0112] Step 302, obtain the probability vector P of the i-th speech frame. t_i , where i is an integer greater than or equal to 1.

[0113] Step 303, determine P t_i The maximum probability P in max_i The category label d corresponding to the highest probability max_i .

[0114] Step 304, determine P max_i Is the threshold Thresh_p greater than the trigger threshold? If yes, proceed to step 306; otherwise, proceed to step 305.

[0115] When every probability in the probability vector corresponding to a speech frame is less than a preset trigger threshold, that speech frame will not be included in the counting of valid frames when each keyword tag is considered the optimal tag, and post-processing of the next speech frame can begin directly. This is equivalent to filtering out the maximum probability P. max_i The prediction results of the speech recognition model in speech frames that have not reached the trigger threshold Thresh_p are used to reduce the amount of computation and ensure that the current speech frame does not affect the count of the number of effective frames when each keyword tag is to be the optimal tag.

[0116] Step 305, increment i by 1. After executing step 305, proceed to step 302.

[0117] After this step, post-processing operations will begin on the probability vector of the next speech frame output by the speech recognition model.

[0118] Step 306, determine d max_i Does it correspond to a background label? If yes, proceed to step 305; otherwise, proceed to step 307.

[0119] This step is performed based on the inclusion of background labels among multiple classification labels. When the background label is detected in the speech frame, the probability corresponding to it is the probability vector P. t_i The maximum probability P in max_iWhen the probability P is greater than the preset trigger threshold Thresh p, the speech frame does not participate in counting of the effective frame number when each keyword label is taken as the optimal label, and the output post-processing of the next speech frame can be directly started. This is equivalent to filtering out the influence of the false prediction that the prediction result is background noise on the prediction result of the speech recognition model, reducing the subsequent calculation amount while ensuring that the current speech frame does not affect the counting result of the effective frame number when each keyword label is taken as the optimal label.

[0120] Step 307, cnt_2_M is increased by 1. After step 307 is executed, the process jumps to step 308. max_i max_i max_i max_i When the probability P is greater than the preset trigger threshold Thresh p, the speech frame does not participate in counting of the effective frame number when each keyword label is taken as the optimal label, and the output post-processing of the next speech frame can be directly started. This is equivalent to filtering out the influence of the false prediction that the prediction result is background noise on the prediction result of the speech recognition model, reducing the subsequent calculation amount while ensuring that the current speech frame does not affect the counting result of the effective frame number when each keyword label is taken as the optimal label.

[0121] In step 307, it is determined whether the maximum probability P max_i is greater than the trigger threshold Thresh p, and the classification label d max_i determined in the current frame is a keyword label. max_i The corresponding second counting value is increased by one counting unit.

[0122] Step 308, it is determined whether cnt_2_M is greater than or equal to the counting threshold Thresh N. If yes, the process jumps to step 309, otherwise, the process jumps to step 305.

[0123] This step is executed after each second counting value cnt_2_M is changed, that is, it is determined whether the changed second counting value cnt_2_M reaches the counting threshold Thresh N. When the second counting value cnt_2_M is greater than or equal to the counting threshold Thresh N, it is determined that the classification label d max_i determined in the current frame is the final optimal label, that is, the final optimal label is selected, and then the feedback of the recognition result can be performed; when the second counting value cnt_2_M is less than the counting threshold Thresh N, the output post-processing of the next speech frame needs to be continued.

[0124] Step 309, the keyword label corresponding to cnt_2_M is output as the predicted optimal label cls_best.

[0125] When the second counting value cnt_2_M is greater than or equal to the counting threshold Thresh N, the keyword label corresponding to the second counting value cnt_2_M reaching the counting threshold Thresh N is determined as the final optimal label cls_best, and the determined optimal label cls_best is fed back as the recognition result.

[0126] The algorithm flow shown in this embodiment has simple rules and is easy to program and implement, and can be used as one of the implementation schemes in the foregoing implementation in which a plurality of second count values are used to count and select the optimal label, and the continuity and accuracy of the prediction result of the speech recognition model are improved.

[0127] An embodiment of the present application also provides a multi-keyword based speech recognition method, as shown in the accompanying drawings, comprising the steps of: Figure 5

[0128] In step 510, a speech signal to be processed is obtained, and feature data available for recognition of the speech signal is calculated.

[0129] The speech signal to be processed can be a signal generated by converting a sound uttered by a user via an audio collector (such as a built-in microphone or an external microphone), or a signal generated by converting a sound uttered by an audio playing device or a video playing device via an audio collector (such as a built-in microphone or an external microphone), or a signal output via a signal port of an electronic module, and the present application does not make strict limitations thereon.

[0130] In some examples, calculating the feature data available for recognition of the speech signal can include: dividing the speech signal into frames according to a set window length and step length to obtain speech signal frame by frame (also referred to as "speech frame signal" herein); then, for each speech frame signal, determining time-frequency features of the speech frame signal by performing short-time Fourier transform on the speech frame signal, i.e., converting features of time domain sampling into time-frequency features; further, determining filter bank based features (fbank) and mel frequency cepstral coefficients (mfcc) based on the time-frequency features, specifically, converting frequency domain coordinates into logarithmic coordinates to obtain fbank features according to human hearing, and performing inverse Fourier transform on the logarithmic coordinates to obtain cepstrum, thereby obtaining mfcc features. The fbank features and the mfcc features can be used as speech recognition features and input into a neural network model in subsequent steps.

[0131] Here, dividing into frames means dividing a speech signal into small segments according to the stationarity of the speech, and the speech in each small segment has similar speech sound rules and has a certain stationarity. Each small segment is referred to as a frame. In specific implementation, considering the continuity of the speech, there is a certain degree of overlap between different frames of signals.

[0132] ​The window length refers to the time length of the segmented speech signal, and the step length refers to the fixed step length in the time dimension for continuously sliding according to the window length to obtain a frame of signal. In the present application, the window length is greater than or equal to the longest pronunciation length of the classification label of the target keyword, and the step length is less than or equal to half of the window length. For example, the length of a frame of signal (i.e., the window length) can be 20-40 ms, and the step length can be 7-16 ms. The window length and the step length can be flexibly selected according to different tasks. Exemplarily, the process of extracting the speech feature can include: first, performing windowing processing on the speech signal to segment the speech signal into frames, and the language signals of the previous frame and the next frame have a time sequence. The power spectrum of each frame is obtained by applying fast Fourier transform (FFT) to the language signal in each frame. Then, the power spectrum is filtered by a filter bank using the Mel-scale. After the power spectrum is transformed into the logarithmic domain, the discrete cosine transform is applied to the speech signal to calculate the MFCC coefficients.

[0133] In step 520, the feature data is input into the neural network model frame by frame, and a probability vector is output.

[0134] The recognizable data features in the extracted speech frame signal (i.e., the speech signal corresponding to the current speech frame) are input into the neural network model frame by frame. After the neural network model processes the feature data, a probability vector in each speech frame is output. The neural network model in the speech recognition model is a model that is pre-trained using training samples including multiple keywords, and thus can be used to determine the corresponding probability that the speech frame signal to be processed in each speech frame is recognized as a classification label, and output the probability vector of the corresponding frame.

[0135] In step 530, the output of the speech recognition model is post-processed according to the output post-processing method described in any one of the embodiments of the present application, and the feedback of the recognition result is realized.

[0136] In the speech recognition, the probability vector output by the speech recognition model is counted and filtered according to the output post-processing method described in any one of the embodiments of the present application, so that the optimal label selection in multi-keyword recognition can be realized, the continuity and accuracy of the prediction result of the speech recognition model are improved, the possibility of repeated prediction or missed detection in the final recognition result of the speech recognition model is reduced, the algorithm is simple, the adaptability of the post-processing algorithm to the accuracy of the speech recognition model is improved, and the demand for the computing power resources of the device is small.

[0137] An embodiment of the present application also provides an output post-processing device for a speech recognition model, as shown in Figure 6 the memory 50 storing a computer program, wherein the processor 60 can realize the output post-processing method for a speech recognition model as described in any one of the embodiments of the present application when executing the computer program.

[0138] An embodiment of the present application further provides a voice recognition device based on multiple keywords, which can also be referred to as Figure 6 , comprising a processor and a memory storing a computer program, wherein the processor can implement the voice recognition method based on multiple keywords as described in any embodiment of the present application when executing the computer program.

[0139] The processor in the voice recognition device and the post-processing device in the above embodiment can be an integrated circuit chip with signal processing capability. For example, the processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or any conventional processor, etc.

[0140] An embodiment of the present application further provides a terminal device, as shown in Figure 7 , comprising:

[0141] An audio collector 1 configured to collect voice signals and output;

[0142] A voice recognition device 2 configured to receive the voice signals and perform voice recognition according to the voice recognition method based on multiple keywords as described in any embodiment of the present application, and output a feedback signal according to the recognition result;

[0143] A driver 3 configured to receive the feedback signal and drive the terminal device or one or more components in the terminal device to perform corresponding functions according to the type of the feedback signal.

[0144] The audio collector 1, the voice recognition device 2 and the driver 3 in the terminal device of the present embodiment can be physically independent or integrated in any combination in one device. For example, the audio collector 1, the voice recognition device 2 and the driver 3 can be integrated in one device such as a mobile phone, a sound system, a car, etc. to implement the voice recognition method based on multiple keywords of the present embodiment. For another example, two of the audio collector 1, the voice recognition device 2 and the driver 3 can be integrated in one device, and the other one can be a separate device. The driver 3 can be any type of single component or a collection of multiple components with driving capability, such as a circuit module or a device providing driving signals such as voltage and current, a device providing mechanical force, etc. The present application does not make strict limitations on this.

[0145] The terminal device of the embodiment can perform the output post-processing method of the speech recognition model of any embodiment of the present application, and / or the multi-keyword based speech recognition method, which can realize optimal label selection in multi-keyword recognition, improve the continuity and accuracy of the prediction result of the speech recognition model, and has a simple algorithm and small demand for the computing power resources of the device.

[0146] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program can realize the output post-processing method of the speech recognition model as described in any embodiment of the present application or the multi-keyword based speech recognition method as described in any embodiment of the present application when executed by a processor.

[0147] An embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program can realize the output post-processing method of the speech recognition model as described in any embodiment of the present application or the multi-keyword based speech recognition method as described in any embodiment of the present application when executed by a processor.

[0148] The above-mentioned embodiments of the present application provide an output post-processing method of a speech recognition model, a speech recognition method and a device, which can realize the selection of the optimal label corresponding to the target keyword in multi-keyword recognition by a simple algorithm, and the continuity of the prediction result is high, can replace the traditional output post-processing method of speech recognition, and has at least one of the following advantages:

[0149] The possibility of repeated prediction or missed detection of the final recognition result caused by the discontinuity of the prediction result of the speech recognition model is reduced;

[0150] The adaptability of the output post-processing algorithm to the model precision is improved;

[0151] Only simple counting is needed, and the algorithm overhead is extremely small.

[0152] In any one or more of the example embodiments described above, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to non-transitory computer- readable storage media or communication media (e.g., electromagnetic signals, radio waves, etc.). Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this application. A computer program product can include a computer-readable medium.

[0153] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0154] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be fully implemented in one or more circuit or logic elements.

[0155] The technical solutions of the embodiments of this application can be implemented in a wide variety of devices or equipment, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The embodiments of this application describe various components, modules, or units to emphasize functional aspects of a device configured to perform the described techniques, but they do not necessarily need to be implemented through different hardware units. Rather, as described above, the various units can be combined in codec hardware units or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with suitable software and / or firmware.

[0156] Finally, it should be noted that the above embodiments are merely examples for clearly illustrating this application and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.

Claims

1. A post-processing method for the output of a speech recognition model, comprising: The maximum probability and the corresponding classification label in the probability vector output by the speech recognition model are determined frame by frame. The probability vector represents the probability that the speech frame signal to be processed is identified as multiple classification labels, including multiple keyword labels corresponding to multiple keywords. The number of effective frames for each keyword tag among the various keyword tags that is proposed as the optimal tag is counted. Each keyword tag participates in the counting process when its corresponding probability is the maximum probability in the probability vector and is greater than a preset trigger threshold. The optimal tag is determined from multiple keyword tags according to a preset counting threshold, and the determined optimal tag is fed back as the recognition result. The step of counting the number of valid frames for each keyword tag among the multiple keyword tags that is intended to be the optimal tag includes: The effective number of frames for each keyword tag among the various keyword tags that is proposed as the optimal tag is counted using multiple second count values. Each second count value is configured to increase by one count unit when its corresponding keyword tag is determined to be able to participate in the counting process.

2. The output post-processing method according to claim 1, wherein, The counting of the number of valid frames for each keyword tag among the multiple keyword tags that is intended to be the optimal tag includes: When the first count value is in the initial state, the optimal tag is updated to the latest determined keyword tag that can participate in the counting, and the first count value is increased by one count unit, wherein the first count value represents the number of valid frames for each keyword tag to be the optimal tag; When the first count value is not in the initial state, the first count value is increased / decreased according to the latest determined keyword tags that can participate in the counting.

3. The output post-processing method according to claim 2, wherein, The step of increasing / decreasing the first count value based on the latest determined keyword tags that can participate in the counting includes: If the newly determined keyword tag that can participate in the counting is the same keyword tag as the current optimal tag, then the first count value is increased by one count unit; or If the newly determined keyword tag that can participate in the counting is a different type of keyword tag from the current optimal tag, the first count value is reduced by one count unit.

4. The output post-processing method according to claim 2 or 3, wherein, The step of determining the optimal tag from multiple keyword tags based on a preset counting threshold includes: When the first count value reaches the preset count threshold, the keyword tag that is currently intended to be the optimal tag is determined as the optimal tag.

5. The output post-processing method according to claim 1, wherein, The step of determining the optimal tag from multiple keyword tags based on a preset counting threshold includes: The keyword tag corresponding to the second count value that first reaches the preset counting threshold is determined as the optimal tag.

6. The output post-processing method according to claim 1, wherein, The multiple category labels also include at least one background label; When the probability corresponding to each background label is the maximum probability in the probability vector and is greater than the preset trigger threshold, the post-processing of the next audio frame output will start directly.

7. The output post-processing method according to claim 1, wherein, After feeding back the determined optimal label as the recognition result, the process also includes: After K consecutive speech frames, the number of valid frames for each keyword tag among the various keyword tags to be used as the optimal tag is counted to determine the optimal tag for the next time. K is a positive integer.

8. A speech recognition method based on multiple keywords, comprising: Acquire the speech signal to be processed and calculate the identifiable feature data of the speech signal; The feature data is input frame by frame into the neural network model, and a probability vector is output. The output of the speech recognition model is post-processed according to any one of claims 1 to 7 to achieve feedback of the recognition result.

9. A post-processing device for the output of a speech recognition model, wherein, include: The processor and the memory storing a computer program, wherein the processor, when executing the computer program, is capable of implementing the output post-processing method of the speech recognition model as described in any one of claims 1 to 7.

10. A speech recognition device based on multiple keywords, wherein, include: The processor and the memory storing a computer program, wherein the processor, when executing the computer program, is capable of implementing the multi-keyword-based speech recognition method as described in claim 8.

11. A terminal device, wherein, include: An audio acquisition device is configured to acquire and output audio signals. A speech recognition device is configured to receive the speech signal, perform speech recognition according to the multi-keyword-based speech recognition method as described in claim 8, and output a feedback signal based on the recognition result.

12. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by the processor, it can implement the post-processing method for the output of the speech recognition model as described in any one of claims 1 to 7, or implement the speech recognition method based on multiple keywords as described in claim 8.

Images