A hot word enhancement method, device and medium for speech recognition

Abstract

The present application discloses a method, device and medium for hot word enhancement for speech recognition. The method includes: obtaining an audio file of the hot word through an encoder, extracting features based on the audio file, sending the extracted features to a CTC decoder, and obtaining a streaming recognition result through the CTC decoder; inputting the streaming recognition result into a language model for shallow fusion, so as to bias the recognition result according to a prefix tree to obtain a search graph; performing shallow fusion through WFST to obtain the best path according to the search graph, sending the best path to an attention decoder, and obtaining accurate results through an attention decoder to complete the enhancement of the hot word. The present application improves the recognition accuracy of out-of-domain (OOD) audio through a hot word enhancement method that combines shallow fusion of WFST, deep bias based on a prefix tree and a language model. The recognition accuracy of hot words is significantly improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and medium for hot word enhancement for speech recognition. Background Art

[0002] With the development of technology, end-to-end automatic speech recognition models are becoming a popular choice for streaming speech recognition. Compared with traditional speech recognition models, end-to-end models are more efficient and accurate in terms of training data, but in some special application scenarios beyond the scope of the training set, the accuracy of end-to-end models is greatly reduced. Summary of the invention

[0003] In order to solve the above problems, the present application proposes a hot word enhancement method for speech recognition, including: obtaining an audio file of the hot word through an encoder, extracting features based on the audio file, and sending the extracted features to a CTC decoder to obtain a streaming recognition result through the CTC decoder; inputting the streaming recognition result into a language model for shallow fusion to bias the recognition result according to a prefix tree to obtain a search graph; performing shallow fusion through WFST to obtain an optimal path according to the search graph, sending the optimal path to an attention decoder, and obtaining an accurate result through the attention decoder to complete the enhancement of the hot word.

[0004] In one example, before shallow fusion is performed through WFST, the method further includes: determining a decoding stage of the WFST, establishing a dynamic bias class in the decoding stage according to the pre-set hot words; determining a segment level according to the hot words, determining an input tag of the WFST according to the segment level, and setting a context prefix for the WFST to narrow the search range of the WFST.

[0005] In one example, a dynamic bias class is established, specifically comprising: establishing an input state and a transfer path of the dynamic bias class, so as to perform a deterministic operation and a minimization operation on the dynamic bias class according to the input state and the transfer path; judging the label state of the input label and the output label in the WFST, and if the label state is an empty state, deleting the transfer path corresponding to the empty state, so as to perform an empty transfer removal operation on the dynamic bias class.

[0006] In one example, setting a context prefix for the WFST specifically includes: determining the pre-information of the hot word according to the segment level, and determining the context of the segment level according to the pre-information; and setting the search path of the WFST according to the context to narrow the search scope of the WFST.

[0007] In one example, the method further includes: determining a reward score of the WFST according to the context prefix, and determining a backoff arc of the WFST to determine a cumulative reward score of the backoff arc, and removing the cumulative reward score.

[0008] In one example, the streaming recognition result is input into a language model for shallow fusion. Before the recognition result is biased according to a prefix tree, the method further includes: determining a segment level according to the hot words, encoding the prefix tree according to the segment level to obtain a query table; determining a bias module according to the query table, and adding the bias module to the language model.

[0009] In one example, the segment level is set by the following formula:

[0010] ;

[0011] in, represents the segment level, Indicates that the corresponding word segment of the hot word starts to be identified in the query table, Indicates continuing to search for the complete hot word in the prefix tree.

[0012] In one example, an audio file of a hot word is obtained through an encoder, and features are extracted based on the audio file, and the extracted features are sent to a CTC decoder to obtain a streaming recognition result through the CTC decoder, specifically including: encoding the audio file through the encoder to obtain the features, recognizing the features through the CTC decoder to obtain multiple initial recognition results; determining multiple weight values ​​of the multiple initial recognition results, scoring the multiple weight values ​​to determine an intermediate recognition result with the highest weight value; and re-recognizing the intermediate recognition result to obtain the streaming recognition result.

[0013] On the other hand, the present application also proposes a hot word enhancement device for speech recognition, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the hot word enhancement device for speech recognition to execute: a method as described in any of the above examples.

[0014] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, characterized in that the computer executable instructions are configured as: a method as described in any of the above examples.

[0015] This application improves the recognition accuracy of out-of-domain (OOD) audio by combining shallow fusion of WFST, deep bias based on prefix tree and language model. This method is generally used to train an end-to-end model on the data set, and after applying the hot word enhancement method to the model, the recognition accuracy of hot words is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flowchart of a hot word enhancement method for speech recognition in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of the structure of the speech recognition system in the embodiment of the present application;

[0019] Figure 3 This is a flow chart of the speech recognition system in the embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of the data structure of the prefix tree in the embodiment of the present application;

[0021] Figure 5 A flowchart of a query table in an embodiment of the present application;

[0022] Figure 6 This is a schematic diagram of the dynamic replacement of WFST in the embodiment of this application;

[0023] Figure 7 This is a schematic diagram of a hot word enhancement device for speech recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0025] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0026] like Figure 1 As shown, a hot word enhancement method for speech recognition provided by an embodiment of the present application is applied to a speech recognition system, such as Figure 2As shown, the speech recognition system includes a shared encoder, a CTC decoder and an attention decoder. The speech recognition system is a two-way hybrid model, including a speech recognition model of CTC and attention mechanism. The method includes:

[0027] S101: Obtain an audio file of a hot word through an encoder, extract features according to the audio file, and send the extracted features to a CTC decoder to obtain a streaming recognition result through the CTC decoder.

[0028] like Figure 3 As shown, the audio of the hot word is input into the speech recognition system, and the audio file is encoded using an encoder to obtain features. The features are sent to the streaming CTC decoder to obtain a rough streaming recognition result.

[0029] In one embodiment, the voice information is encoded by a shared encoder to generate a sequence of features. The feature needs to be shared by the CTC decoder and the shared encoder later. The feature is recognized by the CTC decoder to generate multiple initial recognition results. The weight values ​​corresponding to the multiple initial recognition results are determined, and the multiple weight values ​​are scored to find the highest weight value. The initial recognition result with the highest weight value is determined as the intermediate recognition result corresponding to the feature. Since in the process of generating the coding sequence, each feature generated can generate an intermediate recognition result correspondingly, and the intermediate recognition result is output, thereby realizing streaming voice recognition and obtaining streaming recognition results, so as to present the intermediate recognition results to the user in a timely manner.

[0030] S102: Input the streaming recognition result into a language model for shallow fusion, so as to bias the recognition result according to a prefix tree to obtain a search graph.

[0031] The recognition result of the CTC decoder is shallowly fused with the language model, and the language model is added with a bias module based on a prefix tree, so that the TLG search graph is obtained through the language model.

[0032] In one embodiment, a bias module based on a prefix tree is provided, wherein the prefix tree is a dictionary tree, which is a standard data structure, for example: Figure 4As shown, four phrases - "industrial technology", "industrial technology research institute", "VIPshop", and "only" are used to build a prefix tree. If "go" is identified first, the prefix tree bias that has been encoded in advance will have two paths: "_industry" and "_only". If "industry" is identified, then "technology" or "research institute" will be identified in sequence. If "only" is identified first, then "VIPshop" or "one" will be identified later. The hot words are split to obtain the word segment level. The bias module encodes the set hot words into an untrainable prefix tree data structure through the word segment level of the hot words. In terms of implementation ideas, the prefix tree structure is represented as a query table. The time complexity of the query is O(1), and the time complexity of establishing the prefix tree is O(N).

[0033] In one embodiment, the query table is constructed according to a preset segment level, which is set by the following formula:

[0034] .

[0035] in, Indicates the segment level, Indicates that the corresponding word segment of the hot word in the hot word list of the query table begins to be identified, and h_continue indicates that the complete hot word continues to be searched in the prefix tree. For example: [_go, _industry], the complete query table search process is as follows Figure 5 As shown, Previous Token represents the previous token, which is different from the following token [_industry, _only]. Given a Previous Token, the bias module returns a binary vector , A value of 1 is assigned to words that can be initially recognized in the hot word list, such as "go". A value of 1 is assigned to all words that can be searched in the hot word list, such as "industry" and "only". A bias module is established based on the complete query table, and the bias module is added to the language model to obtain more accurate results based on the bias module.

[0036] S103: shallow fusion is performed through WFST to obtain the best path according to the search graph, and the best path is sent to the attention decoder, and accurate results are obtained through the attention decoder to complete the enhancement of the hot words.

[0037] The WFST shallow fusion method with dynamic permutation is applied to obtain the best path according to the search graph, and the best path is sent to the attention decoder. The recognition results and the features encoded by the shared encoder are sent to the attention decoder. The results obtained by the CTC decoder are scored by the attention decoder to obtain a more accurate result, so that hot words can be accurately matched.

[0038] In one embodiment, a common WFST shallow fusion method is to combine the N-gram language model with the end-to-end network through shallow fusion, and use WFST to search for the best path. Before shallow fusion through WFST, the present application provides a dynamically replaced WFST. In the decoding stage of WFST, a dynamic bias class is established according to pre-set hot words. The dynamic bias class refers to the classification of hot words based on the co-occurrence relationship of hot words, using some statistical algorithms, which can effectively alleviate the problem of static bias occupying too much memory. In addition, when the WFST link needs to be modified, there is no need to rebuild the entire search graph, and the operation is more convenient. Figure 6 As shown, Indicates input label and output label. According to the segment level determined by the hot word, the input label is matched to any segment language symbol. In the figure, 0 indicates the starting point of the search, and the double-framed circle indicates the end point of the search. The arrows pointing forward along the number mark in the figure are forward connections, and the arrows pointing backward are fallback arcs. Arrows indicate state transitions. From the previous state to the next state, the label on each arrow contains the input , Output and the weight of the state transition (not shown in the figure). The above method can be conveniently combined with the context prefix, where the context prefix refers to the preceding information. For example, if "play" is recognized, then the next is likely to be "play movie" or "play music", thereby narrowing the candidate range, which can better help improve the recognition accuracy, narrow the search range of the WFST, and without loss of generality, enhance the generalization ability of the model, so that the model can achieve better results on different data.

[0039] In one embodiment, a dynamic bias class is dynamically created by determinizing, minimizing, and removing empty transitions. WFST is a finite automaton (FA). A FA has a finite set of states and jumps between states, where each jump has at least one label. FA is divided into deterministic FA (DFA) and nondeterministic FA (NFA). Determinizing is the most important optimization operation of FA. Determinizing is an important advantage of using FA to solve sequence recognition and transformation problems. Because the speed of deterministic DFA equivalent to the original FA is much faster than that of nondeterministic NFA. When designing an FA to solve a problem, it is usually easier to understand to use NFA. However, in terms of computer running efficiency, NFA is usually not as good as DFA. Therefore, determinizing is very useful in speeding up the execution of FA. In the deterministic algorithm, if there are multiple jumps from a state to a certain input symbol, they will be merged into one jump. The terminal states of multiple jumps will also be merged into a new state. Therefore, a state in this deterministic FA is a collection of states of the original FA. When a new state that has been determined is given an input symbol, each of the old states in the set corresponding to this state will try to process this input symbol, and then these terminal states will form a new set (new state). Determine the dynamic bias class starting from a certain state, given an input state, and find a uniquely determined transfer path to complete the deterministic operation. Minimization is to find the DFA with the smallest number of states among all DFAs (excluding NFA) equivalent to the original FA. Using the least states to express the original WFST equivalently is similar to simplification. Judge the label state of the input label and output label in the WFST. If the label state is an empty state, delete the transfer path corresponding to the empty state to perform an empty transfer removal operation on the dynamic bias class.

[0040] In one embodiment, the pre-information of the hot word is determined according to the word segment level, and the context of the word segment level is determined according to the pre-information. The search path of the WFST is set according to the context to narrow the search scope of the WFST.

[0041] In one embodiment, during the decoding process, when a corresponding prefix is ​​matched, a corresponding score reward will be obtained. Figure 6 As shown in the figure, the backoff arc will backoff the previously accumulated reward scores to an arc at one time, causing the cost of the arc to be particularly large and thus pruned. Therefore, before pruning, we need to remove the cost generated by the backoff arc.

[0042] like Figure 7 As shown, the embodiment of the present application also provides a hot word enhancement device for speech recognition, including:

[0043] at least one processor; and,

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the hot word enhancement device for speech recognition to execute the method described in any one of the above embodiments.

[0046] An embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: a method as described in any one of the above embodiments.

[0047] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0048] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0049] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0050] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0054] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0055] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0056] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0057] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A hot word enhancement method for speech recognition, characterized in that: include: Obtaining an audio file of a hot word through an encoder, extracting features according to the audio file, and sending the extracted features to a CTC decoder to obtain a streaming recognition result through the CTC decoder; Shallowly fusing the streaming recognition result with a language model to bias the recognition result according to a prefix tree to obtain a search graph; Perform shallow fusion through WFST to obtain the best path according to the search graph, send the best path to the attention decoder, obtain accurate results through the attention decoder, and complete the enhancement of the hot words; Before performing shallow fusion through WFST, the method further includes: Determining a decoding stage of the WFST, and establishing a dynamic bias class according to the preset hot words in the decoding stage; A segment level is determined according to the hot words, an input tag of the WFST is determined according to the segment level, and a context prefix is ​​set for the WFST to narrow the search scope of the WFST.

2. The method according to claim 1, characterized in that Establish a dynamic bias class, including: Determining an input state and a transfer path of the dynamic bias class to perform a deterministic operation and a minimization operation on the dynamic bias class according to the input state and the transfer path; The label states of the input label and the output label in the WFST are judged, and if the label state is an empty state, the transition path corresponding to the empty state is deleted to perform an empty transition removal operation on the dynamic bias class.

3. The method according to claim 1, characterized in that The context prefix is ​​set for the WFST, specifically including: Determine the pre-information of the hot word according to the word segment level, and determine the context of the word segment level according to the pre-information; The search path of the WFST is set according to the context to narrow the search scope of the WFST.

4. The method according to claim 1, characterized in that: The method further comprises: A reward score of the WFST is determined according to the context prefix, and a backoff arc of the WFST is determined to determine a cumulative reward score of the backoff arc, and the cumulative reward score is removed.

5. The method according to claim 1, characterized in that Before inputting the streaming recognition result into a language model for shallow fusion and biasing the recognition result according to a prefix tree, the method further includes: Determine a segment level according to the hot words, and encode the prefix tree according to the segment level to obtain a query table; A bias module is determined according to the query table, and the bias module is added to the language model.

6. The method according to claim 5, characterized in that The segment level is set by the following formula: ； in, represents the segment level, Indicates that the corresponding word segment of the hot word starts to be identified in the query table, Indicates continuing to search for the complete hot word in the prefix tree.

7. The method according to claim 1, characterized in that The audio file of the hot word is obtained through the encoder, and features are extracted according to the audio file, and the extracted features are sent to the CTC decoder to obtain the streaming recognition result through the CTC decoder, which specifically includes: Encoding the audio file by the encoder to obtain the features, and recognizing the features by the CTC decoder to obtain a plurality of initial recognition results; Determining multiple weight values ​​of the multiple initial recognition results, and scoring the multiple weight values ​​to determine the intermediate recognition result with the highest weight value; The intermediate recognition result is recognized again to obtain the streaming recognition result.

8. A hot word enhancement device for speech recognition, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the hot-word enhancement device for speech recognition to execute: the method according to any one of claims 1-7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as follows: a method as described in any one of claims 1 to 7.

Images