A low memory consumption and efficient offline command word recognition system and modeling method
By compressing model parameters and optimizing the decoding network, the problem of misrecognition of command words under environmental noise in low memory conditions was solved, achieving efficient and accurate voice command recognition while reducing memory consumption and computational complexity.
Patent Information
- Application Number
- CN202111061384.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-10
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-10
AI Technical Summary
Existing offline command word recognition systems struggle to achieve efficient and accurate voice command recognition under conditions of low memory and low computing resources, and are prone to misrecognition, especially in the presence of environmental noise.
The model parameter compression and decoding network optimization techniques are adopted, including mapping the neural network model parameters to the range of [-127, 127] and storing them with 8-bit integers, replacing floating-point multiplication with 16-bit integer multiplication, and combining the improved Viterbi algorithm and filler branch decoding network to optimize node merging and confidence judgment during the decoding process.
It achieves 99% command word recognition accuracy under extremely low memory and computing resource conditions, while reducing memory consumption by 70%-90% and computational complexity, and reducing the probability of false recognition.
Smart Images

Figure CN113963690B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a high-efficiency off-line command word recognition system with low memory consumption and a modeling method, belonging to the technical field of speech recognition. Background Art
[0002] Command word recognition technology is an artificial intelligence technology that enables machines to recognize and understand voice commands and convert corresponding voice commands into corresponding computer commands; this technology includes voice endpoint detection (VAD), acoustic modeling, weighted finite state machine (WFST) network construction and speech decoding.
[0003] In recent years, with the rapid development of wearable devices, smart homes, and smart cars, more and more terminals can use command word recognition technology to offline recognize simple commands issued by users, and then automatically complete the commands conveyed by users. However, due to the limitations of battery storage capacity, standby time, power consumption management, and cost control, command word recognition usually faces very stringent computing and storage resources. For example, in order to reduce system power consumption and increase the standby time of smart watches and Bluetooth headsets, product designers may even require the memory usage of the entire command word recognition system to be compressed to about 100kB.
[0004] When acoustic modeling is performed on traditional offline command words, the trained neural network model is compressed using 16-bit fixed-point model compression technology to convert the matrix parameters in the acoustic model from floating-point numbers to 16-bit integers, thereby reducing the model's memory and computing resource consumption. A WFST decoding network is created based on application requirements, and then the audio features are carefully extracted. The extracted features are fed into the model for forward operation, and the output of the forward operation is fed into the decoder. Viterbi decoding is used to synchronously search for the command word text that best matches the audio frame by frame. Traditional Viterbi decoding needs to generate a decoding grid during the decoding process to store the active nodes at each moment, so as to backtrack and find the best matching path at the termination moment.
[0005] Considering that the input audio of the command word system is not all voice commands issued by the user, it is quite likely to be environmental noise, the sound of other people talking, etc. Therefore, in order to prevent misrecognition, that is, the input audio is environmental noise, the command word recognition still outputs the command word text, and finally the confidence level needs to be calculated based on the decoding result to measure the system's trust in the current decoding result. The decoding result will be output only when the confidence level is higher than a certain threshold. Summary of the invention
[0006] In view of the above-mentioned technical problems, the purpose of the present invention is to propose an efficient offline command word recognition system and modeling method with low memory consumption, which greatly reduces the computational complexity and memory consumption of the command word recognition system, and ensures that the accuracy of command words can still reach 99% under quiet conditions with extremely low computing resources and memory resources (about 100kB).
[0007] The technical solution of the present invention is achieved as follows: a highly efficient offline command word recognition system with low memory consumption, comprising:
[0008] Model parameter compression module: maps the parameters of the trained VAD neural network model and command word recognition neural network model to the numerical range of [-127, 127], and uses 8-bit integer data type to save the model parameters;
[0009] Compression model forward operation module: extract features from input audio and use them as input to the neural network; map input features and the input of each layer of the neural network model to the numerical range of [-127, 127], store them in 8-bit integer data, and use 16-bit integer multiplication instead of floating-point multiplication to complete the forward operation of the compression neural network model;
[0010] Decoding network generation module: Generates phoneme sequences from general recognition training texts, performs statistics on the phoneme sequences, removes diphone sequences existing in command word phoneme sequences, and obtains the N diphone sequences with the highest probability; these N diphone sequences are connected in parallel and embedded in the command word decoding network as the filler branch of the final decoding network to decode background noise;
[0011] Command word decoding module: The forward operation result of the command word compression model obtained by the compression model forward operation module is used in conjunction with the decoding network obtained by the decoding network generation module to perform decoding using the improved Viterbi algorithm; during the decoding process, the active nodes at time t of the generated decoding grid are merged with the active nodes at time t-1; and after the nodes are merged, the memory space corresponding to all active nodes at time t-1 in the decoding grid is cancelled, and then the confidence is calculated, and a decision is made whether to output the matched word sequence based on the confidence and the decoding result.
[0012] An efficient offline command word recognition modeling method with low memory consumption includes the following steps:
[0013] Step 101: Use the following formula to train the matrix of the command word neural network model and the VAD neural network model for the l-layer neural network matrix W l Each line of the command is compressed:
[0014] W′ l =[W l , bl ]
[0015] R l (i) = 127 / w′ li,max
[0016] w″ l (i, j) = round(R l (i) w′ l (i, j))
[0017] Where W l Neural network layer l matrix, b l Represents the bias of the lth layer of the neural network, R l (i) indicates W l The fixed-point coefficients of the i-th row, w′ l,i,max Denote the matrix W′ l The maximum absolute value of the data in the i-th row, w′ l (i, j) represents the matrix W′ l The value of the i-th row and j-th column in w l ″(i, j) represents the matrix W l The compressed data of row i and column j; after the above formula, the matrix W′ l The parameters of each row are mapped to the value range of [-127, 127], and the compressed parameter w" after mapping l (i, j) can be directly stored using an 8-bit integer; the VAD neural network model and the command word recognition model are compressed using the above method to obtain a VAD neural network compression model and a command word recognition neural network compression model;
[0018] Step 201: Use the following formula to calculate the input vector h of the neural network layer l l To compress parameters:
[0019]
[0020] Q l =127 / h′ l,max
[0021] h″ l (j) = round(Q l h′ l (j))
[0022] where h′ l,max Represents the vector h′ l Take the maximum absolute value, h′ l (j) represents the vector h′ l The jth value in h″ l (j) represents the vector h′ lThe jth parameter after compression; after the above formula, the vector h′ l It is mapped to the value range of [-127, 127], and the compression parameter after mapping is h″ l (j) The compression parameter can be directly stored using an 8-bit integer;
[0023] Step 202: Use the following formula to complete the matrix operation of the lth layer in the neural network:
[0024]
[0025] The above formula successfully converts the matrix floating-point multiplication operation into a 16-bit integer multiplication operation;
[0026] Step 301: Generate a universal recognition phoneme sequence from the annotated text of the universal recognition data and the dictionary; perform statistics on the phoneme sequences to obtain M diphone sequences with the highest probability; generate a command word phoneme sequence from the command word text using the same method, and perform statistics on the diphone sequences therein; check the M diphone sequences obtained from the universal recognition phoneme sequence, remove the diphone sequences included in the command word phoneme sequence, and finally select N diphone sequences with the highest probability;
[0027] Step 302: Generate a decoding network for the command word part according to the text of the command word; connect the N diphone sequences obtained in step 201 in parallel with the command word decoding network as the filler branch of the command word decoding network, which is used to model the speech of non-command words; when the input audio is a command word, the decoding will match the path of the command word part; when the input audio is background noise, the decoding will tend to match the path of the filler part;
[0028] Step 401: extract Mel-frequency cepstrum features after frame-by-frame windowing of the input audio, use the features as input of the VAD neural network compression model, perform forward operation of the VAD neural network compression model using steps 201-202, and determine whether the current audio segment is a speech segment based on the calculation results;
[0029] Step 402: if it is determined to be a speech segment, the Mel-frequency cepstrum feature is re-extracted from the speech segment according to the feature configuration of the command word model, and the feature is used as the input of the command word recognition neural network compression model, and the forward operation of the command word neural network compression model is completed by using steps 201-202;
[0030] Step 501: Use the forward operation of the command word neural network model obtained in step 402 and the decoding network obtained in step 302 to perform decoding using the improved Viterbi algorithm; when decoding at time t, select the node with the largest score from all active nodes at time t-1 connected to the active node at time t for merging, and the merged node will store the input and output sequences of the two nodes, and the remaining nodes at time t-1 that are not involved in the merger will be cancelled in the memory;
[0031] Step 502: read the next frame of data output by the neural network, and repeat the improved Viterbi algorithm decoding in step 501 until the last frame;
[0032] Step 503: Detect the final decoded word sequence. If the output sequence is a command word, calculate the confidence. If the confidence is greater than a preset threshold, output the word sequence. If it is less than a threshold, do not output it. If the output sequence is a diphone sequence of a filler path, do not output the word sequence.
[0033] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:
[0034] The present invention proposes an improved model compression method, which converts the floating-point parameters of the matrix in the model into 8-bit integer data; in order to offset the performance loss caused by the reduction in the number of storage bits, the numerical dynamic range of the traditional fixed-point technology for the matrix is changed from the entire matrix to each row, and parameter compression is performed on each row of the matrix separately; at the same time, the matrix floating-point multiplication operation of the model forward operation is converted into a 16-bit integer multiplication operation.
[0035] The present invention merges the nodes in the decoding grid in real time, greatly reducing the number of nodes in the decoding grid and significantly reducing the memory consumption in the decoding process without reducing the decoding performance; finally, in order to reduce the probability of misrecognition of command words, the present invention adds a filler branch in the traditional WFST decoding network to recognize background noise speech, and combines the decoding confidence to significantly reduce the probability of misrecognition of command words. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:
[0037] Attached Figure 1 A schematic diagram of a high-efficiency offline command word recognition system with low memory consumption according to the present invention;
[0038] Attached Figure 2 The decoding network generated by the decoding network generation module in step 302 of the present invention;
[0039] Attached Figure 3A decoding grid generated by a conventional Viterbi decoding algorithm in step 502 of the present invention;
[0040] Attached Figure 4 A decoding grid generated by improving the Viterbi algorithm for the command word decoding module in step 302 of the present invention;
[0041] Attached Figure 5 This is a flow chart of an implementation of a high-efficiency offline command word recognition system with low memory consumption according to the present invention. DETAILED DESCRIPTION
[0042] The present invention will be described below in conjunction with the accompanying drawings.
[0043] As attached Figure 1-5 As shown, the present invention provides a low memory consumption and efficient offline command word recognition system, comprising:
[0044] Model parameter compression module: merge the matrices and biases of each layer in the trained VAD neural network model and command word recognition neural network model into one matrix; map each row parameter of the merged matrix to the numerical range of [-127, 127] separately, and use 8-bit integer data type to store the mapped parameters, reducing the model parameter memory consumption by 70% to 75%.
[0045] Compression model forward operation module: extract features from the input audio and use the features as the input of the neural network model compressed by the model parameter compression module; map the input of each layer of the neural network model to the numerical range of [-127, 127] (the input layer features are [-32767, 32767]), and use 8-bit integer data storage (input layer features use 16-bit storage), and use 16-bit (input layer is 32-bit) integer multiplication operations instead of floating-point multiplication operations to complete the forward operation of the compressed neural network model, greatly reducing the computational complexity of the command word recognition system.
[0046] Decoding network generation module: Generate phoneme sequences from general recognition training texts, perform statistics on the phoneme sequences, remove the diphone sequences in the command word phoneme sequences, and obtain the N diphone sequences with the highest probability; these N diphone sequences are connected in parallel and embedded in the command word decoding network as the filler branch of the final decoding network for decoding background noise. In this way, the decoding network has the ability to distinguish between command words and background noise, which can greatly reduce the probability of misrecognition by the command word recognition system.
[0047] Command word decoding module: The forward operation result of the command word compression model obtained by the compression model forward operation module is used in conjunction with the decoding network obtained by the decoding network generation module to perform decoding using the improved Viterbi algorithm. During the decoding process, the active nodes at time t of the generated decoding grid are merged with the active nodes at time t-1. After the nodes are merged, the nodes created at time t-1 in the decoding grid are cancelled in the memory, which can reduce the memory consumption of decoding by 75% to 90% without changing the decoding performance. The confidence is then calculated, and a decision is made whether to output the matched word sequence based on the confidence and the decoding result.
[0048] The modeling method of the above-mentioned low memory consumption and efficient offline command word recognition system includes the following steps:
[0049] Step 101: Use the following formula to train the matrix of the command word neural network model and the VAD neural network model for the l-layer neural network matrix W l Each line of the command is compressed:
[0050] W′ l =[W l , b l ]
[0051] R l (i) = 127 / w′ l,i,max
[0052] w″ l (i, j) = round(R l (i) w′ l (i, j))
[0053] Where W l Neural network layer l matrix, b l represents the bias of the lth layer of the neural network, R l (i) indicates W l The fixed-point coefficients of the i-th row, w′ l,imax Denote the matrix W′ l The maximum absolute value of the data in the i-th row, w′ l (i, j) represents the matrix W′ l The value of the i-th row and j-th column in w l ″(i, j) represents the matrix W l The compressed data of row i and column j; after the above formula, the matrix W′ l The parameters of each row are mapped to the value range of [-127, 127]. The compressed parameter w after mapping is l ″(i, j) can be directly stored using 8-bit integers, which greatly reduces the storage consumption of the system. lParameter mapping is performed to reduce the quantization error generated during parameter compression, thereby reducing the degradation of model performance caused by the quantization error. The above method is used to compress the parameters of the VAD neural network model and the command word recognition model to obtain the VAD neural network compression model and the command word recognition neural network compression model.
[0054] Step 201: Use the following formula to calculate the input vector h of the neural network layer l l To compress parameters:
[0055]
[0056] Q l =127 / / h′ l,max
[0057] h″ l (j) = round(Q l h′ l (j))
[0058] where h′ l,max Represents the vector h′ l Take the maximum absolute value, h′ l (j) represents the vector h′ l The jth value in h″ l (j) represents the vector h′ l The jth parameter after compression. After the above formula, the vector h′ l It is mapped to the numerical range of [-127, 127] (the input layer feature is [-32767, 32767]), and the compression parameter after mapping is h″ l (j) The compression parameter can be directly stored using an 8-bit (the input layer feature is 16-bit) integer.
[0059] Step 202: Use the following formula to complete the matrix operation of the lth layer in the neural network:
[0060]
[0061] The above formula successfully converts the matrix floating-point multiplication operation into a 16-bit (32-bit for the input layer) integer multiplication operation, greatly reducing the computational complexity of the command word recognition system.
[0062] Step 301: Generate a universal recognition phoneme sequence from the annotated text and dictionary of the universal recognition data. Count the phoneme sequences to obtain the M diphone sequences with the highest probability. Generate a command word phoneme sequence from the command word text in the same way, and count the diphone sequences therein. Check the M diphone sequences obtained from the universal recognition phoneme sequence, remove the diphone sequences contained in the command word phoneme sequence, and finally select the N diphone sequences with the highest probability.
[0063] Step 302: Generate a decoding network for the command word part according to the text of the command word. The N diphone sequences obtained in step 201 are connected in parallel with the command word decoding network as the filler branch of the command word decoding network, which is used to model the speech of non-command words to reduce the probability of misrecognition of the command word system. Figure 2 The decoding network with filler branches is shown. The upper half of the network is the command word branch, and the lower half is the filler branch composed of N=2 diphone sequences. When the input audio is a command word, the decoding will match the path of the command word part. When the input audio is background noise, the decoding will tend to match the path of the filler part.
[0064] Step 401: extract Mel-frequency cepstrum features after frame-by-frame windowing of the input audio, use the features as input to the VAD neural network compression model, complete the forward operation of the VAD neural network compression model using steps 201-202, and determine whether the current audio segment is a speech segment based on the calculation results.
[0065] Step 402: If it is determined to be a speech segment, the Mel-frequency cepstrum feature is re-extracted from the speech segment according to the feature configuration of the command word model, and the feature is used as the input of the command word recognition neural network compression model, and the forward operation of the command word neural network compression model is completed using steps 201-202.
[0066] Step 501: Use the forward operation of the command word neural network model obtained in step 402 and the decoding network obtained in step 302 to decode using the improved Viterbi algorithm; different from the traditional Viterbi decoding algorithm, the improved Viterbi algorithm will select the node with the largest score from all active nodes at time t-1 connected to the active node at time t when decoding at time t, and merge them. The merged node will store the input and output sequences of the two nodes, and the remaining nodes that did not participate in the merge at time t-1 will be cancelled in the memory.
[0067] Step 502: Traditional Viterbi decoding needs to save the active nodes of each frame and the connection relationship between the active nodes to generate a decoding grid. Figure 3 A simple decoding grid at time t=5 under traditional Viterbi decoding is shown. The first character above the arrow in the network represents the input phoneme, and the second character represents the output word. The path of ace is the path with the largest score. The output sequence on this path is "A∈Cεε" or "AC" (ε represents the null symbol and needs to be removed) which is the final output word sequence of the decoding. Since the active nodes at each moment need to be stored in memory, the decoding grid will take up more memory space in actual decoding. Figure 4Demonstrated in Figure 3 Under the conditions of the same input audio and decoding network, the changes in the decoding grid from t=3 to t=5 in step 401 are adopted. Nodes 6 and 7 at t=3 are merged with the active node with the highest score connected to t-1 (that is, node 4). The merged nodes 6 and 7 store the best input sequence and the best output sequence corresponding to the node at the current moment. After the merger is completed, nodes 4 and 5 will be cancelled in the memory. Then, at the next t=4 moment, the merged nodes 6 and 7 are connected to nodes 8 and 9 respectively, and the merger with nodes 8 and 9 is completed respectively, and nodes 6 and 7 are cancelled in the memory. This cycle is repeated until the last frame. Figure 3 contrast, Figure 4 At time t=5, the node data of the decoding grid is much smaller than that of the traditional Viterbi decoding, which can greatly reduce the memory consumption in the decoding process.
[0068] Step 503: Detect the final decoded word sequence. If the output sequence is a command word, calculate the confidence. If the confidence is greater than a preset threshold, output the word sequence. If it is less than a preset threshold, do not output it. If the output sequence is a filler path diphone sequence, do not output the word sequence.
[0069] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the content of the present invention and implement it. They cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A low memory consumption and efficient offline command word recognition modeling method, characterized in that: The following steps are involved: Step 101: Use the following formula to train the matrix of the command word neural network model and the VAD neural network model for the l-layer neural network matrix W l Each line of the command is compressed: W' l =[W l ,b l ] R l (i)=127 / w′ l,i,max w” l (i,j)=round(R l (i)w’ l (i,j)) Where W l Neural network layer l matrix, b l Represents the bias of the lth layer of the neural network, R l (i) indicates W l The fixed-point coefficients of the i-th row, w′ l,i,max Denote the matrix W′ l The maximum absolute value of the data in the i-th row, W′ l (i, j) represents the matrix W′ l The value of the i-th row and j-th column in , w″ l (i, j) represents the matrix W l The compressed data of row i and column j; after the above formula, the matrix W′ l The parameters of each row are mapped to the value range of [-127, 127], and the compressed parameters W" after mapping l (i, j) can be directly stored using an 8-bit integer; the VAD neural network model and the command word recognition model are compressed using the above method to obtain a VAD neural network compression model and a command word recognition neural network compression model; Step 201: Use the following formula to calculate the input vector h of the neural network layer l l To compress parameters: Q l =127 / h′ l,max h″ l (j)=round(Q l h′ l (j)) where h′ l,max Represents the vector h′ l Take the maximum absolute value, h′ l (j) represents the vector h′ l The jth value in h″ l (j) represents the vector h′ l The jth parameter after compression; after the above formula, the vector h′ l It is mapped to the value range of [-127, 127], and the compression parameter after mapping is h″ l (j) The compression parameter can be directly stored using an 8-bit integer; Step 202: Use the following formula to complete the matrix operation of the lth layer in the neural network: The above formula successfully converts the matrix floating-point multiplication operation into a 16-bit integer multiplication operation; Step 301: Generate a universal recognition phoneme sequence from the annotated text of the universal recognition data and the dictionary; perform statistics on the phoneme sequences to obtain M diphone sequences with the highest probability; generate a command word phoneme sequence from the command word text using the same method, and perform statistics on the diphone sequences therein; check the M diphone sequences obtained from the universal recognition phoneme sequence, remove the diphone sequences included in the command word phoneme sequence, and finally select N diphone sequences with the highest probability; Step 302: Generate a decoding network for the command word part according to the text of the command word; connect the N diphone sequences obtained in step 201 in parallel with the command word decoding network as the filler branch of the command word decoding network, which is used to model the speech of non-command words; when the input audio is a command word, the decoding will match the path of the command word part; when the input audio is background noise, the decoding will tend to match the path of the filler part; Step 401: extract Mel-frequency cepstrum features after frame-by-frame windowing of the input audio, use the features as input of the VAD neural network compression model, perform forward operation of the VAD neural network compression model using steps 201-202, and determine whether the current audio segment is a speech segment based on the calculation results; Step 402: if it is determined to be a speech segment, the Mel-frequency cepstrum feature is re-extracted from the speech segment according to the feature configuration of the command word model, and the feature is used as the input of the command word recognition neural network compression model, and the forward operation of the command word neural network compression model is completed by using steps 201-202; Step 501: Use the forward operation of the command word neural network model obtained in step 402 and the decoding network obtained in step 302 to perform decoding using the improved Viterbi algorithm; when decoding at time t, select the node with the largest score from all active nodes at time t-1 connected to the active node at time t for merging, and the merged node will store the input and output sequences of the two nodes, and the remaining nodes at time t-1 that are not involved in the merger will be cancelled in the memory; Step 502: read the next frame of data output by the neural network, and repeat the improved Viterbi algorithm decoding in step 501 until the last frame; Step 503: Detect the final decoded word sequence. If the output sequence is a command word, calculate the confidence. If the confidence is greater than a preset threshold, output the word sequence. If it is less than a threshold, do not output it. If the output sequence is a diphone sequence of a filler path, do not output the word sequence.
