Method for improving two-stage voice wake-up
By using the stream convolution enhancement deformer model and connection timing classification forced alignment method using self-attention mechanism in the voice wake-up system, the problems of false triggering and missed wake-up in the prior art are solved, and the recognition rate and system reliability are improved , adapt to the needs of complex scenarios.
Patent Information
- Application Number
- CN202411304286.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing voice wake-up systems are prone to problems of mistriggering and miss wake-up in complex scenarios, and traditional models are difficult to balance power consumption and performance in the context of environmental noise and complex voice, resulting in a decrease in recognition rate.
The streaming convolution enhancement deformer model based on self-attention mechanism is used for all-weather monitoring in the first stage, and the connection timing classification forced alignment method is introduced in the training stage to ensure the consistency of inputs in the training and inference stages, and by maximizing audio inputs The alignment probability with the target sequence is obtained by obtaining the timestamp of each tag, and the valid feature interval is intercepted for secondary verification.
It significantly improves the accuracy of voice wake-up and the reliability of the system, reduces power consumption, reduces false triggering, enhances the robustness and generalization capabilities of the model, and adapts to a variety of complex application scenarios.
Smart Images

Figure CN118968972A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a method for improving two-stage speech wake-up. Background Art
[0002] With the popularity of smart devices, voice wake-up technology, as an important means of human-computer interaction, has received more and more attention. In daily life, users can wake up devices with simple voice commands and achieve natural interaction with smart devices. However, existing voice wake-up systems are prone to false triggering and missed wake-ups in complex scenarios, affecting user experience and system reliability. Existing voice wake-up technology mainly uses a single model for wake-up detection, which has some obvious defects and shortcomings.
[0003] First, most existing single models use traditional acoustic features and machine learning algorithms. These models are easily disturbed when facing environmental noise and complex speech backgrounds, resulting in a high false trigger rate. In addition, traditional models are difficult to achieve high recognition rates while ensuring low power consumption, especially in scenarios with high real-time requirements, where it is even more difficult to strike a balance between power consumption and performance.
[0004] Secondly, the prior art usually adopts a two-stage wake-up mode. In the first stage, a low-power model is used for all-weather monitoring, and in the second stage, a high-performance model is used to perform secondary verification of possible wake-up signals, thereby effectively reducing the false trigger rate. However, due to the mismatch between training and reasoning, that is, the secondary model receives the acoustic features of the complete sentence during training, which usually has reserved segments of uncertain time at the beginning and end, while the valid segments intercepted by the primary model are received during reasoning. This inconsistency leads to a decrease in the performance of the wake-up model. Therefore, how to provide a method to improve two-stage voice wake-up is an urgent problem that technicians in this field need to solve. Summary of the invention
[0005] This patent successfully solves the performance degradation problem caused by inconsistency between training and reasoning by introducing a method to maintain feature alignment during training. Specifically, this patent introduces a connection temporal classification forced alignment (CTC Force alignment) method in training, using the connection temporal classification probability distribution and label sequence output by the first-level model to accurately obtain the timestamp of each label, intercept the valid interval of the first-level model output through these start and end timestamps, and input it into the second-level model. This method ensures the consistency of the input of the second-level model during the training and reasoning stages, thereby significantly improving the computational efficiency and recognition rate.
[0006] A method for improving two-stage voice wake-up according to an embodiment of the present invention includes the following steps:
[0007] S1. Use a low-power consumption model to perform the first stage of all-weather monitoring. The low-power consumption model uses a streaming convolution-enhanced deformer model based on a self-attention mechanism for real-time wake-up detection and performs preliminary processing and feature extraction on the detected audio signal.
[0008] S2. When the low power consumption model detects a possible wake-up signal, roll back some historical feature segments from the circular cache and delay some feature segments to obtain a complete wake-up signal segment, wherein the historical feature segment specifically includes audio data before and after the wake-up signal to ensure complete voice input;
[0009] S3. In the training phase, the connection time classification forced alignment method is introduced to obtain the timestamp of each label by maximizing the alignment probability between the audio input and the target sequence;
[0010] S4. According to the acquired timestamp, intercept the valid feature interval output by the first-stage model, ensure that the intercepted audio contains the complete features of the wake-up signal, and remove irrelevant background noise and interference;
[0011] S5, inputting the intercepted valid feature interval into the second stage for secondary verification, wherein the second stage model adopts an offline convolution-enhanced deformer model based on a self-attention mechanism to accurately analyze and identify the input audio feature segment;
[0012] S6. In the reasoning stage, when the first-stage model detects the wake-up state, it obtains the valid acoustic feature fragment from the circular cache, inputs the valid acoustic feature fragment into the second-stage model to perform self-attention reasoning on the whole sentence, obtains a new wake-up word probability, and makes a final decision.
[0013] Further, the S1 specifically includes:
[0014] S11. A deformer model enhanced by streaming convolution based on a self-attention mechanism, wherein the model comprises a plurality of convolutional layers and a self-attention layer, wherein the self-attention layer captures the global context information of the audio features by calculating the attention weights of the input audio features;
[0015] S12, dividing the detected audio signal into multiple time frames, each time frame has a length of T milliseconds, inputting the time frame into the deformer model for feature extraction, and extracting a feature vector x i ;
[0016] S13. For each time frame i, calculate the output of the self-attention mechanism:
[0017]
[0018] Where Q = W Q ·x i is the query matrix, K = W K ·x i is the key matrix, V = W V ·x i is the value matrix, W Q , W K , W V is the trainable weight matrix, d k is the dimension of the key vector, and the softmax function is used to calculate the attention weight;
[0019] S14. Multiple different attention weights are calculated in parallel through the multi-head self-attention mechanism, the output of each head is concatenated, and the final feature representation is obtained through linear transformation:
[0020] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ;
[0021] Among them, head i represents the output of the i-th attention head, W O represents a linear transformation matrix;
[0022] S15. The output of the multi-head self-attention mechanism is further subjected to feature extraction through a convolutional layer. The output of the convolutional layer is processed through residual connection and layer normalization to obtain the final feature representation of each time frame:
[0023] Output i =LaerNorm(x i +Conv(x i ));
[0024] Among them, Conv represents the convolution operation, LayerNorm represents the layer normalization operation, and Output i represents the final feature representation of the i-th frame;
[0025] S16: Input the processed feature representation into the subsequent processing module for real-time wake-up detection and generate a preliminary wake-up probability. For a specific wake-up word w, assume that the target sequence of w is represented by y w , we need to calculate the output y given an input sequence x w The probability of . The formula is as follows:
[0026]
[0027] Among them, B -1 (y w) represents all the symbols that can be used to generate y by removing blank symbols and merging repeated symbols. w The alignment path of
[0028] S17, the output features of the first-stage model are stored in the circular cache according to the maximum window length. When a possible wake-up signal is detected, valid features are extracted from the circular cache and sent to the second-stage model for inference.
[0029] Furthermore, the S3 specifically includes:
[0030] S31. Preprocess the training data and convert the original audio signal into an acoustic feature vector sequence:
[0031] X={x1,x2,…,x T};
[0032] Among them, X represents the acoustic feature vector sequence, x t represents the acoustic feature vector at time t, T represents the total number of time frames;
[0033] S32, generate target tag sequence:
[0034] Y={y1,y2,…,y L};
[0035] Among them, Y represents the target label sequence, y l represents the label at position l, and L represents the length of the target sequence;
[0036] S33, application connection time series classification loss function L CTC , calculate the alignment probability P(Y|X) between the audio input and the target label sequence. The specific formula is:
[0037]
[0038] Among them, L CTC represents the connection time series classification loss function, π represents all possible alignment paths, B represents the mapping function to remove duplicate labels and blank symbols, represents the path π at time t t The probability of
[0039] S34, calculate the probability distribution of the alignment path through the forward-backward algorithm, specifically including the forward probability α t (s) and the backward probability β t The recursive formula of (s):
[0040]
[0041] Among them, s represents the label state, S represents the set of all possible label states, α t(s) represents the forward probability, β t (s) represents the backward probability;
[0042] S35. Calculate each label y according to the probability distribution of the alignment path l Timestamp t l , the timestamp calculation formula is:
[0043]
[0044] Among them, t l Represents each label y l The timestamp, P(s t =y l |X) represents the label y at time t l The alignment probability of
[0045] S36, obtain the complete timestamp {t1, t2, ..., t L}.
[0046] Furthermore, the S5 specifically includes:
[0047] S51, the intercepted effective feature interval is an acoustic feature vector, wherein the acoustic feature vector represents feature information of the audio signal in different time frames;
[0048] S52, inputting the acoustic feature vector into a second-stage model, where the second-stage model is an offline convolution-enhanced deformer model based on a self-attention mechanism, and the model includes a multi-layer convolutional neural network and a multi-head self-attention mechanism;
[0049] S53, in the second stage model, extracting local features of the acoustic feature vector through a multi-layer convolutional neural network to obtain a high-dimensional feature representation;
[0050] S54, input the high-dimensional feature representation into the multi-head self-attention mechanism, calculate the correlation between each feature frame, and form a global feature map. The calculation formula of the self-attention mechanism is as follows:
[0051]
[0052] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k represents the dimension of the key matrix;
[0053] S55, inputting the generated final feature representation into a decoding layer, and the decoding layer obtains the probability distribution of each possible wake-up word based on the calculation method of S16;
[0054] S56, normalize the output probability distribution to obtain the final wake-up word probability, and determine whether to trigger wake-up according to a preset threshold;
[0055] S57. If the probability of the wake-up word is greater than the preset threshold, the wake-up is confirmed to be successful and a corresponding operation is triggered; if the probability of the wake-up word is less than or equal to the preset threshold, it is judged as a wake-up failure and no operation is performed.
[0056] Furthermore, the S6 specifically includes:
[0057] S61, when the first-stage model detects the wake-up state, start the reading operation of the circular cache, and extract the audio feature data containing 1.5 to 2 seconds before and after the wake-up signal, wherein the extraction time range of the audio feature data is adjusted according to the length of the wake-up signal and the context information to ensure that the acquired feature fragments are complete and have high recognition;
[0058] S62, input the preprocessed audio feature data into the second stage model, the second stage model uses an offline convolution enhanced deformer model based on the attention mechanism to accurately analyze and identify the input feature fragments, specifically including:
[0059] By inputting the local features into the convolution layer layer by layer, the model's perception of local details is improved. Then, through the multi-head self-attention mechanism, the correlation between feature frames is calculated to construct a global feature map. The correlation calculation formula is as follows:
[0060]
[0061] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the scaling factor used to balance the dot product results of the query and the key;
[0062] S63: Input the high-dimensional feature representation into the decoder, generate the probability distribution of the wake-up word through self-attention reasoning, calculate the final probability of the wake-up signal, and make a final decision on whether to trigger the wake-up according to a preset threshold.
[0063] The beneficial effects of the present invention are:
[0064] (1) Reduce power consumption: The streaming convolution-enhanced deformer model based on the self-attention mechanism is used as the low-power model in the first stage, which can achieve all-weather monitoring and real-time wake-up detection while ensuring low power consumption. This design effectively reduces the overall power consumption of the system, making voice wake-up technology more suitable for battery-powered portable smart devices.
[0065] (2) Improve wake-up accuracy: In the training phase, the connection time classification forced alignment method is introduced to obtain the timestamp of each label by maximizing the alignment probability between the audio input and the target sequence. This method ensures the consistency of input in the training and inference phases, significantly improving the recognition performance and wake-up accuracy of the model.
[0066] (3) Reduce false triggers: By rolling back and delaying the historical feature segments, the integrity of the voice input is ensured and the possibility of false triggers is reduced. The second-stage model uses an offline convolution-enhanced deformer model based on the self-attention mechanism to perform secondary verification on the input audio feature segments, effectively filtering out false trigger signals and improving the reliability of the system.
[0067] (4) Improved recognition performance: In the inference stage, by obtaining effective acoustic feature fragments from the circular cache and inputting them into the second-stage model for self-attention reasoning of the entire sentence, it is possible to fully utilize contextual information for accurate recognition, further improving the recognition accuracy of the wake-up word.
[0068] (5) Improve the robustness of the system: The multi-head self-attention mechanism enables the model to effectively capture global context information when processing long-term continuous speech signals, thereby enhancing the generalization ability and robustness of the model and adapting to a variety of complex application scenarios.
[0069] (6) Efficient processing: Through the combination of multi-layer convolutional neural networks and multi-head self-attention mechanisms, multi-level and multi-scale feature extraction and analysis of audio features are performed, significantly improving the efficiency of feature extraction and recognition. The local feature extraction capability of the convolutional layer is combined with the global feature capture capability of the self-attention mechanism, enabling the model to maintain efficient processing capabilities in different speech backgrounds.
[0070] (7) Simplifying the model training and reasoning process: By introducing a forced alignment method and a joint optimization strategy, during the training phase, training audio containing only the first-level model and training audio containing the combined first-level and second-level models are randomly selected in proportion for training. This simplifies the model training and reasoning process, ensures the collaborative work of the two-level models, and improves the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0072] Figure 1 This is an overall flow chart of a method for improving two-stage voice wake-up proposed by the present invention;
[0073] Figure 2A training framework diagram of a cascade structure of a method for improving two-stage voice wake-up proposed by the present invention;
[0074] Figure 3 A two-stage voice wake-up reasoning process of a method for improving two-stage voice wake-up proposed by the present invention. DETAILED DESCRIPTION
[0075] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0076] refer to Figure 1-3 , a method for improving two-stage voice wake-up, comprising the following steps:
[0077] S1. Use a low-power consumption model to perform the first stage of all-weather monitoring. The low-power consumption model uses a streaming convolution-enhanced deformer model based on a self-attention mechanism for real-time wake-up detection and performs preliminary processing and feature extraction on the detected audio signal.
[0078] S2. When the low power consumption model detects a possible wake-up signal, roll back some historical feature segments from the circular cache and delay some feature segments to obtain a complete wake-up signal segment, wherein the historical feature segment specifically includes audio data before and after the wake-up signal to ensure complete voice input;
[0079] S3. In the training phase, the connection time classification forced alignment method is introduced to obtain the timestamp of each label by maximizing the alignment probability between the audio input and the target sequence;
[0080] S4. According to the acquired timestamp, intercept the valid feature interval output by the first-stage model, ensure that the intercepted audio contains the complete features of the wake-up signal, and remove irrelevant background noise and interference;
[0081] S5, inputting the intercepted valid feature interval into the second stage for secondary verification, wherein the second stage model adopts an offline convolution-enhanced deformer model based on a self-attention mechanism to accurately analyze and identify the input audio feature segment;
[0082] S6. In the reasoning stage, when the first-stage model detects the wake-up state, it obtains the valid acoustic feature fragment from the circular cache, inputs the valid acoustic feature fragment into the second-stage model to perform self-attention reasoning on the whole sentence, obtains a new wake-up word probability, and makes a final decision.
[0083] In this implementation, S1 specifically includes:
[0084] S11. A deformer model enhanced by streaming convolution based on a self-attention mechanism, wherein the model comprises a plurality of convolutional layers and a self-attention layer, wherein the self-attention layer captures the global context information of the audio features by calculating the attention weights of the input audio features;
[0085] S12, dividing the detected audio signal into multiple time frames, each time frame has a length of T milliseconds, inputting the time frame into the deformer model for feature extraction, and extracting a feature vector x i ;
[0086] S13. For each time frame i, calculate the output of the self-attention mechanism:
[0087]
[0088] Where Q = W Q ·x i is the query matrix, K = W K ·x i is the key matrix, V = W V ·x i is the value matrix, W Q , W K , W V is the trainable weight matrix, d k is the dimension of the key vector, and the softmax function is used to calculate the attention weight;
[0089] S14. Multiple different attention weights are calculated in parallel through the multi-head self-attention mechanism, the output of each head is concatenated, and the final feature representation is obtained through linear transformation:
[0090] MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O ;
[0091] Among them, head i represents the output of the i-th attention head, W O represents a linear transformation matrix;
[0092] S15. The output of the multi-head self-attention mechanism is further subjected to feature extraction through a convolutional layer. The output of the convolutional layer is processed through residual connection and layer normalization to obtain the final feature representation of each time frame:
[0093] Output i =LayerNorm(x i +Conv(x i ));
[0094] Among them, Conv represents the convolution operation, LayerNorm represents the layer normalization operation, and Output i represents the final feature representation of the i-th frame;
[0095] S16: Input the processed feature representation into the subsequent processing module for real-time wake-up detection and generate a preliminary wake-up probability. For a specific wake-up word w, assume that the target sequence of w is represented by y w , we need to calculate the number of y generated given an input sequence x w The probability of . The formula is as follows:
[0096]
[0097] Among them, B -1 (y w ) represents all the symbols that can be used to generate y by removing blank symbols and merging repeated symbols. w The alignment path of
[0098] S17, the output features of the first-stage model are stored in the circular cache according to the maximum window length. When a possible wake-up signal is detected, valid features are extracted from the circular cache and sent to the second-stage model for inference.
[0099] In this implementation, S3 specifically includes:
[0100] S31. Preprocess the training data and convert the original audio signal into an acoustic feature vector sequence:
[0101] X={x1,x2,…,x T};
[0102] Among them, X represents the acoustic feature vector sequence, x t represents the acoustic feature vector at time t, T represents the total number of time frames;
[0103] S32, generate target tag sequence:
[0104] Y={y1,y2,…,y L};
[0105] Among them, Y represents the target label sequence, y l represents the label at position l, and L represents the length of the target sequence;
[0106] S33, application connection time series classification loss function L CTC , calculate the alignment probability P(Y|X) between the audio input and the target label sequence. The specific formula is:
[0107]
[0108] Among them, L CTC represents the connection time series classification loss function, π represents all possible alignment paths, B represents the mapping function to remove duplicate labels and blank symbols, represents the path π at time t t The probability of
[0109] S34, calculate the probability distribution of the alignment path through the forward-backward algorithm, specifically including the forward probability α t (s) and the backward probability β t The recursive formula of (s):
[0110]
[0111] Among them, s represents the label state, S represents the set of all possible label states, α t (s) represents the forward probability, β t (s) represents the backward probability;
[0112] S35. Calculate each label y according to the probability distribution of the alignment path l Timestamp t l , the timestamp calculation formula is:
[0113]
[0114] Among them, t l Represents each label y l The timestamp, P(s t =y l |X) represents the label y at time t l The alignment probability of
[0115] S36, obtain the complete timestamp {t1, t2, ..., t L}.
[0116] In this implementation manner, S5 specifically includes:
[0117] S51, the intercepted effective feature interval is an acoustic feature vector, wherein the acoustic feature vector represents feature information of the audio signal in different time frames;
[0118] S52, inputting the acoustic feature vector into a second-stage model, where the second-stage model is an offline convolution-enhanced deformer model based on a self-attention mechanism, and the model includes a multi-layer convolutional neural network and a multi-head self-attention mechanism;
[0119] S53, in the second stage model, extracting local features of the acoustic feature vector through a multi-layer convolutional neural network to obtain a high-dimensional feature representation;
[0120] S54, input the high-dimensional feature representation into the multi-head self-attention mechanism, calculate the correlation between each feature frame, and form a global feature map. The calculation formula of the self-attention mechanism is as follows:
[0121]
[0122] Among them, Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k represents the dimension of the key matrix;
[0123] S55, inputting the generated final feature representation into a decoding layer, and the decoding layer obtains the probability distribution of each possible wake-up word based on the calculation method of S16;
[0124] S56, normalize the output probability distribution to obtain the final wake-up word probability, and determine whether to trigger wake-up according to a preset threshold;
[0125] S57. If the probability of the wake-up word is greater than the preset threshold, the wake-up is confirmed to be successful and a corresponding operation is triggered; if the probability of the wake-up word is less than or equal to the preset threshold, it is judged as a wake-up failure and no operation is performed.
[0126] In this implementation manner, S6 specifically includes:
[0127] S61, when the first-stage model detects the wake-up state, start the reading operation of the circular cache, and extract the audio features of 1.5 to 2 seconds before and after the wake-up signal, wherein the extraction time range of the audio features is adjusted according to the length of the wake-up signal and the context information to ensure that the obtained audio clip is complete and has high recognition;
[0128] S62, input the audio feature segment into the second stage model, the second stage model uses an offline convolution enhanced deformer model based on the attention mechanism to accurately analyze and identify the input feature segment, specifically including:
[0129] By inputting the local features into the convolution layer layer by layer, the model's ability to perceive local details is improved. Then, through the multi-head self-attention mechanism, the correlation between the feature frames of the audio data is calculated to construct a global feature map. The correlation calculation formula is as follows:
[0130]
[0131] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the scaling factor used to balance the dot product results of the query and the key;
[0132] S63: Input the high-dimensional feature representation into the decoder, generate the probability distribution of the wake-up word through self-attention reasoning, calculate the final probability of the wake-up signal, and make a final decision on whether to trigger the wake-up according to a preset threshold.
[0133] Embodiment 1:
[0134] The speech wake-up model adopted in the present invention adopts a cascade model, and both stages adopt a convolution-enhanced deformer (Conformer) model based on the self-attention mechanism. The model mainly uses multiple layers of multi-head self-attention to learn different attention weights in parallel. The offline model usually considers the contribution value of the acoustic features of all time frame sequences of a single sentence to the features of a certain frame, while the streaming model usually considers the contribution value of local information to the features of a certain frame.
[0135] The two models are well integrated in a cascade manner. The first level adopts a streaming attention model, which can meet the real-time wake-up requirements while ensuring a certain recognition rate. The second level adopts an offline attention model, which has better recognition performance than the streaming method and can suppress the probability of false touches without reducing the recognition rate.
[0136] During the training process, we randomly train models that only contain one level and two levels merged at a certain ratio, and the training audio is usually a whole sentence of audio, which usually contains head and tail reserved segments of indefinite length. Since the neural network is a data-driven model structure, it is necessary to keep it consistent with the training in the actual reasoning stage, otherwise the recognition rate will be greatly reduced.
[0137] However, in actual deployment, the first stage is in the online recognition stage. When the awakening state is detected, the feature segment of the first stage output before awakening (usually 1.5 to 2 seconds) is directly sent to the secondary model for secondary confirmation. In this way, the segment intercepted by the first stage is a valid wake-up speech feature segment and does not contain the head and tail reserved segments that match the training.
[0138] Faced with this mismatch between training and inference, we not only need to send valid wake-up fragments to the secondary wake-up engine, but also need to roll back some historical feature fragments and delay some features from being sent to the secondary model at the end. Only then can the problem of secondary verification failure be alleviated. This not only increases the calculation time of the secondary verification, but also may cause the risk of failing the secondary verification due to insufficient reserved fragments before and after.
[0139] In order to make the secondary verification model recognize only valid segments in the inference stage, the present invention introduces a Force Alignment method based on CTC (Connectionist Temporal Classification) in the training stage. The principle is to find the best alignment path by maximizing the alignment probability between the audio input and the target sequence (for example, text transcription). The maximum alignment probability path is found through context recursion and forward-backward algorithms. The final generated alignment path indicates the specific text position corresponding to each audio frame, thereby achieving alignment of audio and text.
[0140] By directly introducing the forced alignment method at the first-level output, the output of the first level is combined with the label probability distribution output by the CTC affine transformer and the already marked sequence labels to find the specific timestamp of each sequence label. In this way, the starting and ending positions of the features can be obtained, and the intercepted feature fragments can be sent to the secondary model during training.
[0141] In the actual training process, only the valid feature intervals are sent to the second-level model. This method ensures consistency with the inference stage.
[0142] The training of the convolution-enhanced deformer acoustic model is based on a hybrid CTC / Attention framework. During training, the model parameters are updated in a certain proportion with and without a cascade structure. The online model parameters of the two working modes are always shared during the training process. The calculation of the offline convolution-enhanced deformer is introduced in the part with the cascade structure. Before being sent to the second-level model, the forced alignment method is used to capture the effective feature fragments.
[0143] In the reasoning stage, the first level uses a streaming model to calculate the probability of the wake-up word. When the probability of the wake-up word exceeds a certain threshold, the effective acoustic feature fragment is obtained from the circular cache and sent to the second-level wake-up engine for self-attention reasoning of the whole sentence to obtain a new wake-up word probability. When the probability of the wake-up word exceeds a certain threshold, the final decision is made.
[0144] The effectiveness of the method was further verified by experiments, which compared the inference test results of the models trained without using the method of the present invention (the column for the number of training rollback frames is indicated by "-") and with using the method of the present invention with different numbers of rollback frames under the same training conditions of initial model, data set, data augmentation method, and number of iterations.
[0145] Table 1 Experimental training model reasoning test data comparison table
[0146]
[0147] It can be seen from the experimental results that under the same test conditions, the model that is not trained using the patented method has a high dependence on the input rollback frame number during inference. A larger number of rollback frames is required to ensure an acceptable recognition rate, and as the number of rollback frames decreases, the recognition rate drops significantly. However, the recognition rate of the model trained using the patented method remains basically stable under different input rollback frame numbers. In particular, the model that uses a rollback frame number of 3 during training can reduce the number of rollback frames required for inference from 20 to 3 frames while ensuring a high recognition rate, greatly reducing the amount of calculation and output delay during inference.
[0148] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A method for improving two-stage voice wake-up, characterized in that: The steps include: S1. Use a low-power consumption model to perform the first stage of all-weather monitoring. The low-power consumption model uses a streaming convolution-enhanced deformer model based on a self-attention mechanism for real-time wake-up detection and performs preliminary processing and feature extraction on the detected audio signal. S2. When the low power consumption model detects a possible wake-up signal, roll back some historical feature segments from the circular cache and delay some feature segments to obtain a complete wake-up signal segment, wherein the historical feature segment specifically includes audio data before and after the wake-up signal to ensure complete voice input; S3. In the training phase, the connection time classification forced alignment method is introduced to obtain the timestamp of each label by maximizing the alignment probability between the audio input and the target sequence; S4. According to the acquired timestamp, intercept the valid feature interval output by the first-stage model, ensure that the intercepted audio contains the complete features of the wake-up signal, and remove irrelevant background noise and interference; S5, inputting the intercepted valid feature interval into the second stage model training, wherein the second stage model adopts an offline convolution enhanced deformer model based on a self-attention mechanism to accurately analyze and identify the input audio feature segment; S6. In the reasoning stage, when the first-stage model detects the wake-up state, it obtains the valid acoustic feature fragment from the circular cache, inputs the valid acoustic feature fragment into the second-stage model to perform self-attention reasoning on the whole sentence, obtains a new wake-up word probability, and makes a final decision.
2. A method for improving two-stage voice wake-up according to claim 1, characterized in that: The S1 specifically includes: S11. A deformer model enhanced by streaming convolution based on a self-attention mechanism, the model comprising a plurality of convolutional layers and a self-attention layer, wherein the self-attention layer captures local context information of the audio features by calculating the attention weights of the input audio features; S12, dividing the detected audio signal into multiple time frames, each time frame has a length of T milliseconds, inputting the time frame into the deformer model for feature extraction, and extracting a feature vector x i ; S13. For each time frame i, calculate the output of the self-attention mechanism: Where Q = W Q ·x i is the query matrix, K = W K ·x i is the key matrix, V = W V ·x i is the value matrix, W Q , W K , W V is the trainable weight matrix, d k is the dimension of the key vector, and the softmax function is used to calculate the attention weight; S14. Multiple different attention weights are calculated in parallel through the multi-head self-attention mechanism, the output of each head is concatenated, and the final feature representation is obtained through linear transformation: MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O ; Among them, head i represents the output of the i-th attention head, W O represents a linear transformation matrix; S15. The output of the multi-head self-attention mechanism is further subjected to feature extraction through a convolutional layer. The output of the convolutional layer is processed through residual connection and layer normalization to obtain the final feature representation of each time frame: Output i =LayerNorm(x i +Conv(x i )); Among them, Conv represents the convolution operation, LayerNorm represents the layer normalization operation, and Output i represents the final feature representation of the i-th frame; S16: Input the processed feature representation into the subsequent processing module for real-time wake-up detection and generate a preliminary wake-up probability. For a specific wake-up word w, assume that the target sequence of w is represented by y w , we need to calculate the number of y generated given an input sequence x w The probability of . The formula is as follows: Among them, B -1 (y w ) represents all the symbols that can be used to generate y by removing blank symbols and merging repeated symbols. w The alignment path of S17, the output features of the first-stage model are stored in the circular cache according to the maximum window length. When a possible wake-up signal is detected, valid features are extracted from the circular cache and sent to the second-stage model for inference.
3. The method for improving two-stage voice wake-up according to claim 1, characterized in that: The S3 specifically includes: S31. Preprocess the training data and convert the original audio signal into an acoustic feature vector sequence: X={x1,x2,...,x T }; Among them, X represents the acoustic feature vector sequence, x t represents the acoustic feature vector at time t, T represents the total number of time frames; S32, generate target tag sequence: Y={y1,y2,...,y L }; Among them, Y represents the target label sequence, y l represents the label at position l, and L represents the length of the target sequence; S33, application connection time series classification loss function L CTC , calculate the alignment probability P(Y|X) between the audio input and the target label sequence. The specific formula is: Among them, L CTC represents the connection time series classification loss function, π represents all possible alignment paths, B represents the mapping function to remove duplicate labels and blank symbols, represents the path π at time t t The probability of S34, calculate the probability distribution of the alignment path through the forward-backward algorithm, specifically including the forward probability α t (s) and the backward probability β t The recursive formula of (s): Among them, s represents the label state, S represents the set of all possible label states, α t (s) represents the forward probability, β t (s) represents the backward probability; S35. Calculate each label y according to the probability distribution of the alignment path l Timestamp t l , the timestamp calculation formula is: Among them, t l Represents each label y l The timestamp, P(s t =y l |X) represents the label y at time t l The alignment probability of S36, obtain the complete timestamp {t1, t2, ..., t L }.
4. The method for improving two-stage voice wake-up according to claim 1, characterized in that: The S5 specifically includes: S51, the intercepted effective feature interval is an acoustic feature vector, wherein the acoustic feature vector represents feature information of the audio signal in different time frames; S52, inputting the acoustic feature vector into a second-stage model, where the second-stage model is an offline convolution-enhanced deformer model based on a self-attention mechanism, and the model includes a multi-layer convolutional neural network and a multi-head self-attention mechanism; S53, in the second stage model, extracting local features of the acoustic feature vector through a multi-layer convolutional neural network to obtain a high-dimensional feature representation; S54, inputting the high-dimensional feature representation into the multi-head self-attention mechanism, calculating the correlation between each feature frame, and forming a global feature map; S55, inputting the generated final feature representation into a decoding layer, and the decoding layer obtains the probability distribution of each possible wake-up word based on the calculation method of S16; S56, normalize the output probability distribution to obtain the final wake-up word probability, and determine whether to trigger wake-up according to a preset threshold; S57. If the probability of the wake-up word is greater than the preset threshold, the wake-up is confirmed to be successful and a corresponding operation is triggered; if the probability of the wake-up word is less than or equal to the preset threshold, it is judged as a wake-up failure and no operation is performed.
5. The method for improving two-stage voice wake-up according to claim 1, characterized in that: The S6 specifically includes: S61, when the first-stage model detects the wake-up state, start the read operation of the circular cache, and extract the first-stage model output including 1.5 to 2 seconds before and after the wake-up signal, wherein the extraction time range of the model output is adjusted according to the length of the wake-up signal and the context information to ensure that the obtained audio clip is complete and has high recognition; S62, inputting the valid output fragments of the first stage into the second stage model, wherein the second stage model uses an offline convolution-enhanced deformer model based on an attention mechanism to accurately analyze and identify the input audio feature fragments, specifically including: By inputting the local features into the convolution layer layer by layer, the model's ability to perceive local details is improved. Then, through the multi-head self-attention mechanism, the correlation between the feature frames of the audio data is calculated to construct the global feature map. The correlation calculation formula is as follows: Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and d k is the scaling factor used to balance the dot product results of the query and the key; S63: Input the high-dimensional feature representation into the decoder, generate the probability distribution of the wake-up word through self-attention reasoning, calculate the final probability of the wake-up signal, and make a final decision on whether to trigger the wake-up according to a preset threshold.
