End-to-end wake-up word detection method and device
By inserting virtual mute nodes between target pronunciation units, building a target-time relationship matrix and calculating the cumulative probability, the problem of insufficient detection accuracy and false awakening rate in the prior art is solved, and a higher detection rate is achieved and lower false wake-up rate.
Patent Information
- Application Number
- CN201910067818.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-01-24
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-01-24
AI Technical Summary
The existing end-to-end wake-up word detection methods have shortcomings in wake-up word detection rate and false wake-up rate, and it is difficult to improve the accuracy of the detection results.
By inserting virtual mute nodes between target pronunciation units, a target-time relationship matrix is constructed, and the cumulative probability is calculated frame by frame to determine the optimal path, thereby improving the accuracy of wake-up word detection and reducing the false wake-up rate.
It effectively improves the detection rate of wake-up words, reduces false awakening, saves human resources in model training time, and improves model training efficiency.
Smart Images

Figure CN111554271B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of voice wake-up technology, and in particular to an end-to-end wake-up word detection method and device. Background Art
[0002] Voice wake-up technology refers to the process of putting a smart device into a waiting state and then entering the voice intelligent interaction process when it detects a user's specific voice (usually a set wake-up word) in a dormant state. Voice wake-up has a wide range of applications, such as robots, speakers, and cars. The main indicators for evaluating the effect of voice wake-up are the wake-up rate and the false wake-up rate. The performance of the decoder in the voice recognition process plays a key role in this process.
[0003] At present, the wake-up detection method used by most smart devices is a wake-up word detection method based on end-to-end technology. In the traditional end-to-end model, the output of the acoustic model in the decoder generally corresponds to a pronunciation unit, which can be a word, a character, or a syllable. The acoustic model is mainly used to calculate the likelihood between the speech feature and each pronunciation template. Its input is the speech feature and the output is the posterior probability of the target pronunciation unit. For example, a speech is input, which contains the wake-up word "Hello Wangzai". The output of the end-to-end acoustic model based on the character is: the probability of the six outputs of "you", "good", "Wang", "Zi", "sil" and "other", where "sil" represents a silent output and "other" represents other speech outputs other than "Hello Wangzai".
[0004] The existing end-to-end wake-up word detection method dynamically plans the target pronunciation unit output by the acoustic model, then finds the optimal value, determines whether the optimal value of a certain path exceeds the preset threshold, and then decides whether to wake up. This detection method has at least the following disadvantages: the wake-up word detection rate needs to be improved, and there is a certain degree of false wake-up. Summary of the invention
[0005] Embodiments of the present invention provide an end-to-end wake-up word detection method and device to improve the accuracy of detection results and reduce the false wake-up rate.
[0006] To this end, the present invention provides the following technical solutions:
[0007] An end-to-end wake-up word detection method, the method comprising:
[0008] Receiving the voice to be detected;
[0009] Extracting the acoustic features of each speech frame in the speech to be detected in sequence;
[0010] Inputting the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of the target pronunciation unit in each speech frame output by the acoustic model;
[0011] Each target pronunciation unit is regarded as a node, and virtual silent nodes are inserted before and after the target pronunciation unit, and a target-time relationship matrix is obtained according to the posterior probability of the target pronunciation unit;
[0012] Calculating the cumulative probability of each node in the target-time relationship matrix frame by frame;
[0013] Determining an optimal path according to the cumulative probability of each node in the matrix;
[0014] The wake-up word detection result is determined according to the optimal path.
[0015] Optionally, the acoustic model is an end-to-end acoustic model based on the target pronunciation unit in the wake-up word.
[0016] Optionally, the target pronunciation unit is a syllable, a character, or a word.
[0017] Optionally, the method further comprises constructing the acoustic model in the following manner:
[0018] Collect wake-up word data and non-wake-up word data;
[0019] Time-marking the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data;
[0020] The acoustic model is trained using the frame-level label data.
[0021] Optionally, the time-stamping the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data includes:
[0022] Determine the label mapping relationship between wake-up words and non-wake-up words;
[0023] Aligning the wake-up word data and the non-wake-up word data respectively to obtain a corresponding relationship between each word in the wake-up word data and the non-wake-up word data and the speech frame occupied by the word;
[0024] According to the label mapping relationship and the corresponding relationship, the wake-up word data and the non-wake-up word data are respectively mapped into label forms to obtain frame-level label data.
[0025] Optionally, determining the label mapping relationship between the wake-up word and the non-wake-up word includes:
[0026] The start time period and the end time period of the wake-up word are indicated by silence;
[0027] For each word in the wake-up word, set a label corresponding to the word in order of position;
[0028] For words or characters other than silent and wake-up words, set the corresponding labels to 0.
[0029] Optionally, aligning the wake-up word data and the non-wake-up word data includes:
[0030] The wake-up word data and the non-wake-up word data are aligned respectively using a pre-established alignment model.
[0031] Optionally, the frame-by-frame calculation of the cumulative probability of each node in the target-time relationship matrix includes:
[0032] Determine the cumulative probability of the best path among all paths that can reach the node;
[0033] The cumulative probability of the optimal path is added to the probability of the node to obtain the cumulative probability of the node.
[0034] Optionally, determining the optimal path according to the cumulative probability of each node in the matrix includes:
[0035] The score of each path is calculated according to the cumulative probability of each node in the matrix, and the path with the largest score is taken as the optimal path.
[0036] Optionally, determining the wake-up word detection result according to the optimal path includes:
[0037] If the cumulative probabilities corresponding to the target pronunciation units on the optimal path are all greater than the set maximum probability threshold, it is determined that the wake-up word is detected.
[0038] Optionally, determining the wake-up word detection result according to the optimal path includes:
[0039] Determine the starting position and the ending position of each target pronunciation unit on the optimal path, and calculate the length and the average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position;
[0040] Determining whether a set condition is met according to the length and / or average probability of each target pronunciation unit on the optimal path;
[0041] If so, it is determined that the wake-up word is detected.
[0042] Optionally, the calculating the length and average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position includes:
[0043] Subtracting the end position of the target pronunciation unit from the start position to obtain the length of the target pronunciation unit;
[0044] The cumulative probability corresponding to the end position is subtracted from the cumulative probability corresponding to the start position, and then divided by the length of the target pronunciation unit to obtain the average probability of the target pronunciation unit.
[0045] Optionally, the setting conditions include:
[0046] The length of each target pronunciation unit within the set interval is greater than the set length threshold; and / or
[0047] The average probability of each target pronunciation unit within the set interval is greater than the set average probability threshold.
[0048] An end-to-end wake-up word detection device, the device comprising:
[0049] A receiving module, used for receiving the voice to be detected;
[0050] A feature extraction module, used to sequentially extract the acoustic features of each speech frame in the speech to be detected;
[0051] An acoustic detection module, used to input the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of the target pronunciation unit in each speech frame output by the acoustic model;
[0052] A matrix construction module, used to treat each target pronunciation unit as a node, insert virtual silent nodes before and after the target pronunciation unit, and obtain a target-time relationship matrix according to the posterior probability of the target pronunciation unit;
[0053] A calculation module, used for calculating the cumulative probability of each node in the target-time relationship matrix frame by frame;
[0054] An optimal path determination module, used to determine the optimal path according to the cumulative probability of each node in the matrix;
[0055] A detection module is used to determine a wake-up word detection result according to the optimal path.
[0056] Optionally, the acoustic model is an end-to-end acoustic model based on the target pronunciation unit in the wake-up word.
[0057] Optionally, the target pronunciation unit is a syllable, a character, or a word.
[0058] Optionally, the device further comprises: a model building module, used to build the acoustic model; the model building module comprises:
[0059] A data collection unit, used to collect wake-up word data and non-wake-up word data;
[0060] A marking unit, used to time-mark the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data;
[0061] A training unit is used to train the acoustic model using the frame-level label data.
[0062] Optionally, the marking unit comprises:
[0063] A mapping relationship determination unit, used to determine a label mapping relationship between a wake-up word and a non-wake-up word;
[0064] An alignment unit, used to align the wake-up word data and the non-wake-up word data respectively, to obtain a corresponding relationship between each word in the wake-up word data and the non-wake-up word data and the speech frame occupied by the word;
[0065] A mapping unit is used to map the wake-up word data and the non-wake-up word data into label forms according to the label mapping relationship and the corresponding relationship, so as to obtain frame-level label data.
[0066] Optionally, the mapping relationship determination unit is specifically used to represent the start time period and end time period of the wake-up word using silence; for each character in the wake-up word, set the label corresponding to the character in order of position; for other words or characters other than silence and the wake-up word, set the corresponding label to 0.
[0067] Optionally, the alignment unit is specifically configured to align the wake-up word data and the non-wake-up word data respectively using a pre-established alignment model.
[0068] Optionally, the calculation module is specifically used to determine the cumulative probability of the best path among all paths that can reach the node, and add the cumulative probability of the best path to the probability of the node to obtain the cumulative probability of the node.
[0069] Optionally, the optimal path determination module is specifically configured to calculate the score of each path according to the cumulative probability of each node in the matrix, and take the path with the largest score as the optimal path.
[0070] Optionally, the detection module is specifically used to determine whether the cumulative probabilities corresponding to each target pronunciation unit on the optimal path are greater than a set maximum probability threshold. If so, it is determined that the wake-up word is detected.
[0071] Optionally, the detection module includes:
[0072] A determination unit, used to determine the starting position and the ending position of each target pronunciation unit on the optimal path, and calculate the length and the average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position;
[0073] A judgment unit is used to determine whether a set condition is met based on the length and / or average probability of each target pronunciation unit on the optimal path, and if so, determine that a wake-up word is detected.
[0074] Optionally, the determination unit subtracts the end position from the start position of the target pronunciation unit to obtain the length of the target pronunciation unit; subtracts the cumulative probability corresponding to the end position from the cumulative probability corresponding to the start position, and then divides it by the length of the target pronunciation unit to obtain the average probability of the target pronunciation unit.
[0075] Optionally, the setting conditions include:
[0076] The length of each target pronunciation unit within the set interval is greater than the set length threshold; and / or
[0077] The average probability of each target pronunciation unit within the set interval is greater than the set average probability threshold.
[0078] A computer device, comprising: one or more processors, a memory;
[0079] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the aforementioned method.
[0080] A readable storage medium stores instructions thereon, wherein the instructions are executed to implement the above-mentioned method.
[0081] The end-to-end wake-up word detection method and device provided by the embodiment of the present invention inserts virtual silent nodes before and after each target pronunciation unit, obtains the target-time relationship matrix according to the posterior probability of the target pronunciation unit; calculates the cumulative probability of each node in the target-time relationship matrix frame by frame; determines the optimal path according to the cumulative probability of each node in the matrix; and determines the wake-up word detection result according to the optimal path. Since virtual silent nodes are added between the target pronunciation units, it makes it more consistent with the normal pronunciation rules, effectively improves the accuracy of the detection results, improves the detection rate of the wake-up word, and suppresses the false wake-up phenomenon.
[0082] In the dynamic time planning process, considering that other words or characters other than silence and wake-up words do not play an obvious role in the dynamic time planning and may compete with silence, the present invention represents other words or characters other than silence and wake-up words as silence, thus avoiding the impact on the expression of silence and further improving the accuracy of the detection results.
[0083] Furthermore, when training the acoustic model, the alignment model is used to align the wake-up word data and the non-wake-up word data respectively to obtain the time information of the training data, and then obtain the frame-level label data. The acoustic model is trained using the frame-level labeled data. Compared with the prior art of manual time information labeling, this greatly saves human resources and improves the efficiency of model training.
[0084] Furthermore, by limiting conditions such as the target pronunciation unit length and / or average probability, false awakening is effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0086] Figure 1 is a flow chart of constructing an acoustic model in an embodiment of the present invention;
[0087] Figure 2 is a flow chart of an end-to-end wake-up word detection method according to an embodiment of the present invention;
[0088] Figure 3 is a schematic diagram of end-to-end HMM transfer logic in an embodiment of the present invention;
[0089] Figure 4 is another flow chart of the end-to-end wake-up word detection method according to an embodiment of the present invention;
[0090] Figure 5 This is a structural block diagram of an end-to-end wake-up word detection device according to an embodiment of the present invention;
[0091] Figure 6 is a structural block diagram of a model building module in an embodiment of the present invention;
[0092] Figure 7 is another structural block diagram of an end-to-end wake-up word detection device according to an embodiment of the present invention;
[0093] Figure 8 is a block diagram of a device for an end-to-end wake-up word detection method according to an exemplary embodiment;
[0094] Fig. 9 It is a schematic diagram of the structure of the server in the embodiment of the present invention. DETAILED DESCRIPTION
[0095] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.
[0096] An embodiment of the present invention provides an end-to-end wake-up word detection method and device. According to the posterior probability of each target pronunciation unit output by the acoustic model, virtual silent nodes are inserted before and after the target pronunciation unit when performing dynamic time planning, and the cumulative probability of each target pronunciation unit is calculated frame by frame. Then, the optimal path is determined according to the cumulative probability, and the wake-up word detection result is determined according to the optimal path.
[0097] The following first describes in detail the training process of the acoustic model in the embodiment of the present invention.
[0098] like Figure 1 FIG. 1 is a flowchart of constructing an acoustic model in an embodiment of the present invention, comprising the following steps:
[0099] Step 101, collect wake-up word data and non-wake-up word data.
[0100] The wake-up word data refers to the voice data containing the set wake-up word, and the non-wake-up word data refers to the voice data not containing the wake-up word. It is usually required that the total duration of the non-wake-up word data is the same as or greater than the total duration of the wake-up word data, such as a duration ratio of 2:1 or 3:1.
[0101] Step 102 , time-marking the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data.
[0102] First, the label mapping relationship between the wake-up word and the non-wake-up word needs to be determined. In an embodiment of the present invention, the start time period and the end time period of the wake-up word are represented by silence; for each character in the wake-up word, the label corresponding to the character is set in order of position, for example, for the wake-up word "Hello, Wangzai", the labels corresponding to each character are 1, 2, 3, and 4, respectively, and the corresponding label mapping relationship is (1-2-3-4); for other words or characters other than silence and wake-up words, the corresponding label is set to 0, and the corresponding label mapping relationship is (-0 / 0- / -0-).
[0103] Secondly, the wake-up word data and non-wake-up word data are aligned respectively to obtain the corresponding relationship between each word in the wake-up word data and non-wake-up word data and the speech frames occupied by the word, that is, which frames each word occupies.
[0104] In an embodiment of the present invention, the wake-up word data and the non-wake-up word data can be aligned respectively using a pre-established alignment model. The alignment model is a state-level speech recognition model, and specifically can be a neural network model, such as a DNN-HMM model, trained using the wake-up word data and the non-wake-up word data.
[0105] Finally, according to the label mapping relationship and the corresponding relationship, the wake-up word data and the non-wake-up word data are mapped into label forms respectively to obtain frame-level label data.
[0106] Step 103: The acoustic model is trained using the frame-level label data.
[0107] The acoustic model may adopt a neural network model, such as a DNN-HMM model. The input of the acoustic model is the acoustic features extracted from the current frame, and the output is the posterior probability of the target pronunciation unit in the current frame.
[0108] Taking "Hello Wangzai" as an example, the target pronunciation units are (0-1-2-3-4), and the output of the acoustic model is the posterior probabilities corresponding to these target pronunciation units.
[0109] The wake-up word data and non-wake-up word data are aligned using an alignment model. Compared with the prior art method of manually labeling time information, this can greatly save human resources and improve model training efficiency.
[0110] By using the acoustic model, the posterior probability of each target pronunciation unit in each speech frame of the speech to be detected can be obtained.
[0111] like Figure 2 , which is a flow chart of an end-to-end wake-up word detection method according to an embodiment of the present invention, comprising the following steps:
[0112] Step 201: Receive the speech to be detected.
[0113] For example, the signal may be received through a microphone, and the microphone may be disposed on the device to be awakened, or on a controller of the device to be awakened, such as a remote controller.
[0114] Step 202: extracting the acoustic features of each speech frame in the speech to be detected in sequence.
[0115] The received speech to be detected needs to be framed. In addition, the framed speech data can also be pre-emphasized to increase the high-frequency resolution of the speech.
[0116] The acoustic features may be MFCC (Mel-Frequency Cepstral Coefficients) features, or PLP (Linear Predictive Coding) features, or Filterbank features, etc. These acoustic features may be extracted using existing technologies and will not be described in detail herein.
[0117] Step 203: Input the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of each target pronunciation unit in each speech frame output by the acoustic model.
[0118] In an embodiment of the present invention, the acoustic model is an end-to-end acoustic model based on the target pronunciation unit in the wake-up word, and specifically can adopt DNN (Deep Neural Networks), such as FFNN (Feed Forward Neural Network), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network) or a combination of one or more. The input of the acoustic model is the acoustic features extracted from the current frame, and the output is the posterior probability of the target pronunciation unit in the current frame.
[0119] The acoustic model can be trained using a large amount of collected speech data, and the specific training process will be described in detail later.
[0120] The target pronunciation unit refers to a pronunciation unit of the wake-up word, which can be a syllable, a character, or a word.
[0121] Step 204: Take each target pronunciation unit as a node, insert virtual silent nodes before and after the target pronunciation unit, and obtain a target-time relationship matrix according to the posterior probability of the target pronunciation unit.
[0122] The target-time relationship matrix is a DTW (Dynamic Time Warping) matrix, whose vertical axis is each node. Taking "Hello Wangzai" as an example, its corresponding label is 1-2-3-4. After inserting virtual silent nodes before and after each target pronunciation unit, the form is (0-1-0-2-0-3-0-4-0); the horizontal axis is the time in frames, and the intersection of the vertical axis and the horizontal axis is the value of each node, which is the product of the posterior probability of the node and its weight. For the convenience of description, the value of each node in the matrix is referred to as the probability of the node.
[0123] Step 205 , calculating the cumulative probability of each node in the target-time relationship matrix frame by frame.
[0124] In an embodiment of the present invention, a dynamic programming method is used to calculate the cumulative probability of each node (including virtual silent nodes and target pronunciation units) in the target-time relationship matrix, and the cumulative probability of each node is the sum of the cumulative probability of the optimal path among all paths that can reach the node and the probability of the current node. Specifically, the cumulative probability of the optimal path among all paths that can reach the node can be determined first, and then the cumulative probability of the optimal path is added to the probability of the node to obtain the cumulative probability of the node.
[0125] like Figure 3 As shown, due to the addition of virtual silent nodes, during dynamic planning, the path that can reach each target pronunciation unit includes not only the path from the previous target pronunciation unit to the target pronunciation unit, but also the path from the previous virtual silent node to the target pronunciation unit.
[0126] Assuming that state[] represents the node set, that is, the label set, the calculation formula is as follows:
[0127] 1) For a virtual silent node, that is, when sil_state = 0,
[0128] dp[i][t]=max(dp[i][t-1],dp[i-1][t-1])+out[i][t];
[0129] Among them, dp[i][t] represents the cumulative probability of the i-th node (the node is a virtual silent node) in the t-th frame, dp[i][t-1] represents the cumulative probability of the i-th node (the node is a virtual silent node) in the t-1th frame, dp[i-1][t-1] represents the cumulative probability of the i-1th node (the node is the target pronunciation unit) in the t-1th frame, and out[i][t] represents the probability of the i-th node (the node is a virtual silent node) in the t-th frame.
[0130] That is to say, the maximum value of the cumulative probability of the virtual silent node in the previous frame and the cumulative probability of the previous target pronunciation unit node in the previous frame is selected as the cumulative probability of the virtual silent node as of the previous frame, and the cumulative probability of the virtual silent node as of the previous frame is added to the probability of the virtual silent node in the current frame as the cumulative probability of the virtual silent node as of the current frame.
[0131] 2) For the first target pronunciation unit node, that is, sil_state≠0, and i=1, the calculation formula is similar to the above.
[0132] That is to say, the maximum value of the cumulative probability of the target pronunciation unit node in the previous frame and the cumulative probability of the previous virtual silent node in the previous frame is selected as the cumulative probability of the target pronunciation unit node as of the previous frame, and the cumulative probability of the target pronunciation unit node as of the previous frame plus the probability of the target pronunciation unit node in the current frame is taken as the cumulative probability of the target pronunciation unit node as of the current frame.
[0133] 3) For other target pronunciation unit nodes, that is, sil_state≠0, and i>1,
[0134] dp[i][t]=max{max(dp[i][t-1],dp[i-1][t-1]),dp[i-2][t-1]}+out[i][t];
[0135] Among them, dp[i][t] represents the cumulative probability of the i-th node (the node is the target pronunciation unit node) in the t-th frame, dp[i][t-1] represents the cumulative probability of the i-th node (the node is the target pronunciation unit node) in the t-1th frame, dp[i-1][t-1] represents the cumulative probability of the i-1th node (the node is the virtual silent unit) in the t-1th frame, dp[i-2][t-1] represents the cumulative probability of the i-2th node (the node is the target pronunciation unit) in the t-1th frame, and out[i][t] represents the probability of the i-th node (the node is the target pronunciation unit node) in the t-th frame.
[0136] That is to say, the maximum value of the cumulative probability of the target pronunciation unit node in the previous frame, the cumulative probability of the previous target pronunciation unit node in the previous frame, and the cumulative probability of the previous virtual silent node in the previous frame is selected as the cumulative probability of the target pronunciation unit node as of the previous frame, and the cumulative probability of the target pronunciation unit node as of the previous frame plus the probability of the target pronunciation unit node in the current frame is taken as the cumulative probability of the target pronunciation unit node as of the current frame.
[0137] Step 206: determine the optimal path according to the cumulative probability of each node in the matrix.
[0138] Specifically, the score of each path may be calculated according to the cumulative probability of each node in the matrix, and the path with the largest score is taken as the optimal path.
[0139] Step 207: determine the wake-up word detection result according to the optimal path.
[0140] For example, it can be determined whether the cumulative probabilities corresponding to the target pronunciation units on the optimal path are all greater than the set maximum probability threshold. If so, it is determined that the wake-up word is detected; otherwise, it is determined that the wake-up word is not detected.
[0141] The end-to-end wake-up word detection method provided by the embodiment of the present invention inserts virtual silent nodes before and after each target pronunciation unit to obtain a target-time relationship matrix; according to the posterior probability of the target pronunciation unit in each speech frame output by the acoustic model, the cumulative probability of each node in the target-time relationship matrix is calculated frame by frame; the optimal path is determined according to the cumulative probability of each node in the matrix; and the wake-up word detection result is determined according to the optimal path. Since virtual silent nodes are added between the target pronunciation units, it makes it more consistent with the normal pronunciation rules, effectively improves the accuracy of the detection results, improves the detection rate of the wake-up words, and suppresses the false wake-up phenomenon.
[0142] In the dynamic time planning process, considering that other words or characters other than silence and wake-up words do not play an obvious role in the dynamic time planning and will form a competitive relationship with silence, all words or characters other than silence and wake-up words are represented as silence, avoiding the impact on the expression of silence and further improving the accuracy of the detection results.
[0143] like Figure 4 FIG. 2 is another flow chart of an end-to-end wake-up word detection method according to an embodiment of the present invention, comprising the following steps:
[0144] Step 401: Receive the speech to be detected.
[0145] Step 402: extracting the acoustic features of each speech frame in the speech to be detected in sequence.
[0146] Step 403: Input the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of each target pronunciation unit in each speech frame output by the acoustic model.
[0147] Step 404: construct a target-time relationship matrix and determine the optimal path using a dynamic programming algorithm.
[0148] The construction method of the target-time relationship matrix and the determination process of the optimal path can be found in Figure 2 Steps 204 to 206 are shown and will not be described in detail here.
[0149] Step 405, determining the starting position and the ending position of each target pronunciation unit on the optimal path, and calculating the length and the average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position.
[0150] In the dynamic programming process, after the cumulative probability is calculated to the last frame, each frame is traced back frame by frame to determine whether the corresponding target pronunciation unit is included. Specifically, if the probability of the target pronunciation unit in the current frame is greater than the set output threshold, it is determined that the target pronunciation unit is included in the current frame. In this way, after tracing back to the starting frame, all frames containing the target pronunciation unit can be obtained, and then the starting position and ending position of each target pronunciation unit can be obtained. The length of the target pronunciation unit can be obtained by subtracting the ending position of the target pronunciation unit from the starting position.
[0151] Accordingly, for each target pronunciation unit, the cumulative probability corresponding to its end position is subtracted from the cumulative probability corresponding to its start position, and then divided by the length of the target pronunciation unit to obtain the average probability of the target pronunciation unit.
[0152] Step 406 , determining whether a set condition is met based on the length and / or average probability of each target pronunciation unit on the optimal path; if yes, executing step 407 ; otherwise, executing step 408 .
[0153] In an embodiment of the present invention, the set condition may be: the length of each target pronunciation unit within the set interval is greater than a set length threshold; and / or the average probability of each target pronunciation unit within the set interval is greater than a set average probability threshold.
[0154] Step 407, determine whether the wake-up word is detected.
[0155] Step 408, determine that the wake-up word is not detected.
[0156] The end-to-end wake-up word detection method provided by the embodiment of the present invention can effectively reduce the false wake-up rate by further limiting the length and average probability of each target pronunciation unit.
[0157] Of course, in actual applications, you can also limit the total length of non-silence (for example, you can determine the length threshold of the wake-up word according to different speaking speeds) and the average probability, the total length of silence, the length of silence between characters or words, etc., to ensure that the wake-up operation is performed in a more normal speaking style.
[0158] The end-to-end wake-up word detection method provided by the embodiment of the present invention can be applied to a variety of smart devices, such as home appliances, speakers, tablet computers, mobile phones, wearable devices, robots, toys, etc., and can enable the smart devices to accurately detect the user's voice commands, that is, wake-up words, even in sleep or locked states, so that the device in sleep state can directly enter the waiting command state or directly execute the operation corresponding to the voice command.
[0159] Accordingly, an embodiment of the present invention further provides an end-to-end wake-up word detection device, such as Figure 5Shown is a structural block diagram of the device.
[0160] In this embodiment, the end-to-end wake-up word detection device includes the following modules:
[0161] The receiving module 501 is used to receive the speech to be detected;
[0162] The feature extraction module 502 is used to sequentially extract the acoustic features of each speech frame in the speech to be detected; specifically, the speech received by the receiving module 501 can be framed to obtain each speech frame, and then the acoustic features of each speech frame can be extracted. The acoustic features can be MFCC features, or PLP features, or Filterbank features, etc. The extraction of these acoustic features can adopt existing technologies, which will not be repeated here;
[0163] The acoustic detection module 503 is used to input the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of the target pronunciation unit in each speech frame output by the acoustic model; the target pronunciation unit refers to a pronunciation unit of the wake-up word, which can be a syllable, a character, or a word;
[0164] A matrix construction module 504 is used to treat each target pronunciation unit as a node, insert virtual silent nodes before and after the target pronunciation unit, and obtain a target-time relationship matrix according to the posterior probability of the target pronunciation unit;
[0165] A calculation module 505 is used to calculate the cumulative probability of each node in the target-time relationship matrix frame by frame;
[0166] The optimal path determination module 506 is used to determine the optimal path according to the cumulative probability of each node in the matrix; specifically, the score of each path can be calculated according to the cumulative probability of each node in the matrix, and the path with the largest score is taken as the optimal path;
[0167] The detection module 507 is used to determine the wake-up word detection result according to the optimal path; for example, it can be determined whether the cumulative probabilities corresponding to each target pronunciation unit on the optimal path are greater than the set maximum probability threshold. If so, it is determined that the wake-up word is detected; otherwise, it is determined that the wake-up word is not detected.
[0168] In an embodiment of the present invention, the acoustic model is an end-to-end acoustic model based on the target pronunciation unit in the wake-up word, and specifically can adopt DNN, such as a combination of one or more of FFNN, CNN, and RNN. The input of the acoustic model is the acoustic features extracted from the current frame, and the output is the probability of the target pronunciation unit in the current frame.
[0169] The acoustic model can be pre-trained by a corresponding model building module using a large amount of collected speech data, wherein the speech data includes wake-up word data and non-wake-up word data. The model building module can be integrated into the device or can be independent of the device, which is not limited in this embodiment of the present invention.
[0170] A structural block diagram of the model building module is as follows Figure 6 As shown, it includes the following units:
[0171] A data collection unit 61, used to collect wake-up word data and non-wake-up word data;
[0172] A marking unit 62, configured to time-mark the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data;
[0173] The training unit 63 is used to train the acoustic model using the frame-level label data.
[0174] The marking unit 62 may specifically include: a mapping relationship determination unit, an alignment unit, and a mapping unit.
[0175] The mapping relationship determination unit is used to determine the label mapping relationship between the wake-up word and the non-wake-up word. In an embodiment of the present invention, the start time period and the end time period of the wake-up word can be represented by silence; for each character in the wake-up word, the label corresponding to the character is set in order of position; for other words or characters except silence and the wake-up word, the corresponding label is set to 0, and the traditional other label is no longer set.
[0176] The alignment unit is used to align the wake-up word data and the non-wake-up word data respectively, for example, using a pre-established alignment model to align the wake-up word data and the non-wake-up word data respectively, to obtain a corresponding relationship between each word in the wake-up word data and the non-wake-up word data and the speech frame occupied by the word;
[0177] The mapping unit is used to map the wake-up word data and the non-wake-up word data into label forms according to the label mapping relationship and the corresponding relationship, so as to obtain frame-level label data.
[0178] In an embodiment of the present invention, the wake-up word data and the non-wake-up word data are aligned using an alignment model, which can greatly save human resources and improve model training efficiency compared to the method of manually labeling time information in the prior art. The alignment model can adopt a neural network model, such as a DNN-HMM model, and is trained using the wake-up word data and the non-wake-up word data.
[0179] The acoustic model may adopt a neural network model, such as a DNN-HMM model. The input of the acoustic model is the acoustic features extracted from the current frame, and the output is the posterior probability of the target pronunciation unit in the current frame.
[0180] The posterior probability of each target pronunciation unit obtained by using the acoustic model, Figure 5 When calculating the cumulative probability of each node in the target-time relationship matrix, the calculation module 505 first determines the cumulative probability of the optimal path among all paths that can reach the node, and then adds the cumulative probability of the optimal path to the probability of the node to obtain the cumulative probability of the node. The specific calculation method can refer to the description in the above embodiment of the method of the present invention, which will not be repeated here.
[0181] The end-to-end wake-up word detection device provided by the embodiment of the present invention inserts virtual silent nodes before and after each target pronunciation unit to obtain a target-time relationship matrix; calculates the cumulative probability of each node in the target-time relationship matrix frame by frame according to the posterior probability of the target pronunciation unit in each speech frame output by the acoustic model; determines the optimal path according to the cumulative probability of each node in the matrix; and determines the wake-up word detection result according to the optimal path. Since virtual silent nodes are added between the target pronunciation units, it makes it more consistent with the normal pronunciation rules, effectively improves the accuracy of the detection results, improves the detection rate of the wake-up word, and suppresses the false wake-up phenomenon.
[0182] In the dynamic time planning process, considering that other words or characters other than silence and wake-up words do not play an obvious role in the dynamic time planning and will form a competitive relationship with silence, all words or characters other than silence and wake-up words are represented as silence, avoiding the impact on the expression of silence and further improving the accuracy of the detection results.
[0183] like Figure 7 , is another structural block diagram of an end-to-end wake-up word detection device according to an embodiment of the present invention.
[0184] and Figure 5 The difference between the illustrated embodiment is that, in this embodiment, the detection module 507 includes: a determination unit 571 and a judgment unit 572. Among them:
[0185] The determining unit 571 is used to determine the starting position and the ending position of each target pronunciation unit on the optimal path, and calculate the length and the average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position.
[0186] The determination unit 571 can specifically determine whether the corresponding target pronunciation unit is included in each frame after the calculation module 505 calculates the cumulative probability of each node to the last frame. Specifically, if the probability of the target pronunciation unit in the current frame is greater than the set output threshold, it is determined that the target pronunciation unit is included in the current frame. In this way, after tracing back to the starting frame, all frames containing the target pronunciation unit can be obtained, and then the starting position and ending position of each target pronunciation unit can be obtained. The length of the target pronunciation unit can be obtained by subtracting the ending position of the target pronunciation unit from the starting position. In addition, for each target pronunciation unit, the cumulative probability corresponding to its ending position is subtracted from the cumulative probability corresponding to its starting position, and then divided by the length of the target pronunciation unit, the average probability of the target pronunciation unit can be obtained.
[0187] The judgment unit 572 is used to determine whether the set conditions are met according to the length and / or average probability of each target pronunciation unit on the optimal path, and if so, determine that the wake-up word is detected.
[0188] In an embodiment of the present invention, the set condition may be: the length of each target pronunciation unit within the set interval is greater than a set length threshold; and / or the average probability of each target pronunciation unit within the set interval is greater than a set average probability threshold.
[0189] The end-to-end wake-up word detection device provided by the embodiment of the present invention can effectively reduce the false wake-up rate by further limiting the length and average probability of each target pronunciation unit.
[0190] Of course, in actual applications, you can also limit the total length of non-silence (for example, you can determine the length threshold of the wake-up word according to different speaking speeds) and the average probability, the total length of silence, the length of silence between characters or words, etc., to ensure that the wake-up operation is performed in a more normal speaking style.
[0191] The end-to-end wake-up word detection device provided in the embodiment of the present invention can be applied to a variety of smart devices, such as home appliances, speakers, tablet computers, mobile phones, wearable devices, robots, toys, etc., and can enable the smart devices to accurately detect the user's voice commands, that is, wake-up words, even in sleep or locked states, so that the device in sleep state can directly enter the waiting command state or directly execute the operation corresponding to the voice command.
[0192] It should be noted that, in the description of the embodiments of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" means two or more.
[0193] Figure 88 is a block diagram of an apparatus 800 for an end-to-end wake-up word detection method according to an exemplary embodiment. For example, the apparatus 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0194] Reference Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0195] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0196] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0197] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0198] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0199] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0200] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0201] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0202] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0203] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0204] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0205] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute part or all of the steps in the above-mentioned method embodiments to reduce the false wake-up rate.
[0206] Fig. 91 is a schematic diagram of the structure of the server in an embodiment of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0207] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0208] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0209] It should be understood that the present invention is not limited to the precise structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An end-to-end wake-up word detection method, characterized in that: The method comprises: Receiving the voice to be detected; Extracting the acoustic features of each speech frame in the speech to be detected in sequence; The extracted acoustic features are input into a pre-built acoustic model to obtain a posterior probability of a target pronunciation unit in each speech frame output by the acoustic model; the target pronunciation unit is a pronunciation unit of the wake-up word; Each target pronunciation unit is taken as a node, and virtual silent nodes are inserted before and after the target pronunciation unit, and a target-time relationship matrix is obtained according to the posterior probability of the target pronunciation unit; the target-time relationship matrix is a dynamic time warping matrix constructed with the node as the vertical axis and the time as the horizontal axis, and the intersection of the vertical axis and the horizontal axis is the product of the posterior probability of the node and its weight; Calculating the cumulative probability of each node in the target-time relationship matrix frame by frame; Determining an optimal path according to the cumulative probability of each node in the matrix; The wake-up word detection result is determined according to the optimal path.
2. The method according to claim 1, characterized in that The method further comprises constructing the acoustic model in the following manner: Collect wake-up word data and non-wake-up word data; Time-marking the wake-up word data and the non-wake-up word data respectively to obtain frame-level label data; The acoustic model is trained using the frame-level label data.
3. The method according to claim 2, characterized in that The time-stamping of the wake-up word data and the non-wake-up word data to obtain frame-level label data includes: Determine the label mapping relationship between wake-up words and non-wake-up words; Aligning the wake-up word data and the non-wake-up word data respectively to obtain a corresponding relationship between each word in the wake-up word data and the non-wake-up word data and the speech frame occupied by the word; According to the label mapping relationship and the corresponding relationship, the wake-up word data and the non-wake-up word data are respectively mapped into label forms to obtain frame-level label data.
4. The method according to claim 3, characterized in that Determining the label mapping relationship between the wake-up word and the non-wake-up word includes: The start time period and the end time period of the wake-up word are indicated by silence; For each word in the wake-up word, set a label corresponding to the word in order of position; For words or characters other than silent and wake-up words, set the corresponding labels to 0.
5. The method according to claim 1, characterized in that The frame-by-frame calculation of the cumulative probability of each node in the target-time relationship matrix includes: Determine the cumulative probability of the best path among all paths that can reach the node; The cumulative probability of the optimal path is added to the probability of the node to obtain the cumulative probability of the node.
6. The method according to any one of claims 1 to 5, characterized in that: Determining the wake-up word detection result according to the optimal path includes: Determine the starting position and the ending position of each target pronunciation unit on the optimal path, and calculate the length and the average probability of the target pronunciation unit according to the cumulative probability and the starting position and the ending position; Determining whether a set condition is met according to the length and / or average probability of each target pronunciation unit on the optimal path; If so, it is determined that the wake-up word is detected.
7. The method according to claim 6, characterized in that The setting conditions include: The length of each target pronunciation unit within the set interval is greater than the set length threshold; and / or The average probability of each target pronunciation unit within the set interval is greater than the set average probability threshold.
8. An end-to-end wake-up word detection device, characterized in that: The device comprises: A receiving module, used for receiving the voice to be detected; A feature extraction module, used to sequentially extract the acoustic features of each speech frame in the speech to be detected; An acoustic detection module, used to input the extracted acoustic features into a pre-built acoustic model to obtain the posterior probability of a target pronunciation unit in each speech frame output by the acoustic model; the target pronunciation unit is a pronunciation unit of the wake-up word; A matrix construction module is used to treat each target pronunciation unit as a node, insert virtual silent nodes before and after the target pronunciation unit, and obtain a target-time relationship matrix according to the posterior probability of the target pronunciation unit; the target-time relationship matrix is a dynamic time warping matrix constructed with the node as the vertical axis and the time as the horizontal axis, and the intersection of the vertical axis and the horizontal axis is the product of the posterior probability of the node and its weight; A calculation module, used for calculating the cumulative probability of each node in the target-time relationship matrix frame by frame; An optimal path determination module, used to determine the optimal path according to the cumulative probability of each node in the matrix; A detection module is used to determine a wake-up word detection result according to the optimal path.
9. A computer device, characterized in that: include: One or more processors, memory; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the method according to any one of claims 1 to 7.
10. A readable storage medium having instructions stored thereon, wherein the instructions are executed to implement the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Voice wake-up method and device
CN105096939A
Voice wakeup method and voice wakeup device based on artificial intelligence
CN106611597A
Speech recognition method and wake-up word detection method and device
CN109192210A
Method and device for waking up via speech based on artificial intelligence and computer device
US20180336886A1
Method and apparatus for waking up via speech
US20190139545A1
