A voice wake-up method and system based on hybrid precision quantization neural networks

Abstract

This invention discloses a voice wake-up method and system based on a mixed-precision quantization neural network. The method includes: acquiring a voice signal to be processed using a microphone of a voice wake-up device; extracting features from the voice signal to obtain log-Mel spectrum features; inputting the log-Mel spectrum features into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and calculation simplification for the parameters and operations of each layer in the network, based on the required precision of parameter calculation for each layer; if the classification result corresponding to the log-Mel spectrum features is a wake-up word and a set time limit has been exceeded, waking up the voice wake-up device.

Description

Technical Field

[0001] This application relates to voice control technology, and more particularly to a voice wake-up method and system based on a hybrid precision quantization neural network. Background Technology

[0002] Currently, voice wake-up services are used in various devices, including in-vehicle devices, smart home devices, and smart control devices. These devices share the common characteristics of limited memory and computing power; therefore, voice wake-up services must simultaneously meet the requirements of high accuracy, low memory usage, and minimal computational load.

[0003] To ensure the accuracy of voice wake-up, voice wake-up services require high-performance neural network models, such as high-precision deep convolutional neural network models. However, these models are highly complex and computationally intensive, typically requiring a large amount of device memory, making them difficult to deploy on devices with limited memory.

[0004] Therefore, how to reduce the number of parameters and computational complexity of voice wake-up services while ensuring wake-up accuracy is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a voice wake-up method and system based on a mixed-precision quantization neural network. Addressing the varying precision requirements of parameter calculation at each layer of the network, different quantization bits are used for layer-by-layer quantization. This significantly reduces the number of model parameters and computational complexity while maintaining wake-up accuracy, thereby mitigating the shortcomings of voice wake-up services in terms of memory usage, power consumption, and wake-up latency. In practical applications, this provides a novel low-power voice wake-up approach and chip design concept.

[0006] To address the aforementioned technical problems, a first aspect of the present invention discloses a voice wake-up method based on a hybrid precision quantization neural network, the method comprising:

[0007] The microphone of the voice-activated device is used to collect the voice signal to be processed.

[0008] Feature extraction is performed on the speech signal to be processed to obtain log-Mel spectrum features;

[0009] The log-Mel spectrum features are input into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and operation simplification for the parameters and operations of each layer in the network, based on the accuracy requirements of parameter calculation for each layer.

[0010] If the classification result corresponding to the log-Mel spectrum feature is a wake word and the set time limit has been exceeded, the voice wake-up device is activated.

[0011] Preferably, the hybrid precision quantization neural network includes: a first convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, more than twelve residual blocks, a max pooling layer, and a fully connected layer connected in sequence.

[0012] Each residual block includes: a second convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, and a shortcut;

[0013] In each residual block, the shortcut connects the input of the second convolutional layer in the current residual block to the output of the BN layer, which is used to copy the input of the second convolutional layer and add it to the output of the BN layer to obtain a sum; the sum is used as the input of the ReLU activation function in the current residual block and the input of the next residual block; finally, the output of the residual block enters the max pooling layer for downsampling and is then input to the fully connected layer to calculate the probability;

[0014] The first convolutional layer and the last fully connected layer are quantized with 8 bits; all second convolutional layers in the residual block are quantized with 1 bit; and the feature inputs and feature outputs of all BN layers in the residual block are quantized with 4 bits.

[0015] Preferably, the first convolutional layer and the last fully connected layer use 8-bit quantization, specifically including:

[0016] For the first convolutional layer and the last fully connected layer, the full-precision input is quantized to 8 bits using the formula q = clamp(round(r*S―Z)) to obtain the unsigned 8-bit feature input, its quantization factor, and zero offset; wherein, the full-precision input in the first convolutional layer is the log-Mel spectrum feature input corresponding to the speech dataset;

[0017] Where q is the quantized unsigned parameter, and the number of bits follows the range of the truncation function; clamp is the truncation function, ranging from [0, 255]; round is the rounding function; r is the full-precision input, 32 bits; S is the quantization factor, S = 2. a , quantize_range is the range of values ​​after quantization, float_range is the range of values ​​before quantization, float_range = (rmax - rmin), where max and rmin are the maximum and minimum values ​​of the 32-bit full-precision input r, respectively; Z is the zero offset, Z = int(rmin * S), the range of Z is limited to [0, 255], and int is the rounding function;

[0018] The weights are quantized to 8 bits using the formula to obtain unsigned 8-bit weights, their quantization factors, and zero offsets; wherein each channel of the unsigned 8-bit weights has its own quantization factor and zero offset.

[0019] Preferably, after the first convolutional layer and the last fully connected layer are quantized with 8-bit, the method further includes:

[0020] Based on the unsigned 8-bit feature input, its quantization factor, and zero-point offset, and the unsigned 8-bit weights, their quantization factors, and zero-point offsets, the multiplication calculation in the first convolutional layer and the last fully connected layer is simplified as follows:

[0021] ((Qa+Za) / Sa)*((Qw+Zw) / Sw)=(Qa*Qw+Za*Zw+Za*Qw+Zw*Qa) / (Sa*Sw); where Qa represents the unsigned 8-bit feature input, Za represents the zero-point offset corresponding to the unsigned 8-bit feature input, Sa represents the quantization factor corresponding to the unsigned 8-bit feature input, Qw represents the unsigned 8-bit weight, Zw represents the zero-point offset corresponding to the unsigned 8-bit weight, and Sw represents the quantization factor corresponding to the unsigned 8-bit weight.

[0022] Qa*Qw, Za*Zw, Za*Qw, and Zw*Qa can all be implemented using an 8-bit unsigned multiplier, while Sa*Sw is implemented using a shift operation. The accumulation calculation is implemented using an 8-bit unsigned adder.

[0023] Preferably, all second convolutional layers in the residual block are quantized using 1-bit quantization, specifically including:

[0024] For each of the second convolutional layers, the sign function is used to quantize the part of the feature input and weight of the second convolutional layer that is greater than 0 to 1, and the part that is less than 0 to -1, so as to obtain 1 bit feature input and 1 bit weight.

[0025] Preferably, after performing 1-bit quantization on all second convolutional layers in the residual block, the method further includes:

[0026] For each of the second convolutional layers, based on the 1-bit feature input and the 1-bit weights, the formula y is used. r =popcount(xnor(a b w b )) Calculate the feature output y for each of the second convolutional layers r ; where a b w bThese represent 1-bit feature input and 1-bit weight after 1-bit quantization, respectively; xnor operation is an XNOR operation; popcount operation is a bitwise operation.

[0027] Preferably, the feature inputs and feature outputs of all BN layers in the residual block are quantized using 4-bit quantization, specifically including:

[0028] The formula is used to quantize the feature inputs and feature outputs of all BN layers in the residual block using 4-bit quantization; at this time, the truncation range of the clamp is [0, 63].

[0029] Preferably, the step of performing 4-bit quantization on the feature inputs and feature outputs of all BN layers in the residual block using the formula specifically includes:

[0030] For each feature input of the BN layer, the quantization factor and zero offset corresponding to the feature input of the BN layer are calculated, and then the feature input of the BN layer is quantized to 4 bits using the formula.

[0031] The quantization factor corresponding to the feature input of the BN layer is used as the quantization factor corresponding to the parameter involved in the calculation. The corresponding zero offset is calculated, and then the parameter involved in the calculation is quantized by 4 bits using the formula. Thus, the calculation of the feature output is simplified to the subtraction and multiplication operations of 4-bit unsigned numbers.

[0032] Preferably, the mixed-precision quantization neural network is trained in the following manner:

[0033] Obtain the speech dataset;

[0034] The neural network model is trained and tested using the speech dataset combined with mixed-precision quantization to obtain the mixed-precision quantization neural network.

[0035] A second aspect of the present invention discloses a voice wake-up system based on a hybrid precision quantization neural network, comprising:

[0036] The microphone unit of the voice wake-up device is used to collect the voice signal to be processed;

[0037] The feature extraction unit is used to extract features from the speech signal to be processed to obtain log-Mel spectrum features;

[0038] The model processing unit is used to input the log-Mel spectrum features into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and operation simplification for the parameters and operations of each layer in the network, based on the accuracy requirements of parameter calculation for each layer.

[0039] The determination module is used to wake up the voice wake-up device if the classification result corresponding to the log-Mel spectrum feature is a wake word and a set time limit has been exceeded.

[0040] A third aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0041] A fourth aspect of the present invention discloses a voice wake-up device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0042] Through one or more technical solutions of the present invention, the present invention has the following beneficial effects or advantages:

[0043] This invention discloses a voice wake-up method and system based on a hybrid precision quantization neural network. In order to reduce the memory and computational overhead of the voice wake-up service while ensuring wake-up accuracy, this embodiment adopts a hybrid precision quantization neural network. For the parameter accuracy requirements of each layer in the network, the parameters (weights, input parameters, output parameters) and operations of each layer are quantized and the operations are simplified by using different quantization bits, thereby greatly reducing the number of parameters and computational load in the neural network model, significantly improving the wake-up speed and reducing the power consumption of the voice wake-up system.

[0044] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0045] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0046] In the attached diagram:

[0047] Figure 1 A flowchart illustrating a voice wake-up method based on a hybrid precision quantization neural network according to an embodiment of the present invention is shown.

[0048] Figure 2 A schematic diagram of the network structure of a hybrid precision quantization neural network according to an embodiment of the present invention is shown;

[0049] Figure 3A schematic diagram of a voice wake-up system based on a hybrid precision quantization neural network according to an embodiment of the present invention is shown. Detailed Implementation

[0050] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0051] See Figure 1 This invention discloses a voice wake-up method based on a mixed-precision quantization neural network, comprising the following steps:

[0052] Step 101: Use the microphone of the voice wake-up device to collect the voice signal to be processed.

[0053] In this embodiment, the voice wake-up device can be applied to multiple fields such as smart homes, in-vehicle services, and computer control. Taking in-vehicle services as an example, the microphone of the voice wake-up device can collect wake-up voices such as "Xiao X" issued by the car owner, and respond to the car owner with voices such as "Hey" or "Yes" after successful recognition.

[0054] In this embodiment, the voice wake-up device has N or more wake-up words, where N ≥ 1 and is a positive integer. For example, the voice wake-up device has 10 wake-up words.

[0055] Step 102: Extract features from the speech signal to be processed to obtain log-Mel spectrum features.

[0056] In this embodiment, when extracting features, the speech signal to be processed undergoes pre-emphasis, frame windowing, short-time Fourier transform (STFT), Mel filtering, and logarithmic calculation in sequence to obtain log-Mel spectrum features.

[0057] Step 103: Input the log-Mel spectrum features into the mixed precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features.

[0058] The input to the mixed-precision quantization neural network is a log-Melogram spectral feature. The output of the mixed-precision quantization neural network includes the probabilities of non-keywords and keywords. For example, the output of the mixed-precision quantization neural network is divided into 12 categories, including non-keywords, silence, and 10 keywords.

[0059] Mixed-precision quantization refers to using different quantization bit depths for parameter quantization and computation simplification at each layer of the network, based on the precision requirements of each layer. Specifically, for each layer's parameters and computations, the appropriate quantization bit depth is selected from 1-bit (single-bit) quantization, 4-bit quantization, and 8-bit quantization for parameter quantization and computation simplification.

[0060] In this embodiment, the initial model of the mixed-precision quantization neural network is a residual neural network ResNet14.

[0061] See the network structure of the mixed-precision quantization neural network. Figure 2 It includes: a first convolutional layer, a BN (batch normalization) layer and a ReLU (Rectified Linear Unit) activation function connected in sequence, more than twelve residual blocks, a max pooling layer, and a fully connected layer.

[0062] The first convolutional layer is also known as the first convolutional layer.

[0063] More than twelve residual blocks are connected sequentially, with the output of the previous residual block serving as the input of the next. Each residual block includes: a second convolutional layer, a batch normalization (BN) layer, a ReLU activation function, and a shortcut.

[0064] In each residual block, a shortcut connects the input of the second convolutional layer and the output of the batch normalization (BN) layer. This shortcut copies the input of the second convolutional layer and adds it to the output of the BN layer to obtain a sum. This sum serves as the input to the ReLU activation function in the current residual block and the input to the next residual block. For example, the shortcut copies the input x of the second convolutional layer and adds it to the output f(x) of the BN layer to obtain the input F(x) of the ReLU activation function, i.e., F(x) = f(x) + x. F(x) will then be used as the input to the next residual block, and this process is repeated. Finally, the output of the residual block is downsampled by a max pooling layer and then input to a fully connected layer to calculate the probability.

[0065] The first convolutional layer and the last fully connected layer are quantized with 8 bits; all second convolutional layers in the residual block are quantized with 1 bit; and the feature inputs and feature outputs of all BN layers in the residual block are quantized with 4 bits.

[0066] Specifically, the first convolutional layer and the last fully connected layer use 8-bit quantization. This means that for the first convolutional layer and the last fully connected layer, the full-precision input and weights are quantized to 8 bits respectively, and the multiplication calculation is simplified. Then, the corresponding unsigned multiplication, addition and shift operations are used to calculate to obtain the same calculation result as before.

[0067] In this design, both the first convolutional layer and the last fully connected layer are quantized using 8-bit quantization. This is because the first convolutional layer, as the input layer, needs to minimize information loss, hence the use of high-precision 8-bit quantization; the last fully connected layer, as the output layer, directly affects wake-up accuracy, therefore also requires ensuring output precision, hence the use of 8-bit quantization. Furthermore, to mitigate the decrease in wake-up accuracy caused by quantization, training-time quantization is employed.

[0068] The following explanation uses the first convolutional layer as an example. The last fully connected layer uses the same method for 8-bit quantization.

[0069] Before entering the first convolutional layer, the full-precision input is quantized to 8 bits using the formula q = clamp(round(r*S―Z)) to obtain the unsigned 8-bit feature input, its quantization factor, and zero offset; for the first convolutional layer, its full-precision input is the log-Mel spectrum feature input corresponding to the speech dataset;

[0070] Where q is the quantized unsigned parameter, and the number of bits changes according to the range of the clamp function. In this case, the obtained q is the quantized unsigned 8-bit parameter; clamp is the truncation function, with a range between [0, 255]; round is the rounding function; r is the full-precision input, 32 bits; S is the quantization factor, S = 2. a , `quantize_range` is the range of values ​​after quantization, which can be 255. `float_range` is the range of values ​​before quantization, where `float_range = (rmax - rmin)`. `max` and `rmin` are the maximum and minimum values ​​of the 32-bit full-precision input `r`, respectively. During training, `rmax` and `rmin` can be determined using a moving average method. `Z` is the zero-point offset, where `Z = int(rmin * S)`. The range of `Z` is limited to [0, 255], and `int` is the rounding function.

[0071] Similarly, the weights in the first convolutional layer are quantized to 8 bits using the above formula to obtain unsigned 8-bit weights, their quantization factors, and zero-point offsets. It is worth noting that the quantization factors and zero-point offsets of the unsigned 8-bit weights are calculated per channel; that is, each channel of the unsigned 8-bit weight has its own quantization factor and zero-point offset.

[0072] Finally, before computation in the first convolutional layer, the following processing yields: unsigned 8-bit feature input Qa, the quantization factor Sa corresponding to unsigned 8-bit feature input Qa, the zero-point offset Za corresponding to unsigned 8-bit feature input Qa, the unsigned 8-bit weight Qw, the quantization factor Sw corresponding to unsigned 8-bit weight Qw, and the zero-point offset Zw corresponding to unsigned 8-bit weight Qw. Specifically, for the unsigned 8-bit feature input Qa, its corresponding unsigned 8-bit quantization factor Sa and zero-point offset Za are both 1*1*1*1 dimensional values ​​(i.e., a single number); for the unsigned 8-bit weight Qw, its corresponding quantization factor Sw and zero-point offset Zw are both n*1*1*1 values ​​(i.e., a row number), where n is the number of channels in the unsigned 8-bit weight Qw.

[0073] After applying 8-bit quantization to the first convolutional layer and the last fully connected layer, based on the unsigned 8-bit feature input Qa and its quantization factor Sa and zero offset Za, and the unsigned 8-bit weight Qw and its quantization factor Sw and zero offset Zw, the multiplication calculation in the first convolutional layer is simplified to:

[0074] ((Qa+Za) / Sa)*((Qw+Zw) / Sw)=(Qa*Qw+Za*Zw+Za*Qw+Zw*Qa) / (Sa*Sw); where Qa, Qw, Za, and Zw are all unsigned 8-bit parameters, so Qa*Qw, Za*Zw, Za*Qw, and Zw*Qa can all be implemented using 8-bit unsigned multipliers in hardware, and their power consumption will be much lower than that of 32-bit signed multiplication. Sa*Sw, being a power of 2, can be implemented using shift operations, with negligible power consumption. The accumulation calculation in convolution is implemented using an 8-bit unsigned adder.

[0075] The final fully connected layer can be operated in the same way as described above, so it will not be repeated here.

[0076] Prior to this embodiment, the first convolutional layer and the last fully connected layer of the neural network required storing 32-bit signed parameters. In this embodiment, however, the 32-bit signed parameters can be quantized into 8-bit unsigned parameters according to the required precision, significantly reducing system memory usage. Furthermore, the original 32-bit signed multiplication and addition are simplified to 8-bit unsigned multiplication, addition, and shift operations, greatly simplifying computation and reducing system power consumption.

[0077] In this embodiment, all convolutional layers in the twelve residual blocks of the mixed-precision quantized neural network are quantized using 1-bit quantization. For each second convolutional layer, the following operation can be performed:

[0078] The sign function (Sign) is used to quantize the positive parts of the feature input and weights of the second convolutional layer to 1, and the negative parts to -1, resulting in 1-bit feature input and 1-bit weights. Based on these 1-bit feature inputs and 1-bit weights, the formula y is used... r =popcount(xnor(a b w b )) Calculate the feature output y of each second convolutional layer r ; where a b w b These represent 1-bit feature input and 1-bit weight after 1-bit quantization, respectively; xnor operation is an XNOR operation; popcount operation is a bitwise operation.

[0079] Specifically, after single-bit quantization, the feature input and weights of the second convolutional layer only contain the values ​​1 and -1, so the feature output y r It can be obtained through simple xnor and popcount operations, thus avoiding the large number of floating-point multiplication and addition operations required for convolution, which simplifies the calculation and improves the speed of voice wake-up.

[0080] In this embodiment, 4-bit quantization is used for the feature inputs and outputs of all BN layers in the mixed-precision quantization neural network. That is, the formula is used to perform 4-bit quantization on the feature inputs and outputs of all BN layers in the residual block; at this time, the truncation range of the clamp is [0, 63].

[0081] Specifically, for each feature input of the BN layer, the quantization factor and zero offset corresponding to the feature input of the BN layer are calculated, and then the feature input of the BN layer is quantized to 4 bits using the formula. The quantization factor corresponding to the feature input of the BN layer is used as the quantization factor corresponding to the parameter participating in the calculation, and the corresponding zero offset is calculated. Then, the parameter participating in the calculation is quantized to 4 bits using the formula, thereby simplifying the calculation of the feature output to a 4-bit unsigned number subtraction and multiplication operation.

[0082] In this embodiment, the feature inputs and outputs of all BN layers in the residual block are processed using the same method (formula) as 8-bit quantization. The difference lies in that the quantized value range is now quantize_range = 63, so the range of the truncation function clamp is [0, 63]. Because the truncation function clamp has been changed, the result of the formula changes from 8-bit parameters to 4-bit parameters, which can be used for 4-bit quantization. Therefore, it can be seen that the formula used for 4-bit quantization and 8-bit quantization is the same; the only difference is the range of the truncation function clamp. 4-bit quantization and 8-bit quantization can be selectively performed according to different needs.

[0083] To facilitate the explanation of this embodiment, specific examples are used below.

[0084] For a traditional Batch Normalization (BN) layer, the output is calculated as: y' = γ·x' + β; where, x' is the normalized feature input, obtained through a single feature input x. i 'Subtract its mean μ B Divide by its standard deviation We obtain: γ and β are trainable parameters; y' is the feature output.

[0085] To simplify calculations and reduce memory usage, this embodiment fixes γ and β to 1 and 0 respectively before quantization.

[0086] Where, x i '、μ B , All use the same scaling factor for 4-bit unsigned quantization. Specifically, for the feature input x... i ', its value range before quantization is float_range and its value range after quantization is quantize_range, at this time quantize_range = 63; then use the formula S = 2 to calculate the quantization factor. a , The quantization factor corresponding to the feature input is obtained, and the zero offset Z corresponding to the feature input is obtained accordingly. Then, the feature input is quantized by 4 bits according to the formula q = clamp(round(r*S―Z)). At this time, the value range after quantization is quantize_range = 63, so the range of the truncation function clamp is [0, 63].

[0087] Furthermore, regarding the parameter μ involved in the calculation B and The same quantization factor as the feature input is used to derive the zero-point offset Z. Then, the above formula is used to quantize each parameter to 4 bits, simplifying the normalization calculation to 4-bit unsigned number subtraction and multiplication operations. Combining this with the feature output calculation formula y'=γ·x'+β of the BN layer, all calculations in the BN layer can be simplified to 4-bit unsigned number subtraction and multiplication operations. Ultimately, the entire mixed-precision quantization neural network contains only 1-bit, 4-bit, and 8-bit parameters and calculations, significantly reducing the memory footprint and power consumption of the voice wake-up system.

[0088] The above describes different mixed-precision quantization methods to meet varying computational accuracy requirements. Specifically, 1-bit quantization simplifies the original 32-bit multiplication and addition operations into xnor and popcount operations, while 4-bit and 8-bit quantization simplify 32-bit multiplication and addition operations into 8-bit multiplication and addition operations plus the shift operations required for the quantization factor. These simplifications significantly reduce the computational complexity of the mixed-precision quantization neural network, thereby lowering the power consumption and wake-up latency of the voice wake-up system.

[0089] Based on the above network structure, the mixed-precision quantization neural network is trained as follows:

[0090] Obtain the speech dataset;

[0091] Based on the above network structure, a mixed-precision quantization neural network model is trained using a speech dataset.

[0092] Specifically, mixed-precision quantization neural networks are mainly divided into training and testing phases. The speech dataset contains training speech data and test speech data.

[0093] During the training phase, log-Melogram spectral features are extracted from the training speech data and input into a mixed-precision quantization neural network. This network is obtained by selectively performing 1-bit, 4-bit, and 8-bit mixed-precision quantization on the classic ResNet14 residual network according to the computational precision requirements of each layer. This quantization is then combined with labels from the training speech data to train the model, updating the network weights and obtaining the required quantization factors and zero-point offsets for 4-bit and 8-bit quantization, as well as the weights of each layer after 1-bit, 4-bit, and 8-bit quantization. The resulting trained model is used as a voice wake-up classifier. During the testing phase, the trained voice wake-up classifier is used to infer and determine the wake-up word extracted from the test speech data. If the word is a wake-up word, the system is activated; otherwise, it is not.

[0094] In this embodiment, the classic residual neural network ResNet14 is used as the initial model. During the neural network training phase, each network layer is selectively subjected to mixed-precision quantization of 1-bit (single-bit) quantization, 4-bit quantization, and 8-bit quantization. The quantization factor, zero-point offset, and weights after training are saved for use in the actual application phase of voice wake-up. Ultimately, this significantly reduces the number of parameters and computational load of the voice wake-up service, substantially improves the wake-up speed, and reduces the power consumption of the voice wake-up system.

[0095] After the above training, a mixed-precision quantization neural network is obtained, which processes the log-Mel spectrum features corresponding to the speech signal to be processed, and outputs the class with the highest probability as the classification result. In this embodiment, the classification result of the mixed-precision quantization neural network is divided into "N keywords", "non-keywords", and "silence", with each keyword as a class.

[0096] Step 104: If the classification result corresponding to the log-Mel spectrum feature is a wake word and the set time limit has been exceeded, wake up the voice wake-up device.

[0097] If the classification result corresponding to the log-Mel spectrum feature is a wake-up word, then it is determined whether the wake-up time corresponding to the log-Mel spectrum feature exceeds the set time limit. If so, the voice wake-up device is activated. At this time, the voice wake-up device can respond to the user with voice, such as "here" or "I'm here," indicating that the voice wake-up device has been successfully activated.

[0098] If the classification result corresponding to the log-Mel spectrum feature is "non-keyword" or "silent", then the voice wake-up device should not respond.

[0099] In practical applications, each frame of audio signal can be processed according to the above scheme.

[0100] To verify the effectiveness of the mixed-precision quantization neural network disclosed in this embodiment, this embodiment uses a mixed-precision quantization neural network and a full-precision ResNet14 network to process speech signals. The mixed-precision quantization neural network performs calculations after performing mixed-precision quantization on the 32-bit full-precision input. The full-precision ResNet14 network directly performs calculations on the 32-bit full-precision input. The number of parameters (bits), floating-point computation time (FLOPs), and accuracy used by both are shown in Table 1.

[0101] Table 1

[0102] Model Parameter size (bits) FLOPs Accuracy (%) Full-precision ResNet14 25.32M 38M 95.4 The hybrid precision quantization neural network of the present invention 0.94M 3.3M 95.1

[0103] As shown in Table 1, the voice wake-up service based on a mixed-precision quantized neural network in this invention achieves a wake-up accuracy of 95.1%. Compared to the traditional voice wake-up service based on a full-precision ResNet14 network, the accuracy loss is only 0.3%, while reducing floating-point computation by 91.3% and saving 96.3% of the parameters. Therefore, the solution in this invention significantly reduces the number of parameters and computational load of the neural network model in the voice wake-up service, thereby reducing the memory usage, power consumption, and wake-up latency of the voice wake-up service. This provides a novel low-power voice wake-up design approach and chip design concept.

[0104] Based on the same inventive concept as in the foregoing embodiments, this invention also discloses a voice wake-up system based on a hybrid precision quantization neural network. This system can be deployed in a voice chip or in a voice wake-up device. See also... Figure 3 The system includes:

[0105] The microphone unit 301 of the voice wake-up device is used to acquire the voice signal to be processed.

[0106] Feature extraction unit 302 is used to extract features from the speech signal to be processed to obtain log-Mel spectrum features;

[0107] The model processing unit 303 is used to input the log-Mel spectrum features into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and operation simplification for the parameters and operations of each layer in the network, based on the accuracy requirements of parameter calculation for each layer.

[0108] The determination module 304 is used to wake up the voice wake-up device if the classification result corresponding to the log-Mel spectrum feature is a wake word and a set time limit has been exceeded.

[0109] Based on the same inventive concept as in the foregoing embodiments, this embodiment of the invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0110] Based on the same inventive concept as in the foregoing embodiments, this embodiment of the invention also discloses a voice wake-up device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0111] Compared with traditional voice wake-up services that use full-precision neural networks as classifiers, this invention uses mixed-precision quantization to quantize the original 32-bit parameters (feature input, feature output, and weights) in the network into 1 / 4 / 8-bit parameters respectively, which greatly reduces the number of parameters in the neural network and significantly reduces the memory usage of the voice wake-up service.

[0112] Compared with traditional quantization (taking the most commonly used 8-bit quantization as an example), the hybrid precision quantization in this invention adopts different quantization bit bits to meet the different precision requirements of parameter calculation in each layer of the network. While ensuring the precision, it further reduces the number of network parameters and computational complexity. It also successfully combines 1-bit quantization with multi-bit quantization methods such as 4-bit and 8-bit, providing a brand-new low-power voice wake-up approach and chip design concept.

[0113] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0114] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0115] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0116] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0117] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0118] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components of the gateway, proxy server, or system according to embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0119] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A voice wake-up method based on a hybrid precision quantization neural network, characterized in that, The method includes: The microphone of the voice-activated device is used to collect the voice signal to be processed. Feature extraction is performed on the speech signal to be processed to obtain log-Mel spectrum features; The log-Mel spectrum features are input into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and operation simplification for the parameters and operations of each layer in the network, based on the accuracy requirements of parameter calculation for each layer. If the classification result corresponding to the log-Mel spectrum feature is a wake word and the set time limit has been exceeded, the voice wake-up device is activated. The hybrid precision quantization neural network includes: a first convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, more than twelve residual blocks, a max pooling layer, and a fully connected layer connected in sequence. Each residual block includes: a second convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, and a shortcut; In each residual block, the shortcut connects the input of the second convolutional layer in the current residual block to the output of the BN layer, which is used to copy the input of the second convolutional layer and add it to the output of the BN layer to obtain a sum; the sum is used as the input of the ReLU activation function in the current residual block and the input of the next residual block; finally, the output of the residual block enters the max pooling layer for downsampling and is then input to the fully connected layer to calculate the probability; The first convolutional layer and the last fully connected layer are quantized with 8 bits; all second convolutional layers in the residual block are quantized with 1 bit; and the feature inputs and feature outputs of all BN layers in the residual block are quantized with 4 bits.

2. The method as described in claim 1, characterized in that, The first convolutional layer and the last fully connected layer use 8-bit quantization, specifically including: For the first convolutional layer and the last fully connected layer, the formula is used. The full-precision input is quantized to 8 bits to obtain an unsigned 8-bit feature input, its quantization factor, and zero offset; wherein, the full-precision input in the first convolutional layer is the log-Mel spectrum feature input corresponding to the speech dataset; Where q is the quantized unsigned parameter, and the number of bits follows the range of the truncation function; clamp is the truncation function, ranging from [0, 255]; round is the rounding function; r is the full-precision input, 32 bits; and S is the quantization factor. , `quantize_range` is the range of values ​​after quantization, `float_range` is the range of values ​​before quantization, and `float_range = (rmax - rmin)`, where `max` and `rmin` are the maximum and minimum values ​​of the 32-bit full-precision input `r`, respectively; `Z` is the zero-point offset. Z is limited to the range [0, 255], and int is the integer function; The weights are quantized to 8 bits using the formula to obtain unsigned 8-bit weights, their quantization factors, and zero offsets; wherein each channel of the unsigned 8-bit weights has its own quantization factor and zero offset.

3. The method as described in claim 2, characterized in that, After the first convolutional layer and the last fully connected layer are quantized with 8 bits, the method further includes: Based on the unsigned 8-bit feature input, its quantization factor, and zero-point offset, and the unsigned 8-bit weights, their quantization factors, and zero-point offsets, the multiplication calculation in the first convolutional layer and the last fully connected layer is simplified as follows: Wherein, Qa represents the unsigned 8-bit feature input, Za represents the zero offset corresponding to the unsigned 8-bit feature input, Sa represents the quantization factor corresponding to the unsigned 8-bit feature input, Qw represents the unsigned 8-bit weight, Zw represents the zero offset corresponding to the unsigned 8-bit weight, and Sw represents the quantization factor corresponding to the unsigned 8-bit weight. Both can be implemented using an 8-bit unsigned multiplier. It is implemented using shift operations, and the accumulation calculation is implemented using an 8-bit unsigned adder.

4. The method as described in claim 1, characterized in that, All second convolutional layers in the residual block are quantized with 1 bit, specifically including: For each of the second convolutional layers, the sign function is used to quantize the part of the feature input and weight of the second convolutional layer that is greater than 0 to 1, and the part that is less than 0 to -1, so as to obtain 1 bit feature input and 1 bit weight.

5. The method as described in claim 4, characterized in that, After quantizing all second convolutional layers in the residual block with 1 bit, the method further includes: For each of the second convolutional layers, based on the 1-bit feature input and the 1-bit weights, the formula is used... Calculate the feature output y of each of the second convolutional layers. r ;in, , These represent 1-bit feature input and 1-bit weight after 1-bit quantization, respectively; xnor operation is an XNOR operation; popcount operation is a bitwise operation.

6. The method as described in claim 3, characterized in that, The feature inputs and outputs of all BN layers in the residual block are quantized using 4-bit quantization, specifically including: The formula is used to quantize the feature inputs and feature outputs of all BN layers in the residual block using 4-bit quantization; at this time, the truncation range of the clamp is [0, 63].

7. The method as described in claim 6, characterized in that, The step of performing 4-bit quantization on the feature inputs and feature outputs of all BN layers in the residual block using the aforementioned formula specifically includes: For each feature input of the BN layer, the quantization factor and zero offset corresponding to the feature input of the BN layer are calculated, and then the feature input of the BN layer is quantized to 4 bits using the formula. The quantization factor corresponding to the feature input of the BN layer is used as the quantization factor corresponding to the parameter involved in the calculation. The corresponding zero offset is calculated, and then the parameter involved in the calculation is quantized by 4 bits using the formula. Thus, the calculation of the feature output is simplified to the subtraction and multiplication operations of 4-bit unsigned numbers.

8. The method according to any one of claims 1-7, characterized in that, The mixed-precision quantization neural network is trained in the following manner: Obtain the speech dataset; The neural network model is trained and tested using the speech dataset combined with mixed-precision quantization to obtain the mixed-precision quantization neural network.

9. A voice wake-up system based on a hybrid precision quantization neural network, characterized in that, include: The microphone unit of the voice wake-up device is used to collect the voice signal to be processed; The feature extraction unit is used to extract features from the speech signal to be processed to obtain log-Mel spectrum features; The model processing unit is used to input the log-Mel spectrum features into a mixed-precision quantization neural network to determine the classification result corresponding to the log-Mel spectrum features; wherein, the mixed-precision quantization refers to using different quantization bits for parameter quantization and operation simplification for the parameters and operations of each layer in the network, based on the accuracy requirements of parameter calculation for each layer. The determination module is used to wake up the voice wake-up device if the classification result corresponding to the log-Mel spectrum feature is a wake word and a set time limit has been exceeded. The hybrid precision quantization neural network includes: a first convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, more than twelve residual blocks, a max pooling layer, and a fully connected layer connected in sequence. Each residual block includes: a second convolutional layer, a batch normalized (BN) layer, a linear rectified unit (ReLU) activation function, and a shortcut; In each residual block, the shortcut connects the input of the second convolutional layer in the current residual block to the output of the BN layer, which is used to copy the input of the second convolutional layer and add it to the output of the BN layer to obtain a sum; the sum is used as the input of the ReLU activation function in the current residual block and the input of the next residual block; finally, the output of the residual block enters the max pooling layer for downsampling and is then input to the fully connected layer to calculate the probability; The first convolutional layer and the last fully connected layer are quantized with 8 bits; all second convolutional layers in the residual block are quantized with 1 bit; and the feature inputs and feature outputs of all BN layers in the residual block are quantized with 4 bits.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-8.

11. A voice wake-up device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-8.

Images