Voice wake-up methods, electronic devices, storage media, and computer program products

Abstract

This application discloses a voice wake-up method, electronic device, storage medium, and computer program product, relating to the field of signal processing technology. The method includes: acquiring a sound signal; extracting features from the sound signal to obtain at least two-dimensional signal features for at least two channels, wherein the two dimensions of the two-dimensional signal features are time and frequency; performing convolution processing on the two-dimensional signal features of each channel to obtain a feature map for at least one channel, wherein the convolution processing includes at least one depthwise separable convolution process, which includes two-dimensional depthwise convolution and two-dimensional pointwise convolution; and identifying the voice wake-up word based on the feature maps of each channel. This application significantly reduces the number of parameters and computational load of the algorithm and improves the performance of edge-side voice wake-up in complex scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a voice wake-up method, an electronic device, a storage medium and a computer program product. BACKGROUND

[0002] In recent years, with the popularity of intelligent terminal devices (such as smart watches, AR / VR glasses, wireless earphones, etc.), end-side voice wake-up (KeyWord Spotting, KWS) technology has become a key entry for human-computer interaction. Traditional voice wake-up systems usually rely on cloud processing, but due to network delay, privacy security and power consumption limitations, end-side voice wake-up has gradually become a research hotspot. Current voice wake-up can be divided into traditional machine learning algorithms and deep learning algorithms. In traditional machine learning algorithms, the core process usually includes feature extraction, keyword matching and other steps. This algorithm has low computational complexity, but has limitations in dealing with complex scene tasks. Deep learning-based methods use feedforward neural networks, convolutional neural networks and recurrent neural networks to enhance voice wake-up performance, achieving excellent performance. However, these models have high parameter quantity and computational complexity, making it difficult to run on end-side devices.

[0003] In actual scenarios, especially on low-power embedded devices and mobile terminals, the deployment of voice wake-up systems is usually limited by computing resources and storage space. Deep neural networks have excellent performance, but are difficult to apply under low latency and low power consumption requirements. Therefore, under the condition of limited resources, how to design an efficient KWS system is still a challenge that needs to be broken through.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a voice wake-up method, an electronic device, a storage medium and a computer program product, aiming to improve the voice wake-up performance of the voice wake-up (KWS) task in complex scenarios on end-side devices under the condition of low parameter quantity and low computing resources.

[0006] To achieve the above purpose, the present application provides a voice wake-up method, which comprises:

[0007] obtaining a sound signal, performing feature extraction on the sound signal to obtain two-dimensional signal features of at least two channels, wherein the two dimensions of the two-dimensional signal features are time dimension and frequency dimension;

[0008] The two-dimensional signal features of each channel are convolved to obtain a feature map of at least one channel. The convolution process includes at least one depth-separable convolution process, which includes two-dimensional depthwise convolution and two-dimensional pointwise convolution.

[0009] The voice wake-up word recognition result is obtained based on the feature map recognition of each channel.

[0010] Optionally, the step of convolving the two-dimensional signal features of each channel to obtain a feature map of at least one channel includes:

[0011] The two-dimensional signal features of each channel are convolved using at least two pre-defined, sequentially connected depthwise separable convolutional structures to obtain feature maps of at least one channel. The input data of the first depthwise separable convolutional structure is the two-dimensional signal features of each channel, and the input data of each depthwise separable convolutional structure after the first depthwise separable convolutional structure is the output data of the previous depthwise separable convolutional structure.

[0012] Optionally, for each of the depthwise separable convolutional structures, the step of performing convolution processing on the input data using the depthwise separable convolutional structure includes:

[0013] The input data is input into the depth-separable convolutional structure for convolution processing to obtain the processing result;

[0014] The input data is added to the processing result to obtain the output data of the depth-separable convolutional structure.

[0015] Optionally, the depth-separable convolutional structure includes a depthwise convolutional module, a first normalized activation module, a pointwise convolutional module, and a second normalized activation module connected in sequence; the depthwise convolutional module includes a first type of convolutional kernel corresponding to the two-dimensional features of each channel in the input data, the first type of convolutional kernel being used to perform convolution processing on the two-dimensional features of the corresponding channel; the pointwise convolutional module includes at least one second type of convolutional kernel, the second type of convolutional kernel having a size of 1×1×M, where M is the number of output channels of the first normalized activation module.

[0016] Optionally, the step of performing convolution processing on the two-dimensional signal features of each channel using at least two pre-defined, sequentially connected depthwise separable convolutional structures to obtain a feature map of at least one channel includes:

[0017] The two-dimensional signal features of each channel are convolved using at least two pre-defined, sequentially connected depthwise separable convolutional structures to obtain the output data corresponding to each depthwise separable convolutional structure.

[0018] The output data are summed to obtain a feature map of at least one channel.

[0019] Optionally, the step of extracting features from the sound signal to obtain two-dimensional signal features for at least two channels includes:

[0020] The FBANK feature is obtained by extracting features from the sound signal;

[0021] The FBANK features are expanded in channels and transformed in dimensions to obtain two-dimensional signal features with at least two channels.

[0022] Optionally, the step of obtaining the voice wake-up word recognition result based on the feature map recognition of each channel includes:

[0023] The feature maps of each channel are input into a preset classification layer for classification to obtain probability values;

[0024] If the probability value is greater than a preset threshold, the recognition result of the sound signal containing a voice wake-up word is obtained;

[0025] If the probability value is less than or equal to the preset threshold, then the recognition result is that the sound signal does not contain a voice wake-up word.

[0026] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the voice wake-up method as described above.

[0027] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the voice wake-up method described above.

[0028] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the voice wake-up method described above.

[0029] One or more technical solutions proposed in this application have at least the following technical effects:

[0030] By extracting features from the sound signal, at least two-dimensional signal features of two channels are obtained, with the two dimensions being time and frequency, providing a data foundation for fully capturing detailed information in the time and frequency domains. The two-dimensional signal features of each channel are then convolved, including at least one depthwise separable convolution. The depthwise separable convolution includes two-dimensional depthwise convolution and two-dimensional pointwise convolution. The feature map of at least one channel obtained from the convolution is then recognized to obtain the voice wake-up word recognition result. Compared with ordinary two-dimensional convolution, this method greatly reduces the number of algorithm parameters and computational load while ensuring full capture of detailed information in the time and frequency domains, and improves the performance of voice wake-up on the device side in complex scenarios. Ultimately, it achieves accurate voice command triggering in the standby state of the device side. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating the first embodiment of the voice wake-up method of this application;

[0034] Figure 2 A schematic diagram of two-dimensional depthwise convolution provided in one embodiment of this application;

[0035] Figure 3 A schematic diagram of two-dimensional pointwise convolution provided in one embodiment of this application;

[0036] Figure 4 A schematic diagram of a neural network structure provided in one embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice wake-up method in the embodiments of this application.

[0038] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0040] It should be noted that in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0041] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0042] In real-world scenarios, especially in low-power embedded devices and mobile terminals, the deployment of voice wake-up systems is often limited by computing resources and storage space. While deep neural networks offer excellent performance, they are difficult to apply under the requirements of low latency and low power consumption. Therefore, designing an efficient KWS system under limited resources remains a challenge that urgently needs to be overcome.

[0043] This application provides a solution to improve the voice wake-up (KWS) performance in complex scenarios on edge devices with low parameter count and low computing resources. It extracts features from the sound signal to obtain at least two-dimensional signal features for two channels, with the two dimensions being time and frequency, providing a data foundation for fully capturing detailed information in the time and frequency domains. The two-dimensional signal features of each channel are then convolved, including at least one depthwise separable convolution, which includes two-dimensional depthwise convolution and two-dimensional pointwise convolution. The feature maps of at least one channel obtained from the convolution are then recognized to obtain the voice wake-up word recognition result. Compared to ordinary two-dimensional convolution, this solution significantly reduces the number of parameters and computational load while ensuring full capture of detailed information in the time and frequency domains, and improves the performance of edge-side voice wake-up in complex scenarios, ultimately achieving accurate voice command triggering in the standby state of the edge device.

[0044] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication, and program execution functions, such as AR (Augmented Reality) / VR (Virtual Reality) glasses, helmets, or other head-mounted devices, or devices such as tablets, personal computers, and mobile phones. (See also...) Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the voice wake-up method of this application. In this embodiment, the voice wake-up method includes steps S10 to S30:

[0045] Step S10: Acquire an audio signal and perform feature extraction on the audio signal to obtain two-dimensional signal features for at least two channels, wherein the two dimensions of the two-dimensional signal features are time dimension and frequency dimension.

[0046] The sound signal can be acquired through a built-in or external microphone; this embodiment does not limit the method of acquiring the sound signal. In one feasible embodiment, to ensure the timeliness of voice wake-up, the sound signal acquired by the microphone in real time can be acquired and processed in real time. In another feasible embodiment, to improve the accuracy of voice wake-up, voice activity detection (VAD) can be performed on the sound signal acquired by the microphone. Voice wake-up recognition is only performed when the sound signal contains speech; that is, the acquired sound signal can be a sound signal containing speech.

[0047] The purpose of feature extraction from audio signals is to extract key parameters from the original waveform that can effectively represent the speech content, so as to accurately identify the voice wake-up word in subsequent steps. In this embodiment, feature extraction is performed on the audio signal to obtain two-dimensional signal features of at least two channels. Let C represent the number of channels, T represent the time dimension, and F represent the frequency dimension. The dimension of the two-dimensional signal features of at least two channels can be expressed as C×T×F, where C≥2.

[0048] In a specific implementation, to obtain two-dimensional signal features with time and frequency dimensions, the time-domain audio signal can be framed, and each frame can be subjected to Fourier transform processing to obtain a time-varying spectrum, which serves as the basis for subsequent feature extraction. This embodiment does not limit the feature extraction method for the audio signal; for example, Mel-Frequency Cepstral Coefficients (MFCCs) can be extracted as signal features.

[0049] In this embodiment, there are no restrictions on the method of acquiring at least two channels of two-dimensional signal features. For example, the single-channel audio signal obtained by feature extraction can be extended to form at least two channels of two-dimensional signal features, and the method of channel extension is not limited here; or, a microphone array including at least two microphones can be used to acquire at least two channels of audio signals, and feature extraction can be performed on the audio signals of each channel to obtain the single-channel two-dimensional signal features corresponding to each channel, which are then combined to obtain at least two channels of two-dimensional signal features; or, at least two feature extraction methods can be used to extract features from the single-channel audio signal to obtain at least two types of two-dimensional signal features, with each type of two-dimensional signal feature serving as a channel, resulting in at least two channels of two-dimensional signal features.

[0050] In one feasible implementation, to improve the accuracy of voice wake-up, the FBANK (Filter Bank Energy) of the sound signal can be extracted as a signal feature. Step S10 includes S101~S102:

[0051] Step S101: Extract features from the sound signal to obtain FBANK features.

[0052] In a specific implementation, the audio signal can be segmented to obtain a series of short-time frames (e.g., each frame is 20-40ms, with a frame shift of 10-20ms); the spectrum of each frame signal is obtained by performing STFT (Short-Time Fourier Transform), and then passed through a set of Mel filter banks, each filter corresponding to a specific frequency band (the center frequency is non-linearly distributed according to the Mel scale, with dense low frequencies and sparse high frequencies), each Mel filter outputs an energy value, and the logarithm of this energy value is taken; the resulting FBANK feature is a two-dimensional matrix of T'×F', where T' is the number of frames, F' is the number of Mel filters, and the element in the i-th row and j-th column is the result of taking the logarithm of the energy value output by the j-th Mel filter in the i-th frame.

[0053] It should be noted that the FBANK-based feature extraction method compresses spectral energy onto a frequency band (Mel band) that simulates the auditory characteristics of the human ear. Each filter output represents the "intensity" of that frequency band. Compared to the original STFT spectrogram, the Mel filter bank greatly reduces the frequency dimension (e.g., from several hundred to 40 or 80), while retaining the most important auditory information and conforming to the nonlinear characteristics of frequency perception. This reduces computational complexity and improves the accuracy of voice wake-up.

[0054] Step S102: Perform channel expansion and dimension transformation on the FBANK features to obtain two-dimensional signal features with at least two channels.

[0055] The extracted FBANK features can be represented as 1×T'×F', that is, a single-channel T'×F' two-dimensional matrix. Channel expansion and dimensional transformation can be performed on the FBANK features to obtain C×T×F signal features, where C≥2. The relationship between T and T', and between F and F', can be set as needed and are not restricted here.

[0056] The order of channel expansion and dimensionality transformation is not limited in this embodiment; that is, channel expansion can be performed before dimensionality transformation, or dimensionality transformation can be performed before channel expansion, depending on the needs. Through channel expansion and dimensionality transformation, the signal features are transformed to a number of channels and dimensions more suitable for subsequent convolutional processing, while reducing redundant information between frequencies, thereby improving the expressive power of the features and enhancing the accuracy of voice wake-up.

[0057] There are many ways to implement channel expansion, and this implementation does not impose any limitations. For example, taking the channel expansion of the FBANK feature as an example, a convolutional kernel of shape C×1×1×1 can be used to convolve the FBANK feature, expanding it into a C×T'×F signal feature, and then performing a dimensionality transformation to obtain a C×T×F signal feature. There are many ways to implement dimensionality transformation, and this implementation does not impose any limitations. For example, a linear layer can be set to perform a linear transformation on the FBANK feature to obtain a T×F two-dimensional matrix, and then the T×F two-dimensional matrix can be channel expanded to obtain a C×T×F signal feature.

[0058] Step S20: Perform convolution processing on the two-dimensional signal features of each channel to obtain a feature map of at least one channel, wherein the convolution processing includes at least one depth-separable convolution processing, and the depth-separable convolution processing includes two-dimensional depth-wise convolution processing and two-dimensional point-wise convolution processing.

[0059] Two-dimensional depthwise convolution processing involves performing a two-dimensional convolution on each input channel individually, with the number of input channels being the same as the total number of input channels. For example... Figure 2 This paper presents a schematic diagram of two-dimensional depthwise convolution processing. Taking a three-channel two-dimensional feature input as an example, the diagram uses three 3×3 convolution kernels to perform individual two-dimensional convolutions, outputting three-channel two-dimensional features. Two-dimensional convolution refers to the convolution kernel moving along both the time and frequency dimensions to perform convolution processing. Two-dimensional depthwise convolution processing can effectively reduce the number of model parameters and computational cost. Furthermore, compared to one-dimensional convolution, two-dimensional convolution can jointly capture local patterns (such as formant trajectories and phoneme boundaries) in the two-dimensional time-frequency structure of speech, significantly improving feature discrimination and noise robustness. Two-dimensional pointwise convolution processing is a two-dimensional convolution with the input feature dimension unchanged, and the convolution kernel size is 1×1×M, where M is the number of output channels of the previous layer.

[0060] Since 2D depthwise convolution performs convolution operations independently on each channel of the input layer, it does not effectively utilize the feature information of different channels at the same spatial location. To fully utilize the feature information between different channels, 2D pointwise convolution is used to perform weighted summation of the feature maps output by 2D depthwise convolution in the depth direction, thereby effectively utilizing the feature information of different channels at the same spatial location. For example... Figure 3 The diagram illustrates two-dimensional pointwise convolution processing. Taking a three-channel two-dimensional feature as an example, the diagram shows three 1×1×3 convolution kernels used for convolution processing, and the output is a three-channel two-dimensional feature.

[0061] Compared to ordinary two-dimensional convolution, depthwise separable convolution ensures the capture of detailed information in the time and frequency domain features. This can greatly reduce the number of parameters and computational load of the algorithm, and improve the performance of edge voice wake-up in complex scenarios.

[0062] In one feasible implementation, to enhance the modeling capability and the ability to capture information at different levels, the two-dimensional signal features of each channel are convolved using at least two stacked depthwise separable convolutional structures. Specifically, step S20 includes:

[0063] Step S201: At least two depthwise separable convolutional structures connected in sequence are used to perform convolution processing on the two-dimensional signal features of each channel to obtain feature maps of at least one channel. The input data of the first depthwise separable convolutional structure is the two-dimensional signal features of each channel, and the input data of each depthwise separable convolutional structure after the first depthwise separable convolutional structure is the output data of the previous depthwise separable convolutional structure.

[0064] The depthwise separable convolutional structure includes depthwise convolution modules and pointwise convolution modules for performing two-dimensional depthwise convolution and two-dimensional pointwise convolution processing on the input data. The number of depthwise separable convolutional structures can be preset as needed and is not limited in this embodiment. The depthwise separable convolutional structures are connected sequentially. That is, the two-dimensional signal features of each channel are convolved by the first depthwise separable convolutional structure to obtain output data (also called intermediate feature maps). This output data is then input into the second depthwise separable convolutional structure for processing to obtain output data, and so on, until the output data obtained by the last depthwise separable convolutional structure is obtained. At least the feature map of at least one channel of the final output is obtained based on the output data of the last depthwise convolutional structure.

[0065] In this embodiment, the stacking of multiple layers of depthwise separable convolutional structures increases the receptive field of the model, enhancing its ability to model contextual information and thus further improving the accuracy of voice wake-up.

[0066] Step S30: Based on the feature map recognition of each channel, the voice wake-up word recognition result is obtained.

[0067] After obtaining the feature maps of at least one channel, the voice wake-up word recognition result can be obtained based on the feature maps of each channel. The voice wake-up word recognition result can be a recognition result indicating whether the sound signal contains a voice wake-up word. In this embodiment, there are no restrictions on the method of recognizing the voice wake-up word result based on the feature maps.

[0068] In one feasible embodiment, step S30 includes S301 to S303:

[0069] Step S301: Input the feature maps of each channel into a preset classification layer for classification to obtain probability values.

[0070] Step S302: If the probability value is greater than a preset threshold, the recognition result of the voice wake-up word contained in the sound signal is obtained.

[0071] Step S303: If the probability value is less than or equal to the preset threshold, then the recognition result that the sound signal does not contain a voice wake-up word is obtained.

[0072] The classification layer can be implemented using a linear layer; in this embodiment, the implementation method of the classification layer is not limited. The preset threshold can be set as needed, for example, to 0.8. The feature maps of each channel are input into the classification layer for classification, and a probability value in the range of 0-1 is output. This probability value is compared with the preset threshold. If it is greater than the preset threshold, the output is a recognition result indicating that the sound signal includes the wake-up word; otherwise, the output is a recognition result indicating that the sound signal does not include the wake-up word.

[0073] It should be noted that this embodiment does not impose any restrictions on the specific training method for neural network models containing linear layers, depthwise separable convolutional structures, and classification layers. Training samples can use sound signals containing the target wake-up word as positive samples and sound signals containing background noise, similar words, or irrelevant speech as negative samples.

[0074] In this embodiment, at least two-dimensional signal features of two channels are obtained by feature extraction of the sound signal, with the two dimensions being the time dimension and the frequency dimension, providing a data foundation for fully capturing the detailed information of time-frequency domain features. The two-dimensional signal features of each channel are convolved, and the convolution processing includes at least one depth-separable convolution processing, which includes two-dimensional depthwise convolution processing and two-dimensional pointwise convolution processing. The feature map of at least one channel obtained by the convolution processing is recognized to obtain the voice wake-up word recognition result. Compared with ordinary two-dimensional convolution, while ensuring the full capture of detailed information of time-frequency domain features, the algorithm's parameter quantity and computational quantity are greatly reduced, and the performance of terminal voice wake-up in complex scenarios is improved, ultimately achieving accurate voice command triggering in the standby state of the terminal device.

[0075] Based on the first embodiment described above, a second embodiment of the voice wake-up method of this application is proposed. In this embodiment, content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. In this embodiment, for each of the depthwise separable convolutional structures, the step S201 of using the depthwise separable convolutional structure to perform convolution processing on the input data includes S2011~S2012:

[0076] Step S2011: Input the input data into the depth-separable convolutional structure and perform convolution processing to obtain the processing result.

[0077] It should be noted that for the first depthwise separable convolutional structure, the two-dimensional signal features of each channel are input into the depthwise separable convolutional structure for convolution processing to obtain its output processing result; for the second and subsequent depthwise separable convolutional structures, the output data of the previous depthwise separable convolutional structure is used as the data input and input into the depthwise separable convolutional structure for convolution processing to obtain the processing result.

[0078] Step S2012: Add the input data to the processing result to obtain the output data of the depth-separable convolutional structure.

[0079] To better preserve input information and improve training stability, the input data of each depthwise separable convolutional structure is added to the processing result obtained by convolving the input data, and the result of the addition is used as the output data of that depthwise separable convolutional structure.

[0080] In a specific implementation, to facilitate the addition of input data and processing results, the size of the input data can be the same as the size of the processing result, for example, both being C×T×F. Then, the input data and processing result can be added element-wise, and the size of the resulting output data is also C×T×F. This can be achieved by designing the size of the convolutional layers in a depthwise separable convolutional structure so that both the input and output data have the same size of C×T×F.

[0081] In one feasible implementation, the depth-separable convolutional structure includes a depthwise convolutional module, a first normalized activation module, a pointwise convolutional module, and a second normalized activation module connected in sequence; the depthwise convolutional module includes a first type of convolutional kernel corresponding to the two-dimensional features of each channel in the input data, the first type of convolutional kernel being used to perform convolution processing on the two-dimensional features of the corresponding channel; the pointwise convolutional module includes at least one second type of convolutional kernel, the second type of convolutional kernel having a size of 1×1×M, where M is the number of output channels of the first normalized activation module.

[0082] It should be noted that the convolution kernels in the depthwise convolution module are referred to as the first type of convolution kernels, and the convolution kernels in the pointwise convolution module are referred to as the second type of convolution kernels, for distinction. The size of the first type of convolution kernels can be set as needed and is not limited in this embodiment; for example, a 3×3 convolution kernel can be used. The number of first type convolution kernels is the same as the number of channels in the input data, and each convolution kernel is responsible for convolution processing the two-dimensional features of one channel in the input data. The size of the second type of convolution kernel is 1×1×M, and it is responsible for convolution processing the two-dimensional features of each channel of the input data, outputting the processing result of one channel. Multiple second type convolution kernels can process multiple channels to obtain the processing results. The number of second type convolution kernels can be set as needed and is not limited in this embodiment; for example, to ensure that the size of both the input and output data is C×T×F, the number of second type convolution kernels can be C.

[0083] Both the first and second normalized activation modules include a normalization module and an activation module. The normalization module is used for batch normalization, and the activation module is used for activation, such as through the ReLU activation function (Rectified Linear Unit). Through batch normalization and activation, the data distribution can be stabilized, and nonlinearity can be introduced to improve the model's generalization ability.

[0084] Based on the above depthwise separable convolutional structure, the process for processing the input data can include: first, inputting the input data into a depthwise convolutional module, where each channel of the input data is convolved by a first type of convolutional kernel, and the result is called the first intermediate output data for differentiation; second, inputting the first intermediate output data into a first normalization activation module, where batch normalization and activation processing are performed, and the result is called the second intermediate output data for differentiation; third, inputting the second intermediate output data into a pointwise convolutional module, where convolution is performed by a second type of convolutional kernel, and the result is called the third intermediate output data for differentiation; fourth, inputting the third intermediate output data into a second normalization activation module, where batch normalization and activation processing are performed, and the result is the processing result of the depthwise separable convolutional structure; finally, this processing result can be added to the input data to obtain the output data of the depthwise separable convolutional structure.

[0085] In one feasible embodiment, step S201 includes S2013~S2014:

[0086] Step S2013: At least two depthwise separable convolutional structures connected in sequence are used to perform convolution processing on the two-dimensional signal features of each channel to obtain the output data corresponding to each depthwise separable convolutional structure.

[0087] Step S2014: Add the output data together to obtain a feature map of at least one channel.

[0088] To further enhance the ability to capture hidden features in audio signals, in this embodiment, the two-dimensional signal features of each channel can be convolved based on at least two sequentially connected depthwise separable convolutional structures. After obtaining the output data corresponding to each depthwise separable convolutional structure, the output data of each depthwise separable convolutional structure are added together to obtain a feature map of at least one channel. Specifically, if the size of the output data of each depthwise separable convolutional structure is C×T×F, then the output data can be added element-wise to obtain a feature map of size C×T×F.

[0089] By summing the output features of separable convolutional structures at different depths, fully utilizing the hidden layer features at different levels, and then performing recognition, the voice wake-up word recognition result can be obtained, which can further improve the accuracy of voice wake-up.

[0090] In one feasible implementation, the following can be adopted: Figure 4 The neural network structure shown processes the two-dimensional signal features extracted from the sound signal. Figure 4In this architecture, at least two channels of two-dimensional signal features are transformed by a linear layer and then input into N stacked depthwise separable convolutional structures for convolution processing to obtain feature maps for at least one channel. These feature maps are then input into a classification layer for classification, yielding probability values. The voice wake-up word recognition result is obtained by comparing these probability values ​​with a threshold. Each depthwise separable convolutional structure includes a depthwise convolution module, a first normalized activation module, a pointwise convolution module, and a second normalized activation module. The input data and processing result of each depthwise separable convolutional structure are summed to obtain its output data. The output data of all depthwise separable convolutional structures are summed to obtain the feature map for at least one channel input to the classification layer. This neural network structure realizes a time-frequency depthwise separable convolutional voice wake-up architecture, which significantly reduces the number of model parameters and computational cost, ultimately achieving accurate voice command triggering in the standby state of the device.

[0091] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the voice wake-up method described in the above embodiments.

[0092] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0093] like Figure 5As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0094] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0095] The electronic device provided in this application adopts the voice wake-up method in the above embodiments. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the voice wake-up method provided in the above embodiments. Moreover, the other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0096] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0098] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the voice wake-up method in the above embodiments.

[0099] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0100] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0101] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.

[0102] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0105] The readable storage medium provided in this application embodiment is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described voice wake-up method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the voice wake-up method provided in the above-described embodiments, and will not be repeated here.

[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the voice wake-up method described above.

[0107] Compared with the prior art, the beneficial effects of the computer program product provided in this application embodiment are the same as the beneficial effects of the voice wake-up method provided in the above embodiments, and will not be repeated here.

[0108] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A voice wake-up method, characterized by, The voice wake-up method comprises: obtaining a sound signal, and performing feature extraction on the sound signal to obtain two-dimensional signal features of at least two channels, wherein two dimensions of the two-dimensional signal features are a time dimension and a frequency dimension; performing convolution processing on the two-dimensional signal features of each channel to obtain a feature map of at least one channel; obtaining a voice wake-up word recognition result based on the feature map of each channel; the step of performing convolution processing on the two-dimensional signal features of each channel to obtain a feature map of at least one channel comprises: using at least two preset sequentially connected depth separable convolution structures to perform convolution processing on the two-dimensional signal features of each channel to obtain output data corresponding to each depth separable convolution structure, wherein the input data of the first depth separable convolution structure is the two-dimensional signal features of each channel, the input data of each depth separable convolution structure after the first depth separable convolution structure is the output data of the previous depth separable convolution structure, for each depth separable convolution structure, the input data is input into the depth separable convolution structure to obtain a processing result, and the input data and the processing result are added to obtain the output data of the depth separable convolution structure; adding the output data of each channel to obtain a feature map of at least one channel; wherein the depth separable convolution structure comprises sequentially connected a depth-wise convolution module, a first normalization activation module, a point-wise convolution module, and a second normalization activation module; the depth-wise convolution module comprises a first type of convolution kernel corresponding to the two-dimensional features of each channel in the input data, and the first type of convolution kernel is used for convolution processing on the two-dimensional features of the corresponding channel; the point-wise convolution module comprises at least one second type of convolution kernel, and the size of the second type of convolution kernel is 1x1xM, and M is the output channel number of the first normalization activation module; the step of obtaining a sound signal and performing feature extraction on the sound signal to obtain two-dimensional signal features of at least two channels comprises: using a microphone array comprising at least two microphones to collect sound signals of at least two channels, performing feature extraction on the sound signal of each channel to obtain two-dimensional signal features corresponding to each channel, and combining to obtain two-dimensional signal features of at least two channels; or, using at least two feature extraction methods to perform feature extraction on a single-channel sound signal to obtain at least two two-dimensional signal features, each two-dimensional signal feature being a channel, and obtaining two-dimensional signal features of at least two channels.

2. The voice wake-up method of claim 1, wherein, the step of performing feature extraction on the sound signal to obtain two-dimensional signal features of at least two channels comprises: performing feature extraction on the sound signal to obtain FBANK features; performing channel expansion and dimension transformation on the FBANK features to obtain two-dimensional signal features of at least two channels.

3. The voice wake-up method of any one of claims 1 to 2, wherein, the step of obtaining a voice wake-up word recognition result based on the feature map of each channel comprises: inputting the feature map of each channel into a preset classification layer to perform classification and obtain a probability value; If the probability value is greater than a preset threshold, a recognition result that the voice signal contains a voice wake-up word is obtained. If the probability value is less than or equal to the preset threshold, a recognition result that the voice signal does not contain a voice wake-up word is obtained.

4. An electronic device, comprising: The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the voice wake-up method according to any one of claims 1 to 3.

5. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the voice wake-up method according to any one of claims 1 to 3.

6. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by a processor to implement the steps of the voice wake-up method according to any one of claims 1 to 3.

Images