Voice activity detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of voice signal processing, and discloses a voice activity detection method and device, electronic equipment and a storage medium. The voice activity detection method comprises the steps that a microphone signal and a bone voiceprint signal are obtained, and the microphone signal and the bone voiceprint signal are signals collected through a microphone and a bone voiceprint sensor at the same time for the same scene; respectively carrying out feature extraction on the microphone signal and the bone voiceprint signal; splicing the extracted microphone features and bone voiceprint features to obtain input features; the input features are processed through a pre-trained model, the detection result of the voice activity is obtained, and the model is obtained through training of training data obtained by conducting feature extraction and feature splicing on signals collected through a microphone and a bone voiceprint sensor at the same time in the same scene. The accuracy of voice activity detection can be improved at least.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of speech signal processing, and in particular to a speech activity detection method, device, electronic device and storage medium. Background Art

[0002] Voice Activity Detection (VAD) is a basic module in many audio applications. It is used to detect whether there is voice in the audio and locate the start and end of the voice, which is beneficial to subsequent keyword detection, speech recognition, speech enhancement and other modules. In practical applications, subsequent modules can only perform subsequent processing when voice activity is detected, which can effectively reduce power consumption.

[0003] However, existing voice activity detection methods still have some problems such as false detection and missed detection, resulting in low accuracy of voice activity detection. Summary of the invention

[0004] The embodiments of the present application provide a voice activity detection method, device, electronic device, and storage medium, which are at least helpful in improving the accuracy of voice activity detection.

[0005] According to some embodiments of the present application, on the one hand, the embodiments of the present application provide a voice activity detection method, including: obtaining a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene; performing feature extraction on the microphone signal and the bone voiceprint signal respectively; splicing the extracted microphone features and bone voiceprint features to obtain input features; processing the input features using a pre-trained model to obtain a detection result of voice activity, wherein the model is trained using training data obtained by performing feature extraction and feature splicing on signals collected by a microphone and a bone voiceprint sensor at the same scene.

[0006] In some embodiments, the extracted microphone features and bone voiceprint features are spliced ​​to obtain input features, including: dimensionality compression of the microphone features and taking logarithms to obtain N-dimensional microphone features; intercepting the low-frequency band of the bone voiceprint features, taking logarithms and translating them to obtain N-dimensional bone voiceprint features; splicing the N-dimensional microphone features and the N-dimensional bone voiceprint features to obtain the input features.

[0007] In some embodiments, before using a pre-trained model to process the input features to obtain a detection result of voice activity, the method also includes: obtaining a training microphone signal and a training bone voiceprint signal, wherein the training microphone signal and the training bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor for the same scene at the same time; performing feature extraction on the training microphone signal and the training bone voiceprint signal respectively; splicing the extracted training microphone features and training bone voiceprint features to obtain training feature data; determining a label for the training feature data, wherein the label is used to indicate whether there is voice activity in the corresponding training feature data; performing model training according to the training feature data and the corresponding label to obtain the pre-trained model.

[0008] In some embodiments, the collection scene of the training microphone signal and the training bone voiceprint signal has a quiet environment; the determining of the label of the training feature data includes: determining the voice activity detection result of the microphone signal as the label of the corresponding training feature data.

[0009] In some embodiments, the difference between the duration of voice activity and the duration of no voice activity in the training microphone signal and the training bone voiceprint signal does not exceed a preset value.

[0010] In some embodiments, after the input features are processed using a pre-trained model to obtain the detection result of the voice activity, the method further includes: after determining the voice end point based on the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames, maintaining the voice activity detection of the microphone signal and the bone voiceprint signal for a subsequent preset number of frames; and / or, performing a sliding average on the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames to update the detection result of the voice activity.

[0011] In some embodiments, the model includes two one-dimensional convolutional layers, two gated recurrent units, two dense layers, and an activation function layer connected in sequence from the input end to the output end.

[0012] According to some embodiments of the present application, the second aspect of the embodiments of the present application further provides a voice activity detection device, including: an input module, used to obtain a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene; a feature extraction module, used to perform feature extraction on the microphone signal and the bone voiceprint signal respectively; a splicing module, used to splice the extracted microphone features and bone voiceprint features to obtain input features; a detection module, used to process the input features using a pre-trained model to obtain a detection result of voice activity, wherein the model is trained using training data obtained by performing feature extraction and feature splicing on signals collected by a microphone and a bone voiceprint sensor at the same scene.

[0013] According to some embodiments of the present application, a third aspect of the embodiments of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a voice activity detection method as described in any one of the first aspects.

[0014] According to some embodiments of the present application, a fourth aspect of the embodiments of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the voice activity detection method as described in any one of the first aspects.

[0015] The technical solution provided in the embodiment of the present application has at least the following advantages:

[0016] After collecting the microphone signal and bone voiceprint signal through the microphone and bone voiceprint sensor for the same scene at the same time, feature extraction is performed on them respectively, and the extracted microphone features and bone voiceprint features are spliced ​​to obtain input features. Therefore, the microphone features and bone voiceprint signal features in the input features can provide reference and compensation for each other, so that the input features can more comprehensively, completely, accurately and reliably reflect the speech features in the corresponding scene, so that the detection results of speech activities obtained by processing the input features with a pre-trained model will be more accurate and reliable, and can reduce the occurrence of false detection, missed detection and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 This is a schematic diagram comparing the complete speech signal and the bone voiceprint signal;

[0019] Figure 2 The process of the voice activity detection method provided in the embodiment of the present application is Figure 1 ;

[0020] Figure 3 It is a structural diagram of a model used in the voice activity detection method provided in the embodiment of the present application;

[0021] Figure 4 The process of the voice activity detection method provided in the embodiment of the present application is Figure 2 ;

[0022] Figure 5 is a flow chart of model training involved in the voice activity detection method provided in an embodiment of the present application;

[0023] Figure 6 is a flow chart of a model evaluation process involved in the voice activity detection method provided in an embodiment of the present application;

[0024] Figure 7 The process of the voice activity detection method provided in the embodiment of the present application is Figure 3 ;

[0025] Figure 8 It is a flowchart of a splicing process involved in the voice activity detection method provided in an embodiment of the present application;

[0026] Fig. 9 is a processing schematic diagram of a one-dimensional convolution involved in the voice activity detection method provided in an embodiment of the present application;

[0027] Fig.10 is a comparison chart of the results after simulating the voice activity detection method provided in the embodiment of the present application;

[0028] Fig.11 is another result comparison diagram after simulating the voice activity detection method provided in the embodiment of the present application;

[0029] Fig.12 is a structural diagram of a voice activity detection device provided in an embodiment of the present application;

[0030] Fig.13 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] As can be seen from the background technology, the existing voice activity detection method still has the problem of low accuracy.

[0032] After analysis, it was found that the reasons for the above problems are at least that: most of the existing voice activity detection systems are based on the energy and zero-crossing rate of the voice recorded by the microphone, which are not effective in noisy conditions and are prone to false detection and missed detection, affecting subsequent voice processing. Although some audio-related detection schemes use the bone voiceprint signal collected by the bone voiceprint sensor to resist external interference and detect through the bone voiceprint signal, they all use the bone voiceprint signal alone for processing. However, a single bone voiceprint signal can only record the voiced part of the vocal cord vibration. The specific comparison between the bone voiceprint signal and the complete human voice signal is as follows: Figure 1 As shown, please refer to the part selected in the red box for details, such as Figure 1 As shown in the signal diagram above, the complete human voice signal contains both unvoiced and voiced sounds, and is rich in content. Figure 1 The signal shown below, the bone voiceprint sensor only includes the voiced part, and the content is missing (that is, the unvoiced part is missing), making its detection inaccurate. Figure 1 The horizontal axis of the signal spectrum shown in represents time, and the vertical axis represents frequency in Mel scale.

[0033] Based on this, the embodiments of the present application provide a voice activity detection method, device, electronic device and storage medium, which splices the features extracted from the microphone signal and the bone voiceprint signal to produce a fusion. On the one hand, the bone voiceprint signal highlights the voiced sound component to avoid the adverse effects caused by the voiced sound in the microphone signal being drowned in the noise. On the other hand, the microphone signal compensates for the missing unvoiced part of the bone voiceprint signal to avoid the adverse effects caused by the missing signal, thereby effectively improving the accuracy of VAD in complex environments, especially in large noise and with external interference speech. In addition, since a pre-trained model is used, the detection efficiency is high.

[0034] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0035] The division of the following embodiments is for the convenience of description and shall not constitute any limitation on the specific implementation of the present application. The embodiments may be combined with each other and referenced to each other without contradiction.

[0036] On the one hand, an embodiment of the present application provides a voice activity detection method. It is applied to any electronic device or electronic device cluster with computing and processing capabilities, such as a server, a computer, a wearable device, etc. It improves the accuracy of detection by fusing bone voiceprint signals and microphone signals, and also has the advantages of anti-noise, anti-interference, and strong adaptability. For ease of understanding, the voice activity detection method provided by the embodiment of the present application will be described below in combination with different implementation processes.

[0037] In some embodiments, Figure 2 As shown, the process of the voice activity detection method may include the following steps:

[0038] Step 201 : Acquire a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor simultaneously for the same scene.

[0039] Step 202: extract features from the microphone signal and the bone voiceprint signal respectively.

[0040] Step 203, concatenating the extracted microphone features and bone voiceprint features to obtain input features.

[0041] Step 204, using a pre-trained model to process the input features to obtain a detection result of the voice activity, wherein the model is trained with training data obtained by extracting and splicing features of signals collected by a microphone and a bone voiceprint sensor for the same scene.

[0042] In this way, after collecting the microphone signal and bone voiceprint signal through the microphone and bone voiceprint sensor for the same scene at the same time, feature extraction is performed on them respectively, and the extracted microphone features and bone voiceprint features are spliced ​​to obtain input features. Therefore, the microphone features and bone voiceprint signal features in the input features can provide reference and compensation for each other, so that the input features can more comprehensively, completely, accurately and reliably reflect the speech features in the corresponding scene, so that the detection results of speech activities obtained by processing the input features with a pre-trained model will be more accurate and reliable, and can reduce the occurrence of false detection, missed detection and the like.

[0043] In order to facilitate those skilled in the art to better understand the above embodiment, the steps are explained below.

[0044] In step 201, a microphone signal and a bone voiceprint signal are obtained, wherein the microphone signal and the bone voiceprint signal are signals collected by the microphone and the bone voiceprint sensor for the same scene at the same time. Among them, the embodiment of the present application does not limit the specific method of obtaining the microphone signal and the bone voiceprint signal. For example, the microphone signal and the bone voiceprint signal can be obtained in real time by a sensor set on a wearable device. For example, the microphone signal and the bone voiceprint signal can be obtained by intercepting the corresponding audio, etc. Of course, the above are only examples. According to different application scenarios, requirements, etc., the methods of obtaining microphone signals and bone voiceprint signals can be adaptable to different needs, and they will not be listed one by one here.

[0045] In step 202, feature extraction is performed on the microphone signal and the bone voiceprint signal respectively. Among them, the embodiment of the present application does not limit the method of extracting features from the signal. It can be understood that the method of extracting features will also be different in adaptability according to different application requirements, hardware performance, real-time requirements, etc. For example, in the case of high real-time requirements, the feature extraction process should be simple and easy to implement, such as extracting the amplitude spectrum characteristics of the signal. For example, in the case of excellent hardware performance of the device, feature extraction can apply a more complex but more intuitive and accurate processing method that reflects the signal characteristics. It can also be understood that in the process of feature extraction, some signal processing can be introduced, such as filtering, segmentation, DC offset removal, etc. Of course, the above is only an example description of feature extraction. In some embodiments, one or more of the speech signal feature extraction algorithms such as Mel Frequency Cepstrum Coefficient (MFCC) algorithm, filter bank (Fbank) algorithm, Log-Fbank algorithm, etc. can be used to implement feature extraction, which will not be repeated here.

[0046] In step 203, the extracted microphone features and bone voiceprint features are spliced ​​to obtain input features. Among them, the embodiment of the present application does not limit the splicing method. For example, the aforementioned extracted microphone features and bone voiceprint features can be simply merged and spliced ​​directly, so that the splicing efficiency is higher and the real-time detection is good. For example, the aforementioned extracted microphone features and bone voiceprint features can be processed and then spliced. In this way, the effect of feature splicing can be optimized through processing to obtain input features with better effects, so that subsequent detection based on the input features can obtain better results.

[0047] For example, in some embodiments, the extracted microphone features and bone voiceprint features are spliced ​​to obtain input features, which can be achieved in the following way: compress the microphone features and take the logarithm to obtain N-dimensional microphone features; intercept the low-frequency band of the bone voiceprint features, take the logarithm and translate them to obtain N-dimensional bone voiceprint features; splice the N-dimensional microphone features and the N-dimensional bone voiceprint features to obtain input features.

[0048] In this way, the differences between the microphone signal and the bone voiceprint signal (including numerical differences, dimensional differences, etc.) are combined, the logarithms of the microphone features and the bone voiceprint features are taken, and the bone voiceprint signal is additionally shifted, so that the numerical values ​​of the two types of features, the microphone features and the bone voiceprint features, are converted to a relatively close state, avoiding the difference in numerical values ​​causing the microphone signal and the bone voiceprint signal to have different weights in the detection, which affects the accuracy of the detection. At the same time, the microphone features are dimensionally compressed, so that the data dimension of the microphone feature can be the same as the dimension of the bone voiceprint feature in the input feature, and avoiding the difference in data dimension causing the microphone signal and the bone voiceprint signal to have different weights in the detection, which affects the accuracy of the detection. Finally, by balancing the weights of the microphone signal and the bone voiceprint signal on the detection, the fusion effect of the microphone signal and the bone voiceprint signal is further improved, and the accuracy and reliability of the detection are improved.

[0049] In particular, in the above examples, the bone voiceprint features are intercepted in the low-frequency band, mainly considering the characteristics of the bone voiceprint sensor itself, the low-pass characteristics of bone conduction transmission, and the fact that the bone voiceprint sensor can only record the low-frequency part of the voice. In this way, the input feature data obtained will be valid data, and at the same time, the dimension of the input feature can be reduced, the amount of calculation can be reduced, and the detection efficiency can be improved. In some examples, the low-frequency band can be a frequency band below 1KHz. Of course, in some embodiments, before feature extraction, only the part of the bone voiceprint signal below 1KHz can be feature extracted, etc., which will not be repeated here.

[0050] It should be noted that the dimensions of the relevant data and the number of parameters of the model configuration in the above processing can be flexibly adjusted according to actual needs, noise environment, hardware computing power, etc.

[0051] Of course, the above example is only an example of a splicing method. In some embodiments, different weights can be assigned to data of different dimensions of the input feature to improve the fusion effect of the microphone signal and the bone voiceprint signal, so as to further improve the accuracy and reliability of the detection. Alternatively, in the above example, the dimensional compression processing and the numerical processing can also be applied separately, and the splicing methods will not be illustrated one by one here.

[0052] In step 204, the input features are processed using a pre-trained model to obtain the detection results of the voice activity, wherein the model is trained using training data obtained by extracting features and splicing features of signals collected by a microphone and a bone voiceprint sensor for the same scene. Among them, the embodiment of the present application emphasizes that the training data used for model training and the aforementioned input features are obtained through the same processing method and have the same characteristics. Therefore, through training, the model will learn how to output corresponding detection results based on the input features. At the same time, it can be understood that the model can be implemented using different structures based on different application requirements, hardware performance, etc. For example, the model can be a deep learning model based on a convolution-loop architecture.

[0053] In some examples, such as Figure 3 As shown in the figure, the model includes two one-dimensional convolutional layers (Conv1D), two gated recurrent units (GRU), two dense layers (Dense), and one activation function layer (Sigmoid) from the input to the output. In other words, the model uses one-dimensional convolution and gated recurrent units to process input features, thereby improving the detection effect through one-dimensional convolution and gated recurrent units.

[0054] Specifically, one-dimensional convolution is used to process input features to further extract features, so that in scenarios where real-time performance is not strictly required, the receptive field of the model can be increased by increasing the convolution step size, thereby improving the accuracy of the model. For example, when the convolution step size is set to 5, the current frame, the past two frames, and the future two frames can be sensed through one-dimensional convolution, so that only a small number of parameters and a certain delay are required to obtain a quasi-real-time model with higher accuracy, thereby achieving real-time and accurate detection. In addition, for voice activity detection scenarios, since the beginning and end of speech are gradual processes and are not completed within one frame, and the energy at the beginning and end is very small, it is difficult to determine whether speech is contained by only one frame. Therefore, by adding one-dimensional convolution, the detection of the current frame can refer to the previous and next frames to consider the influence of speech gradients, and ultimately further improve the accuracy of detection. The use of gated recurrent units can, on the one hand, utilize the characteristics of the recurrent neural network (RNN) to which it belongs to solve the gradient vanishing problem of the standard RNN and model the time series, thereby improving the detection effect. At the same time, compared with other RNN implementation schemes, such as long short-term memory (LSTM), it has lower computational complexity, higher efficiency, and better real-time performance, thereby further improving the real-time performance of detection.

[0055] In some embodiments, two or more models may be set. For example, two models, a unidirectional recurrent network and a bidirectional recurrent network, may be constructed, so that in scenarios such as offline speech recognition, the use of a bidirectional recurrent network is supported to further improve the accuracy of speech activity detection, and in real-time scenarios such as headphones, a unidirectional recurrent network is supported to ensure a causal system.

[0056] Of course, the above are only examples provided for the model. In some embodiments, the structure of the model and the number of models can be flexibly adjusted, which will not be listed here one by one.

[0057] In addition, the training of the model can be achieved by extracting and splicing the features of the signals collected by the microphone and the bone voiceprint sensor in the same scene to obtain the training data. In order to facilitate those skilled in the art to better understand the training process, a model training process further optimized for related technologies will be provided for illustration, but this does not mean that the training of the model must or can only be achieved as described below. In some embodiments, the model training scheme recorded in the related technology can also be used for implementation, which will not be listed here one by one.

[0058] In some embodiments, Figure 4 As shown, the process of the voice activity detection method may also include the following steps:

[0059] Step 401 : obtaining a training microphone signal and a training bone voiceprint signal, wherein the training microphone signal and the training bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor simultaneously for the same scene.

[0060] Step 402: extract features from the training microphone signal and the training bone voiceprint signal respectively.

[0061] Step 403: splice the extracted training microphone features and training bone voiceprint features to obtain training feature data.

[0062] Step 404: determine a label of the training feature data, wherein the label is used to indicate whether there is speech activity in the corresponding training feature data.

[0063] Step 405: Perform model training based on the training feature data and corresponding labels to obtain a pre-trained model.

[0064] Step 406, obtaining a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor simultaneously for the same scene.

[0065] Step 407: extract features from the microphone signal and the bone voiceprint signal respectively.

[0066] Step 408: concatenate the extracted microphone features and bone voiceprint features to obtain input features.

[0067] Step 409, using a pre-trained model to process the input features to obtain a detection result of the voice activity, wherein the model is trained by extracting and concatenating the training data obtained by performing feature extraction and feature splicing on the signals collected by the microphone and the bone voiceprint sensor for the same scene.

[0068] Among them, steps 401-403 are roughly the same as steps 201-203 in the aforementioned embodiment, and the main difference is that steps 401-403 occur in the model training stage, while steps 201-203 occur in the model application stage, and the steps are substantially the same. Steps 406-409 are roughly the same as steps 201-204 in the aforementioned embodiment, and the main difference is that steps 401-405 are also executed before steps 406-409, and steps 406-409 will not be repeated here one by one. For ease of understanding, the following mainly describes the difference between steps 401-403 and steps 201-203 in the aforementioned embodiment, and explains steps 404-405.

[0069] In step 401-step 403, in order to better implement model training, a large amount of training feature data needs to be constructed here. Therefore, in some embodiments, data can be read from a location with microphone audio resources and corresponding bone voiceprint audio resources such as the network or an existing database. Of course, it can also be obtained by real-time acquisition of microphone signals and bone voiceprint signals (such as recording with headphones containing bone voiceprint sensors and microphones).

[0070] In addition, considering that the quality of the signal will affect the training effect of the model, in some embodiments, the collection scene of the training microphone signal and the training bone voiceprint signal has a quiet environment, that is, the microphone signal and the bone voiceprint signal are recorded in a quiet environment. Thus, the accuracy and reliability of the trained model can be improved by improving the quality of the signal.

[0071] In some embodiments, data enhancement can be performed on microphone signals and / or bone voiceprint signals before feature extraction, such as amplitude scaling, data noise addition, etc., so as to enhance the generalization of the model by randomly scaling the amplitude of the microphone signal and randomly mixing it with different environmental noises and interfering voices; on the basis of a certain degree of amplitude scaling of the bone voiceprint data, further adding noise such as teeth chewing, so that the bone voiceprint sensor can effectively resist external interference, especially internal interference that cannot be overcome by amplitude scaling. Ultimately, the robustness of the model is improved to improve the reliability of detection and enhance the wide applicability of voice activity detection provided by the embodiments of the present application.

[0072] In some embodiments, training feature data can also be constructed by training microphone signals and training bone voiceprint signals in which the difference between the duration of voice activity and the duration of no voice activity does not exceed a preset value. In this way, by controlling the duration of voice activity and the duration of no voice activity to be equal, the training data is balanced, which is conducive to better training models, obtaining more accurate and reliable models, and thus improving the detection accuracy and reliability when the model is applied.

[0073] Of course, the above are only some examples. In some embodiments, other processing can be combined to construct training feature data, which will not be listed here one by one.

[0074] In step 404, the label of the training feature data is determined. Among them, the embodiment of the present application does not limit the method of determining the label. For example, in the case of acquiring the training microphone signal and the training bone voiceprint signal by real-time acquisition, some time periods can be artificially set to have voice activities and some time periods can be artificially set to have no voice activities, and then the label of the training feature data of the corresponding time can be determined by reading the start time and end time of these time periods. For example, the training microphone signal can be detected for voice activity by some existing accurate and reliable algorithms and models (such as the existing voice activity detection system based on the energy and zero-crossing rate of the microphone signal or the existing deep learning system that can realize voice activity detection based on the microphone signal) to determine the label of the corresponding training feature data. Among them, the existing scheme can be used for voice activity detection mainly because the current processing is the processing before the model is applied. Even if the efficiency is low, it does not affect the efficiency of subsequent users to realize voice activity detection through the model in the model application stage, and it will not affect the user experience. In this way, the label is obtained by the audio synchronously recorded by the microphone, making full use of the existing information, which is accurate and reliable. In addition, the training bone voiceprint signal is not used here to determine the label mainly because the bone voiceprint sensor can only record the voiced part of the vocal cord vibration as the bone voiceprint signal, that is, the training bone voiceprint signal is missing content and may be inaccurate.

[0075] In some embodiments, when the voice activity detection system based on the energy and zero-crossing rate of the microphone signal determines the label, in order to improve the reliability of the label, the microphone signal can also be set to be recorded in a quiet environment to ensure the accuracy of the label. In other words, determining the label of the training feature data can be achieved in the following way: determining the voice activity detection result of the microphone signal as the label of the corresponding training feature data. In this way, using the voice detection activity of the microphone signal as a label in a quiet environment, the label can be automatically generated, and the generation efficiency is high, accurate and reliable, and during the training process, it will not affect the detection efficiency of the trained model.

[0076] It should be noted that the embodiment of the present application does not limit the labeling method of the label. For example, the label of the training feature data with voice activity can be set to 1, otherwise it can be set to 0. Of course, the above is only an example, and other labeling methods can be used in some embodiments, which will not be listed here one by one.

[0077] In step 405, model training is performed based on the training feature data and the corresponding labels to obtain a pre-trained model. It can be understood that the labels are obtained above, so supervised training can be performed. For example, the model can be trained using binary cross entropy (BCE) loss and Adam optimizer to achieve accurate detection of voice activities. The model training process here is as follows: Figure 5 After the bone voiceprint sensor and microphone collect the corresponding signals, they are used as the recording data set, and then feature extraction is performed on the recording data set for model training. The microphone recording can also be used as a label extraction and input into the model training stage for labeled supervised training. Finally, the trained model is saved for use in the application stage. Figure 2 The voice activity detection process shown in the figure implements voice activity detection. Figure 2 The feature stitching process is not shown in the figure, but this does not mean that there is no stitching process.

[0078] It should be noted that the above embodiment mainly introduces the training process to illustrate the voice activity detection method provided in the embodiment of the present application. It can be understood that in addition to model training, in order to ensure the accuracy and reliability of the trained model, model evaluation can also be introduced, that is, while obtaining the above training feature data for model training, the training feature data that does not participate in model training is also used for model evaluation. After the evaluation is passed, the model is stored and called in the application stage to perform voice activity detection based on the input features. Among them, the model evaluation can be specifically referred to Figure 6 The process diagram shown is Figure 5 The training process diagrams shown are roughly the same, with the main difference being that after feature extraction, the model is evaluated and output. During model evaluation, the model is mainly used to process the input feature data (after splicing) and then the model is evaluated based on the model output results, and the model evaluation results are output. When the evaluation results reach a satisfactory level, the training is considered complete, otherwise it is necessary to train and evaluate again until a satisfactory model is obtained.

[0079] In some embodiments, Figure 7 As shown, the process of the voice activity detection method may also include the following steps:

[0080] Step 701, obtaining a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor simultaneously for the same scene.

[0081] Step 702: extract features from the microphone signal and the bone voiceprint signal respectively.

[0082] Step 703: concatenate the extracted microphone features and bone voiceprint features to obtain input features.

[0083] Step 704, using a pre-trained model to process the input features to obtain a detection result of the voice activity, wherein the model is trained with training data obtained by extracting and splicing features of signals collected by a microphone and a bone voiceprint sensor for the same scene.

[0084] Step 705, after determining the speech end point based on the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames, maintain the voice activity detection of the microphone signal and the bone voiceprint signal for a subsequent preset number of frames; and / or, perform sliding averaging on the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames to update the detection results of the voice activity.

[0085] In this way, based on the above-mentioned embodiment, detection is maintained after the end point is detected, so that the detection of the end point can be more accurate, and the sliding average method is used to improve the accuracy of detection.

[0086] To facilitate understanding of the above embodiment, its steps will be explained below, wherein steps 701 to 704 are substantially the same as steps 201 to 204 in the above embodiment, the main difference being that step 705 is further executed after step 704, and steps 701 to 704 will not be described in detail here.

[0087] In step 705, after the voice end point is determined based on the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames, the voice activity detection of the microphone signal and the bone voiceprint signal of the subsequent preset number of frames is maintained; and / or, the detection results of the voice activity of the microphone signal and the bone voiceprint signal of several consecutive frames are sliding averaged to update the detection results of the voice activity. In this way, after obtaining the probability of the voice activity detection output of each frame of audio, a smoother detection result can be obtained by performing numerical averaging within a certain window as post-processing. In addition, since the voice does not exist continuously during speaking, when the end of the voice segment is detected, a certain holding window can also be set. If no voice is detected within the window, the voice end point is determined, which can avoid excessive interruption of normal voice and affect the subsequent possible voice enhancement and other processing.

[0088] It should be noted that Figure 4 and Figure 7 The embodiment shown is mainly in Figure 2 The description is based on the illustrated embodiments, in some cases, Figure 4 and Figure 7 The illustrated embodiments may also be combined with each other, for example, step 705 is performed after step 409, etc., which will not be described in detail here.

[0089] To help those skilled in the art better understand the voice activity detection method provided in the above embodiment, an example is given below.

[0090] First, the training microphone signal and the training bone voiceprint signal are obtained. Specifically, the bone voiceprint sensor and microphone data are collected synchronously in a quiet environment.

[0091] Secondly, feature extraction. This process also involves data preprocessing. Preprocessing includes DC bias removal, filtering, framing, etc. Specifically, the training bone voiceprint signal is high-pass filtered to remove DC bias and filter low-frequency interference; if the air conduction data recorded by the microphone (i.e., the training microphone signal) has DC bias, it also needs to be high-pass filtered to enhance the accuracy of the model; the recorded long audio is segmented into 15s to obtain multiple frames of signals; the audio sampling rate is set to 16KHz, the audio frame length is 32ms, and the frame shift is 16ms, and features are extracted for each frame signal to obtain feature data composed of 512 sampling points in each frame signal (i.e., the data is 512-dimensional); the subsequent processing is as follows Figure 8 As shown in the figure, each 512-dimensional feature data is windowed and processed by Fast Fourier Transform (FFT) to obtain the amplitude spectrum, and the corresponding 257-dimensional bone voiceprint features and microphone features are obtained; the low frequency (below 1KHz) is cut off from the bone voiceprint feature to obtain 30-dimensional data, and the amplitude spectrum is applied to the microphone feature using the Fbank filter group with an equivalent rectangular bandwidth (ERB) scale to compress the 30-dimensional data; since the value of the bone voiceprint feature is usually much smaller than the value of the microphone feature, the 30-dimensional data obtained are logarithmically scaled and the dynamic range is compressed, and a certain offset (i.e., amplitude spectrum amplification) is applied to the 30-dimensional logarithmic spectrum corresponding to the bone voiceprint signal to balance the data, and finally the two are spliced ​​to obtain 60-dimensional features as training feature data.

[0092] Then, determine the label for data annotation. Among them, since the bone voiceprint sensor can only record the voiced part of the vocal cord vibration, the signal recorded by the microphone needs to be obtained using a traditional voice activity detection system such as one based on energy and zero-crossing rate. The label of the training feature data corresponding to the part with voice signal is set to 1, otherwise it is 0; since the microphone data is recorded in a quiet environment, the accuracy of the target label can be guaranteed by using the existing voice activity detection system. At this time, the label is determined according to the following expression:

[0093]

[0094] Wherein, F(n) is the probability of the training feature data corresponding to the microphone signal of the nth frame being output after being processed by the existing voice activity detection system, T is a preset value, and VAD(n) is the value of the label.

[0095] Next, the model training is performed. Specifically, on the one hand, the following Figure 3 The model shown in the figure includes a one-dimensional convolutional layer, a gated recurrent unit layer, a dense layer, and a sigmoid layer. The output of the model is the probability of speech activity. The probability is compared with the preset value to obtain the result of whether the speech in the current frame exists. Among them, the convolution step size of the one-dimensional convolutional layer can be set to 5, so the convolutional layer can sense the current frame, the past two frames, and the next two frames. Only a small amount of parameters and a certain delay are needed to obtain a quasi-real-time system with higher accuracy. For example, Fig. 9 As shown in the figure, when the current frame is the 5th frame, the past 3rd and 4th frames, as well as the future 6th and 7th frames can be sensed through the one-dimensional convolution layer. Since the beginning and end of speech are gradual processes, not completed in one frame, and the energy at the beginning and end is very small, it is difficult to determine whether speech is contained by only one frame, so adding a one-dimensional convolution layer can improve the accuracy of the model, especially for speech recognition scenarios. On the other hand, the previously obtained training feature data is used as input and the label is used as the target. The model is trained through gradient descent and back propagation to obtain a trained model.

[0096] Then, the trained model is applied to perform voice activity detection. Among them, assuming that the bone voiceprint signal and microphone signal for voice activity detection are audio of a certain length, the same processing as the aforementioned training microphone signal and training bone voiceprint signal is used to obtain input features, and the input features are input into the model to obtain the probability of voice activity in each frame of bone voiceprint signal and the corresponding frame of microphone signal. By comparing with the preset value, it can be determined whether there is voice activity in the scene corresponding to each frame of bone voiceprint signal and the corresponding frame of microphone signal. In particular, after obtaining the probability of the model output and before determining whether there is language activity, a sliding window average can be used for processing, that is, based on the probability of the model output corresponding to the signal of multiple consecutive frames, a window of a certain length is set, and the probability corresponding to the window at the current time is updated to the result obtained by averaging the probability value in the window, and then compared with the preset value, so that the result of the model output for audio of a certain length will be smoother. In particular, the detection results before and after can be combined to adapt to the continuity of voice activity, thereby improving the accuracy of judgment. Further, by comparing the final judgment results of whether there is voice activity corresponding to the continuous multiple frames of signals, the start and end times of speech can be determined in audio of a certain length.

[0097] Furthermore, in the above scheme, if the model is quantized into 8-bit integers, the above scheme can be applied to the voice activity detection scenario of small wearable devices. For example, if the above scheme is configured in the earphones of the wearable device, personalized voice activity detection can be achieved without user registration.

[0098] A simulation experiment was conducted on the above scheme. When the relevant signal was collected in an environment with large noise, the following results were obtained: Fig.10 The comparison diagram of the input audio and the detection result shown in FIG. 1 shows that when the relevant signal is collected in an environment with interfering speech, the following will be obtained: Fig.11 The input audio and detection result comparison diagram shown in FIG. Fig.10 and Fig.11 The upper graph in the figure is the input audio, and the lower graph is the detection result. The signals shown in the figure are all based on time as the horizontal axis and frequency as the vertical axis. Fig.10 and Fig.11 It can be clearly seen that in both cases, the speech start time and end time in the input audio are accurately detected, thereby obtaining audio with valid speech content.

[0099] It can be seen that the voice activity detection method provided by the embodiment of the present application can greatly improve the accuracy of voice activity detection by integrating bone voiceprint and microphone voice. It utilizes the characteristics that the user's bone voiceprint information collected by the bone voiceprint sensor can effectively resist the influence of environmental noise or interfering voice, while retaining the high-frequency part of the voice captured by the microphone. This system uses deep learning technology and has been trained with a large amount of data to effectively improve the adaptability of the system in different scenarios, which is beneficial to subsequent voice processing such as keyword detection, voice recognition, and voice enhancement. Specifically:

[0100] 1. Using deep learning technology, the accuracy of voice activity detection can be greatly improved by integrating bone voiceprint and microphone voice features through large-scale data training.

[0101] 2. Data enhancement is performed before feature extraction to improve the robustness of language activity detection. In particular, adding interfering speech to the microphone signal can effectively resist the interference of speech around the speaker.

[0102] 3. The deep learning model using the convolution-recurrent architecture can effectively model the frequency and time dimensions and improve the overall accuracy of the system.

[0103] 4. The model can be quantized to 8-bit integers and applied to small wearable devices, enabling personalized voice activity detection without user registration.

[0104] 5. Since the bone voiceprint sensor only records the wearer's voice and does not collect the echo propagated in the air, after adopting the voice activity detection scheme provided in the embodiment of the present application, the device used by the scheme has a certain acoustic echo cancellation (Acoustic Echo Cancel, AEC) function, that is, when the wearer stops speaking, the microphone will not record the sound of the speaker and produce an echo.

[0105] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0106] Another aspect of the present application embodiment further provides a voice activity detection device, such as Fig.12 As shown, including:

[0107] An input module, used to obtain a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene;

[0108] A feature extraction module is used to extract features from microphone signals and bone voiceprint signals respectively;

[0109] A splicing module is used to splice the extracted microphone features and bone voiceprint features to obtain input features;

[0110] The detection module is used to process input features using a pre-trained model to obtain detection results of voice activities, wherein the model is trained by extracting and splicing features from signals collected by a microphone and a bone voiceprint sensor in the same scene to obtain training data.

[0111] It is not difficult to find that this embodiment is a device embodiment corresponding to the method embodiment, and this embodiment can be implemented in conjunction with the method embodiment. The relevant technical details mentioned in the method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment.

[0112] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0113] Another aspect of the present application embodiment further provides an electronic device, such as Fig.13 As shown, it includes: at least one processor 1301; and a memory 1302 that is communicatively connected to the at least one processor 1301; wherein the memory 1302 stores instructions that can be executed by the at least one processor 1301, and the instructions are executed by the at least one processor 1301 so that the at least one processor 1301 can execute the voice activity detection method described in any of the above method embodiments.

[0114] Among them, the memory 1302 and the processor 1301 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 1301 and the memory 1302 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 1301 is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor 1301.

[0115] The processor 1301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 1302 can be used to store data used by the processor 1301 when performing operations.

[0116] Another aspect of the present application embodiment further provides a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0117] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0118] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A voice activity detection method, characterized in that: include: Acquire a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene; Performing feature extraction on the microphone signal and the bone voiceprint signal respectively; The extracted microphone features and bone voiceprint features are concatenated to obtain input features; The input features are processed using a pre-trained model to obtain a detection result of the voice activity, wherein the model is trained using training data obtained by extracting and splicing features from signals collected by a microphone and a bone voiceprint sensor in the same scene.

2. The voice activity detection method according to claim 1, characterized in that: The extracted microphone features and bone voiceprint features are spliced ​​to obtain input features, including: Compressing the microphone feature dimensionally and taking its logarithm to obtain an N-dimensional microphone feature; After intercepting the low-frequency band of the bone voiceprint feature, taking the logarithm and performing translation, an N-dimensional bone voiceprint feature is obtained; The N-dimensional microphone feature and the N-dimensional bone voiceprint feature are concatenated to obtain the input feature.

3. The voice activity detection method according to claim 1 or 2, characterized in that: Before using the pre-trained model to process the input features to obtain the detection result of the voice activity, the method further includes: Acquire a training microphone signal and a training bone voiceprint signal, wherein the training microphone signal and the training bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene; Performing feature extraction on the training microphone signal and the training bone voiceprint signal respectively; The extracted training microphone features and training bone voiceprint features are spliced ​​to obtain training feature data; Determining a label of the training feature data, wherein the label is used to indicate whether there is speech activity in the corresponding training feature data; Model training is performed according to the training feature data and the corresponding labels to obtain the pre-trained model.

4. The voice activity detection method according to claim 3, characterized in that: The collection scene of the training microphone signal and the training bone voiceprint signal has a quiet environment; The determining of the label of the training feature data comprises: The voice activity detection result of the microphone signal is determined as the label of the corresponding training feature data.

5. The voice activity detection method according to claim 3, characterized in that: The difference between the duration of voice activity and the duration of no voice activity in the training microphone signal and the training bone voiceprint signal does not exceed a preset value.

6. The voice activity detection method according to claim 1 or 2, characterized in that: After the input feature is processed by the pre-trained model to obtain the detection result of the voice activity, the method further includes: After determining a speech end point based on the detection results of the speech activity of the microphone signal and the bone voiceprint signal for a plurality of consecutive frames, maintaining the speech activity detection for a subsequent preset number of frames of the microphone signal and the bone voiceprint signal; and / or, A sliding average is performed on the voice activity detection results of the microphone signal and the bone voiceprint signal of several consecutive frames to update the voice activity detection result.

7. The voice activity detection method according to claim 1 or 2, characterized in that: The model includes two one-dimensional convolutional layers, two gated recurrent units, two dense layers, and an activation function layer connected in sequence from the input end to the output end.

8. A voice activity detection device, characterized in that: include: An input module, used to obtain a microphone signal and a bone voiceprint signal, wherein the microphone signal and the bone voiceprint signal are signals collected by a microphone and a bone voiceprint sensor at the same time for the same scene; A feature extraction module, used to extract features from the microphone signal and the bone voiceprint signal respectively; A splicing module is used to splice the extracted microphone features and bone voiceprint features to obtain input features; The detection module is used to process the input features using a pre-trained model to obtain a detection result of the voice activity, wherein the model is trained by extracting and splicing features from the signals collected by the microphone and the bone voiceprint sensor in the same scene.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the voice activity detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the voice activity detection method according to any one of claims 1 to 7 is implemented.

Images