Voice activity detection method and device, computer equipment and storage medium

Abstract

The invention relates to a voice activity detection method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring acoustic characteristics of each audio frame of audio sampling data; inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network and a second intermediate feature tensor output by a gating cycle unit; inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, and weighting the first output feature to obtain a first enhanced feature; inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, and weighting the second output feature to obtain a second enhanced feature; performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; and performing voice activity detection according to the aggregated features. By adopting the method, the voice activity detection capability can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of voice activity processing, and in particular to a voice activity detection method, apparatus, computer equipment and storage medium. Background Art

[0002] With the advent of the mobile Internet era, people's demand for intelligence has promoted the great development of intelligent voice technology. More and more smart devices have begun to use intelligent voice technology to complete interaction, such as smart cars, smart phones and smart speakers. Voice activity detection technology is one of the key steps in voice interaction. Voice activity detection technology is one of the core technologies in the fields of voice processing systems, consumer electronics, multimedia systems, etc. Voice activity detection not only involves the problem of digital signal processing, but also involves auditory perception characteristics and human voice characteristics. At the same time, the diversity of noise increases the difficulty of voice activity detection. It is very difficult to judge the starting point and end point of a voice signal from a noisy voice signal.

[0003] At present, voice activity detection technologies include energy-based methods, zero-crossing rate-based methods, statistical model-based methods, and deep learning-based methods. Among them, both the energy-based method and the zero-crossing rate-based method need to determine a heuristic threshold, and then compare the feature value with the threshold to obtain an estimated result, thereby realizing voice activity detection. Although this method is easy to implement, it does not work well in a noisy environment. The statistical model-based method not only needs to assume the distribution characteristics of speech and non-speech, but also needs to determine whether the Markov assumption is satisfied. This method is not conducive to processing unseen data. The deep learning-based method usually uses the structure of a convolutional recurrent neural network to model audio timing information. It achieves a good detection effect on non-stationary noise signals, but it is still not accurate enough for voice activity detection. Summary of the invention

[0004] Based on this, it is necessary to provide a voice activity detection method, device, computer equipment and storage medium to address the above technical problems, which can effectively improve the ability of voice activity detection, so as to more effectively determine the starting point and end point of the voice signal under steady or non-steady noise interference.

[0005] A voice activity detection method comprises: obtaining acoustic features of each audio frame of audio sampling data; inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network and a second intermediate feature tensor output by a gated recurrent unit in the voice activity detection model; inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weighting the first output feature to obtain a first enhanced feature; inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weighting the second output feature to obtain a second enhanced feature; performing feature aggregation on the first enhanced feature and the second enhanced feature based on vectorized attention weights to obtain an aggregated feature; and performing voice activity detection according to the aggregated feature.

[0006] In some embodiments, the first intermediate feature tensor is input into a first multi-head self-attention module to obtain a first output feature, including: obtaining a first position embedding vector; converting the first intermediate feature tensor into a first embedding vector according to the first position embedding vector and using vector accumulation; and inputting the first embedding vector into the first multi-head self-attention module to obtain a first output feature.

[0007] In some embodiments, the second intermediate feature tensor is input into a second multi-head self-attention module to obtain a second output feature, including: obtaining a second position embedding vector; converting the second intermediate feature tensor into a second embedding vector according to the second position embedding vector and using a vector accumulation method; and inputting the second embedding vector into a second multi-head self-attention module to obtain a second output feature.

[0008] In some embodiments, the first enhanced feature and the second enhanced feature are feature aggregated based on the vectorized attention weight to obtain an aggregated feature, including: obtaining a weight vector of the attention weight; determining a first weight of the first enhanced feature and a second weight of the second enhanced feature according to the weight vector; and performing feature aggregation according to the first weight, the first enhanced feature, the second weight, and the second enhanced feature by feature accumulation to obtain an aggregated feature.

[0009] In some embodiments, the weight vector is determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

[0010] In some embodiments, voice activity detection is performed based on aggregated features, including: performing fully connected layer mapping and normalization processing on the aggregated features to obtain the probability of each audio frame; and determining the result of voice activity detection based on the probability of each audio frame.

[0011] In some embodiments, a training method for a voice activity detection model includes: acquiring voice data and vehicle noise data; synthesizing noisy audio data with different signal-to-noise ratios based on the voice data and vehicle noise data; optimizing parameters of the voice activity detection model using the voice data and noisy audio data and minimizing a cross-entropy loss function; and training the voice activity detection model using the voice data and noisy audio data and an adaptive torque estimation optimization algorithm.

[0012] A voice activity detection device comprises: an acquisition module for acquiring acoustic features of each audio frame of audio sampling data; a first input module for inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network and a second intermediate feature tensor output by a gated recurrent unit in the voice activity detection model; a second input module for inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weighting the first output feature to obtain a first enhanced feature; a third input module for inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weighting the second output feature to obtain a second enhanced feature; an aggregation module for performing feature aggregation on the first enhanced feature and the second enhanced feature based on a vectorized attention weight to obtain an aggregated feature; and a detection module for performing voice activity detection according to the aggregated feature.

[0013] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned voice activity detection methods when executing the computer program.

[0014] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any of the above-mentioned voice activity detection methods when executed by a processor.

[0015] The above-mentioned voice activity detection method, device, computer equipment and storage medium obtain the acoustic features of each audio frame of audio sampling data; input the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network in the voice activity detection model and a second intermediate feature tensor output by a gated recurrent unit; input the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weight the first output feature, and obtain a first enhanced feature; input the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weight the second output feature, and obtain a second enhanced feature; perform feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; and perform voice activity detection according to the aggregated feature.

[0016] Therefore, we can make full use of the advantages of the multi-head self-attention mechanism in emphasizing the dependency of different parts of the input, establish contextual dependencies, weight the importance of different parts, and model the information of different subspaces. Finally, we can aggregate the features from different network structures based on the vectorized attention weights. This method can improve the effect of voice activity detection, is more practical, and can be applied to various practical applications related to voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a voice activity detection method in some embodiments;

[0018] Figure 2 A schematic diagram of a process of performing voice activity detection using a convolutional recurrent neural network model based on a multi-head self-attention mechanism in some embodiments;

[0019] Figure 3 is a schematic diagram of a workflow of voice activity detection in some embodiments;

[0020] Figure 4 is a structural block diagram of a voice activity detection device in some embodiments;

[0021] Figure 5 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] The voice activity detection method provided in the present application can be used in a computer device. It should be noted that its execution subject can be a configuration device for virtual network card resources, which can be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware. The computer device can be a terminal, a client, or a server. The server can be a single server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a vehicle-mounted terminal, a smart phone, a smart speaker, a personal computer, a tablet computer, a wearable device, an intelligent robot, and other intelligent hardware devices. In the following method embodiments, the execution subject is a computer device as an example for explanation.

[0024] In some embodiments, Figure 1 As shown, a voice activity detection method is provided, which is described by taking the method applied to a vehicle-mounted terminal as an example, and includes the following steps:

[0025] S102: Acquire acoustic features of each audio frame of audio sampling data.

[0026] In this embodiment, the vehicle terminal obtains audio sampling data through audio sampling. The audio sampling data includes one or more audio frames. Generally, the audio sampling data includes multiple audio frames. Acoustic features are extracted from each audio frame to obtain acoustic features of each audio frame.

[0027] The acoustic features are used to represent the acoustic characteristics of the audio. For example, the acoustic features may include one or more of timbre energy, audio frequency, audio intensity, and audio power. The acoustic features of different audio frames can identify whether the corresponding audio frame is a speech frame.

[0028] In some embodiments, the above-mentioned obtaining of the acoustic features of each audio frame of the audio sampling data includes: performing frame processing on the input sampled audio to obtain a plurality of computable audio frames, and extracting acoustic features from each audio frame to obtain the acoustic features of each audio frame.

[0029] S104, inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network in the voice activity detection model and a second intermediate feature tensor output by a gated recurrent unit.

[0030] In this embodiment, the voice activity detection model can be a convolutional recurrent neural network model based on a multi-head self-attention mechanism. Figure 2 As shown in the figure. The model consists of two convolutional layers with convolution kernels of 5×5 and 3×3. Each convolutional layer is followed by a batchnorm operation, a RELU (rectified linear unit) activation function, and a 2×2 maximum pooling layer. A single-layer LSTM (Long Short Term Memory) structure is attached to the output of the last convolutional layer, and a single-layer GRU (Gate Recurrent Unit) structure is attached to the output of the last convolutional layer to enhance the temporal dependency of the model. Then, a multi-head self-attention module is inserted after the single-layer LSTM, and a multi-head self-attention module is inserted after the single-layer GRU. Next, a feature aggregation module based on vectorized attention weights is used, followed by a full connection operation and a tanh activation function. Finally, a full connection operation is performed and a softmax (normalized exponential function) output is obtained.

[0031] Among them, the first intermediate feature tensor is as follows Figure 2The vector output by the long short-term memory network (LSTM) shown in the figure, the second intermediate feature tensor is as follows Figure 2 The vector output by the gated recurrent unit (GRU) shown in FIG. Specifically, the acoustic feature is input into the convolutional recurrent neural network model to obtain an intermediate feature expression from LSTM and an intermediate feature expression from GRU.

[0032] S106, input the first intermediate feature tensor into the first multi-head self-attention module to obtain a first output feature, weight the first output feature, and obtain a first enhanced feature.

[0033] S108, input the second intermediate feature tensor into the second multi-head self-attention module to obtain a second output feature, weight the second output feature, and obtain a second enhanced feature.

[0034] In this embodiment, Figure 2 As shown, the multi-head self-attention mechanism is constructed by using the relationship between different parts in the convolutional recurrent neural network model, and the two intermediate features generated by the convolutional recurrent neural network model are weighted, and finally the two feature expressions after the enhancement of different parts are obtained by weighting. Specifically, the first intermediate feature tensor is input into the first multi-head self-attention module, and the second intermediate feature tensor is input into the second multi-head self-attention module, so that the self-attention importance of the two intermediate features is calculated, and the tensor expression after the importance of different input parts is weighted is obtained, that is, the first enhanced feature and the second enhanced feature are obtained.

[0035] Among them, Figure 2 As shown in the figure, in the multi-head self-attention module of the convolutional recurrent neural network model based on the multi-head self-attention mechanism, a mechanism is introduced to consider the correlation between different parts of the input, and the model's ability to model contextual dependencies is enhanced. The multi-head self-attention module can filter and emphasize important information in the global information, which is also a kind of attention mechanism. In addition, the multi-head mechanism of the multi-head self-attention module can also model information in different subspaces.

[0036] In some embodiments, the first intermediate feature tensor is input into a first multi-head self-attention module to obtain a first output feature, including: obtaining a first position embedding vector; converting the first intermediate feature tensor into a first embedding vector according to the first position embedding vector and using vector accumulation; and inputting the first embedding vector into the first multi-head self-attention module to obtain a first output feature.

[0037] In some embodiments, the second intermediate feature tensor is input into a second multi-head self-attention module to obtain a second output feature, including: obtaining a second position embedding vector; converting the second intermediate feature tensor into a second embedding vector according to the second position embedding vector and using a vector accumulation method; and inputting the second embedding vector into a second multi-head self-attention module to obtain a second output feature.

[0038] Specifically, assume that the intermediate feature tensor output by a single-layer LSTM is v t , that is, the first intermediate feature tensor is v t , the first position embedding vector is p t , the first embedding vector obtained by conversion is e t , which is expressed as follows:

[0039] e t =v t +p t ;

[0040] Among them, t is the position index of the input sequence, and the first position embedding vector p t It can be expressed as:

[0041]

[0042] here k takes values ​​of 0, 1, ... (d / 2-1), d is the first position embedding vector p t The dimension of . Among them, d = 160.

[0043] The first embedding vector is e t The sequence E = [e1, e2, ..., e t ,…,e L ], L is the length of the input sequence. The output of a single-layer multi-head attention mechanism can be expressed as:

[0044] MultiHead=Concat(head0,...,head H-1 )W O ;

[0045] in, in are learnable parameters in the multi-head attention mechanism.

[0046] The final output of the entire first multi-head attention module after the LSTM structure can be expressed as:

[0047]

[0048] That is, the first output feature is f lstm .

[0049] Referring to the processing method of the above LSTM structure, the final output of the entire second multi-head attention module after the GRU structure can be expressed as:

[0050]

[0051] That is, the second output feature is f gru .

[0052] S110, performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature.

[0053] In this embodiment, features from different network structures are aggregated based on vectorized attention weights. Specifically, the first enhanced features and the second enhanced features are aggregated into a set of intermediate feature vectors, namely, aggregated features, based on the vectorized attention weights.

[0054] In some embodiments, the first enhanced feature and the second enhanced feature are feature aggregated based on the vectorized attention weight to obtain an aggregated feature, including: obtaining a weight vector of the attention weight; determining a first weight of the first enhanced feature and a second weight of the second enhanced feature according to the weight vector; and performing feature aggregation according to the first weight, the first enhanced feature, the second weight, and the second enhanced feature by feature accumulation to obtain an aggregated feature.

[0055] In which, the weight vector may be determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

[0056] Specifically, the input vector of the feature aggregation process based on vectorized attention weights is defined as:

[0057]

[0058] Among them, the first output feature is f lstm , the second output feature is f gru .

[0059] The weight vector a of the attention weights for each element of the input vector i =[ω i1 ,…,ω ij ,…,ω im ] can be expressed as:

[0060] a i =τ(W2F(W1h i +b1)+b2) T ;

[0061] Among them, the shape of W1 is [d, 2], b1 is a d-dimensional vector, the shape of W2 is [2, d], b2 is a two-dimensional vector. F is the RELU activation function. τ is the softmax function, and the expression is:

[0062]

[0063] The aggregation result of the first output feature and the second output feature can be expressed as f = [F1, F2, ..., F i , …, f n ],in

[0064] S112: Perform voice activity detection based on the aggregated features.

[0065] In this embodiment, the probability value of each audio frame being a speech frame can be determined based on the aggregated features, and the speech activity of the audio sample data can be determined based on the probability value of each audio frame to achieve speech activity detection on the audio sample data.

[0066] In some embodiments, voice activity detection is performed based on aggregated features, including: performing fully connected layer mapping and normalization processing on the aggregated features to obtain the probability of each audio frame; and determining the result of voice activity detection based on the probability of each audio frame.

[0067] Specifically, Figure 2 As shown in the figure, after feature aggregation based on vectorized attention weights, a full connection operation and tanh activation function are performed; finally, a full connection operation is performed and the softmax processing output is obtained. Softmax processing is normalization processing. In this way, the probability of judging a speech frame or a non-speech frame is obtained. Finally, the probability of a speech frame is compared with the threshold to obtain the result of voice activity detection.

[0068] The above-mentioned voice activity detection method obtains the acoustic features of each audio frame of the audio sampling data; inputs the acoustic features of each audio frame into the pre-trained voice activity detection model to obtain the first intermediate feature tensor output by the long short-term memory network in the voice activity detection model and the second intermediate feature tensor output by the gated recurrent unit; inputs the first intermediate feature tensor into the first multi-head self-attention module to obtain the first output feature, weights the first output feature, and obtains the first enhanced feature; inputs the second intermediate feature tensor into the second multi-head self-attention module to obtain the second output feature, weights the second output feature, and obtains the second enhanced feature; based on the vectorized attention weight, the first enhanced feature and the second enhanced feature are feature aggregated to obtain the aggregated feature; and voice activity detection is performed based on the aggregated feature. Therefore, the advantage of the multi-head self-attention mechanism emphasizing the dependency of different input parts can be fully utilized, the context dependency relationship can be established, the importance of different parts can be weighted, and the information of different subspaces can be modeled, and finally the features from different network structures can be aggregated together based on the vectorized attention weight. This method can improve the effect of voice activity detection, is more practical, and can be applied to various practical applications related to voice interaction.

[0069] In one embodiment, the training method of the above-mentioned voice activity detection model includes the steps of: acquiring voice data and vehicle noise data; synthesizing noisy audio data with different signal-to-noise ratios based on the voice data and the vehicle noise data; optimizing the parameters of the voice activity detection model using the voice data and the noisy audio data and minimizing the cross-entropy loss function; and training the voice activity detection model using the voice data and the noisy audio data and an adaptive torque estimation optimization algorithm.

[0070] Specifically, the speech data is a speech signal, and the noise data is a noise signal. A pure speech signal and five types of vehicle noise signals are obtained, and noisy audio signals are synthesized with different signal-to-noise ratios. The pure speech signal and the synthesized noisy audio signal are paired as training data for a speech activity detection model based on a multi-head self-attention mechanism. For example, the five types of vehicle noise signals include: the noise of open windows on the road, the noise of closed windows on the road 1, the noise of closed windows on the road 2, the noise of open windows in the parking lot, and the noise of closed windows in the parking lot. The noise of closed windows on the road 1 and the noise of closed windows on the road 2 are different noises, mainly referring to the different recorded sections.

[0071] The voice activity detection model is a convolutional recurrent neural network model based on a multi-head self-attention mechanism and feature aggregation. In the training process of the voice activity detection model, the model parameters are optimized according to the minimization of the cross entropy loss function, and the model is trained by the adam (adaptive moment estimation) optimization algorithm, and the trained model is used for voice activity detection.

[0072] The above-mentioned voice activity detection method mainly integrates the feature expression of convolutional recurrent neural network on the basis of multi-head self-attention mechanism and feature aggregation, screens and emphasizes important information from the feature expressions of different parts, and also models the information of different subspaces. Finally, based on the vectorized attention weight method, the features from different network structures are aggregated together, thereby improving the accuracy of voice activity detection.

[0073] In a specific embodiment, the voice activity detection method adopts a voice activity detection model that adds a multi-head self-attention mechanism that considers the correlation of different parts on the basis of the convolutional recurrent neural network model, and screens and emphasizes important information from different part features. The voice activity detection model also models the information of different subspaces, and finally aggregates the features from different network structures based on the vectorized attention weights.

[0074] Specifically, Figure 3 As shown, the above-mentioned voice activity detection method includes: inputting audio sampling data, framing the input audio sampling data, extracting acoustic features of each audio frame, inputting the extracted acoustic features into a convolutional recurrent neural network module based on a multi-head self-attention mechanism and feature aggregation, and outputting the probability of the audio frame being a voice frame and a threshold comparison. Specifically, it includes the following steps:

[0075] Step 1: Obtain a clean speech signal and five types of vehicle noise signals, synthesize noisy audio signals with different signal-to-noise ratios, and pair the clean speech signal with the synthesized noisy speech signal as training data for a voice activity detection model based on a convolutional recurrent neural network with multi-head self-attention mechanism and feature aggregation.

[0076] Step 2: Build a voice activity detection model based on a convolutional recurrent neural network with multi-head self-attention mechanism and feature aggregation.

[0077] Step three: optimize the model parameters by minimizing the cross entropy loss function.

[0078] Step 4: Train the model using the Adam optimization algorithm and use the trained model for voice activity detection.

[0079] Step 5: perform frame processing on the input audio sample data to obtain computable audio frames, where the audio frames may be speech frames.

[0080] Step six, extracting acoustic features of the obtained speech frames.

[0081] Step 7: Input the acoustic features into the convolutional recurrent neural network to obtain an intermediate feature expression from LSTM and an intermediate feature expression from GRU.

[0082] Step eight, using the relationship between different parts in the voice activity detection model constructed above, construct a multi-head self-attention mechanism, weight the two intermediate feature expressions generated by the convolutional recurrent neural network model, and finally weight them to obtain two different feature expressions enhanced by different parts.

[0083] Step nine, the two feature expressions enhanced in different parts are subjected to feature aggregation based on vectorized attention weights, mapped through a fully connected layer, and processed through a softmax layer to obtain the probability value of the audio frame being a speech frame;

[0084] Step ten: compare the probability value of the audio frame being a speech frame with a threshold value to obtain a result of speech activity detection.

[0085] The above-mentioned voice activity detection method constructs a voice activity detection model of a convolutional recurrent neural network based on a multi-head self-attention mechanism and feature aggregation. On the basis of the convolutional recurrent neural network, a multi-head self-attention mechanism is introduced, which can model the context dependency and then weight the importance of different parts. At the same time, the multi-head mechanism in the voice activity detection model also models the information of different subspaces. Further, on the basis of the multi-head self-attention mechanism, the feature expressions of heterogeneous networks (LSTM and GRU) are integrated, and important information is screened and emphasized from different feature expressions based on the vectorized attention weights. Therefore, voice activity detection is realized based on the constructed voice activity detection model, not only by fusing the CNN and LSTM network structures to generate more meaningful feature expressions, but also by considering the correlation between different parts in the input, a new modeling with context dependency is generated by assigning different weights to different parts, and then the information of different subspaces is modeled. Finally, the feature expressions of the heterogeneous networks are aggregated, and the results of voice activity detection under different network structures are weighted, which effectively improves the performance of separating voice segments and non-voice segments in the signal.

[0086] It should be understood that although Figure 1 , Figure 2 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 2 and Figure 3At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0087] In some embodiments, Figure 4 As shown, a voice activity detection device is provided, including: an acquisition module 402, a first input module 404, a second input module 406, a third input module 408, an aggregation module 410 and a detection module 412, wherein: the acquisition module 402 is used to acquire the acoustic features of each audio frame of the audio sampling data; the first input module 404 is used to input the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network in the voice activity detection model and a second intermediate feature tensor output by a gated recurrent unit; the second input module 406 is used to input the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weight the first output feature, and obtain a first enhanced feature; the third input module 408 is used to input the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weight the second output feature, and obtain a second enhanced feature; the aggregation module 410 is used to perform feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; the detection module 412 is used to perform voice activity detection according to the aggregated feature.

[0088] In some embodiments, the first intermediate feature tensor is input into a first multi-head self-attention module to obtain a first output feature, including: obtaining a first position embedding vector; converting the first intermediate feature tensor into a first embedding vector according to the first position embedding vector and using vector accumulation; and inputting the first embedding vector into the first multi-head self-attention module to obtain a first output feature.

[0089] In some embodiments, the second intermediate feature tensor is input into a second multi-head self-attention module to obtain a second output feature, including: obtaining a second position embedding vector; converting the second intermediate feature tensor into a second embedding vector according to the second position embedding vector and using a vector accumulation method; and inputting the second embedding vector into a second multi-head self-attention module to obtain a second output feature.

[0090] In some embodiments, the first enhanced feature and the second enhanced feature are feature aggregated based on the vectorized attention weight to obtain an aggregated feature, including: obtaining a weight vector of the attention weight; determining a first weight of the first enhanced feature and a second weight of the second enhanced feature according to the weight vector; and performing feature aggregation according to the first weight, the first enhanced feature, the second weight, and the second enhanced feature by feature accumulation to obtain an aggregated feature.

[0091] In some embodiments, the weight vector is determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

[0092] In some embodiments, voice activity detection is performed based on aggregated features, including: performing fully connected layer mapping and normalization processing on the aggregated features to obtain the probability of each audio frame; and determining the result of voice activity detection based on the probability of each audio frame.

[0093] In some embodiments, a training method for a voice activity detection model includes: acquiring voice data and vehicle noise data; synthesizing noisy audio data with different signal-to-noise ratios based on the voice data and vehicle noise data; optimizing parameters of the voice activity detection model using the voice data and noisy audio data and minimizing a cross-entropy loss function; and training the voice activity detection model using the voice data and noisy audio data and an adaptive torque estimation optimization algorithm.

[0094] For the specific definition of the voice activity detection device, please refer to the definition of the voice activity detection method above, which will not be repeated here. Each module in the above-mentioned voice activity detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0095] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a voice activity detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0096] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0097] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining acoustic features of each audio frame of audio sampling data; inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network and a second intermediate feature tensor output by a gated recurrent unit in the voice activity detection model; inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weighting the first output feature, and obtaining a first enhanced feature; inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weighting the second output feature, and obtaining a second enhanced feature; performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; and performing voice activity detection according to the aggregated feature.

[0098] In some embodiments, when the processor executes a computer program to input the first intermediate feature tensor into the first multi-head self-attention module to obtain the first output feature, the following steps are specifically implemented: obtaining the first position embedding vector; converting the first intermediate feature tensor into the first embedding vector based on the first position embedding vector and using vector accumulation; inputting the first embedding vector into the first multi-head self-attention module to obtain the first output feature.

[0099] In some embodiments, when the processor executes a computer program to input the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, the following steps are specifically implemented: obtaining a second position embedding vector; converting the second intermediate feature tensor into a second embedding vector based on the second position embedding vector and using vector accumulation; and inputting the second embedding vector into the second multi-head self-attention module to obtain a second output feature.

[0100] In some embodiments, the processor executes a computer program to implement feature aggregation of the first enhancement feature and the second enhancement feature based on the vectorized attention weight. When obtaining the aggregated feature, the following steps are specifically implemented: obtaining a weight vector of the attention weight; determining the first weight of the first enhancement feature and the second weight of the second enhancement feature according to the weight vector; and performing feature aggregation according to the first weight, the first enhancement feature, the second weight, and the second enhancement feature by means of feature accumulation to obtain the aggregated feature.

[0101] In some embodiments, the weight vector is determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

[0102] In some embodiments, when the processor executes a computer program to implement voice activity detection based on aggregated features, the following steps are specifically implemented: the aggregated features are subjected to fully connected layer mapping and normalization processing to obtain the probability of each audio frame; and the result of the voice activity detection is determined based on the probability of each audio frame.

[0103] In some embodiments, when the processor executes a computer program to implement a training method for a voice activity detection model, the following steps are specifically implemented: acquiring voice data and vehicle noise data; synthesizing noisy audio data with different signal-to-noise ratios based on the voice data and vehicle noise data; optimizing parameters of the voice activity detection model using the voice data and noisy audio data and minimizing the cross-entropy loss function; and training the voice activity detection model using the voice data and noisy audio data and an adaptive torque estimation optimization algorithm.

[0104] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining acoustic features of each audio frame of audio sampling data; inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network in the voice activity detection model and a second intermediate feature tensor output by a gated recurrent unit; inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, weighting the first output feature, and obtaining a first enhanced feature; inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, weighting the second output feature, and obtaining a second enhanced feature; performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; and performing voice activity detection according to the aggregated feature.

[0105] In some embodiments, the computer program is executed by a processor to input the first intermediate feature tensor into the first multi-head self-attention module to obtain the first output feature, and the following steps are specifically implemented: obtaining the first position embedding vector; converting the first intermediate feature tensor into the first embedding vector based on the first position embedding vector and using vector accumulation; inputting the first embedding vector into the first multi-head self-attention module to obtain the first output feature.

[0106] In some embodiments, the computer program is executed by the processor to input the second intermediate feature tensor into the second multi-head self-attention module to obtain the second output feature, and the following steps are specifically implemented: obtaining the second position embedding vector; converting the second intermediate feature tensor into a second embedding vector based on the second position embedding vector and using vector accumulation; inputting the second embedding vector into the second multi-head self-attention module to obtain the second output feature.

[0107] In some embodiments, a computer program is executed by a processor to implement feature aggregation of the first enhanced feature and the second enhanced feature based on the vectorized attention weight. When obtaining the aggregated feature, the following steps are specifically implemented: obtaining a weight vector of the attention weight; determining the first weight of the first enhanced feature and the second weight of the second enhanced feature according to the weight vector; and performing feature aggregation according to the first weight, the first enhanced feature, the second weight, and the second enhanced feature by feature accumulation to obtain the aggregated feature.

[0108] In some embodiments, the weight vector is determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

[0109] In some embodiments, when a computer program is executed by a processor to implement voice activity detection based on aggregated features, the following steps are specifically implemented: the aggregated features are subjected to fully connected layer mapping and normalization processing to obtain the probability of each audio frame; and the result of voice activity detection is determined based on the probability of each audio frame.

[0110] In some embodiments, when the computer program is executed by a processor to implement a training method for a voice activity detection model, the following steps are specifically implemented: acquiring voice data and vehicle noise data; synthesizing noisy audio data with different signal-to-noise ratios based on the voice data and vehicle noise data; optimizing the parameters of the voice activity detection model using the voice data and noisy audio data and minimizing the cross-entropy loss function; and training the voice activity detection model using the voice data and noisy audio data and an adaptive torque estimation optimization algorithm.

[0111] Those skilled in the art can understand that all or part of the processes in the methods for implementing the above embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0112] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] In addition, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the characters in this article generally indicate that the associated objects before and after are in an "or" relationship.

[0114] The terms such as “first” and “second” in the present application are only used to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual contextual relationship or sequence between these entities or operations.

[0115] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A voice activity detection method, the method comprising: Acquiring acoustic features of each audio frame of the audio sampling data; Inputting the acoustic features of each audio frame into a pre-trained voice activity detection model to obtain a first intermediate feature tensor output by a long short-term memory network and a second intermediate feature tensor output by a gated recurrent unit in the voice activity detection model; Inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, and weighting the first output feature to obtain a first enhanced feature; Inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, and weighting the second output feature to obtain a second enhanced feature; Performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; Voice activity detection is performed according to the aggregated features.

2. The method according to claim 1, characterized in that The step of inputting the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature includes: Get the first position embedding vector; According to the first position embedding vector and by using a vector accumulation method, the first intermediate feature tensor is converted into a first embedding vector; The first embedding vector is input into the first multi-head self-attention module to obtain the first output feature.

3. The method according to claim 1, characterized in that The step of inputting the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature includes: Get the second position embedding vector; According to the second position embedding vector and by means of vector accumulation, the second intermediate feature tensor is converted into a second embedding vector; The second embedding vector is input into the second multi-head self-attention module to obtain the second output feature.

4. The method according to claim 1, characterized in that The step of performing feature aggregation on the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature includes: Get the weight vector of attention weights; Determine a first weight of the first enhancement feature and a second weight of the second enhancement feature according to the weight vector; According to the first weight, the first enhanced feature, the second weight and the second enhanced feature, feature aggregation is performed in a feature accumulation manner to obtain the aggregated feature.

5. The method according to claim 4, characterized in that The weight vector is determined based on a set dimension vector and a rectified linear unit activation function and a normalized exponential function.

6. The method according to claim 1, characterized in that The performing voice activity detection according to the aggregated features includes: Performing a fully connected layer mapping process and a normalization process on the aggregated features to obtain the probability of each audio frame; The result of voice activity detection is determined according to the probability of each audio frame.

7. The method according to claim 1, characterized in that The training method of the voice activity detection model includes: Acquire voice data and vehicle noise data; synthesizing noisy audio data according to the voice data and the vehicle noise data with different signal-to-noise ratios; Optimizing parameters of the voice activity detection model by using the voice data and the noisy audio data and minimizing a cross entropy loss function; The voice activity detection model is trained using the voice data, the noisy audio data and an adaptive moment estimation optimization algorithm.

8. A voice activity detection device, the device comprising: An acquisition module, used to acquire acoustic features of each audio frame of the audio sampling data; A first input module, configured to input the acoustic features of each audio frame into a pre-trained voice activity detection model, to obtain a first intermediate feature tensor output by a long short-term memory network in the voice activity detection model and a second intermediate feature tensor output by a gated recurrent unit; A second input module, configured to input the first intermediate feature tensor into a first multi-head self-attention module to obtain a first output feature, and weight the first output feature to obtain a first enhanced feature; A third input module is used to input the second intermediate feature tensor into a second multi-head self-attention module to obtain a second output feature, and weight the second output feature to obtain a second enhanced feature; an aggregation module, configured to aggregate the first enhanced feature and the second enhanced feature based on the vectorized attention weight to obtain an aggregated feature; A detection module is used to perform voice activity detection according to the aggregated features.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images