Voice activity detection method and apparatus, computer device, and storage medium

Abstract

The application relates to a voice activity detection method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring acoustic features of each audio frame of audio sampling data; obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network; obtaining a second tensor according to the acoustic features and a pre-trained gate recurrent neural network; fusing the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and detecting voice activity according to the fused tensor. The method can improve the accuracy of voice activity detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent speech, in particular to a voice activity detection method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of intelligent speech technology, more and more intelligent devices begin to use intelligent speech technology to complete voice interaction, such as smart cars, smart phones and smart speakers.

[0003] Voice activity detection technology is one of the key steps of intelligent speech technology, and its purpose is to determine the voice signal from an audio file containing voice. Voice activity detection not only involves digital signal processing, but also involves auditory perception characteristics and human voice features. At the same time, the diversity of noise also increases the difficulty of voice activity detection.

[0004] In the traditional technology, the traditional recurrent neural network structure is used to model the audio timing information, so as to perform voice activity detection. However, although the recurrent neural network based on deep learning achieves good detection effect on non-stationary noise signal, it has the problems of too single model and insufficient diversity of feature expression, so the detection of voice activity is still not accurate enough. SUMMARY

[0005] Therefore, it is necessary to provide a voice activity detection method, device, computer equipment and storage medium capable of improving the accuracy of voice activity detection in view of the above technical problems.

[0006] A voice activity detection method, the method comprising:

[0007] obtaining acoustic features of each audio frame of audio sampling data;

[0008] obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network;

[0009] obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network;

[0010] fusing the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and

[0011] detecting voice activity according to the fused tensor.

[0012] In some embodiments, the first tensor and the second tensor are fused based on the soft attention weight distribution mechanism to obtain a fused tensor after feature fusion, including: performing parity addition processing on the first tensor and the second tensor to obtain a third tensor; performing global pooling processing on the third tensor to obtain a pooling output; performing convolution change processing on the pooling output through two group convolution layers to obtain a convolution output; obtaining a fusion weight according to the convolution output and the soft attention weight distribution mechanism; and performing weighted summation processing on the first tensor, the second tensor and the fusion weight to obtain the fused tensor.

[0013] In some embodiments, the global pooling processing is performed on the third tensor to obtain the pooling output, including: obtaining a global feature map of the third tensor, determining the height and width of the global feature map; and performing global pooling processing on the height, the width and the third tensor to obtain the pooling output.

[0014] In some embodiments, the fusion weight is obtained according to the convolution output and the soft attention weight distribution mechanism, including: inputting the convolution output into a normalization activation function layer; performing normalization processing on the convolution output based on the normalization activation function layer, the soft attention mechanism and an exponential function; and obtaining the fusion weight.

[0015] In some embodiments, the first tensor is obtained according to the acoustic feature and a pre-trained long short-term memory network, including: inputting the acoustic feature into the long short-term memory network to obtain an output tensor output by the long short-term memory network; and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the first tensor.

[0016] In some embodiments, the second tensor is obtained according to the acoustic feature and a pre-trained gate recurrent neural network, including: inputting the acoustic feature into the gate recurrent neural network to obtain an output tensor output by the gate recurrent neural network; and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the second tensor.

[0017] In some embodiments, voice activity detection is performed according to the fused tensor, including: performing dimension compression processing on the fused tensor; performing full connection layer mapping and normalization processing on the fused tensor after the dimension compression processing to obtain a probability for judging whether each audio frame is a voice frame; and performing voice activity detection according to the probability.

[0018] In some embodiments, the method further comprises: acquiring in-vehicle noise, the in-vehicle noise comprising at least one of noise when the vehicle is driving on a highway with windows open, noise when the vehicle is driving on a highway with windows closed, noise when the vehicle is in a parking lot environment with windows open, and noise when the vehicle is in a parking lot environment with windows closed; generating an audio sample set according to the in-vehicle noise; and performing parameter optimization on the long short-term memory network and the gated recurrent neural network using the audio sample set and according to minimization of a cross-entropy loss function; and training the long short-term memory network and the gated recurrent neural network using the audio sample set and an adaptive moment estimation optimization algorithm.

[0019] A voice activity detection apparatus, the apparatus comprising:

[0020] An acoustic feature acquisition module configured to acquire acoustic features of each audio frame of an audio sample;

[0021] A first tensor acquisition module configured to obtain a first tensor according to the acoustic features and a pre-trained long short-term memory network;

[0022] A second tensor acquisition module configured to obtain a second tensor according to the acoustic features and a pre-trained gated recurrent neural network;

[0023] A tensor fusion module configured to fuse the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and

[0024] A voice activity detection module configured to detect voice activity according to the fused tensor.

[0025] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the voice activity detection methods described above when executing the computer program.

[0026] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the voice activity detection methods described above.

[0027] The voice activity detection method, apparatus, computer device, and storage medium described above process acoustic features of each audio frame of an audio sample data through a long short-term memory network and a gated recurrent neural network respectively, fuse the processing structures of the two different neural networks, and further combine a soft attention mechanism to reasonably distribute feature weights, thereby achieving feature expression diversification and weighting important features. Therefore, using the fused tensor for voice activity detection can achieve the purpose of improving voice activity detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1A flowchart of a voice activity detection method in some embodiments;

[0029] Figure 2 A flowchart of a step of fusing a first tensor and a second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion in some embodiments;

[0030] Figure 3 A model structure diagram of voice activity detection based on a hybrid neural network structure in some embodiments;

[0031] Figure 4 A block diagram of a voice activity detection device in some embodiments;

[0032] Figure 5 An internal structure diagram of a computer device in some embodiments. DETAILED DESCRIPTION

[0033] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed descriptions of the present application are given below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0034] The voice activity detection method provided by the present application can be used in a computer device. It should be noted that the execution subject can be a configuration device of a virtual network card resource. The device can be implemented by software, hardware or a combination of software and hardware to become part or all of the computer device. The computer device can be a terminal or a client or a server. The server can be a single server or a server cluster composed of multiple servers. The terminal in the embodiments of the present application can be a vehicle terminal, a smart phone, a smart speaker, a personal computer, a tablet computer, a wearable device, a smart robot and other smart hardware devices. In the following method embodiments, the execution subject is taken as an example of a computer device.

[0035] In some embodiments, as shown in Figure 1 A voice activity detection method is provided. The method is taken as an example of application in a vehicle terminal and includes the following steps:

[0036] Step S102: Acquire acoustic features of each audio frame of audio sampling data.

[0037] The audio sampling data refers to data sampled from the audio to be detected. The audio to be detected can be real-time collected or pre-recorded audio. The acoustic features refer to features containing acoustic information.

[0038] Specifically, the vehicle terminal can perform speech signal processing on the audio sample, for example, can perform speech framing and the like, thereby obtaining at least one audio frame, and then performs acoustic feature extraction on each audio frame to obtain acoustic features corresponding to each audio frame.

[0039] In some embodiments, the acoustic features of each audio frame of the audio sample data are obtained by: obtaining the audio sample data; performing framing on the audio sample data to obtain at least one audio frame; and performing feature extraction on each audio frame to obtain acoustic features corresponding to each audio frame.

[0040] Step S104: obtaining a first tensor according to the acoustic features and a pre-trained long short term memory network.

[0041] The long short term memory network (LSTM) is a kind of time recurrent neural network. The first tensor refers to an output tensor obtained by inputting the acoustic features into the pre-trained long short term memory network.

[0042] Specifically, a two-layer long short term memory network can be constructed and trained, and the acoustic features are converted into the first tensor by the trained long short term memory network. For training of the long short term memory network, for example, a neural network-based deep learning training can be performed based on an audio sample set generated based on vehicle noise.

[0043] In some embodiments, the first tensor is obtained according to the acoustic features and the long short term memory network, including: inputting the acoustic features into the long short term memory network to obtain an output tensor output by the long short term memory network, and performing expansion processing on the output tensor according to a preset batch size, sequence length and dimension of the hidden layer to obtain the first tensor.

[0044] For example, the acoustic features are input into the constructed and trained two-layer long short term memory network, and the dimension of the tensor output by the long short term memory network is expanded, and the long short term memory network is dimensionally expanded by a preset batch size (batch_size), sequence length (seq_len) and dimension of the hidden layer (hidden_dim) to match the dimension requirement of subsequent calculation. For example, a network with a two-layer LSTM structure is constructed, and the dimension of the tensor output by the LSTM is expanded to obtain X, which has a shape of [batch_size, 1, seq_len, hidden_dim], where batch_size is the batch size, seq_len is the sequence length, and hidden_dim is the dimension of the hidden layer.

[0045] Step S106: obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network.

[0046] The gated recurrent neural network (GRU) can better capture the dependence of time steps with large distance in the time sequence, and it controls the flow of information through the learnable gate. The second tensor refers to the output tensor obtained after inputting the acoustic feature into the pre-trained gated recurrent neural network.

[0047] Specifically, a two-layer GRU network can be constructed and trained, and the acoustic feature is converted into the second tensor through the trained GRU network. For the pre-training of the GRU network, for example, a neural network-based deep learning training can be performed based on an audio sample set generated based on the vehicle noise.

[0048] In some embodiments, the second tensor is obtained according to the acoustic feature and the pre-trained gated recurrent neural network, including: inputting the acoustic feature into the gated recurrent neural network to obtain an output tensor output by the gated recurrent neural network, and performing expansion processing on the output tensor according to a preset batch size, sequence length and dimension of the hidden layer to obtain the second tensor.

[0049] For example, the acoustic feature is input into the trained two-layer GRU network, and the dimension of the tensor output by the GRU network is expanded, and the dimension of the GRU network is expanded through the preset batch size (batch_size), sequence length (seq_len) and dimension of the hidden layer (hidden_dim), so as to match the dimension requirement of subsequent calculation. For example, a two-layer GRU network structure model is constructed, and the dimension of the tensor output by the GRU network is expanded to obtain Y, and the shape is [batch_size, 1, seq_len, hidden_dim], wherein batch_size is the batch size, seq_len is the sequence length, and hidden_dim is the dimension of the hidden layer.

[0050] Step S108: The first tensor and the second tensor are fused based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion.

[0051] In this step, the first tensor and the second tensor can be fused, and the fused tensor can be weighted based on the soft attention weight distribution mechanism, so as to improve the diversity and accuracy of feature expression.

[0052] Step S110: Detecting voice activity according to the fused tensor.

[0053] In this step, voice activity detection can be performed according to the fused tensor after feature fusion, that is, it can be judged according to the fused tensor whether the audio frame is a frame containing voice activity.

[0054] The voice activity detection method described above respectively processes the acoustic features of each audio frame of the audio sampling data through the long short-term memory network and the gated recurrent neural network, fuses the processing structures of the two different neural networks, and further combines a soft attention mechanism to realize reasonable allocation of feature weights, thereby realizing feature expression diversification and weighting important features. Therefore, using the fused tensor for voice activity detection can achieve the purpose of improving the accuracy of voice activity detection.

[0055] In some embodiments, reference can be made to Figure 2 , as shown, Figure 2 shows a flowchart of the step of fusing the first tensor and the second tensor based on the soft attention weight distribution mechanism to obtain the fused tensor after feature fusion in some embodiments. The step can specifically include:

[0056] Step S202: Perform same-position addition processing on the first tensor and the second tensor to obtain a third tensor.

[0057] In this step, the tensors output and expanded by the long short-term memory network and the gated recurrent neural network can be taken as inputs for same-position addition operation. For example, the formula can be as follows:

[0058] Z = X + Y

[0059] Wherein, X is the first tensor, Y is the second tensor, and Z is the third tensor.

[0060] Step S204: Perform global pooling processing on the third tensor to obtain a pooling output.

[0061] In this step, the global feature information of the third tensor can be compressed by global pooling processing based on the global feature map, thereby emphasizing important information.

[0062] In some embodiments, the global pooling processing on the third tensor to obtain the pooling output includes: obtaining a global feature map of the third tensor, determining the height and width of the global feature map; and performing global pooling processing on the height, width, and third tensor to obtain the pooling output.

[0063] For example, the output of the previous step is taken as input for global pooling processing, and the formula is as follows:

[0064]

[0065] Wherein, H and W represent the height and width of the global feature map, and are used to represent the size of the global feature map; and s represents the pooling output.

[0066] Step S206: Perform convolutional transformation processing on the pooling output through two group convolution layers to obtain a convolution output.

[0067] In this step, the result of the global pooling processing (pooling output) can be taken as input, and then two group convolution layers are used for convolution transformation to obtain the convolution output g k (s), where k is the index of the algorithm.

[0068] Step S208: obtaining the fusion weight according to the convolution output and the soft attention weight distribution mechanism.

[0069] In this step, the convolution output g k (s) can be taken as the input of the normalization activation function layer through the soft attention weight distribution mechanism to obtain the weight distribution a of the soft attention mechanism, that is, the fusion weight.

[0070] In some embodiments, obtaining the fusion weight according to the convolution output and the soft attention weight distribution mechanism includes: inputting the convolution output into a normalization activation function layer; performing normalization processing on the convolution output based on the normalization activation function layer, the soft attention mechanism, and an exponential function; and obtaining the fusion weight.

[0071] For example, the r-Softmax layer can be used to take g k (s) as input, where r and k represent the same meaning, that is, the index of the algorithm, and a is the weight distribution of the soft attention mechanism, that is, the fusion weight. The formula for calculating a can be referred to as follows:

[0072]

[0073] Step S210: performing weighted summation processing on the first tensor, the second tensor, and the fusion weight to obtain a fusion tensor.

[0074] In this step, the first tensor X and the second tensor Y can be taken as input and weighted summation is performed on the fusion weight a to obtain a fusion tensor. The formula can be referred to as follows:

[0075] H = X·a1 + Y·a2

[0076] Through the above embodiments, the output tensors of different recurrent neural networks can be better fused. Different recurrent neural networks have different feature extraction angles, so the mixed neural network can improve the diversity of feature expression. The fusion weight is calculated based on the soft attention mechanism, and the fused tensor is further weighted to enrich the expression of the features and improve the accuracy of the feature expression.

[0077] In some embodiments, further based on the fusion tensor, the voice activity detection is performed, including: performing dimension compression processing on the fusion tensor; performing full connection layer mapping and normalization processing on the fusion tensor after dimension compression to obtain a probability for judging whether each audio frame is a voice frame; and performing voice activity detection based on the probability.

[0078] In the embodiment, the dimension compression processing can be performed on the fusion tensor. In order to maintain the consistency of the shape, the dimension compression is performed on the fusion tensor H generated by the soft attention mechanism weighting, and the shape of the fusion tensor H can be represented as [batch_size, seq_len, hidden_dim]. Then, a full connection layer and a Softmax layer are used to obtain a probability value representing that the audio frame is a frame containing voice activity. Further, the voice activity detection is performed by judging the size of the probability value and a preset threshold.

[0079] For example, if the probability value is greater than the preset threshold, it is determined that the audio frame is a frame containing voice activity; and if the probability value is less than the preset threshold, it is determined that the audio frame is a frame not containing voice activity.

[0080] More specifically, it can be referred to Figure 3 , as shown in Figure 3 , a model structure diagram of voice activity detection based on a hybrid neural network structure in some embodiments is shown. In the diagram, Figure 3 , the symbol ⊕ between tensors represents addition processing, , and the symbol represents multiplication operation between tensors.

[0081] In some embodiments, the method further includes: obtaining vehicle noise, the vehicle noise including at least one of noise when a vehicle is driving on a highway with a window open, noise when the vehicle is driving on the highway with the window closed, noise when the vehicle is in a parking lot environment with the window open, and noise when the vehicle is in the parking lot environment with the window closed; generating an audio sample set based on the vehicle noise; and performing parameter optimization on the long short-term memory network and the gated recurrent neural network using the audio sample set and according to cross-entropy loss function minimization; and training the long short-term memory network and the gated recurrent neural network using the audio sample set and an adaptive moment estimation optimization algorithm.

[0082] In the embodiment, the LSTM network and the GRU network can be trained in advance by using the training method, so as to improve the adaptability of the entire detection model to vehicle voice activity, and further improve the accuracy of voice activity detection.

[0083] It should be understood that, although Figure 1 and Figure 2The steps in the flowcharts are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order limitation for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1 and Figure 2 At least part of the steps in the flowcharts can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0084] In some embodiments, as shown in Figure 4 a voice activity detection apparatus is provided, comprising: an acoustic feature acquisition module 402, a first tensor acquisition module 404, a second tensor acquisition module 406, a tensor fusion module 408, and a voice activity detection module 410, wherein:

[0085] The acoustic feature acquisition module 402 is configured to acquire acoustic features of each audio frame of an audio sample;

[0086] The first tensor acquisition module 404 is configured to obtain a first tensor according to the acoustic features and a pre-trained long short-term memory network;

[0087] The second tensor acquisition module 406 is configured to obtain a second tensor according to the acoustic features and a pre-trained gated recurrent neural network;

[0088] The tensor fusion module 408 is configured to fuse the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fusion tensor after feature fusion; and

[0089] The voice activity detection module 410 is configured to detect voice activity according to the fusion tensor.

[0090] In some embodiments, the tensor fusion module 408 performs same-position addition processing on the first tensor and the second tensor to obtain a third tensor, performs global pooling processing on the third tensor to obtain a pooling output, performs convolutional transformation processing on the pooling output through two group convolution layers to obtain a convolution output, obtains a fusion weight according to the convolution output and the soft attention weight distribution mechanism, and performs weighted summation processing on the first tensor, the second tensor, and the fusion weight to obtain the fusion tensor.

[0091] In some embodiments, the tensor fusion module 408 acquires a global feature map of the third tensor, determines the height and width of the global feature map, and performs global pooling processing on the third tensor according to the height, the width, and the third tensor to obtain a pooling output.

[0092] In some embodiments, the tensor fusion module 408 inputs the convolution output into a normalization activation function layer; performs normalization processing on the convolution output based on the normalization activation function layer, a soft attention mechanism, and an exponential function; and obtains a fusion weight.

[0093] In some embodiments, the first tensor obtaining module 404 inputs the acoustic feature into a long short-term memory network to obtain an output tensor of a long short-term memory network output, performs expansion processing on the output tensor according to a preset batch size, a sequence length, and a dimension of a hidden layer, and obtains a first tensor.

[0094] In some embodiments, the second tensor obtaining module 406 inputs the acoustic feature into a gated recurrent neural network to obtain an output tensor of a gated recurrent neural network output, performs expansion processing on the output tensor according to a preset batch size, a sequence length, and a dimension of a hidden layer, and obtains a second tensor.

[0095] In some embodiments, the speech activity detection module 410 performs dimension compression processing on the fusion tensor; performs full connection layer mapping and normalization processing on the fusion tensor after the dimension compression processing, to obtain a probability for judging whether each audio frame is a speech frame; and performs speech activity detection according to the probability.

[0096] In some embodiments, the speech activity detection module 410 is further configured to obtain vehicle noise, the vehicle noise including at least one of noise when a vehicle is driving on a highway with a window open, noise when the vehicle is driving on the highway with the window closed, noise when the vehicle is in a parking lot environment with the window open, and noise when the vehicle is in the parking lot environment with the window closed; generate an audio sample set according to the vehicle noise; perform parameter optimization on the long short-term memory network and the gated recurrent neural network by using the audio sample set and according to minimization of a cross-entropy loss function; and train the long short-term memory network and the gated recurrent neural network by using the audio sample set and an adaptive moment estimation optimization algorithm.

[0097] Specific limitations of the speech activity detection apparatus can be referred to the limitations of the speech activity detection method in the foregoing, which will not be repeated here. Each module in the speech activity detection apparatus described above can be realized by software, hardware, and a combination thereof in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0098] In some embodiments, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a voice activity detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0099] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0100] In some embodiments, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the following steps: obtaining acoustic features of each audio frame of audio sample data; obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network; obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network; fusing the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and detecting voice activity according to the fused tensor.

[0101] In some embodiments, when the processor executes the computer program to implement the fusion of the first tensor and the second tensor based on the soft attention weight distribution mechanism to obtain the fused tensor after feature fusion, the following steps are specifically implemented: performing same-position addition processing on the first tensor and the second tensor to obtain a third tensor; performing global pooling processing on the third tensor to obtain a pooling output; performing convolution transformation processing on the pooling output through two group convolution layers to obtain a convolution output; obtaining a fusion weight according to the convolution output and the soft attention weight distribution mechanism; and performing weighted summation processing on the first tensor, the second tensor and the fusion weight to obtain the fused tensor.

[0102] In some embodiments, when the processor executes the computer program to implement global pooling processing on the third tensor to obtain a pooling output, the following steps are implemented: obtaining a global feature map of the third tensor, determining a height and a width of the global feature map; and performing global pooling processing on the third tensor according to the height, the width and the third tensor to obtain the pooling output.

[0103] In some embodiments, when the processor executes the computer program to obtain the fusion weight according to the convolution output and the soft attention weight distribution mechanism, the following steps are implemented: inputting the convolution output into a normalization activation function layer; performing normalization processing on the convolution output based on the normalization activation function layer, the soft attention mechanism and an exponential function; and obtaining the fusion weight.

[0104] In some embodiments, when the processor executes the computer program to obtain the first tensor according to the acoustic feature and the pre-trained long short-term memory network, the following steps are implemented: inputting the acoustic feature into the long short-term memory network to obtain an output tensor output by the long short-term memory network, and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the first tensor.

[0105] In some embodiments, when the processor executes the computer program to obtain the second tensor according to the acoustic feature and the pre-trained gate recurrent neural network, the following steps are implemented: inputting the acoustic feature into the gate recurrent neural network to obtain an output tensor output by the gate recurrent neural network, and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the second tensor.

[0106] In some embodiments, when the processor executes the computer program to perform voice activity detection according to the fusion tensor, the following steps are implemented: performing dimension compression processing on the fusion tensor; performing full connection layer mapping and normalization processing on the fusion tensor after the dimension compression processing to obtain a probability for judging whether each audio frame is a voice frame; and performing voice activity detection according to the probability.

[0107] In some embodiments, when the processor executes the computer program, the following steps are further implemented: obtaining vehicle noise, the vehicle noise including at least one of noise when a vehicle is driving on a highway with a window open, noise when the vehicle is driving on the highway with the window closed, noise when the vehicle is in a parking lot environment with the window open, and noise when the vehicle is in the parking lot environment with the window closed; generating an audio sample set according to the vehicle noise; performing parameter optimization on the long short-term memory network and the gate recurrent neural network by using the audio sample set and according to minimization of a cross-entropy loss function; and training the long short-term memory network and the gate recurrent neural network by using the audio sample set and a self-adaptive moment estimation optimization algorithm.

[0108] In some embodiments, a computer readable storage medium is provided, having stored thereon a computer program, the computer program being executed by a processor to implement the following steps: obtaining acoustic features of each audio frame of audio sample data; obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network; obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network; fusing the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and performing voice activity detection according to the fused tensor.

[0109] In some embodiments, when the computer program is executed by the processor to implement the fusing of the first tensor and the second tensor based on the soft attention weight distribution mechanism to obtain the fused tensor after feature fusion, the following steps are specifically implemented: performing same-position addition processing on the first tensor and the second tensor to obtain a third tensor; performing global pooling processing on the third tensor to obtain a pooling output; performing convolution transformation processing on the pooling output through two group convolution layers to obtain a convolution output; obtaining a fusion weight according to the convolution output and the soft attention weight distribution mechanism; and performing weighted summation processing on the first tensor, the second tensor and the fusion weight to obtain the fused tensor.

[0110] In some embodiments, when the computer program is executed by the processor to implement the global pooling processing on the third tensor to obtain the pooling output, the following steps are specifically implemented: obtaining a global feature map of the third tensor, and determining a height and a width of the global feature map; performing global pooling processing on the third tensor according to the height, the width and the global feature map to obtain the pooling output.

[0111] In some embodiments, when the computer program is executed by the processor to implement the obtaining of the fusion weight according to the convolution output and the soft attention weight distribution mechanism, the following steps are specifically implemented: inputting the convolution output into a normalization activation function layer as an input; performing normalization processing on the convolution output based on the normalization activation function layer, a soft attention mechanism and an exponential function; and obtaining the fusion weight.

[0112] In some embodiments, when the computer program is executed by the processor to implement the obtaining of the first tensor according to the acoustic features and the pre-trained long short-term memory network, the following steps are specifically implemented: inputting the acoustic features into the long short-term memory network to obtain an output tensor output by the long short-term memory network, and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the first tensor.

[0113] In some embodiments, when the computer program is executed by the processor to implement the obtaining of the second tensor according to the acoustic features and the pre-trained gated recurrent neural network, the following steps are specifically implemented: inputting the acoustic features into the gated recurrent neural network to obtain an output tensor output by the gated recurrent neural network, and performing expansion processing on the output tensor according to a preset batch size, a sequence length and a dimension of a hidden layer to obtain the second tensor.

[0114] In some embodiments, when the computer program is executed by the processor to implement the detection of voice activity according to the fusion tensor, the following steps are specifically implemented: performing dimension compression processing on the fusion tensor; performing full connection layer mapping and normalization processing on the fusion tensor after the dimension compression processing, to obtain a probability for judging whether each audio frame is a voice frame; and performing voice activity detection according to the probability.

[0115] In some embodiments, when the computer program is executed by the processor, the following steps are further implemented: obtaining vehicle noise, the vehicle noise including at least one of noise when a vehicle is driving on a highway with a window open, noise when the vehicle is driving on the highway with the window closed, noise when the vehicle is in a parking lot environment with the window open, and noise when the vehicle is in the parking lot environment with the window closed; generating an audio sample set according to the vehicle noise; performing parameter optimization on the long short-term memory network and the gated recurrent neural network by using the audio sample set and according to minimization of a cross-entropy loss function; and training the long short-term memory network and the gated recurrent neural network by using the audio sample set and a self-adaptive moment estimation optimization algorithm.

[0116] A person of ordinary skill in the art can understand that all or part of the processes in the methods for implementing the above embodiments can be completed by using a computer program to instruct related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the computer program can include the processes of the above embodiments. Any reference to a memory, storage, database, or other medium in each embodiment provided in the present application can include a non-volatile and / or volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory can include a random access memory (RAM) or an external cache memory. As an illustration but not limitation, the RAM is available in various forms, such as a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchronous link (Synchlink) DRAM (SLDRAM), a memory bus (Rambus) direct RAM (RDRAM), a direct memory bus dynamic RAM (DRDRAM), and a memory bus dynamic RAM (RDRAM), and the like.

[0117] Each technical feature of the above embodiments can be combined arbitrarily, and to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, it should be considered that they are within the scope of the present disclosure.

[0118] In addition, the term "and / or" in this document merely describes an associated relationship with associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, the characters generally represent the relationship between the associated objects before and after.

[0119] The terms such as "first", "second" in this application are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual front-back relationship or sequence between the entities or operations.

[0120] The above-described embodiments only express several embodiments of the present application, which are described in detail and specifically, but should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A voice activity detection method, the method comprising: obtaining acoustic features of each audio frame of audio sample data; obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network; obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network; fusing the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and detecting voice activity according to the fused tensor. The fusing the first tensor and the second tensor based on the soft attention weight distribution mechanism to obtain the fused tensor after feature fusion comprises:

2. The method of claim 1, wherein, performing a same-position addition process on the first tensor and the second tensor to obtain a third tensor; performing global pooling processing on the third tensor to obtain a pooling output; performing convolutional transformation processing on the pooling output through two grouped convolutional layers to obtain a convolutional output; obtaining a fusion weight according to the convolutional output and the soft attention weight distribution mechanism; performing weighted summation processing on the first tensor, the second tensor and the fusion weight to obtain the fused tensor. The performing global pooling processing on the third tensor to obtain the pooling output comprises:

3. The method of claim 2, wherein, obtaining a global feature map of the third tensor, determining a height and a width of the global feature map; performing global pooling processing on the third tensor according to the height, the width and the global feature map to obtain the pooling output. The obtaining a fusion weight according to the convolutional output and the soft attention weight distribution mechanism comprises:

4. The method of claim 2, wherein, inputting the convolutional output into a normalization activation function layer; performing normalization processing on the convolutional output based on the normalization activation function layer, the soft attention weight distribution mechanism and an exponential function to obtain the fusion weight. The obtaining a first tensor according to the acoustic features and a pre-trained long short-term memory network comprises:

5. The method of claim 1, wherein, inputting the acoustic features into the long short-term memory network to obtain an output tensor output by the long short-term memory network, performing expansion processing on the output tensor according to a preset batch size, sequence length and dimension of a hidden layer to obtain the first tensor; and / or The obtaining a second tensor according to the acoustic features and a pre-trained gated recurrent neural network comprises: inputting the acoustic features into the gated recurrent neural network to obtain an output tensor output by the gated recurrent neural network, performing expansion processing on the output tensor according to a preset batch size, sequence length and dimension of a hidden layer to obtain the second tensor. The detecting voice activity according to the fused tensor comprises:

6. The method according to any one of claims 1 to 5, characterized in that, performing dimension compression processing on the fused tensor; performing fully connected layer mapping and normalization processing on the fused tensor after the dimension compression processing to obtain a probability for judging whether each audio frame is a voice frame; detecting voice activity according to the probability. Further comprising:

7. The method according to any one of claims 1 to 5, characterized in that, obtaining vehicle noise, the vehicle noise comprising at least one of noise when a vehicle is driving on a highway with a window open, noise when the vehicle is driving on the highway with the window closed, noise when the vehicle is in a parking lot environment with the window open, and noise when the vehicle is in the parking lot environment with the window closed. ​ generate an audio sample set according to the in-vehicle noise; perform parameter optimization on the long short-term memory network and the gated recurrent neural network by using the audio sample set and according to cross-entropy loss function minimization; train the long short-term memory network and the gated recurrent neural network by using the audio sample set and an adaptive moment estimation optimization algorithm.

8. A voice activity detection apparatus characterized by comprising: The device comprises: an acoustic feature acquisition module configured to acquire acoustic features of each audio frame of an audio sample; a first tensor acquisition module configured to obtain a first tensor according to the acoustic features and a pre-trained long short-term memory network; a second tensor acquisition module configured to obtain a second tensor according to the acoustic features and a pre-trained gated recurrent neural network; a tensor fusion module configured to fuse the first tensor and the second tensor based on a soft attention weight distribution mechanism to obtain a fused tensor after feature fusion; and a voice activity detection module configured to perform voice activity detection according to the fused tensor.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images