Training and detection method, device, equipment and medium for voice activity detection model

Abstract

The present application relates to the field of audio and live broadcast technology, and provides a method, device, equipment and medium for training and detecting a voice activity detection model. The present application can improve the detection accuracy of a voice activity detection model. It includes: dividing a pure voice signal sample into frames to obtain a plurality of pure voice signal frame samples, obtaining the normalized value of the logarithmic energy corresponding to each of the plurality of signal frame samples, obtaining, for each signal frame sample, the proportion of the first pure voice signal frame sample in the first preset number of signal frame samples before and after the signal frame sample, the first pure voice signal frame sample being a signal frame sample whose normalized value of the logarithmic energy is greater than a threshold, determining the voice activity detection label corresponding to the signal frame sample according to the proportion, obtaining a signal frame sample carrying noise according to the signal frame sample and the preset noise, and using the signal frame sample carrying noise and the voice activity detection label to train the voice activity detection model to be trained.

Description

Technical Field

[0001] The present application relates to the field of audio and live broadcast technology, and in particular to a training method for a voice activity detection model, a voice activity detection method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Voice Activity Detection (VAD) plays a vital role in the field of real-time communication (RTC). It can provide a reference for the voice front-end processing algorithm and a basis for voice data stream transmission.

[0003] However, in the deep learning-based voice activity detection method provided by current technology, the detection accuracy of the deep learning model used is insufficient, the probability of speech missed detection is high, and it is difficult to meet the needs of real-time communication. Summary of the invention

[0004] Based on this, it is necessary to provide a training method for a voice activity detection model, a voice activity detection method, a device, an electronic device and a computer-readable storage medium to address the above technical problems.

[0005] In a first aspect, the present application provides a method for training a voice activity detection model. The method comprises:

[0006] Get a clean speech signal sample;

[0007] Dividing the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtaining normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples;

[0008] For each clean speech signal frame sample, according to the normalized values ​​of the logarithmic energies corresponding to the multiple clean speech signal frame samples, obtain the proportion of the first clean speech signal frame sample in the first preset number of clean speech signal frame samples before and after the clean speech signal frame sample; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of the logarithmic energy is greater than the normalized value threshold;

[0009] Determining a voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample;

[0010] Obtaining a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal;

[0011] The speech activity detection model to be trained is trained using the speech signal frame samples carrying noise and the speech activity detection labels.

[0012] In a second aspect, the present application provides a method for detecting voice activity. The method comprises:

[0013] Obtaining a voice signal frame to be detected;

[0014] Detecting the speech signal frame to be detected using a trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained according to the training method of the speech activity detection model as described above;

[0015] A voice activity detection result of the voice signal frame to be detected is obtained according to the voice activity detection prediction information.

[0016] In a third aspect, the present application also provides a training device for a voice activity detection model. The device comprises:

[0017] A speech sample acquisition module is used to obtain pure speech signal samples;

[0018] A normalization processing module, used for dividing the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtaining a normalized value of the logarithmic energy corresponding to each of the plurality of clean speech signal frame samples;

[0019] a proportion acquisition module, configured to acquire, for each clean speech signal frame sample, a proportion of a first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample according to normalized values ​​of logarithmic energies corresponding to each of the plurality of clean speech signal frame samples; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of logarithmic energy is greater than a normalized value threshold;

[0020] a label determination module, configured to determine a voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample;

[0021] A training sample acquisition module, used to obtain a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal;

[0022] The model training module is used to train the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels.

[0023] In a fourth aspect, the present application further provides a voice activity detection device. The device comprises:

[0024] A frame acquisition module to be detected, used to acquire a voice signal frame to be detected;

[0025] A model detection module, configured to detect the speech signal frame to be detected using a trained speech activity detection model, and obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained using the training device for the speech activity detection model as described above;

[0026] The result obtaining module is used to obtain the voice activity detection result of the voice signal frame to be detected according to the voice activity detection prediction information.

[0027] In a fifth aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0028] Acquire a clean speech signal sample; divide the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtain normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; for each clean speech signal frame sample, obtain the proportion of the first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample according to the normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of logarithmic energy is greater than a normalized value threshold; determine a voice activity detection label corresponding to the clean speech signal frame sample according to the proportion of the first clean speech signal frame sample; obtain a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal; and train a speech activity detection model to be trained using the speech signal frame sample carrying noise and the voice activity detection label.

[0029] In a sixth aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0030] Acquire a speech signal frame to be detected; detect the speech signal frame to be detected using a trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained according to the training method of the speech activity detection model as described above; and obtain a speech activity detection result of the speech signal frame to be detected according to the speech activity detection prediction information.

[0031] In a seventh aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0032] Acquire a clean speech signal sample; divide the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtain normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; for each clean speech signal frame sample, obtain the proportion of the first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample according to the normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of logarithmic energy is greater than a normalized value threshold; determine a voice activity detection label corresponding to the clean speech signal frame sample according to the proportion of the first clean speech signal frame sample; obtain a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal; and train a speech activity detection model to be trained using the speech signal frame sample carrying noise and the voice activity detection label.

[0033] In an eighth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0034] Acquire a speech signal frame to be detected; detect the speech signal frame to be detected using a trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained according to the training method of the speech activity detection model as described above; and obtain a speech activity detection result of the speech signal frame to be detected according to the speech activity detection prediction information.

[0035] The training method, voice activity detection method, device, electronic device and computer-readable storage medium of the voice activity detection model are as follows: a clean voice signal sample is framed to obtain a plurality of clean voice signal frame samples, and the normalized values ​​of the logarithmic energy corresponding to each of the plurality of clean voice signal frame samples are obtained; for each clean voice signal frame sample, the proportion of the first clean voice signal frame sample in a first preset number of clean voice signal frame samples before and after the clean voice signal frame sample is obtained according to the normalized values ​​of the logarithmic energy corresponding to each of the plurality of clean voice signal frame samples; the first clean voice signal frame sample is a clean voice signal frame sample whose normalized value of the logarithmic energy is greater than a normalized value threshold; a voice activity detection label corresponding to the clean voice signal frame sample is determined according to the proportion; a voice signal frame sample carrying noise is obtained according to the clean voice signal frame sample and a preset noise signal; and a voice activity detection model to be trained is trained using the voice signal frame sample carrying noise and the aforementioned voice activity detection label. The scheme makes labels based on pure speech signals, divides the pure speech signals into several speech signal frames by framing, and then accurately obtains the speech activity detection label of each speech signal frame based on the signal energy. The speech signal frames are mixed with noise in a certain proportion to obtain speech signal frames carrying noise to adapt to the actual noise environment. Finally, the speech signal frames carrying noise and the aforementioned speech activity detection labels are used to train the speech activity detection model to be trained, so as to obtain a speech activity detection model with high detection accuracy, improve the detection accuracy of the speech activity detection model, and reduce the probability of speech missed detection, so as to meet the needs of real-time communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an application environment diagram of the relevant method in the embodiments of the present application;

[0037] Figure 2 A flowchart of a method for training a voice activity detection model in an embodiment of the present application;

[0038] Figure 3 A flowchart of the steps of determining a voice activity detection tag in an embodiment of the present application;

[0039] Figure 4 A flowchart of the steps of training a voice activity detection model in an embodiment of the present application;

[0040] Figure 5 Schematic diagram of a voice activity detection model in an embodiment of the present application;

[0041] Figure 6 This is a schematic diagram of a gated two-dimensional convolutional layer in an embodiment of the present application;

[0042] Figure 7 Schematic diagram of the flow of the voice activity detection method in the embodiment of the present application;

[0043] Figure 8 It is a structural block diagram of a training device for a voice activity detection model in an embodiment of the present application;

[0044] Fig. 9 This is a structural block diagram of a voice activity detection device in an embodiment of the present application;

[0045] Fig.10 This is a diagram of the internal structure of an electronic device in an embodiment of the present application;

[0046] Fig.11 This is a diagram of the internal structure of an electronic device in another embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The training method of the voice activity detection model and the voice activity detection method provided in the embodiments of the present application can be applied to the following examples: Figure 1 In the application environment shown, the application environment may include a terminal and a server, wherein the terminal may be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers, and the server may be implemented with an independent server or a server cluster composed of multiple servers, and the terminal may communicate with the server through the Internet. Specifically, the training method of the voice activity detection model provided in the embodiment of the present application may be executed by a server, and the voice activity detection method provided in the embodiment of the present application may be applied to a terminal, wherein the server trains the voice activity detection model to be trained according to the training method of the voice activity detection model of the present application to obtain a trained voice activity detection model, and then the server may send the trained voice activity detection model to the terminal for storage, and the terminal may apply the trained voice activity detection model in audio transmission to perform voice activity detection so as to accurately determine whether to transmit the corresponding voice signal frame, thereby accurately and effectively reducing the transmission bandwidth and reducing the service traffic.

[0049] The following describes the training method of the voice activity detection model and the voice activity detection method provided by the present application in sequence in combination with various embodiments and corresponding drawings.

[0050] In one embodiment, Figure 2 As shown, a method for training a voice activity detection model is provided, and the method may include the following steps:

[0051] Step S201, obtaining a clean speech signal sample.

[0052] Step S202: divide the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtain normalized values ​​of logarithmic energies corresponding to the plurality of clean speech signal frame samples.

[0053] Step S201 and step S202 are related steps for processing the clean speech signal sample. In step S201, a clean speech signal sample is first obtained, and a corresponding speech activity detection label is required when training the speech activity detection model to be trained. The speech activity detection label can be a binary value, that is, 0 or 1, wherein 0 can represent non-speech and 1 can represent speech. The clean speech signal sample is a speech time domain signal without noise. In subsequent steps, a speech activity detection label will be made according to the clean speech signal, that is, the clean speech signal sample. Specifically, in step S202, the clean speech signal sample is first divided into multiple frames by framing, and it is recorded as obtaining multiple clean speech signal frame samples, that is, the multiple clean speech signal frame samples are the results obtained by framing the clean speech signal sample, and then the normalized values ​​of the logarithmic energy corresponding to each of the multiple clean speech signal frame samples are obtained. Specifically, the logarithmic energy corresponding to each of the multiple clean speech signal frame samples can be calculated first and then the normalized value thereof can be obtained.

[0054] In one embodiment, obtaining the normalized values ​​of the logarithmic energies corresponding to the plurality of clean speech signal frame samples in step S202 may include:

[0055] For each clean speech signal frame sample, the clean speech signal frame sample is de-averaged, the corresponding logarithmic energy is calculated according to the clean speech signal frame sample after the de-averaging, and the logarithmic energy is normalized to obtain the normalized value of the logarithmic energy corresponding to each of the multiple clean speech signal frame samples.

[0056] Specifically, after the clean speech signal sample is divided into frames to obtain multiple clean speech signal frame samples, the mean of the clean speech signal frame sample is removed for each clean speech signal frame sample to eliminate the influence of the DC component, and the clean speech signal frame sample after the mean removal is obtained. Then, according to the clean speech signal frame sample after the mean removal, the mean of the signal energy of the clean speech signal frame sample is calculated and then the logarithm is taken to obtain the corresponding logarithmic energy. Then, the logarithmic energy can be normalized by subtracting the mean and dividing by the standard deviation, thereby obtaining the normalized value y of the logarithmic energy corresponding to each clean speech signal frame sample.

[0057] Step S203, for each clean speech signal frame sample, according to the normalized values ​​of the logarithmic energies corresponding to the plurality of clean speech signal frame samples, obtain the proportion of the first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample.

[0058] Step S204: determining a voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample.

[0059] Step S203 and step S204 are related steps for determining the corresponding voice activity detection label based on the normalized value of the logarithmic energy of each clean voice signal frame sample. Specifically, in step S203, the normalized value y of the logarithmic energy of each clean voice signal frame sample has been obtained, and then for each clean voice signal frame sample, a previous and next frame search is performed based on the normalized value y of the logarithmic energy. Specifically, for each clean voice signal frame sample, the proportion z of the first clean voice signal frame sample in the first preset number L1 (L1≥0) of clean voice signal frame samples before and after the clean voice signal frame sample is obtained, wherein the first clean voice signal frame sample refers to a clean voice signal frame sample whose normalized value y of the logarithmic energy is greater than the normalized value threshold t1 (0≤t1≤1). Exemplarily, assuming that L1 is 4, the number of clean voice signal frame samples whose normalized value y of the logarithmic energy is greater than the normalized value threshold t1 can be counted in the 4 clean voice signal frame samples before and after a clean voice signal frame sample and its proportion z is calculated. Then in step S204, for each clean speech signal frame sample, its corresponding voice activity detection label is determined according to the proportion of the first clean speech signal frame sample corresponding to it. For example, if the proportion z of the first clean speech signal frame sample corresponding to the clean speech signal frame sample is greater than the proportion threshold t2 (0≤t2≤1), the voice activity detection label corresponding to the clean speech signal frame sample is determined to be 1, otherwise the voice activity detection label corresponding to the clean speech signal frame sample is determined to be 0, thereby accurately forming the voice activity detection label corresponding to each clean speech signal frame sample.

[0060] Further, in one embodiment, if Figure 3 As shown, determining the voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample in step S204 may further include:

[0061] Step S301 : obtaining an initial voice activity detection label corresponding to the clean voice signal frame sample according to a comparison result of the proportion of the first clean voice signal frame sample and a proportion threshold.

[0062] In this step, for each clean speech signal frame sample, the voice activity detection label corresponding to the clean speech signal frame sample is obtained according to the comparison result of the proportion z of the corresponding first clean speech signal frame sample and the proportion threshold t2. The voice activity detection label is recorded as the initial voice activity detection label, thereby obtaining the initial voice activity detection label corresponding to each clean speech signal frame sample.

[0063] As an embodiment, step S301 may specifically include: if the proportion of the first clean speech signal frame sample is greater than the proportion threshold, determining that the initial voice activity detection label corresponding to the clean speech signal frame sample belongs to the first category of labels; if the proportion of the first clean speech signal frame sample is less than or equal to the proportion threshold, determining that the initial voice activity detection label corresponding to the clean speech signal frame sample belongs to the second category of labels.

[0064] Specifically, in this embodiment, the first type of label can be 1, indicating speech, and correspondingly, the second type of label can be 0, indicating non-speech. That is, for each clean speech signal frame sample, if the proportion z of the corresponding first clean speech signal frame sample is greater than the proportion threshold t2, then the initial speech activity detection label corresponding to the clean speech signal frame sample is first determined to be 1, and if the proportion z of the corresponding first clean speech signal frame sample is less than or equal to the proportion threshold t2, then the initial speech activity detection label corresponding to the clean speech signal frame sample is first determined to be 0, thereby obtaining the initial speech activity detection label corresponding to each clean speech signal frame sample.

[0065] Step S302 , for each clean speech signal frame sample, according to the initial voice activity detection labels corresponding to the plurality of clean speech signal frame samples, obtain the number of second clean speech signal frame samples in a second preset number of clean speech signal frame samples before and after the clean speech signal frame sample.

[0066] Specifically, in this step, the initial voice activity detection label of each clean voice signal frame sample has been obtained, and then for each clean voice signal frame sample, the number n of second clean voice signal frame samples in the second preset number L2 (L2≥0) clean voice signal frame samples before and after the clean voice signal frame sample is obtained, wherein the second clean voice signal frame sample is a clean voice signal frame sample whose initial voice activity detection label belongs to the first category of labels, that is, a clean voice signal frame sample whose initial voice activity detection label is 1. Exemplarily, assuming that L2 is 5, the number n of clean voice signal frame samples whose initial voice activity detection label is 1 can be counted within 5 frames before and after a clean voice signal frame sample.

[0067] Step S303: determining the voice activity detection labels corresponding to the clean speech signal frame samples according to the number of the second clean speech signal frame samples.

[0068] In this step, for each clean speech signal frame sample, it is finally determined whether the voice activity detection label corresponding to the clean speech signal frame sample belongs to the first category label or the second category label according to the number n of the second clean speech signal frame samples corresponding to the clean speech signal frame sample, thereby achieving smoothing processing of the voice activity detection label based on the initial voice activity detection label, and further improving the accuracy of the voice activity detection label.

[0069] In one embodiment, step S303 may specifically include: if the number of the second clean speech signal frame samples is greater than or equal to the second preset number, determining that the voice activity detection label corresponding to the clean speech signal frame sample belongs to the first category of labels; if the number of the second clean speech signal frame samples is less than the second preset number, determining that the voice activity detection label corresponding to the clean speech signal frame sample belongs to the second category of labels.

[0070] Specifically, for each clean speech signal frame sample, if the number n of the corresponding second clean speech signal frame samples is greater than or equal to the aforementioned second preset number L2 (n≥L2), it is determined that the voice activity detection label finally corresponding to the clean speech signal frame sample belongs to the first category label, i.e., 1; if the number n of the corresponding second clean speech signal frame samples is less than the aforementioned second preset number L2, it is determined that the voice activity detection label finally corresponding to the clean speech signal frame sample belongs to the second category label 0.

[0071] Step S205: obtaining a speech signal frame sample carrying noise according to the clean speech signal frame sample and the preset noise signal.

[0072] In order to adapt to the actual noise environment, what is needed to input into the voice activity detection model to be trained for training is a speech signal frame sample carrying noise. To this end, in this step, for each pure speech signal frame sample, the pure speech signal frame sample and the preset noise signal can be mixed according to a certain signal-to-noise ratio to obtain a speech signal frame sample carrying noise.

[0073] Step S206: training the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels.

[0074] This step is to apply the noisy speech signal frame samples and their corresponding voice activity detection labels to the training of the voice activity detection model to be trained. As an example, the noisy speech signal frame samples can be input into the voice activity detection model to be trained, and the voice activity detection model to be trained outputs voice activity detection prediction information corresponding to the sample based on the noisy speech signal frame samples. The voice activity detection prediction information can be a probability value that the sample belongs to speech, and then the model loss is obtained according to the voice activity detection prediction information and the voice activity detection label. The voice activity detection model to be trained is trained based on the model loss, and the model parameters are updated until the obtained model loss is less than or equal to the model loss threshold, so as to obtain a trained voice activity detection model.

[0075] The training method of the voice activity detection model of the present embodiment divides the clean voice signal sample into frames to obtain multiple clean voice signal frame samples, obtains the normalized value of the logarithmic energy corresponding to each of the multiple clean voice signal frame samples, obtains the proportion of the first clean voice signal frame sample in a first preset number of clean voice signal frame samples before and after the clean voice signal frame sample according to the normalized value of the logarithmic energy corresponding to each of the multiple clean voice signal frame samples, the first clean voice signal frame sample is a clean voice signal frame sample whose normalized value of the logarithmic energy is greater than the normalized value threshold, determines the voice activity detection label corresponding to the clean voice signal frame sample according to the proportion, obtains the voice signal frame sample carrying noise according to the clean voice signal frame sample and the preset noise signal, and trains the voice activity detection model to be trained by using the voice signal frame sample carrying noise and the aforementioned voice activity detection label. The scheme makes labels based on pure speech signals, divides the pure speech signals into several speech signal frames by framing, and then accurately obtains the speech activity detection label of each speech signal frame based on the signal energy. The speech signal frames are mixed with noise in a certain proportion to obtain speech signal frames carrying noise to adapt to the actual noise environment. Finally, the speech signal frames carrying noise and the aforementioned speech activity detection labels are used to train the speech activity detection model to be trained, so as to obtain a speech activity detection model with high detection accuracy, improve the detection accuracy of the speech activity detection model, and reduce the probability of speech missed detection, so as to meet the needs of real-time communication.

[0076] In one embodiment, Figure 4 As shown, the step S206 of training the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels may include:

[0077] Step S401, obtaining the Mel power spectrum corresponding to the speech signal frame sample carrying noise.

[0078] In this step, after obtaining the speech signal frame sample carrying noise, the Mel power spectrum corresponding to the speech signal frame sample carrying noise is obtained, so that the Mel power spectrum corresponding to the speech signal frame sample carrying noise is used as input data when training the model, which can achieve the effect of reducing the amount of model calculation. Specifically, for the speech signal frame sample carrying noise, the short-time Fourier transform can be used to convert the speech signal frame sample carrying noise from the time domain to the frequency domain, and then the power spectrum in the frequency domain is obtained, and then the Mel filter group is used to convert the frequency domain power spectrum into the Mel power spectrum, which can be used as the input data when training the model and input into the voice activity detection model to be trained.

[0079] Step S402: inputting the Mel power spectrum corresponding to the speech signal frame sample as an input feature into the voice activity detection model to be trained, and obtaining the voice activity detection prediction information output by the voice activity detection model to be trained according to the Mel power spectrum.

[0080] Specific, combined Figure 5 In this step, the voice activity detection model to be trained may include a gated two-dimensional convolution layer (GConv, Gated Conv-2d), a long short-term memory network (LSTM, Long Short-Term Memory), a fully connected layer (FC, Fully Connected) and an output layer connected in sequence, wherein the Mel power spectrum corresponding to the speech signal frame sample is input as the input feature X to the voice activity detection model to be trained, and is processed in sequence by the gated two-dimensional convolution layer, the long short-term memory network, the fully connected layer and the output layer, and the voice activity detection prediction information is output, and the voice activity detection prediction information can be a probability value Among them, the gated two-dimensional convolution layer can specifically include three layers, and the structure of each gated two-dimensional convolution layer is as follows: Figure 6 As shown, ReLU6 and LReLU are nonlinear activation functions, BatchNorm is batch normalization, the convolution kernel size of the two-dimensional convolution layer Conv2d can be set to (K×K), and the stride can be set to (S×S), where K∈{3,5,7},S∈{1,2}. The number of output channels of the three-layer gated two-dimensional convolution layer can be set to N, N, and 1 respectively, where N is a positive integer power of 2. The long short-term memory network can include two layers. The fully connected layer can use 1 layer. The output layer can use the nonlinear activation function Sigmoid. That is, the input feature X can pass through three layers of gated two-dimensional convolution layers, two layers of long short-term memory networks, fully connected layers, and nonlinear activation function Sigmoid to output a probability value. As prediction information for voice activity detection.

[0081] Step S403: input the voice activity detection prediction information and the voice activity detection label into a preset weighted focus loss function to obtain a model loss.

[0082] This application improves the focus loss function to form a weighted focus loss function. fl , which can be expressed as:

[0083]

[0084] in, is the voice activity detection prediction information, p is the voice activity detection label, γ is an adjustment factor, and γ≥0.

[0085] Based on this, the weighted focal loss function wL formed in this application fl for:

[0086]

[0087] Among them, ReLU is a rectified linear unit. This weighted focal loss function can impose a penalty on the situation where 1 is misjudged as 0, thereby improving the accuracy of voice activity detection.

[0088] Therefore, after obtaining the voice activity detection prediction information and the voice activity detection label, the voice activity detection prediction information and the voice activity detection label can be input into the above-mentioned weighted focal loss function to obtain the corresponding model loss.

[0089] Step S404: training the voice activity detection model to be trained according to the model loss.

[0090] In this step, the voice activity detection model to be trained is trained according to the model loss obtained by the weighted focal loss function, and its model parameters are continuously updated. When the model loss is less than or equal to the model loss threshold, a trained voice activity detection model can be obtained, thereby further improving the voice activity detection accuracy of the voice activity detection model and further reducing the probability of speech missed detection.

[0091] The solution of this embodiment uses the Mel power spectrum as input data when training the model, and uses a lightweight neural network structure, so that the number of parameters and the amount of calculation of the voice activity detection model are relatively small, avoiding the problem that the number of parameters of the deep learning model used in some current technologies is too large, resulting in large memory usage and large amount of calculation and is not suitable for use on PCs and mobile terminals. It can meet the requirements of electronic devices such as PCs and mobile phones for algorithm processing power consumption and real-time performance, and also makes the voice activity detection model have the advantages of high accuracy and low false alarm rate.

[0092] In one embodiment, Figure 7 As shown, a voice activity detection method is provided, which can be applied to Figure 1 The terminal in the method may include the following steps:

[0093] Step S701, obtaining a voice signal frame to be detected;

[0094] Step S702: Detect the speech signal frame to be detected using the trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model.

[0095] The trained voice activity detection model is trained according to the training method of the voice activity detection model provided in the above embodiment of the present application.

[0096] Step S703: obtaining a voice activity detection result of the voice signal frame to be detected according to the voice activity detection prediction information.

[0097] The solution of this embodiment can be applied in scenarios such as live broadcast and voice room, and can provide accurate voice activity detection results for audio algorithms such as noise reduction, echo cancellation, and automatic gain control, which can greatly improve the effect of the audio algorithm, thereby significantly improving the audio experience in scenarios such as live broadcast and microphone connection. It can also be applied to audio data transmission to provide a key reference for transmission. For example, audio data can be transmitted only when the voice activity detection result is 1, and audio data can not be transmitted when the voice activity detection result is 0, thereby accurately and effectively greatly reducing the transmission bandwidth and reducing business traffic, thereby providing important reference information for audio algorithms and transmission in RTC.

[0098] Specifically, the server may train the trained voice activity detection model according to the training method of the voice activity detection model provided in the above embodiment of the present application and then send it to the terminal for storage. In the application, the terminal obtains the voice signal frame to be detected, and then uses the trained voice activity detection model to detect the voice signal frame to be detected, and obtains the voice activity detection prediction information corresponding to the voice signal frame to be detected output by the voice activity detection model. In some embodiments, the terminal may obtain the Mel power spectrum corresponding to the voice signal frame to be detected and input it into the trained voice activity detection model to obtain the voice activity detection prediction information corresponding to the voice signal frame to be detected output by the voice activity detection model according to the Mel power spectrum. The voice activity detection prediction information corresponding to the voice signal frame to be detected may be a probability value. Then, the voice activity detection result of the voice signal frame to be detected can be obtained according to the probability value. As an example, the voice activity detection result r vad It can be expressed as:

[0099]

[0100] Among them, η is a threshold, and 0≤η≤1, r vad =1 means the voice activity detection result is voice, r vad =0 indicates that the voice activity detection result is non-speech.

[0101] Overall, this application combines sub-band signal processing and a lightweight neural network structure, so that the model has a small number of parameters and computational complexity, which can meet the requirements of electronic devices such as PCs and mobile phones for algorithm processing power consumption and real-time performance, and has the advantages of high accuracy in voice activity detection and low false alarm rate.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a related device for implementing the above-mentioned related methods. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more related device embodiments provided below can refer to the limitations on the related methods above, and will not be repeated here.

[0104] In one embodiment, Figure 8 As shown, a training device for a voice activity detection model is provided, and the device 800 may include:

[0105] The speech sample acquisition module 801 is used to acquire a pure speech signal sample;

[0106] A normalization processing module 802 is used to divide the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtain a normalized value of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples;

[0107] The proportion acquisition module 803 is used to obtain, for each clean speech signal frame sample, the proportion of the first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample according to the normalized values ​​of the logarithmic energies corresponding to the multiple clean speech signal frame samples; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of the logarithmic energy is greater than the normalized value threshold;

[0108] A label determination module 804 is used to determine a voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample;

[0109] The training sample acquisition module 805 is used to obtain a speech signal frame sample carrying noise according to the clean speech signal frame sample and the preset noise signal;

[0110] The model training module 806 is used to train the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels.

[0111] In one embodiment, the label determination module 804 is used to obtain an initial voice activity detection label corresponding to the clean voice signal frame sample according to a comparison result of the proportion of the first clean voice signal frame sample and a proportion threshold; for each clean voice signal frame sample, according to the initial voice activity detection labels corresponding to each of the multiple clean voice signal frame samples, obtain the number of second clean voice signal frame samples in a second preset number of clean voice signal frame samples before and after the clean voice signal frame sample; the second clean voice signal frame sample is a clean voice signal frame sample whose initial voice activity detection label belongs to the first category of labels; according to the number of the second clean voice signal frame samples, determine the voice activity detection label corresponding to the clean voice signal frame sample.

[0112] In one embodiment, the label determination module 804 is used to determine that the voice activity detection label corresponding to the clean voice signal frame sample belongs to the first category of labels if the number of the second clean voice signal frame samples is greater than or equal to the second preset number; if the number of the second clean voice signal frame samples is less than the second preset number, determine that the voice activity detection label corresponding to the clean voice signal frame sample belongs to the second category of labels.

[0113] In one embodiment, the label determination module 804 is used to determine that the initial voice activity detection label corresponding to the clean voice signal frame sample belongs to the first category of labels if the proportion of the first clean voice signal frame sample is greater than the proportion threshold; if the proportion of the first clean voice signal frame sample is less than or equal to the proportion threshold, determine that the initial voice activity detection label corresponding to the clean voice signal frame sample belongs to the second category of labels.

[0114] In one embodiment, the normalization processing module 802 is used to perform a de-averaging process on each clean speech signal frame sample, calculate the corresponding logarithmic energy according to the clean speech signal frame sample after the de-averaging process, and normalize the logarithmic energy to obtain the normalized value of the logarithmic energy corresponding to each of the multiple clean speech signal frame samples.

[0115] In one embodiment, the model training module 806 is used to obtain the Mel power spectrum corresponding to the speech signal frame sample carrying noise; input the Mel power spectrum corresponding to the speech signal frame sample as an input feature to the voice activity detection model to be trained, and obtain the voice activity detection prediction information output by the voice activity detection model to be trained according to the Mel power spectrum; wherein the voice activity detection model to be trained includes a gated two-dimensional convolutional layer, a long short-term memory network, a fully connected layer and an output layer connected in sequence; input the voice activity detection prediction information and the voice activity detection label into a preset weighted focal loss function to obtain a model loss; and train the voice activity detection model to be trained according to the model loss.

[0116] In one embodiment, Fig. 9 As shown, a voice activity detection device is provided, and the device 900 may include:

[0117] The to-be-detected frame acquisition module 901 is used to acquire the to-be-detected speech signal frame;

[0118] A model detection module 902 is used to detect the speech signal frame to be detected using a trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained using the training device for the speech activity detection model as described above;

[0119] The result obtaining module 903 is used to obtain the voice activity detection result of the voice signal frame to be detected according to the voice activity detection prediction information.

[0120] Each module in the above-mentioned related devices can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0121] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data such as voice signal samples. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a voice activity detection model is implemented.

[0122] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a voice activity detection method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0123] Those skilled in the art will understand that Fig.10 and Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training a voice activity detection model, characterized in that: The method comprises: Get a clean speech signal sample; Dividing the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtaining normalized values ​​of logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; For each clean speech signal frame sample, according to the normalized values ​​of the logarithmic energies corresponding to the multiple clean speech signal frame samples, obtain the proportion of the first clean speech signal frame sample in the first preset number of clean speech signal frame samples before and after the clean speech signal frame sample; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of the logarithmic energy is greater than the normalized value threshold; Determine the voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample; including: obtaining the initial voice activity detection label corresponding to the clean voice signal frame sample according to the comparison result of the proportion of the first clean voice signal frame sample and the proportion threshold; for each clean voice signal frame sample, according to the initial voice activity detection labels corresponding to each of the multiple clean voice signal frame samples, obtain the number of second clean voice signal frame samples in a second preset number of clean voice signal frame samples before and after the clean voice signal frame sample; the second clean voice signal frame sample is a clean voice signal frame sample whose initial voice activity detection label belongs to the first category of labels; according to the number of the second clean voice signal frame samples, determine the voice activity detection label corresponding to the clean voice signal frame sample; Obtaining a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal; The speech activity detection model to be trained is trained using the speech signal frame samples carrying noise and the speech activity detection labels.

2. The method according to claim 1, characterized in that The step of determining the voice activity detection label corresponding to the clean voice signal frame sample according to the number of the second clean voice signal frame samples includes: If the number of the second clean speech signal frame samples is greater than or equal to a second preset number, determining that the voice activity detection label corresponding to the clean speech signal frame samples belongs to the first category of labels; If the number of the second clean speech signal frame samples is less than a second preset number, it is determined that the voice activity detection label corresponding to the clean speech signal frame samples belongs to the second category label.

3. The method according to claim 1, characterized in that The obtaining, according to a comparison result of the proportion of the first clean voice signal frame sample and the proportion threshold, an initial voice activity detection label corresponding to the clean voice signal frame sample comprises: If the proportion of the first clean voice signal frame samples is greater than the proportion threshold, determining that the initial voice activity detection label corresponding to the clean voice signal frame samples belongs to the first category of labels; If the proportion of the first clean speech signal frame samples is less than or equal to the proportion threshold, it is determined that the initial voice activity detection label corresponding to the clean speech signal frame samples belongs to the second category label.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of normalized values ​​of logarithmic energies corresponding to the plurality of clean speech signal frame samples respectively comprises: For each clean speech signal frame sample, the clean speech signal frame sample is subjected to a de-averaging process, the corresponding logarithmic energy is calculated according to the clean speech signal frame sample after the de-averaging process, and the logarithmic energy is normalized to obtain the normalized value of the logarithmic energy corresponding to each of the multiple clean speech signal frame samples.

5. The method according to any one of claims 1 to 3, characterized in that: The step of training the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels includes: Obtaining the Mel power spectrum corresponding to the speech signal frame sample carrying noise; Inputting the Mel power spectrum corresponding to the speech signal frame sample as an input feature into the voice activity detection model to be trained, and obtaining the voice activity detection prediction information output by the voice activity detection model to be trained according to the Mel power spectrum; wherein the voice activity detection model to be trained comprises a gated two-dimensional convolutional layer, a long short-term memory network, a fully connected layer and an output layer connected in sequence; Inputting the voice activity detection prediction information and the voice activity detection label into a preset weighted focus loss function to obtain a model loss; The voice activity detection model to be trained is trained according to the model loss.

6. A voice activity detection method, characterized in that: The method comprises: Obtaining a voice signal frame to be detected; Detecting the speech signal frame to be detected by using a trained speech activity detection model to obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained according to the training method for a speech activity detection model according to any one of claims 1 to 5; A voice activity detection result of the voice signal frame to be detected is obtained according to the voice activity detection prediction information.

7. A training device for a voice activity detection model, characterized in that: The device comprises: A speech sample acquisition module is used to obtain pure speech signal samples; A normalization processing module, used for dividing the clean speech signal sample into frames to obtain a plurality of clean speech signal frame samples, and obtaining a normalized value of the logarithmic energy corresponding to each of the plurality of clean speech signal frame samples; a proportion acquisition module, configured to acquire, for each clean speech signal frame sample, a proportion of a first clean speech signal frame sample in a first preset number of clean speech signal frame samples before and after the clean speech signal frame sample according to normalized values ​​of logarithmic energies corresponding to each of the plurality of clean speech signal frame samples; the first clean speech signal frame sample is a clean speech signal frame sample whose normalized value of logarithmic energy is greater than a normalized value threshold; A label determination module, for determining a voice activity detection label corresponding to the clean voice signal frame sample according to the proportion of the first clean voice signal frame sample; comprising: obtaining an initial voice activity detection label corresponding to the clean voice signal frame sample according to a comparison result of the proportion of the first clean voice signal frame sample and a proportion threshold; for each clean voice signal frame sample, obtaining the number of second clean voice signal frame samples in a second preset number of clean voice signal frame samples before and after the clean voice signal frame sample according to the initial voice activity detection labels corresponding to each of the multiple clean voice signal frame samples; the second clean voice signal frame sample is a clean voice signal frame sample whose initial voice activity detection label belongs to the first category of labels; according to the number of the second clean voice signal frame samples, determining the voice activity detection label corresponding to the clean voice signal frame sample; A training sample acquisition module, used to obtain a speech signal frame sample carrying noise according to the clean speech signal frame sample and a preset noise signal; The model training module is used to train the voice activity detection model to be trained by using the speech signal frame samples carrying noise and the voice activity detection labels.

8. A voice activity detection device, characterized in that: The device comprises: A frame acquisition module to be detected, used to acquire a voice signal frame to be detected; A model detection module, configured to detect the speech signal frame to be detected using a trained speech activity detection model, and obtain speech activity detection prediction information corresponding to the speech signal frame to be detected output by the speech activity detection model; wherein the trained speech activity detection model is trained using the training device for the speech activity detection model according to claim 7; The result obtaining module is used to obtain the voice activity detection result of the voice signal frame to be detected according to the voice activity detection prediction information.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 or claim 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 or claim 6 are implemented.

Images