Voice activity detection method, storage medium and electronic device

Abstract

This specification discloses a voice activity detection method, storage medium, and electronic device. In embodiments of this specification, the audio features of the audio to be recognized are input into a voice activity detection model to obtain a recognition result output by the voice activity detection model. Classification parameters are then determined based on the recognition scenario type corresponding to the audio to be recognized. Finally, the audio to be recognized is finally recognized based on the recognition result and classification parameters output by the voice activity detection model. This method only requires the deployment of a universal voice activity detection model, eliminating the need to deploy separate voice activity detection models for different recognition scenarios. This effectively reduces the cost of model deployment and iterative upgrades.

Description

Technical Field

[0001] This specification relates to the field of Internet technology, and in particular to a voice activity detection method, storage medium, and electronic device. Background Art

[0002] At present, speech recognition technology has been widely used in various scenarios, such as telephone speech recognition, voice conference recording, etc.

[0003] In existing technologies, speech recognition typically uses a pre-trained machine learning model. Specifically, voice activity detection (VAD) is first performed on the audio to be recognized. This involves inputting the audio to be recognized into a VAD model to identify speech and non-speech segments within the audio. The identified speech segments are then input into a speech recognition model for speech recognition. The VAD model is a binary classification model that classifies the input audio into speech and non-speech segments.

[0004] However, in practice, the sampling frequency and input mode of the audio to be recognized vary across different application scenarios. For example, in telephone speech recognition scenarios, the sampling frequency of the audio to be recognized is typically 8 kHz, and the input mode is streaming, while in voice conference recording scenarios, the sampling frequency is typically 16 kHz, and the input mode is non-streaming. Therefore, for the audio to be recognized in different application scenarios, different voice activity detection models often need to be trained to recognize speech and non-speech segments, which increases the cost of model deployment and iterative upgrades.

[0005] In view of this, how to reduce the cost of model deployment and iterative upgrades has become an urgent problem to be solved. Summary of the Invention

[0006] The embodiments of this specification provide a voice activity detection method, apparatus, storage medium, and electronic device to partially solve the problems existing in the above-mentioned prior art.

[0007] The embodiments of this specification adopt the following technical solutions:

[0008] This specification provides a voice activity detection method, including:

[0009] Get the audio to be recognized;

[0010] Extracting audio features from each audio frame of the audio to be recognized;

[0011] Inputting the audio features of each audio frame into a pre-trained voice activity detection model to obtain a probability that each audio frame output by the voice activity detection model is a speech segment;

[0012] Determining a classification parameter according to a recognition scene type corresponding to the audio to be recognized;

[0013] Whether each audio frame is a speech segment is identified based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter.

[0014] Optionally, obtaining the audio to be recognized specifically includes:

[0015] Get the audio to be recognized from streaming input; or get the audio to be recognized from non-streaming input;

[0016] Input the audio features of each audio frame into a pre-trained voice activity detection model, specifically including:

[0017] When the audio to be recognized is streamed audio, inputting the audio features of each audio frame in the audio to be recognized into a pre-trained voice activity detection model in a streamed manner;

[0018] When the audio to be recognized is non-streaming input audio, the audio features of each audio frame in the audio to be recognized are input into a pre-trained voice activity detection model in a non-streaming input manner.

[0019] Optionally, extracting audio features from each audio frame of the audio to be recognized specifically includes:

[0020] According to the sampling frequency of the audio to be recognized, normalized audio features are extracted from each audio frame of the audio to be recognized; the normalized audio features include: MixBand FBank features.

[0021] Optionally, determining a classification parameter according to the recognition scene type corresponding to the audio to be recognized specifically includes:

[0022] Determining a probability threshold corresponding to the recognition scene type according to the recognition scene type corresponding to the audio to be recognized;

[0023] Identifying whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter specifically includes:

[0024] For each audio frame, if the probability that the audio frame is a speech segment output by the voice activity detection model is greater than the probability threshold, the audio frame is identified as a speech segment; otherwise, the audio frame is identified as a non-speech segment.

[0025] Optionally, obtaining the audio to be recognized specifically includes:

[0026] Receive a recognition request and obtain the audio to be recognized corresponding to the recognition request;

[0027] Determining classification parameters based on the recognition scene type corresponding to the audio to be recognized, specifically including:

[0028] Determining, according to the recognition request, a recognition scene type corresponding to the audio to be recognized;

[0029] Determining a smoothing processing parameter according to a recognition scene type corresponding to the audio to be recognized;

[0030] Identifying whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter specifically includes:

[0031] For each audio frame, identifying whether the audio frame is a speech segment based on the probability of the audio frame being a speech segment output by the voice activity detection model and a predetermined probability threshold;

[0032] The recognition result of each audio frame is smoothed according to the smoothing processing parameter and the recognition result of each audio frame in the audio to be recognized.

[0033] Optionally, the smoothing processing parameters include a minimum speech length n1 and a minimum non-speech length n2;

[0034] According to the smoothing processing parameter and the recognition result of each audio frame in the audio to be recognized, smoothing the recognition result of each audio frame specifically includes:

[0035] Extracting continuous audio frames recognized as speech segments and continuous audio frames recognized as non-speech segments from the audio to be recognized;

[0036] If the number of the consecutive audio frames identified as speech segments is less than n1, all the consecutive audio frames identified as speech segments are regarded as non-speech segments;

[0037] If the number of the continuous audio frames identified as non-speech segments is less than n2, all the continuous audio frames identified as non-speech segments are treated as speech segments.

[0038] Optionally, pre-train a voice activity detection model, specifically including:

[0039] Obtaining a sample audio and annotations of the sample audio, wherein the annotations of the sample audio include speech segments and non-speech segments in the sample audio;

[0040] extracting audio features from each sample audio frame of the sample audio;

[0041] Inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a streaming input manner to obtain a first recognition result output by the voice activity detection model to be trained; inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a non-streaming input manner to obtain a second recognition result output by the voice activity detection model to be trained;

[0042] determining a loss of the voice activity detection model to be trained based on the annotation of the sample audio, the first recognition result, and the second recognition result;

[0043] Taking minimizing the loss as a training objective, model parameters of the voice activity detection model to be trained are adjusted.

[0044] Optionally, determining the loss of the voice activity detection model to be trained according to the annotation of the sample audio, the first recognition result, and the second recognition result specifically includes:

[0045] Determining a difference between the first recognition result and the annotation of the sample audio as a first difference;

[0046] Determining a difference between the second recognition result and the annotation of the sample audio as a second difference;

[0047] Taking the second recognition result as a pseudo-label, determining a difference between the first recognition result and the pseudo-label as a third difference;

[0048] The loss of the voice activity detection model to be trained is determined based on the first difference, the second difference, and the third difference.

[0049] This specification provides a voice activity detection device, the device comprising:

[0050] An acquisition module, used to acquire the audio to be recognized;

[0051] A feature extraction module, configured to extract audio features from each audio frame of the audio to be recognized;

[0052] a recognition module, configured to input the audio features of each audio frame into a pre-trained voice activity detection model to obtain a probability that each audio frame output by the voice activity detection model is a speech segment;

[0053] A post-processing module is used to determine a classification parameter based on the recognition scene type corresponding to the audio to be recognized; and to identify whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter.

[0054] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned voice activity detection method is implemented.

[0055] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned voice activity detection method is implemented.

[0056] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0057] In the embodiments of this specification, the audio features of the audio to be recognized are input into a voice activity detection model to obtain a recognition result output by the voice activity detection model. Classification parameters are then determined based on the recognition scenario type corresponding to the audio to be recognized. Finally, the audio to be recognized is finally recognized based on the recognition result and classification parameters output by the voice activity detection model. This allows only a general voice activity detection model to be deployed, eliminating the need to deploy separate voice activity detection models for different recognition scenarios. This effectively reduces the cost of model deployment and iterative upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0059] Figure 1 A schematic diagram of the voice activity detection process provided in the embodiments of this specification;

[0060] Figure 2 A schematic diagram of the structure of a voice activity detection device provided in an embodiment of this specification;

[0061] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0063] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0064] Figure 1 The voice activity detection process provided in the embodiments of this specification specifically includes the following steps:

[0065] S100: Acquire audio to be recognized.

[0066] The embodiments of this specification aim to deploy only one voice activity detection model to identify audio of various input modes (such as streaming input and non-streaming input) and various sampling frequencies (such as 8KHz, 16KHz) through the only voice activity detection model.

[0067] First, the voice activity detection device obtains the audio to be recognized. The audio to be recognized can be streaming audio or non-streaming audio. Specifically, the device receives a recognition request and receives the audio to be recognized corresponding to the recognition request. The audio to be recognized can be included in the recognition request. The recognition requesting party can also add a tag corresponding to the recognition request to the audio to be recognized, thereby marking the audio to be recognized as the audio to be recognized corresponding to the recognition request.

[0068] The recognition request can be sent by the recognition requesting party to the voice activity detection device during the execution of a service. The recognition requesting party can then send a recognition request carrying the service identifier of the service being executed to the voice activity detection device. Accordingly, based on the service identifier, the voice activity detection device can determine the recognition scenario type corresponding to the audio to be recognized, such as telephone voice recognition, voice conference recording, and other scenarios. Of course, other methods can also be used to enable the voice activity monitoring device to determine the recognition scenario type corresponding to the audio to be recognized, which will not be detailed here.

[0069] The voice activity detection device can pre-store the correspondence between different recognition scene types and input modes and sampling frequencies. Since different recognition scene types usually correspond to different input modes and sampling frequencies of the audio to be recognized, after the voice activity detection device determines the recognition scene type corresponding to the audio to be recognized, it can determine the input mode and sampling frequency of the audio to be recognized. Subsequently, voice activity detection can be performed on the audio to be recognized based on the input mode and sampling frequency of the audio to be recognized.

[0070] S102: Extracting audio features from each audio frame of the audio to be recognized.

[0071] Because the audio features extracted from the audio to be identified at different sampling frequencies are different, the embodiments of this specification require extracting normalized audio features from each audio frame of the audio to be identified based on the sampling frequency of the audio to be identified. The normalized audio features include, but are not limited to, MixBand FBank features. The audio frame described in the embodiments of this specification can specifically be a segment of audio of a specified duration in the audio to be identified, such as a 10ms segment of audio.

[0072] Specifically, for a given audio signal, a Fourier transform is typically used to convert the audio signal from the time domain to the frequency domain. For example, for a 1-second audio signal with a 16 kHz sampling rate, assuming an audio frame is 10 ms, the Fourier transform transforms 16,000 numbers representing audio amplitude values ​​into 100 audio features (audio features are represented by vectors). Each audio feature represents the intensity value of the audio frame at a specific frequency. According to the Nyquist sampling theorem, the resulting frequency range is 0-8000 Hz. That is, the higher the sampling frequency of the audio signal, the easier it is to capture the high-frequency features of the audio signal itself. Generally, assuming the sampling frequency of the audio signal is f, the audio features extracted from the audio signal can only represent the audio frequency of the audio signal itself at a frequency of f / 2. The audio features of each audio frame are filtered using a triangular Mel-shaped filter bank. For example, 40 filters can be used, resulting in 40 eigenvalues. Finally, the 1-second audio signal is converted into a 100×40 feature matrix, which serves as the input for the subsequent voice activity detection model.

[0073] From the above description, it can be seen that for the two audio signals with sampling frequencies of 8KHz and 16KHz, although the above method can extract audio features of 40 eigenvalues ​​(i.e., 40 dimensions) for each audio frame, the physical meanings of the 40-dimensional audio features extracted from the two audio signals are not the same. For the audio signal with a sampling frequency of 8KHz, the 40-dimensional audio features extracted therefrom represent the audio features of the audio signal itself from 0 to 4000Hz, while for the audio signal with a sampling frequency of 16KHz, the 40-dimensional audio features extracted therefrom represent the audio features of the audio signal itself from 0 to 8000Hz. That is, although the dimensions of the audio features are the same, the physical meanings expressed are not the same. Since the present application aims to identify audio to be identified with different sampling frequencies through a voice activity detection model, it is necessary to unify the physical meanings of the input audio features, which also requires normalizing the audio features of the audio to be identified. Specifically, the audio features can be normalized by extracting MixBand FBank features from the audio to be identified.

[0074] Three methods for extracting normalized audio features (MixBand FBank features) are provided below.

[0075] The first method is to process the audio to be identified into audio with a sampling frequency of the standard sampling frequency according to a pre-set standard sampling frequency, and then extract audio features from each audio frame of the processed audio to be identified as normalized audio features. Among them, processing the audio to be identified into audio with a sampling frequency of the standard sampling frequency may include interpolation or filtering, that is, upsampling and downsampling. For example, assuming that the standard sampling frequency is 16KHz, if the audio to be identified is audio with a sampling frequency of 16KHz, then the audio features can be directly extracted as normalized audio features. If the audio to be identified is audio with a sampling frequency of 8KHz, then it can be processed into 16KHz audio by interpolation, and then audio features can be extracted from the audio processed into 16KHz as normalized audio features. Assuming that the standard sampling frequency is 8KHz, if the audio to be identified is audio with a sampling frequency of 16KHz, then it can be processed into 8Hz audio by filtering, and then audio features can be extracted from the audio processed into 8KHz as normalized audio features.

[0076] The second method is to extract normalized audio features by low-dimensional alignment features. Specifically, a standard sampling frequency can be set in advance, and the standard sampling frequency is not lower than the maximum sampling frequency of the audio to be identified. When using a triangular Mel filter group containing m filters (m is a positive integer, for example, it can be 40) to extract audio features, the positive integer k is determined according to the difference between the standard sampling frequency and the sampling frequency of the audio to be identified. The larger the difference, the smaller k, and the smaller the difference, the larger k (wherein, it is necessary to select the first k from the m triangular filters corresponding to the standard sampling frequency so that the frequency range corresponding to these k filters covers the frequency domain of the audio to be identified as much as possible, and after the endpoints are given, the distribution of the sampling frequencies corresponding to the triangular filters is also deterministic. Therefore, after the standard sampling frequency and the sampling frequency of the audio to be identified are given, the value of k can be directly determined). Among the m filters, k filters are used to extract audio features from each audio frame of the audio to be identified. At this time, the extracted audio features are k-dimensional, so it is necessary to add mk eigenvalues ​​to the audio features of these k dimensions to convert the audio features of these k dimensions into audio features of these m dimensions. Specifically, mk eigenvalues ​​can be added by padding with zeros, that is, padding with mk eigenvalues ​​with a value of 0. In this way, regardless of the sampling frequency of the audio to be identified, audio features of m dimensions can be extracted, and the meaning of the audio features of each dimension is unified. The relationship between the above-mentioned difference and the positive integer k can be a linear relationship or a nonlinear relationship.

[0077] For example, assuming the standard sampling frequency is 16 kHz and the sampling frequency of the audio to be recognized is 8 kHz, then for a triangular Mel filter bank containing 40 filters, the positive integer k can be determined to be 30. Using 30 filters, 30-dimensional audio features are extracted from each audio frame of the audio to be recognized, and 10 feature values ​​are added with a value of 0, for a total of 40-dimensional audio features. This is because the audio features of the audio to be recognized with a sampling frequency of 8 kHz can only represent the audio of the audio to be recognized from 0 to 4000 Hz, while the audio features of the audio to be recognized with a sampling frequency of 16 kHz can represent the audio of the audio to be recognized from 0 to 8000 Hz. However, since audio features are not uniformly distributed in frequency, with lower dimensions being denser and higher dimensions being sparser, the first 30 filters of the 16 kHz filter bank correspond to 0-4000 Hz, and the last 10 correspond to 4000-8000 Hz. In other words, the difference between the standard sampling frequency and the sampling frequency of the audio to be recognized has a nonlinear negative correlation with the positive integer k.

[0078] The third method is to use a machine learning model to extract normalized audio features. Specifically, a machine learning model for outputting audio with a preset standard sampling frequency can be pre-trained, and the audio to be identified can be input into the machine learning model to obtain the audio to be identified with the standard sampling frequency outputted by the machine learning model, and then processed using a method similar to the first method of extracting normalized audio features. The standard sampling frequency can also be preset, and a machine learning model for outputting mk-dimensional audio features can be pre-trained. Combined with the second method of extracting normalized audio features, a positive integer k is determined, and k-dimensional audio features are extracted from the audio to be identified. The k-dimensional audio features are then input into the machine learning model to obtain the mk-dimensional audio features outputted by the machine learning model, and the m-dimensional normalized audio features are obtained through the k-dimensional audio features and the mk-dimensional audio features.

[0079] S104: Inputting the audio features of each audio frame into a pre-trained voice activity detection model to obtain the probability that each audio frame output by the voice activity detection model is a speech segment.

[0080] After obtaining the audio features of each audio frame in the audio to be recognized through the above step S102, the audio features of each audio frame can be input into a pre-trained voice activity detection model to obtain the probability that each audio frame output by the voice activity detection model is a speech segment.

[0081] Specifically, when it is determined according to the recognition request that the audio to be recognized is streaming input audio, the audio features of each audio frame in the audio to be recognized are input into the pre-trained voice activity detection model in a streaming input manner; when it is determined according to the recognition request that the audio to be recognized is non-streaming input audio, the audio features of each audio frame in the audio to be recognized are input into the pre-trained voice activity detection model in a non-streaming input manner.

[0082] For streaming input scenarios such as telephone speech recognition, the audio features of every i audio frames can be used as input to the voice activity detection model. For non-streaming input scenarios such as conference recordings, the audio features of every j audio frames can be used as input to the voice activity detection model. Where j is greater than i. For example, if i is 20 and j is 200, the audio features of every 20 audio frames are used as input to the voice activity detection model for streaming input, and every 200 audio frames are used as input for non-streaming input.

[0083] S106: Determine a classification parameter according to the recognition scene type corresponding to the audio to be recognized.

[0084] S108: Identify whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment and the classification parameter output by the voice activity detection model.

[0085] The classification parameters described in the embodiments of this specification include but are not limited to probability thresholds and smoothing processing parameters.

[0086] When the classification parameter is a probability threshold, the recognition scene type can be determined according to the recognition request, and then the probability threshold corresponding to the recognition scene type can be determined according to the recognition scene type. Among them, different recognition scene types correspond to different probability thresholds. Then in step S108, for each audio frame, if the probability that the audio frame output by the voice activity detection model is a voice segment is greater than the probability threshold, the audio frame is identified as a voice segment, otherwise, the audio frame is identified as a non-speech segment. For example, when the recognition scene is a telephone voice recognition scene, since the noise in the scene is usually high (especially the telephone voice recognition scene during the distribution process of the delivery capacity), the probability threshold corresponding to the recognition scene can be set to a higher threshold, and when the recognition scene is a voice conference recording scene, since the noise in the scene is usually low, the probability threshold corresponding to the recognition scene can be set to a lower threshold.

[0087] When the classification parameter is a smoothing parameter, the recognition scene type may be determined based on the recognition request, and then the smoothing parameter corresponding to the recognition scene type may be determined based on the recognition scene type. In step S108, for each audio frame, whether the audio frame is a speech segment may be determined based on the probability of the audio frame being a speech segment output by the voice activity detection model and a predetermined probability threshold. The recognition results of each audio frame are then smoothed based on the smoothing parameter and the recognition results of each audio frame in the audio to be recognized.

[0088] Specifically, the smoothing processing parameters may include a minimum speech length n1 and a minimum non-speech length n2. When performing smoothing processing, continuous audio frames identified as speech segments and continuous audio frames identified as non-speech segments can be extracted from the audio to be identified. If the number of continuous audio frames identified as speech segments is less than n1, the continuous audio frames identified as speech segments will all be treated as non-speech segments. If the number of continuous audio frames identified as non-speech segments is less than n2, the continuous audio frames identified as non-speech segments will all be treated as speech segments.

[0089] Of course, the above-mentioned classification parameters may also include a probability threshold and a smoothing processing parameter at the same time. In this case, step S108 is specifically as follows: for each audio frame, based on the probability that the audio frame is a speech segment output by the voice activity detection model and the probability threshold corresponding to the recognition scene type, if the probability that the audio frame is a speech segment output by the voice activity detection model is greater than the probability threshold, the audio frame is identified as a speech segment; otherwise, the audio frame is identified as a non-speech segment; and then, based on the smoothing processing parameter and the recognition results of each audio frame in the audio to be identified, the recognition results of each audio frame are smoothed.

[0090] Through the above Figure 1 After the method shown identifies the speech segments and non-speech segments of the audio to be recognized, the identified speech segments can be input into a pre-trained speech recognition model to recognize the input speech segments through the speech recognition model, that is, the speech segments can be converted into corresponding text.

[0091] Through the above method, only one general voice activity detection model can be deployed. After obtaining the output results of the model, corresponding post-processing (steps S106 to S108) is performed according to the recognition scenario type, which can adapt to various different recognition scenarios without having to deploy corresponding voice activity detection models for different recognition scenarios. This can effectively reduce the cost of model deployment and iterative upgrades.

[0092] In particular, for the above-mentioned general voice activity detection model with high generalization, the voice activity detection model can process the audio to be recognized regardless of the input mode and sampling frequency.

[0093] In an embodiment of the present specification, a method for training the above-mentioned general voice activity detection model with high generalization may include: obtaining sample audio and annotations of the sample audio, wherein the annotations of the sample audio include speech segments and non-speech segments in the sample audio; extracting audio features from each sample audio frame of the sample audio; inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a streaming input manner to obtain a first recognition result output by the voice activity detection model to be trained; inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a non-streaming input manner to obtain a second recognition result output by the voice activity detection model to be trained; determining the loss of the voice activity detection model to be trained based on the annotations of the sample audio, the first recognition result, and the second recognition result; and adjusting the model parameters of the voice activity detection model to be trained with minimizing the loss as the training goal.

[0094] Among them, the method of extracting audio features from each sample audio frame of the sample audio is the same as Figure 1 The method of extracting normalized audio features (MixBand FBank features) in step S102 is exactly the same as that in step S103, and will not be described in detail here. Figure 1 The method of inputting the audio features of each audio frame in the audio to be recognized into the pre-trained voice activity detection model in a streaming input manner in step S104 is exactly the same as the method of inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a non-streaming input manner. Figure 1 The manner of inputting the audio features of each audio frame in the audio to be recognized into the pre-trained voice activity detection model in a non-streaming input manner in step S104 is exactly the same as that in step S104, and will not be described in detail here.

[0095] It should be noted that when training the voice activity detection model as described above, in the embodiments of this specification, the training can be performed by knowledge distillation. That is, when determining the loss, the difference between the first recognition result and the annotation of the sample audio is determined as the first difference, the difference between the second recognition result and the annotation of the sample audio is determined as the second difference, the second recognition result is used as a pseudo-annotation, and the difference between the first recognition result and the pseudo-annotation is determined as the third difference. The loss of the voice activity detection model to be trained is determined based on the first difference, the second difference, and the third difference. The first difference, the second difference, and the third difference are all positively correlated with the loss. Specifically, the first difference, the second difference, and the third difference can be weighted and summed according to preset weights, and the result of the weighted sum is used as the loss of the voice activity detection model to be trained.

[0096] When determining the third difference, the second recognition result is used as a pseudo-label because, generally, after inputting the model in a non-streaming input manner, the accuracy of the recognition result obtained by the model is higher than that of the streaming input.

[0097] The above-mentioned method for training a general voice activity detection model with high generalization is mainly based on the joint training of streaming input and non-streaming input. In addition to the above-mentioned joint training method, the voice activity detection model can also be trained by other joint training methods of streaming input and non-streaming input.

[0098] In addition to the above-mentioned joint training method, other non-joint methods can also be used to train a general voice activity detection model with higher generalization. For example, a model that supports streaming input is selected as the voice activity detection model, and the voice activity detection model is trained through a supervised training method based on the sample audio and the annotation of the sample audio. The sample audio includes both the sample audio of streaming input and the sample audio of non-streaming input. The model that supports streaming input includes but is not limited to deep neural networks (DNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM). For another example, a model that supports non-streaming input is selected as the voice activity detection model, and the voice activity detection model is trained through a supervised training method based on the sample audio and the annotation of the sample audio. The sample audio includes both the sample audio of streaming input and the sample audio of non-streaming input. The model that supports non-streaming input includes but is not limited to Transformer and LC-LSTM.

[0099] The above is the voice activity detection method provided in the embodiments of this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0100] Figure 2 This is a schematic diagram of the structure of a voice activity device provided in an embodiment of this specification, the device comprising:

[0101] An acquisition module 201 is used to acquire audio to be recognized;

[0102] A feature extraction module 202 is configured to extract audio features from each audio frame of the audio to be recognized;

[0103] The recognition module 203 is configured to input the audio features of each audio frame into a pre-trained voice activity detection model to obtain the probability that each audio frame output by the voice activity detection model is a speech segment;

[0104] The post-processing module 204 is configured to determine a classification parameter based on the recognition scene type corresponding to the audio to be recognized; and identify whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter.

[0105] Optionally, the acquisition module 201 is specifically configured to acquire the audio to be recognized that is input in a streaming manner; or acquire the audio to be recognized that is input in a non-streaming manner;

[0106] The recognition module 203 is specifically configured to, when the audio to be recognized is streamed audio, input the audio features of each audio frame in the audio to be recognized into a pre-trained voice activity detection model in a streaming input manner; and when the audio to be recognized is non-streamed audio, input the audio features of each audio frame in the audio to be recognized into the pre-trained voice activity detection model in a non-streaming input manner.

[0107] Optionally, the feature extraction module 202 is specifically configured to extract normalized audio features from each audio frame of the audio to be identified according to a sampling frequency of the audio to be identified; the normalized audio features include MixBandFBank features.

[0108] Optionally, the post-processing module 204 is specifically configured to determine a probability threshold corresponding to the recognition scene type according to the recognition scene type corresponding to the audio to be recognized;

[0109] The post-processing module 204 is specifically configured to, for each audio frame, identify the audio frame as a speech segment if the probability of the audio frame being a speech segment output by the voice activity detection model is greater than the probability threshold; otherwise, identify the audio frame as a non-speech segment.

[0110] Optionally, the acquisition module 201 is specifically configured to receive a recognition request and acquire the audio to be recognized corresponding to the recognition request;

[0111] The post-processing module 204 is specifically configured to determine, according to the recognition request, a recognition scene type corresponding to the audio to be recognized; and determine a smoothing processing parameter according to the recognition scene type corresponding to the audio to be recognized;

[0112] The post-processing module 204 is specifically used to, for each audio frame, identify whether the audio frame is a speech segment based on the probability of the audio frame being a speech segment output by the voice activity detection model and a predetermined probability threshold; and smooth the recognition results of each audio frame based on the smoothing processing parameters and the recognition results of each audio frame in the audio to be recognized.

[0113] Optionally, the smoothing processing parameters include a minimum speech length n1 and a minimum non-speech length n2;

[0114] The post-processing module 204 is specifically used to extract continuous audio frames identified as speech segments and continuous audio frames identified as non-speech segments from the audio to be identified; if the number of continuous audio frames identified as speech segments is less than n1, then all the continuous audio frames identified as speech segments are regarded as non-speech segments; if the number of continuous audio frames identified as non-speech segments is less than n2, then all the continuous audio frames identified as non-speech segments are regarded as speech segments.

[0115] The device further comprises:

[0116] The training module 205 is used to obtain sample audio and annotations of the sample audio, where the annotations of the sample audio include speech segments and non-speech segments in the sample audio; extract audio features from each sample audio frame of the sample audio; input the audio features of each sample audio frame into the voice activity detection model to be trained in a streaming input manner to obtain a first recognition result output by the voice activity detection model to be trained; input the audio features of each sample audio frame into the voice activity detection model to be trained in a non-streaming input manner to obtain a second recognition result output by the voice activity detection model to be trained; determine the loss of the voice activity detection model to be trained based on the annotations of the sample audio, the first recognition result, and the second recognition result; and adjust the model parameters of the voice activity detection model to be trained with minimizing the loss as the training goal.

[0117] Optionally, the training module 205 is specifically used to determine the difference between the first recognition result and the annotation of the sample audio as a first difference; determine the difference between the second recognition result and the annotation of the sample audio as a second difference; use the second recognition result as a pseudo-annotation, and determine the difference between the first recognition result and the pseudo-annotation as a third difference; and determine the loss of the voice activity detection model to be trained based on the first difference, the second difference, and the third difference.

[0118] This specification also provides a computer-readable storage medium, wherein the storage medium stores a computer program. When the computer program is executed by a processor, it can be used to perform the voice activity detection method provided above.

[0119] Based on the above provided voice activity detection method, this specification embodiment also provides Figure 3 The structural diagram of the electronic device shown in FIG. Figure 3At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its services. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it to implement the above-mentioned voice activity detection method.

[0120] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0122] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0123] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0124] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0125] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0127] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0130] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0131] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0133] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0135] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0136] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A voice activity detection method, characterized in that: include: Get the audio to be recognized; Extracting audio features from each audio frame of the audio to be recognized; Inputting the audio features of each audio frame into a pre-trained voice activity detection model to obtain a probability that each audio frame output by the voice activity detection model is a speech segment; Determining a classification parameter according to a recognition scene type corresponding to the audio to be recognized; identifying whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter; The obtaining of the audio to be recognized specifically includes: Get the audio to be recognized from streaming input; or get the audio to be recognized from non-streaming input; Inputting the audio features of each audio frame into a pre-trained voice activity detection model specifically includes: When the audio to be recognized is streamed audio, inputting the audio features of each audio frame in the audio to be recognized into a pre-trained voice activity detection model in a streamed manner; When the audio to be recognized is non-streaming input audio, inputting the audio features of each audio frame in the audio to be recognized into a pre-trained voice activity detection model in a non-streaming input manner; The step of obtaining the audio to be recognized further specifically includes: Receive a recognition request and obtain the audio to be recognized corresponding to the recognition request; The determining of the classification parameter according to the recognition scene type corresponding to the audio to be recognized specifically includes: Determining, according to the recognition request, a recognition scene type corresponding to the audio to be recognized; Determining a probability threshold and a smoothing processing parameter corresponding to the recognition scene type according to the recognition scene type corresponding to the audio to be recognized; The identifying whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter specifically includes: For each audio frame, identifying whether the audio frame is a speech segment based on the probability of the audio frame being a speech segment output by the voice activity detection model and a predetermined probability threshold; The recognition result of each audio frame is smoothed according to the smoothing processing parameter and the recognition result of each audio frame in the audio to be recognized.

2. The method according to claim 1, wherein The identifying whether each audio frame is a speech segment based on the probability of each audio frame being a speech segment output by the voice activity detection model and the classification parameter further specifically includes: For each audio frame, if the probability that the audio frame is a speech segment output by the voice activity detection model is greater than the probability threshold, the audio frame is identified as a speech segment; otherwise, the audio frame is identified as a non-speech segment.

3. The method according to claim 1, wherein The smoothing processing parameters include a minimum speech length n1 and a minimum non-speech length n2; According to the smoothing processing parameter and the recognition result of each audio frame in the audio to be recognized, smoothing the recognition result of each audio frame specifically includes: Extracting continuous audio frames recognized as speech segments and continuous audio frames recognized as non-speech segments from the audio to be recognized; If the number of the consecutive audio frames identified as speech segments is less than n1, all the consecutive audio frames identified as speech segments are regarded as non-speech segments; If the number of the continuous audio frames identified as non-speech segments is less than n2, all the continuous audio frames identified as non-speech segments are treated as speech segments.

4. The method according to claim 1, wherein Pre-trained voice activity detection model, including: Obtaining a sample audio and annotations of the sample audio, wherein the annotations of the sample audio include speech segments and non-speech segments in the sample audio; extracting audio features from each sample audio frame of the sample audio; Inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a streaming input manner to obtain a first recognition result output by the voice activity detection model to be trained; inputting the audio features of each sample audio frame into the voice activity detection model to be trained in a non-streaming input manner to obtain a second recognition result output by the voice activity detection model to be trained; determining a loss of the voice activity detection model to be trained based on the annotation of the sample audio, the first recognition result, and the second recognition result; Taking minimizing the loss as a training objective, model parameters of the voice activity detection model to be trained are adjusted.

5. The method according to claim 4, wherein Determining the loss of the voice activity detection model to be trained according to the annotation of the sample audio, the first recognition result, and the second recognition result, specifically comprising: Determining a difference between the first recognition result and the annotation of the sample audio as a first difference; Determining a difference between the second recognition result and the annotation of the sample audio as a second difference; Taking the second recognition result as a pseudo-label, determining a difference between the first recognition result and the pseudo-label as a third difference; The loss of the voice activity detection model to be trained is determined based on the first difference, the second difference, and the third difference.

6. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

Images