Voice detection model training method, voice detection method, device and apparatus

Abstract

The present disclosure provides a speech detection model training method, a speech detection method, an apparatus and a device, relates to the technical field of computers, in particular to the technical field of deep learning and speech recognition. The specific implementation scheme is: obtaining a speech sample set, the speech sample set including sample features of speech samples and labeling information of the speech samples; inputting the sample features into an encoder of a speech detection model to obtain intermediate features, and the dimension of the intermediate features is less than the dimension of the sample features; inputting the intermediate features into a decoder of the speech detection model to obtain a predicted confidence of the speech samples, the predicted confidence being used to represent the probability that the speech samples are non-silence; training the speech detection model according to the labeling information and the predicted confidence to obtain a trained speech detection model, the trained speech detection model can be used for speech silence detection, the processing speed of the speech detection model is improved, the speech detection model has good real-time performance, and can meet various scenes.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of deep learning and speech recognition technology, specifically to a training method for a speech detection model, a speech detection method, an apparatus, and a device. Background Technology

[0002] Voice Activity Detection (VAD) is a crucial front-end processing module in speech recognition, primarily responsible for detecting whether a current speech segment is silent or not. In offline speech processing, VAD can be used to segment audio from videos and, in conjunction with speech recognition, generate timestamped audio subtitles. In streaming speech processing, VAD can request a speech recognition service to transcribe speech upon detection, thereby improving the utilization of the speech recognition service and saving costs; however, this places demands on VAD's processing speed. Therefore, improving the processing speed of voice activity detection is receiving increasing attention. Summary of the Invention

[0003] This disclosure provides a method for training a speech detection model, a speech detection method, an apparatus, and a device.

[0004] According to one aspect of the present disclosure, a method for training a speech detection model is provided, comprising: acquiring a speech sample set, the speech sample set including sample features of speech samples and annotation information of speech samples; inputting the sample features into the encoder of the speech detection model to obtain intermediate features, wherein the dimension of the intermediate features is smaller than the dimension of the sample features; inputting the intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech samples, wherein the prediction confidence is used to characterize the probability that the speech sample is non-silent; and training the speech detection model according to the annotation information and the prediction confidence to obtain a trained speech detection model.

[0005] According to another aspect of the present disclosure, a speech detection method is provided, comprising: acquiring a current speech segment in a speech stream to be detected; determining the speech features of the current speech segment; inputting the speech features into a trained speech detection model to obtain the confidence level of the current speech segment, wherein the trained speech detection model is trained according to the method of any of the above embodiments; and determining a first label of the current speech segment based on the confidence level of the current speech segment, wherein the first label is used to determine the position of a non-silent segment in the speech stream to be detected.

[0006] According to another aspect of the embodiments of this disclosure, a training apparatus for a speech detection model is provided, comprising: an acquisition unit for acquiring a speech sample set, the speech sample set including sample features of speech samples and annotation information of speech samples; an encoding unit for inputting the sample features into an encoder of the speech detection model to obtain intermediate features, wherein the dimension of the intermediate features is smaller than the dimension of the sample features; a decoding unit for inputting the intermediate features into a decoder of the speech detection model to obtain a prediction confidence score of the speech sample, wherein the prediction confidence score is used to characterize the probability that the speech sample is non-silent; and a training unit for training the speech detection model based on the annotation information and the prediction confidence score to obtain a trained speech detection model.

[0007] According to another aspect of the present disclosure, a speech detection apparatus is provided, comprising: a segmentation unit for acquiring a current speech segment in a speech stream to be detected; a determination unit for determining current speech features of the current speech segment; a prediction unit for inputting the speech features into a trained speech detection model to obtain a confidence level of the speech segment, wherein the trained speech detection model is trained according to the method of any of the above embodiments; and a processing unit for determining a first label of the current speech segment based on the confidence level of the current speech segment, wherein the first label is used to determine the position of a non-silent segment in the speech stream to be detected.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] The speech detection model training method, speech detection method, apparatus, and device provided in this disclosure acquire a speech sample set, which includes sample features and annotation information of the speech samples; input the sample features into the encoder of the speech detection model to obtain intermediate features, wherein the dimension of the intermediate features is smaller than the dimension of the sample features; input the intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech samples, wherein the prediction confidence is used to characterize the probability that the speech sample is not silent; train the speech detection model according to the annotation information and the prediction confidence to obtain a trained speech detection model. The trained speech detection model can be used for speech silence detection, and since the encoder can reduce the dimension of the output intermediate features, the processing speed of the speech detection model can be improved, so that the speech detection model can be used not only for offline speech detection, but also for streaming speech detection, with good real-time performance, and can meet various scenarios.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 A schematic diagram of the structure of a training method for the speech detection model and a system for the speech detection method applying the embodiments of this disclosure;

[0015] Figure 2 This is a flowchart illustrating the training method for a speech detection model provided according to an embodiment of the present disclosure;

[0016] Figure 3 This is a schematic diagram of the structure of the speech detection model provided according to an embodiment of the present disclosure;

[0017] Figure 4 yes Figure 3 A schematic diagram of the structure of the first convolutional block of the encoder;

[0018] Figure 5 yes Figure 3 A schematic diagram of the structure of the second convolutional block in the encoder;

[0019] Figure 6 yes Figure 3 A schematic diagram of the decoder structure;

[0020] Figure 7 This is a schematic flowchart of a speech detection method provided according to an embodiment of the present disclosure;

[0021] Figure 8This is a schematic diagram of the structure of a training device for a speech detection model provided according to an embodiment of this disclosure;

[0022] Figure 9 This is a schematic diagram of the structure of a voice detection device provided according to an embodiment of the present disclosure;

[0023] Figure 10 This is a block diagram of an electronic device used to implement the speech detection model training method and speech detection method of the embodiments of this disclosure. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] This disclosure provides a method for training a speech detection model, a speech detection method, an apparatus, and a device. Specifically, the speech detection model training method and speech detection method of this disclosure can be executed by an electronic device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart voice interaction device, smart home appliance, wearable smart device, aircraft, smart vehicle terminal, etc. The terminal can also include a client, which can be an audio client, video client, browser client, instant messaging client, or mini-program, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0026] In speech silence detection, the sampling rate varies, including 8kbps, 16kbps, 24kbps, and 48kbps. The length of the detected speech chunks should not be too long or too short, typically 10 / 20 / 30ms. If the chunk is too long, the detection granularity is too coarse, leading to inaccuracy and missed detections. If the chunk is too short, insufficient context can cause classification errors and reduced accuracy. Furthermore, VADs are usually deployed on the terminal side and need to be lightweight and efficient, such as considering the size of the speech detection model.

[0027] A voice stream of <<2MB and 30+ms needs to be processed within 1ms, using only one thread. Therefore, VAD, as the front-end processing module for voice, requires extremely high accuracy and generalization; otherwise, mis-segmentation or missed segmentation may occur.

[0028] In addition, related technologies employ energy-based VAD and Gaussian mixture model (GMM)-based VAD, but both of these methods have poor precision and recall.

[0029] To address at least one of the aforementioned problems, the speech detection model training method, speech detection method, apparatus, and device provided in this disclosure involve: acquiring a speech sample set, which includes sample features and annotation information of the speech samples; inputting the sample features into the encoder of the speech detection model to obtain intermediate features, wherein the dimension of the intermediate features is smaller than the dimension of the sample features; inputting the intermediate features into the decoder of the speech detection model to obtain the predicted confidence of the speech samples, wherein the predicted confidence is used to characterize the probability that the speech sample is not silent; training the speech detection model based on the annotation information and the predicted confidence to obtain a trained speech detection model. The trained speech detection model can be used for speech silence detection, and since the encoder can reduce the dimension of the output intermediate features, the processing speed of the speech detection model can be improved, enabling the speech detection model to be used not only for offline speech detection but also for streaming speech detection, with good real-time performance, and capable of meeting various scenarios.

[0030] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 This is a schematic diagram illustrating the training method of the speech detection model and the system structure of the speech detection method applied in this disclosure. Please refer to... Figure 1 The system includes a terminal 110 and a server 120, etc.; the terminal 110 and the server 120 are connected via a network, such as a wired or wireless network.

[0031] The server 120 can be used to acquire a speech sample set, which includes the sample features and annotation information of the speech samples. The sample features are input into the encoder of the speech detection model to obtain intermediate features, where the dimensionality of the intermediate features is smaller than that of the sample features. The intermediate features are input into the decoder of the speech detection model to obtain the predicted confidence score of the speech sample, which represents the probability that the speech sample is not silent. The speech detection model is trained based on the annotation information and the predicted confidence score to obtain a trained speech detection model. The terminal 110 can be used to display a graphical user interface. This terminal is used to interact with the user through the graphical user interface, such as downloading and installing a corresponding client, running a corresponding app, or displaying a graphical user interface when logging into a website. In this embodiment of the present disclosure, the terminal 110 may be equipped with an audio processing application, and obtain the current speech segment in the speech stream to be inspected through the audio processing application; determine the speech features of the current speech segment; input the speech features into the trained speech detection model to obtain the confidence level of the current speech segment; and determine the first label of the current speech segment based on the confidence level of the current speech segment. The first label is used to determine the position of the non-silent segment in the speech stream to be inspected.

[0032] It should be noted that although the example uses an audio processing application, those skilled in the art should understand that the application can be other suitable programs such as audio playback applications. Furthermore, the application can be installed on a desktop computer, a mobile device, or a mini-program embedded within an application.

[0033] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. On the contrary, the embodiments of this disclosure can be applied to any applicable scenario.

[0034] The following is a detailed description. It should be noted that the order of description of the following embodiments is not intended to limit the priority of the embodiments.

[0035] Figure 2 This is a flowchart illustrating the training method for a speech detection model provided according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of the speech detection model provided according to an embodiment of this disclosure; please refer to... Figure 2 and Figure 3 This disclosure provides a training method 200 for a speech detection model, including the following steps S201 to S204.

[0036] Step S201: Obtain a speech sample set, which includes the sample features of the speech samples and the annotation information of the speech samples.

[0037] Step S202: Input the sample features into the encoder of the speech detection model to obtain intermediate features, and the dimension of the intermediate features is smaller than the dimension of the sample features.

[0038] Step S203: Input the intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech sample. The prediction confidence is used to characterize the probability that the speech sample is not silent.

[0039] Step S204: Train the speech detection model based on the annotation information and prediction confidence to obtain the trained speech detection model.

[0040] A speech sample set can include multiple speech samples, each of which can include sample features and annotation information. Speech samples can be speech segments, and their features can be preprocessed features that can be recognized by a speech detection model. For example... Figure 3 As shown, speech samples can be processed through feature preprocessing and normalization to obtain their sample features. The annotation information for a speech sample can be the true probability that the speech sample is non-silent, for example, 1 or 0.

[0041] The speech detection model is a model built on a neural network, such as... Figure 3 The speech detection model includes an encoder and a decoder.

[0042] The encoder is primarily used to extract high-level audio features from speech samples. It processes the sample features to obtain intermediate features, which are the high-level audio features of the speech samples. Additionally, the encoder can perform downsampling (dimensionality reduction) on the features, making the dimensionality of the intermediate features smaller than that of the sample features. Dimensionality refers to the length of the feature, thus reducing the computational cost of the model.

[0043] The decoder is mainly used to obtain the prediction confidence of a speech sample based on its high-level audio features. The prediction confidence is the probability that the speech sample is not silent, and it can be a value greater than 0 or less than 1.

[0044] The loss function of the speech detection model is calculated using labeled information and prediction confidence. If the loss function does not meet the preset convergence condition, the parameters of the speech detection model are adjusted, such as the encoder and decoder parameters, and steps S202 and S203 are repeated until the loss function meets the preset convergence condition. After the loss function meets the preset convergence condition, the trained speech detection model is obtained. The trained speech detection model can detect the speech stream to be tested, thereby determining the confidence of each speech segment in the speech stream, i.e., the probability that the speech segment is non-silent, and thus determining the location of non-silent segments in the speech stream to be tested.

[0045] This embodiment sets up an encoder and a decoder in the speech detection model, so that the trained speech detection model can be used to detect silence. The encoder can reduce the dimensionality of features, thereby improving the speed of the speech detection model. This makes the speech detection model not only suitable for offline speech detection, but also for streaming speech detection. It has good real-time performance and can meet the needs of various scenarios.

[0046] In some embodiments, the encoder includes N first convolutional blocks connected in sequence, where N is a positive integer greater than or equal to 2; and among the N first convolutional blocks, the remaining first convolutional blocks except for the Nth first convolutional block are used to reduce the dimensionality of the sample features.

[0047] In this embodiment, the encoder may include multiple first convolutional blocks. Sample features can be used as the input of the first first convolutional block, and the output features of the first convolutional block can be used as the input of the second first convolutional block, thereby passing through N first convolutional blocks in sequence.

[0048] It is understandable that, except for the last of the N first convolutional blocks, the dimension of the input features of the other first convolutional blocks is greater than the dimension of the output features. This allows the encoder to reduce the dimension of the sample features, thereby downsampling the sample features and reducing the computational cost of the model.

[0049] The input and output features of the last convolutional block can have the same dimension, thus preserving more complete feature data.

[0050] Figure 4 yes Figure 3 Please refer to the structural diagram of the first convolutional block of the encoder. Figure 4In some embodiments, each of the N first convolutional blocks includes: a first separable convolutional layer, a first residual connection layer, and a first normalization layer; the first separable convolutional layer is used to perform first separable convolution processing on the input features of the first convolutional block to obtain a first feature; the first residual connection layer is used to process the first feature to obtain a second feature, and fuse the second feature with the first feature to obtain a third feature; the first normalization layer is used to normalize the third feature.

[0051] It is understandable that each of the N first convolutional blocks has the same structure, and each can include a first separable convolutional layer, a first residual connection layer, and a first normalization layer.

[0052] The first separable convolutional layer can include depthwise convolution (dw_conv) and pointwise convolution (pw_conv). For example... Figure 4 As shown, dw_conv consists of a 1D convolutional layer (Conv1d), an equivalent (Identity) layer, and a rectified linear activation function (ReLU) layer from bottom to top. pw_conv also consists of a 1D convolutional layer (Conv1d) and an equivalent (Identity) layer from bottom to top. The input features of the first convolutional block can be input into dw_conv and pw_conv from bottom to top, thus performing separable convolution processing to obtain the first feature. By using separable convolution at the beginning of the first convolutional block, the computational cost of the model can be further reduced.

[0053] The name of the first residual connection layer can be... Figure 4 In this context, `proj` (project) can include one-dimensional convolutional (Conv1d) layers. We can understand this as follows: `x0` represents the first feature; after `x0` is input into the one-dimensional convolutional (Conv1d) layer, we obtain the residual, which is the second feature. The output of the first residual connection layer is `x0 + residual`, which is the third feature. By using a residual connection after the first separable convolutional layer, gradient propagation can be facilitated.

[0054] The first normalization layer, from bottom to top, can consist of a one-dimensional convolutional (Conv1d) layer, a one-dimensional batch normalization (Batch_Normalization 1d, BatchNorm1d) layer, and a rectified linear unit (ReLU) layer, named encoder.0, encoder.1, and encoder.2, respectively. The first normalization layer normalizes the output distribution of the first convolutional block.

[0055] It's understandable that a one-dimensional convolutional (Conv1d) layer exists simultaneously in the first separable convolutional layer, the first residual connection layer, and the first normalization layer, but its parameters and settings can be the same or different, depending on the specific situation. Similarly, each Corrected Linear Unit (ReLU) layer is an activation function layer, and its parameters and settings can also be set according to the actual situation. In a neural network, the activation function layer is responsible for transforming the weighted input from a node into the activation of that node or output. The ReLU layer is a piecewise linear function; if the input is positive, it will output directly; otherwise, it will output zero. Using ReLU layers makes the model easier to train and usually achieves better performance.

[0056] In some embodiments, an activation function layer and a dropout layer may be added after the first residual connection layer. The activation function layer may also be a ReLU layer. The dropout layer can reduce overfitting. The output x2 of the dropout layer can be used as the input feature of the first normalization layer.

[0057] like Figure 4 As shown, in terms of the structure of the first convolutional block, it can include a first layer and a first normalization layer. The first layer can further include a convolution block (ConvBlock) and a Dropout layer. The convolution block can include dw_conv, pw_conv, proj, and activation function layers.

[0058] In some embodiments, the first normalization layer is further used to perform a first convolution process on the third feature; and the stride of the first convolution process in the remaining first convolution blocks is a positive integer greater than 1.

[0059] It is understandable that the first normalization layer includes a one-dimensional convolutional (Conv1d) layer. Except for the last first convolutional block, the stride of the one-dimensional convolutional (Conv1d) layer in the other first convolutional blocks can be a positive integer greater than 1, for example, it can be 2. This allows the dimension of the output feature to become half of the input feature, thereby reducing the dimension of the sample feature to achieve downsampling.

[0060] The stride of the one-dimensional convolutional (Conv1d) layer in the first normalization layer of the last first convolutional block can be 1, thus preserving the feature data.

[0061] Figure 5 yes Figure 3 A schematic diagram of the structure of the second convolutional block in the encoder; please refer to... Figure 5In some embodiments, the encoder further includes a second convolutional block connected between the (N-1)th first convolutional block and the Nth first convolutional block; the second convolutional block includes a second separable convolutional layer, a second residual connection layer, and a second normalization layer; the second separable convolutional layer is used to perform second separable convolution processing on the input features of the second convolutional block to obtain a fourth feature; the second residual connection layer is used to process the input features of the second convolutional block to obtain a fifth feature, and fuse the fourth feature and the fifth feature to obtain a sixth feature; the second normalization layer is used to normalize the sixth feature.

[0062] The second convolutional block can be located before the last first convolutional block. In some embodiments, the encoder includes three first convolutional blocks, which can be first convolutional block 1, first convolutional block 2, and first convolutional block 3 in sequence. Sample features can pass through first convolutional block 1, first convolutional block 2, second convolutional block, and first convolutional block 3 in sequence. The structures of first convolutional block 1, first convolutional block 2, and first convolutional block 3 can be referenced. Figure 4 The difference is that the stride of Conv1d in the first normalization layer of the first convolutional block 1 and the first convolutional block 2 is 2, while the stride of Conv1d in the first normalization layer of the first convolutional block 3 is 1.

[0063] The second convolutional block includes a second separable convolutional layer, a second residual connection layer, and a second normalization layer.

[0064] like Figure 5 As shown, the second separable convolutional layer includes dw_conv and pw_conv. dw_conv, from bottom to top, consists of a one-dimensional convolutional (Conv1d) layer, an equivalent (Identity) layer, and a rectified linear unit (ReLU) layer. pw_conv, from bottom to top, consists of a one-dimensional convolutional (Conv1d) layer and an equivalent (Identity) layer. The input feature x of the second convolutional block can pass through dw_conv and pw_conv sequentially from bottom to top, thus undergoing separable convolution processing to obtain the fourth feature. By using separable convolution at the beginning of the second convolutional block, the computational cost of the model can be further reduced.

[0065] The name of the second residual connection layer can be... Figure 5In the model `proj`, which can include one-dimensional convolutional (Conv1d) layers, the residual connection layers in the first and second convolutional blocks are positioned differently. In the second convolutional block, `x` represents the input feature. After `x` is input into the one-dimensional convolutional (Conv1d) layer in the second residual connection layer, a fifth feature is obtained. Furthermore, the fourth and fifth features can be fused, for example, by adding them to obtain a sixth feature, which can then be used as the input to the second normalization layer. By using residual connections after the second separable convolutional layer, gradient propagation can be promoted, which is beneficial for model convergence.

[0066] The second normalization layer, from bottom to top, can consist of a one-dimensional convolutional (Conv1d) layer, a one-dimensional batch normalization (BatchNormalization 1d, BatchNorm1d) layer, and a rectified linear unit (ReLU) layer, named encoder.0, encoder.1, and encoder.2, respectively. The second normalization layer normalizes the output distribution of the first convolutional block.

[0067] It's understandable that a one-dimensional convolutional (Conv1d) layer exists simultaneously in the second separable convolutional layer, the first residual connection layer, and the second normalization layer, but its parameters and settings can be the same or different, depending on the specific situation. Similarly, each Corrected Linear Unit (ReLU) layer is an activation function layer, and its parameters and settings can also be set according to the actual situation. ReLU layers make the model easier to train and usually achieve better performance.

[0068] In some embodiments, an activation function layer and a dropout layer can be added after the second residual connection layer. The activation function layer can also be a ReLU layer. The dropout layer can reduce overfitting. The sixth feature can pass through the activation function layer and the dropout layer in sequence. The output x2 of the dropout layer can be used as the input feature of the first normalization layer.

[0069] like Figure 5 As shown, the second convolutional block can include a convolutional block (ConvBlock), a Dropout layer, and a second normalization layer. The convolutional block can include dw_conv, pw_conv, proj, and an activation function layer.

[0070] Figure 6 yes Figure 3 A schematic diagram of the decoder structure; please refer to... Figure 6In some embodiments, step S203, which involves inputting intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech sample, may include: inputting intermediate features into the recurrent neural network of the decoder to obtain multiple probability values ​​of the speech sample, wherein the speech sample includes multiple speech frames, and the multiple probability values ​​are the probabilities of the multiple speech frames; averaging the multiple probability values ​​to obtain the prediction confidence.

[0071] Encoders can include recurrent neural networks, such as Long Short-Term Memory (LSTM) networks. Figure 6 As shown, in one possible embodiment, the decoder may sequentially include an LSTM, a first activation function layer (ReLU), and a linear layer (such as...). Figure 6 The layers include the Cov1d layer, the second activation function layer (Sigmoid), and the mean layer.

[0072] It is understandable that speech samples can be obtained through a preset sampling rate, which can include multiple sampling points. Frame extraction from these multiple sampling points (which can be achieved through downsampling by the encoder) yields multiple speech frames. That is, the intermediate features output by the encoder are sequentially processed through LSTM, ReLU, Linear, and Sigmoid to obtain multiple probability values. These probability values ​​represent the probabilities of non-silence in the multiple speech frames of the speech sample. An averaging layer is then used to obtain the mean of these multiple probability values, which is then used as the prediction confidence. By combining the probability values ​​from multiple speech frames of the speech sample, the prediction confidence is determined, resulting in a more accurate outcome and improving the precision of the speech detection model and speech detection.

[0073] In some embodiments, method 200 may further include: extracting spectral features of speech samples; normalizing the spectral features to obtain first input features; and concatenating the spectral features and the first input features to obtain sample features.

[0074] like Figure 2As shown, speech samples can be preprocessed using Features. A speech sample can be multiple sampling points obtained using a preset sampling rate. The duration of these sampling points should be the preset speech segment length (chunk_size). When collecting samples, an audio sample can be segmented according to the preset speech segment length to obtain multiple speech samples. However, since the length of an audio sample is variable, the length of the last speech sample may be less than the preset speech segment length. In this case, padding can be performed on the multiple sampling points of the speech sample to make the length of the speech sample equal to the preset speech segment length. For example, if the preset speech segment length is 30ms, the corresponding number of sampling points is 600. If the number of sampling points in the input speech sample is 599, then zeros need to be added after the 599 sampling points, that is, padding the 599 sampling points to make them the preset speech segment length of 600 sampling points.

[0075] Then, the short-time Fourier transform (STFT) can be used to extract the spectral features of the speech samples.

[0076] Next, the spectral features can be normalized using normalization. For example, the spectral features can be processed in the log domain and normalized (e.g., by subtracting the mean, dividing by the variance, etc.) to obtain the first input feature. Then, the spectral feature and the first input feature are concatenated to form the sample feature. For example, if the spectral feature is 'a' and the first input feature is 'b', the concatenated sample feature is [a, b].

[0077] By using the sample features generated by concatenating the spectral features and the first input features as the input features of the encoder, feature enhancement can be achieved, the richness of features can be improved, and the accuracy of the speech detection model can be improved.

[0078] In some embodiments, method 200 may further include: obtaining multiple speech samples for constructing a speech sample set based on a preset sampling rate and multiple preset sampling durations.

[0079] It's understandable that the preset sampling rate is the applicable sampling rate for the speech detection model. This means that all audio input to the speech detection model is obtained at the preset sampling rate, and there's a direct correlation between the speech detection model and the preset sampling rate. Audio obtained at other sampling rates is not suitable for the speech detection model corresponding to this preset sampling rate, and the sampling rate needs to be adjusted to the preset sampling rate.

[0080] Multiple preset sampling durations correspond to multiple preset speech segment lengths, meaning that when selecting speech samples, samples of various chunk sizes can be chosen as the speech sample set. For example, according to the Bernoulli distribution, data of different chunk sizes can be randomly selected for training. For instance, 50% of the speech samples in the speech sample set could be selected with a chunk size of 30ms, 30% with a chunk size of 60ms, and 20% with a chunk size of 100ms. Alternatively, the number of speech samples with chunk sizes of 30ms, 60ms, or 100ms can be randomly selected to construct the speech sample set.

[0081] By training the speech detection model with multiple chunk-size speech samples, the processing power of the speech detection model can be improved, thus enhancing its performance.

[0082] In some embodiments, method 200 may further include: acquiring sample audio; generalizing the sample audio by adding background noise processing to obtain a speech sample set.

[0083] To increase the number of speech samples, each acquired audio sample can be generalized to generate multiple speech samples. Generalization processing can include background noise enhancement. Background noise enhancement can increase data diversity through speech enhancement methods such as noise addition, reverberation addition, and audio encoding / decoding. Additionally, large datasets can be collected to obtain speech samples from multiple languages, speakers, and scenarios.

[0084] Generalization can achieve sample diversity, which helps improve the accuracy of speech detection model results.

[0085] In some embodiments, method 200 may further include: compressing the trained speech detection model to reduce the storage space of the trained speech detection model; and pruning the library of the trained speech detection model to remove some operators in the model inference library that were not involved in the training process of the speech detection model.

[0086] The trained speech detection model can be compressed using Post-Training Quantization (PTQ), for example, by compressing the model to one-quarter of its original size. This reduces the model's storage space and computational cost. Overall, the compressed speech detection model is approximately 1.8MB in size and can include both a first speech detection model suitable for an 8k sampling rate and a second speech detection model suitable for a 16k sampling rate.

[0087] Additionally, the runtime library (model inference library) can be tailored based on the computational operators (kernels) used by the model. It can be understood that the model inference library provides a variety of operators, such as different operators for one-dimensional, two-dimensional, or three-dimensional data. In speech detection, audio is usually one-dimensional data. Therefore, one-dimensional operators involved in the training and use of the speech detection model can be retained, while two-dimensional and three-dimensional operators can be removed, thereby reducing the size of the model inference library and saving storage space.

[0088] By compressing and pruning the trained speech detection model, the storage space of the speech detection model can be reduced, making it more suitable for terminal devices such as mobile phones and achieving lightweight design.

[0089] Figure 7 This is a schematic flowchart of the speech detection method provided according to embodiments of this disclosure; please refer to... Figure 7 This disclosure also provides a speech detection method 700, including the following steps S701 to S704.

[0090] Step S701: Obtain the current speech segment in the speech stream to be inspected.

[0091] Step S702: Determine the speech features of the current speech segment.

[0092] Step S703: Input the speech features into the trained speech detection model to obtain the confidence level of the current speech segment. The trained speech detection model is trained according to the training method of the speech detection model in any of the above embodiments.

[0093] Step S704: Determine the first label of the current speech segment based on the confidence level of the current speech segment. The first label is used to determine the location of non-silent segments in the speech stream to be inspected.

[0094] The speech stream to be inspected can be real-time speech or a recording. Step S701 can obtain the current speech segment in the speech stream to be inspected. It can be understood that the length of the current speech segment is a certain preset speech segment length. During the training of the speech detection model, samples of various chunk_sizes can be selected for training. During the use of the speech detection model, one of the chunk_sizes can be set as the preset speech segment length, and the speech stream to be inspected can be segmented according to the preset speech segment length to obtain the current speech segment.

[0095] Step S702 can obtain the speech features of the current speech segment, in the same way as obtaining the sample features of the speech samples. Alternatively, the current speech segment can be preprocessed using the features; for example, if the number of sampling points is less than the preset speech segment length (chunk_size), padding can be performed on multiple sampling points to make the length of the current speech segment equal to the preset speech segment length. Then, a short-time Fourier transform can be used to extract the first spectral features of the current speech segment.

[0096] Next, the first spectral feature can be normalized using normalization. For example, the first spectral feature can be processed in the log domain and normalized (e.g., by subtracting the mean, dividing by the variance, etc.) to obtain the second input feature. Then, the first spectral feature and the second input feature are concatenated to form the speech feature.

[0097] Step S703 inputs the speech features into the trained speech detection model to obtain the confidence score of the current speech segment. This confidence score is the probability score that the current speech segment is not silent.

[0098] Step S704 determines the first label of the current speech segment based on its confidence level. It is understood that the speech stream to be inspected may include silent segments and non-silent segments, and both silent and non-silent segments can include one or more speech segments. A silent segment can be understood as a segment without human voice or other valid sound, and is a segment that needs to be removed. A non-silent segment can be understood as a speech segment with human voice or valid sound, and is a segment that needs to be retained. A speech stream to be inspected typically includes one or more non-silent segments, and the number of silent segments can be one or more, or there may be no silent segments. The first label determines the position of the current speech segment within the non-silent segments, for example, it could be the first speech segment (i.e., the start segment) in the non-silent segment, or the last speech segment (i.e., the end segment) in the non-silent segment, or the middle segment between the start and end segments, or a speech segment within a silent segment.

[0099] By determining the first label for each current speech segment in the speech stream to be inspected, the location of non-silent segments can be determined, i.e., the start time (the start time of the start segment) and the end time (the end time of the end segment). This allows for the extraction of non-silent segments from the speech stream, resulting in a speech file with silence segments removed. Since the trained speech detection model has a fast processing speed, the real-time performance of the speech detection method can be improved.

[0100] In some embodiments, determining the first label of the current speech segment based on the confidence level of the current speech segment in step S704 may include: determining the second label of the current speech segment based on the confidence level of the current speech segment, wherein the second label is used to indicate whether the speech segment is silent or not; and determining the first label of the current speech segment based on the second label of the current speech segment.

[0101] For each current speech segment, a second label can be determined based on its confidence level. The second label is either silent or non-silent, which indicates whether the current speech segment is a silent speech segment or a non-silent speech segment.

[0102] Based on the second label of the current speech segment, the first label of the current speech segment can be further determined. By using the first labels of multiple speech segments in the speech stream to be detected, non-silent segments in the speech stream can be identified. Since the trained speech detection model has high generalization and accuracy, the accuracy of non-silent segment detection can be improved based on its obtained confidence level.

[0103] In some embodiments, determining a second label for a current speech segment based on the confidence level of the current speech segment includes: determining that the second label for the current speech segment is non-silent if the confidence level of the current speech segment is greater than a first confidence threshold; and determining that the second label for the current speech segment is silent if the confidence level of the current speech segment is less than or equal to the first confidence level.

[0104] In determining the second label based on confidence level, a first confidence threshold can be set. If the confidence level of the current speech segment is greater than the first confidence threshold, then the second label of the current speech segment is determined to be non-silent. Conversely, if the confidence level is less than the first confidence threshold, then the second label of the current speech segment is determined to be silent. The first confidence threshold can be 0.8, 0.9, etc.

[0105] The second label of the current speech segment can be determined by using the first confidence threshold.

[0106] It is understandable that the determination of the second label can rely on a confidence threshold, such as the first confidence threshold. In other embodiments, multiple confidence thresholds can also be used to determine the second label. The following explanation uses the determination of the second label based on two confidence thresholds as an example.

[0107] In some embodiments, determining a second label for the current speech segment based on the confidence level of the current speech segment may include the following steps one to three.

[0108] Step 1: If the confidence level of the current speech segment is greater than the second confidence threshold and the confidence level of the next speech segment is greater than the third confidence threshold, determine that the second label of the current speech segment is non-silent, where the second confidence threshold is less than the third confidence threshold.

[0109] Step 2: If the confidence level of the current speech segment is less than or equal to the second confidence threshold, determine the second label of the current speech segment as silence.

[0110] Step 3: If the confidence level of the current speech segment is greater than the second confidence threshold and the confidence level of the next speech segment is less than or equal to the third confidence threshold, determine the second label of the current speech segment as silence.

[0111] It is understood that this embodiment sets a second confidence threshold and a third confidence threshold, with the second confidence threshold being less than the third confidence threshold. For example, the second confidence threshold is 0.6 and the third confidence threshold is 0.9.

[0112] If the confidence level of the current speech segment is greater than the second confidence level threshold, and the confidence level of the next speech segment is greater than the third confidence level threshold, it indicates that the second label of the current speech segment is non-silent, thus accurately determining the second label of the current speech segment and avoiding misjudgments caused by noise and other interference.

[0113] If the confidence level of the current speech segment is less than or equal to the second confidence threshold, it indicates that the second label of the current speech segment is silence.

[0114] If the confidence level of the current speech segment is greater than the second confidence threshold and the confidence level of the next speech segment is less than or equal to the third confidence threshold, the second label of the current speech segment can also be determined to be silent. That is, the confidence level of the current speech segment may be too high due to noise and other interference, but in this case, the second label of the current speech segment is still silent.

[0115] By setting two confidence thresholds, the second label of each speech segment can be judged more accurately, which in turn improves the accuracy of non-silent segment recognition.

[0116] In some embodiments, determining the first label of the current speech segment based on the second label of the current speech segment may include: determining the first label of the current speech segment as the starting segment of a non-silent segment when the second label of the previous speech segment is silent, the second label of the current speech segment is non-silent, and the second labels of the X consecutive speech segments following the current speech segment are all non-silent, where X is a preset value and X is a positive integer greater than or equal to 1.

[0117] It is understandable that after determining the second label of several consecutive speech segments, the non-silent segments of the speech stream to be inspected can be judged.

[0118] If, after the start of a non-silent segment, the second label of the previous speech segment is silent, while the second label of the current speech segment is non-silent, and the second label of X consecutive speech segments following the current speech segment is also non-silent, then the current speech segment indicates that it is the starting segment of a non-silent segment; that is, the first label of the current speech segment indicates the starting segment of a non-silent segment. Here, X can be a positive integer, which can be a preset value. For example, it can be set to 1, 2, 3, or 4. X can be determined by the maximum start duration parameter (Max_start_duration), which indicates that if multiple consecutive speech segments exceed the first confidence threshold, then the current speech segment is the starting segment of a non-silent segment. Max_start_duration can be equal to X plus 1.

[0119] By determining whether the current speech segment is the start of a non-silent segment by identifying the second label of X consecutive speech segments following it as non-silent, misjudgments due to noise or other interference can be avoided. For example, if the second label of the current speech segment is non-silent, but the following few speech segments are silent, the current speech segment is considered a silent segment. However, if the second labels of the speech segments following the current speech segment are not considered, this situation could potentially be misjudged as a non-silent segment.

[0120] In some embodiments, determining the first label of the current speech segment based on the second label of the current speech segment may further include: determining the current speech segment as the end segment of a non-silent segment when the start segment of a non-silent segment has been detected before the current speech segment, the second label of the current speech segment is non-silent, and the second labels of the Y consecutive speech segments after the current speech segment are all silent, wherein Y is a preset value and Y is a positive integer greater than or equal to 1.

[0121] After the start of a non-silent segment has been detected, if the second label of the current speech segment is non-silent, and the second labels of the following Y consecutive speech segments are all silent, then the current speech segment is determined to be the end of a non-silent segment. After determining the start and end of the non-silent segment, the position of the entire non-silent segment can be determined.

[0122] Where Y can be a positive integer, and it can be a preset value. For example, it can be set to 1, 2, 3, or 4. Y can be determined by the minimum silence duration parameter (Min_silence_duaration), which indicates that if multiple second labels of silence are detected consecutively after the start of a non-silent segment, the number of second labels is equal to or greater than Y, then the first label of the current speech segment is the end of a non-silent segment.

[0123] Max_start_duration can be equal to Y.

[0124] Additionally, if multiple silent second tags are detected consecutively after the start of a non-silent segment, and the number is less than Y, it indicates that the current speech segment and the subsequent speech segments with silent second tags are still in a non-silent segment, and the non-silent segment has not ended, meaning that the current speech segment is located in the middle of a non-silent segment.

[0125] By setting Min_silence_duaration, the end of a non-silent segment can be accurately determined, thus pinpointing the location of the entire non-silent segment.

[0126] In some embodiments, determining the first label of the current speech segment based on the second label of the current speech segment further includes: if the start segment of a non-silent segment has been detected before the current speech segment and the end segment of a non-silent segment has not been detected, determining the first label of the current speech segment as the middle segment of a non-silent segment.

[0127] Method 700 may further include: lowering the confidence threshold used to determine the second label of the current speech segment when the first label of the current speech segment is a non-silent segment in the middle.

[0128] It is understandable that if the start segment of the current speech segment has been detected, but the end segment has not been detected, the first label of the current speech segment can be determined as the middle segment of a non-silent segment.

[0129] Additionally, after detecting an intermediate segment, the confidence threshold used to determine the second label of the current speech segment can be lowered. For example, by setting a lowering parameter (Beam), taking the first confidence threshold as an example, when detecting the beginning of a non-silent segment, the second label of the current speech segment can be determined by the first confidence threshold, for example, the first confidence threshold can be T. This threshold can be slightly higher to avoid false warnings. After detecting the beginning of a non-silent segment, the first confidence threshold can be lowered to T minus Beam. The lowered first confidence threshold is then used to determine the second label of the current speech segment and subsequent speech segments, i.e., to determine whether it is consistently within a non-silent segment, thus avoiding misclassifying low-confidence speech segments within a non-silent segment as silent. It can be understood that after the non-silent segment ends, the first confidence threshold can be restored to T.

[0130] It is understandable that, except for the first label being the start segment, end segment, or middle segment, in all other cases, the first label is the silent segment of the silent segment.

[0131] In some embodiments, method 700 further includes: shifting the start time of the start segment forward by a first preset time period, and using the shifted start time of the start segment as the start time of the non-silent segment; shifting the end time of the end segment forward by a second preset time period, and using the shifted end time of the end segment as the end time of the non-silent segment.

[0132] The first preset time period (Shift left ms, left offset time period) and the second preset time period (Shift right ms, right offset time period) can be set according to the actual situation.

[0133] It is understandable that detecting the beginning of a non-silent segment indicates that the non-silent segment has started, thus the real-time performance of speech detection can be ensured through a first preset time period. The same principle applies to the second preset time period, which can further improve the accuracy of non-silent segment location.

[0134] In some embodiments, obtaining the current speech segment in the speech stream to be inspected in step S701 may include: if the sampling rate of the speech stream to be inspected is greater than a first preset sampling rate, performing downsampling processing on the speech stream to be inspected so that the sampling rate of the speech stream to be inspected becomes the first preset sampling rate; and obtaining the current speech segment in the speech stream to be inspected according to the first preset sampling rate.

[0135] In step S703, the speech features are input into the trained speech detection model to obtain the confidence of the current speech segment. This includes: when the sampling rate of the speech stream to be detected is a first preset sampling rate, the speech features are input into the first speech detection model to obtain the confidence of the current speech segment. The first speech detection model is a trained speech detection model, and the applicable sampling rate of the first speech detection model is the first preset sampling rate.

[0136] The current speech segment can be acquired based on a sampling rate, collecting a speech segment of a preset length as the current speech segment. Common sampling rates are 8k, 16k, 24k, 48k, etc. The first preset sampling rate can be 8k or 16k, etc.

[0137] It is understandable that since the speech detection model uses speech samples with a fixed sampling rate during the training process, the speech detection model can be trained using speech samples obtained with a first preset sampling rate to obtain a trained first speech detection model. The applicable sampling rate of the first speech detection model is the first preset sampling rate.

[0138] If the first preset sampling rate is 16k, in order to apply the first speech detection model, if the sampling rate of the current speech segment is greater than 16k, the sampling rate of the speech stream to be detected can be reduced by downsampling to make it the first preset sampling rate.

[0139] Then, the first speech detection model can be used to predict the speech features of the current speech segment. This allows the speech detection method to be used for speech streams with various sampling rates, improving its versatility and meeting the needs of multiple scenarios.

[0140] In some embodiments, step S703, which involves inputting speech features into a trained speech detection model to obtain the confidence level of the current speech segment, may further include: when the sampling rate of the speech stream to be detected is lower than a second preset sampling rate (which is lower than a first preset sampling rate), inputting speech features into a second speech detection model to obtain the confidence level of the current speech segment. The second speech detection model is a trained speech detection model, and the applicable sampling rate of the second speech detection model is the second preset sampling rate.

[0141] It is understood that in this embodiment, two speech detection models can be set, namely a first speech detection model and a second speech detection model. Both of these speech detection models can be trained by the training method of the speech detection model in any of the above embodiments.

[0142] The first speech detection model uses speech samples collected at a first preset sampling rate during training, while the second speech detection model uses speech samples collected at a second preset sampling rate during training. Therefore, the applicable sampling rate for both the first and second speech detection models is the first preset sampling rate. The first preset sampling rate is greater than the second preset sampling rate. For example, the first preset sampling rate is 16k, and the second preset sampling rate is 8k.

[0143] When the sampling rate of the speech stream to be detected is greater than 16k, it can be downsampled to 16k, and then the speech features of the obtained speech segment can be input into the first speech detection model for prediction.

[0144] When the sampling rate of the speech stream to be detected is 8k, the current speech segment can be directly obtained, and the speech features of the current speech segment can be input into the second speech detection model for prediction.

[0145] For human hearing, 16k is sufficient to represent most speech features. This embodiment, by setting two speech detection models, can cover most speech detection sampling rates, thus having better versatility.

[0146] In one specific embodiment, a lightweight speech detection model and a speech detection method are provided.

[0147] 1. Lightweight speech detection model

[0148] This lightweight speech detection model is mainly divided into four parts, with the following structure: Figure 3 As shown, it includes Feature, Normalization, Encoder, and Decoder.

[0149] (1) Feature: Preprocess the input data (current speech segment) and extract speech features.

[0150] First, downsampling can be performed based on the sampling rate of the input speech stream to be detected and the sampling rate supported by the model (the applicable sampling rate). For example, this lightweight speech detection model can include speech detection models with two sampling rates of 8k and 16k (two models are trained separately).

[0151] Secondly, padding is performed on the input audio sampling points. If the number of sampling points in the input audio (current speech segment) is less than the number of points corresponding to chunk_size, zeros are added to the right.

[0152] Finally, the spectral features of the current audio segment are extracted using STFT.

[0153] (2) Normalization: Normalize the feature data.

[0154] The input spectral features are processed in the log domain and normalized (e.g., by subtracting the mean and dividing by the variance). The input 'a' and the output 'b' are concatenated to form [a, b] and then sent to the next module for processing.

[0155] (3) Encoder: Extracts high-level audio features. The Encoder consists of four convolutional blocks, and its core structure includes the first convolutional block 1, the first convolutional block 2, the second convolutional block, and the first convolutional block 3 in sequence.

[0156] The structures of the first convolutional block 1, the first convolutional block 2, and the first convolutional block 3 are as follows: Figure 4 As shown, the structure of the second convolutional block is as follows: Figure 5 As shown.

[0157] Furthermore, the stride of the last Conv1d layer in the first convolutional block 1 and the first convolutional block 2 is 2, so the model downsamples the length of the input features by 4 times, reducing the computational cost of the model.

[0158] Furthermore, each convolutional block begins with a separable convolution (dw_conv + pw_conv) to further reduce the model's computational cost. Residual connections are then used after the separable convolutions to facilitate gradient propagation.

[0159] The residual connection method of the second convolutional block is different from the others, mainly because the input and output feature lengths are different in the downsampling operations of the first convolutional block 1 and the first convolutional block 2.

[0160] In addition, the Conv1d layer + BatchNorm1d (BN) layer + ReLU layer (encoder.0 + encoder.1 + encoder.2) in each convolutional block can be used to normalize the output distribution of the convolutional block.

[0161] (4) Decoder: Processes the high-level representation and calculates the binary classification probability (confidence): silent / non-silent.

[0162] The decoder contains LSTM, Linear, and Sigmoid, and outputs a probability score (confidence) of silence / non-silence.

[0163] 2. Supports generalization, multisampling, and chunk size.

[0164] (1) Generalization: This is achieved by collecting large datasets (speech sample sets). Speech sample sets contain speech samples of various languages, speakers, and scenarios. The diversity of speech sample data can also be increased through speech enhancement methods such as adding noise, adding reverberation, and audio encoding / decoding.

[0165] (2) Multisampling rate

[0166] This is achieved by setting up a downsampling module within the Feature module. Essentially, a lightweight speech detection model can include two pre-trained speech detection models: one with an 8k sampling rate and the other with a 16k sampling rate. For sampling rates greater than 16k, downsampling is performed, and the 16k model is used for training or prediction. Finally, the 8k and 16k models are merged into a single model, and the choice between using the 16k first speech detection model and the 8k second speech detection model is determined based on the sampling rate of the input speech stream.

[0167] (3) Multiple chunk_size: During training, data of different chunk_sizes are randomly selected according to the Bernoulli distribution (for example, 50% of the speech samples are selected as 30ms, 30% as 60ms, and 20% as 100ms. Or, 30ms, 60ms, and 100ms of data can be randomly selected for training). When using the data, one of the chunk_sizes can be selected to obtain the current speech segment of the speech stream to be examined.

[0168] 3. Lightweight model and efficient deployment

[0169] (1) Model lightweighting: The 8k and 16k models are PTQ (post-training) quantized to compress the model size by 4x and reduce the computational cost. The compressed model size is approximately 1.8MB, and it contains both 8k and 16k speech detection models.

[0170] (2) Library trimming: Trim the size of the runtime library for the computing kernel used by the model to save the overall library size.

[0171] 4. Flow cytometry noise level detection

[0172] The input streaming speech (speech stream to be detected) is collected according to the size of chunk_size and fed into the VAD model (lightweight speech detection model). The VAD model outputs the probability scores (confidence scores) of binary classification.

[0173] Based on the classification, a second label, called a state, can be obtained for each speech segment. By analyzing consecutive state transitions, the start and end points of the speech (non-silent segments) can be determined, thus enabling speech segmentation and the removal of silence.

[0174] The state sequence, known as state alignment, can be used to generate audio timestamps, which are multiple second tags for multiple speech segments.

[0175] The VAD model has the following parameters set to facilitate user control over the VAD's accuracy rate.

[0176] (1) Threshold: After Sigmoid classification, the score (confidence) can be controlled by Threshold (confidence threshold). When the score exceeds Threshold, it is considered that there is non-silence. There can be multiple Thresholds, which can avoid false alarms.

[0177] (2) Max_start_duration: When n(X+1) consecutive speech segments exceed Threshold, the task is the start of a non-silent segment (starting segment).

[0178] (3) Min_silence_duaration: If multiple silent states are detected consecutively after the start of a non-silent segment, but the duration is less than Min_silence_duration, it will still be judged as non-silent.

[0179] (4) Beam: When transitioning from a silent state (silent segment) to a non-silent state (non-silent segment), the threshold is used for judgment. Generally, the threshold is relatively high to avoid false alarms. However, the threshold can be appropriately lowered to determine whether the state remains non-silent after entering a non-silent state, in order to avoid false rejections. The range of this reduction is controlled by beam.

[0180] (5) Shift left ms / Shift right ms: These two parameters can be used to control the duration of the leftward shift of the start and end points of the speech (non-silent segment). This is because when the start point of the speech (non-silent segment) is detected, the actual start point has often already passed.

[0181] The lightweight speech detection model provided in this embodiment is lightweight and efficient, with fast and consistent processing. It boasts strong generalization capabilities, supporting various sampling rates and speech chunk sizes, enabling it to be used in diverse scenarios. Furthermore, due to its lightweight and efficient nature, it can be deployed in multiple scenarios, including low-power edge (terminal) and server-side applications. Moreover, its high processing efficiency allows for streaming silence detection, enabling real-time processing of speech streams, avoiding false alarms and false rejections, and improving accuracy.

[0182] Figure 8 This is a schematic diagram of the structure of a training device for a speech detection model provided according to an embodiment of this disclosure; please refer to... Figure 8 This disclosure provides a training device 800 for a speech detection model, which includes the following units.

[0183] The acquisition unit 801 is used to acquire a speech sample set, which includes the sample features of the speech samples and the annotation information of the speech samples.

[0184] The encoding unit 802 is used to input sample features into the encoder of the speech detection model to obtain intermediate features, and the dimension of the intermediate features is smaller than the dimension of the sample features.

[0185] The decoding unit 803 is used to input intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech sample. The prediction confidence is used to characterize the probability that the speech sample is non-silent.

[0186] Training unit 804 is used to train the speech detection model based on the annotation information and prediction confidence to obtain the trained speech detection model.

[0187] In some embodiments, the encoder 802 includes N first convolutional blocks connected in sequence, where N is a positive integer greater than or equal to 2; and among the N first convolutional blocks, the remaining first convolutional blocks except for the Nth first convolutional block are used to reduce the dimensionality of the sample features.

[0188] In some embodiments, each of the N first convolutional blocks includes: a first separable convolutional layer, a first residual connection layer, and a first normalization layer; the first separable convolutional layer is used to perform a first separable convolutional process on the input features of the first convolutional block to obtain a first feature; the first residual connection layer is used to process the first feature to obtain a second feature, and fuse the second feature with the first feature to obtain a third feature; the first normalization layer is used to normalize the third feature.

[0189] In some embodiments, the first normalization layer is further used to perform a first convolution process on the third feature; and the stride of the first convolution process in the remaining first convolution blocks is a positive integer greater than 1.

[0190] In some embodiments, the encoder 802 further includes a second convolutional block connected between the (N-1)th first convolutional block and the Nth first convolutional block; the second convolutional block includes a second separable convolutional layer, a second residual connection layer, and a second normalization layer; the second separable convolutional layer is used to perform second separable convolution processing on the input features of the second convolutional block to obtain a fourth feature; the second residual connection layer is used to process the input features of the second convolutional block to obtain a fifth feature, and fuse the fourth feature and the fifth feature to obtain a sixth feature; the second normalization layer is used to normalize the sixth feature.

[0191] In some embodiments, the decoding unit 803 is further configured to: input intermediate features into the recurrent neural network of the decoder to obtain multiple probability values ​​of the speech sample, wherein the speech sample includes multiple speech frames, and the multiple probability values ​​are the probabilities of the multiple speech frames respectively; and average the multiple probability values ​​to obtain the prediction confidence.

[0192] In some embodiments, the apparatus 800 further includes: a preprocessing unit for extracting spectral features from speech samples; normalizing the spectral features to obtain first input features; and concatenating the spectral features and the first input features to obtain sample features.

[0193] In some embodiments, the apparatus 800 further includes a sampling unit, configured to acquire multiple speech samples for constructing a speech sample set based on a preset sampling rate and multiple preset sampling durations.

[0194] In some embodiments, the apparatus 800 further includes: a generalization unit for acquiring sample audio; and generalizing the sample audio by adding background noise to obtain a speech sample set.

[0195] In some embodiments, the apparatus 800 further includes: a compression unit for compressing the trained speech detection model to reduce the storage space of the trained speech detection model; and a pruning unit for pruning the library of the trained speech detection model to remove some operators in the model inference library that were not involved in the training process of the speech detection model.

[0196] Figure 9 This is a schematic diagram of the structure of the voice detection device provided according to an embodiment of this disclosure; please refer to... Figure 9 This embodiment provides a voice detection device 900, which includes the following units.

[0197] The segmentation unit 901 is used to obtain the current speech segment in the speech stream to be inspected.

[0198] The determining unit 902 is used to determine the speech features of the current speech segment.

[0199] The prediction unit 903 is used to input speech features into the trained speech detection model to obtain the confidence level of the current speech segment. The trained speech detection model is trained according to the training method of the speech detection model in any of the above embodiments.

[0200] The processing unit 904 determines the first label of the current speech segment based on the confidence level of the current speech segment. The first label is used to determine the location of non-silent segments in the speech stream to be inspected.

[0201] In some embodiments, the processing unit 904 is further configured to: determine a second label for the current speech segment based on the confidence level of the current speech segment, the second label being used to indicate whether the speech segment is silent or not; and determine a first label for the current speech segment based on the second label of the current speech segment.

[0202] In some embodiments, the processing unit 904 is further configured to: determine that the second label of the current speech segment is non-silent if the confidence level of the current speech segment is greater than the first confidence level threshold; and determine that the second label of the current speech segment is silent if the confidence level of the current speech segment is less than or equal to the first confidence level.

[0203] In some embodiments, the processing unit 904 is further configured to: determine that the second label of the current speech segment is non-silent when the confidence level of the current speech segment is greater than a second confidence threshold and the confidence level of the next speech segment is greater than a third confidence threshold, wherein the second confidence threshold is less than the third confidence threshold; determine that the second label of the current speech segment is silent when the confidence level of the current speech segment is less than or equal to the second confidence threshold; and determine that the second label of the current speech segment is silent when the confidence level of the current speech segment is greater than the second confidence threshold and the confidence level of the next speech segment is less than or equal to the third confidence threshold.

[0204] In some embodiments, the processing unit 904 is further configured to: determine that the first label of the current speech segment is the starting segment of a non-silent segment when the second label of the previous speech segment is silent, the second label of the current speech segment is non-silent, and the second labels of X consecutive speech segments after the current speech segment are all non-silent, wherein X is a preset value and X is a positive integer greater than or equal to 1.

[0205] In some embodiments, the processing unit 904 is further configured to: determine that the current speech segment is the end segment of a non-silent segment when the start segment of a non-silent segment has been detected before the current speech segment, the second label of the current speech segment is non-silent, and the second labels of the Y consecutive speech segments after the current speech segment are all silent, wherein Y is a preset value and Y is a positive integer greater than or equal to 1.

[0206] In some embodiments, the processing unit 904 is further configured to: determine that the first label of the current speech segment is the middle segment of a non-silent segment when the start segment of a non-silent segment has been detected before the current speech segment and the end segment of a non-silent segment has not been detected; the device 900 further includes: an adjustment unit configured to lower the confidence threshold for determining the second label of the current speech segment when the first label of the current speech segment is the middle segment of a non-silent segment.

[0207] In some embodiments, the apparatus further includes: an adjustment unit, configured to shift the start time of the start segment forward by a first preset time period and use the shifted start time of the start segment as the start time of the non-silent segment; shift the end time of the end segment forward by a second preset time period and use the shifted end time of the end segment as the end time of the non-silent segment.

[0208] In some embodiments, the segmentation unit 901 is further configured to: perform downsampling processing on the speech stream to be inspected when the sampling rate of the speech stream to be inspected is greater than the first preset sampling rate, so that the sampling rate of the speech stream to be inspected becomes the first preset sampling rate; and obtain the current speech segment in the speech stream to be inspected according to the first preset sampling rate; the prediction unit 903 is further configured to: input the speech features into the first speech detection model when the sampling rate of the speech stream to be inspected is the first preset sampling rate, and obtain the confidence of the current speech segment, wherein the first speech detection model is a trained speech detection model, and the applicable sampling rate of the first speech detection model is the first preset sampling rate.

[0209] In some embodiments, the prediction unit 903 is further configured to: input speech features into a second speech detection model to obtain the confidence level of the current speech segment when the sampling rate of the speech stream to be detected is lower than the second preset sampling rate of the first preset sampling rate, wherein the second speech detection model is a trained speech detection model and the applicable sampling rate of the second speech detection model is the second preset sampling rate.

[0210] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0211] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0212] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0213] This disclosure also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the training method and speech detection method of the speech detection model described in any of the above embodiments.

[0214] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the training method and speech detection method of the speech detection model described in any of the above embodiments.

[0215] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the training method for the speech detection model and the speech detection method described in any of the above embodiments.

[0216] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0217] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0218] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0219] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method for a speech detection model and the speech detection method. For example, in some embodiments, the training method for a speech detection model and the speech detection method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the training method for a speech detection model and the speech detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for the speech detection model and a speech detection method.

[0220] The various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), computed-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can execute on a programmable system including at least one programmable processor and / or...

[0221] Alternatively, the programmable processor may be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0226] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0227] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0228] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a speech detection model, comprising: Obtain a speech sample set, which includes the sample features of the speech samples and the annotation information of the speech samples; The sample features are input into the encoder of the speech detection model to obtain intermediate features, and the dimension of the intermediate features is smaller than the dimension of the sample features; the encoder includes multiple first convolutional blocks; each first convolutional block includes: a first separable convolutional layer, a first residual connection layer and a first normalization layer; The first separable convolutional layer is used to perform a first separable convolutional process on the input features of the first convolutional block to obtain a first feature; The first residual connection layer is used to process the first feature to obtain the second feature, and then fuse the second feature with the first feature to obtain the third feature; The first normalization layer is used to normalize the third feature; The intermediate features are input into the decoder of the speech detection model to obtain the prediction confidence of the speech sample, which is used to characterize the probability that the speech sample is non-silent. The speech detection model is trained based on the annotation information and the prediction confidence to obtain a trained speech detection model.

2. The method according to claim 1, wherein, The encoder includes N first convolutional blocks connected in sequence, where N is a positive integer greater than or equal to 2; and among the N first convolutional blocks, the remaining first convolutional blocks except for the Nth first convolutional block are used to reduce the dimensionality of the sample features.

3. The method according to claim 2, wherein, The first normalization layer is also used to perform a first convolution process on the third feature; Furthermore, the stride of the first convolution process in the remaining first convolution blocks is a positive integer greater than 1.

4. The method according to any one of claims 1-3, wherein, The encoder further includes a second convolutional block connected between the (N-1)th first convolutional block and the Nth first convolutional block; The second convolutional block includes a second separable convolutional layer, a second residual connection layer, and a second normalization layer; The second separable convolutional layer is used to perform a second separable convolution process on the input features of the second convolutional block to obtain a fourth feature; The second residual connection layer is used to process the input features of the second convolutional block to obtain the fifth feature, and to fuse the fourth feature with the fifth feature to obtain the sixth feature; The second normalization layer is used to normalize the sixth feature.

5. The method according to any one of claims 1-3, wherein, The intermediate features are input into the decoder of the speech detection model to obtain the prediction confidence of the speech sample, including: The intermediate features are input into the recurrent neural network of the decoder to obtain multiple probability values ​​of the speech sample, wherein the speech sample includes multiple speech frames, and the multiple probability values ​​are the probabilities of the multiple speech frames respectively; The prediction confidence level is obtained by averaging the multiple probability values.

6. The method according to any one of claims 1-3, further comprising: Extract the spectral features of the speech samples; The spectral features are normalized to obtain the first input features; The spectral features and the first input features are concatenated to obtain the sample features.

7. The method according to any one of claims 1-3, further comprising: Based on a preset sampling rate and multiple preset sampling durations, multiple speech samples are obtained to construct the speech sample set.

8. The method according to any one of claims 1-3, further comprising: Obtain sample audio; By adding background noise, the sample audio is generalized to obtain the speech sample set.

9. The method according to any one of claims 1-3, further comprising: The trained speech detection model is compressed to reduce its storage space. The trained speech detection model is then pruned to remove operators from the model inference library that were not used during the training of the speech detection model.

10. A speech detection method, comprising: Get the current speech segment in the speech stream to be inspected; Determine the speech features of the current speech segment; The speech features are input into the trained speech detection model to obtain the confidence level of the current speech segment. The trained speech detection model is trained by the method according to any one of claims 1-9. Based on the confidence level of the current speech segment, a first label is determined for the current speech segment. The first label is used to determine the location of non-silent segments in the speech stream to be inspected.

11. The method according to claim 10, wherein, Based on the confidence level of the current speech segment, a first label for the current speech segment is determined, including: Based on the confidence level of the current speech segment, a second label is determined for the current speech segment, which is used to indicate whether the speech segment is silent or not silent; The first tag of the current speech segment is determined based on the second tag of the current speech segment.

12. The method according to claim 11, wherein, Based on the confidence level of the current speech segment, a second label for the current speech segment is determined, including: If the confidence level of the current speech segment is greater than the first confidence threshold, the second label of the current speech segment is determined to be non-silent; If the confidence level of the current speech segment is less than or equal to the first confidence level, the second label of the current speech segment is determined to be silent.

13. The method according to claim 11, wherein, Based on the confidence level of the current speech segment, a second label for the current speech segment is determined, including: If the confidence level of the current speech segment is greater than the second confidence level threshold and the confidence level of the next speech segment of the current speech segment is greater than the third confidence level threshold, then the second label of the current speech segment is determined to be non-silent, wherein the second confidence level threshold is less than the third confidence level threshold. If the confidence level of the current speech segment is less than or equal to the second confidence threshold, the second label of the current speech segment is determined to be silent; If the confidence level of the current speech segment is greater than the second confidence level threshold and the confidence level of the next speech segment is less than or equal to the third confidence level threshold, the second label of the current speech segment is determined to be silent.

14. The method according to any one of claims 11-13, wherein, Determining the first tag of the current speech segment based on the second tag of the current speech segment includes: If the second label of the previous speech segment is silent, the second label of the current speech segment is non-silent, and the second labels of the X consecutive speech segments following the current speech segment are all non-silent, then the first label of the current speech segment is determined to be the starting segment of the non-silent segment, where X is a preset value and X is a positive integer greater than or equal to 1.

15. The method according to claim 14, wherein, Determining the first label of the current speech segment based on the second label of the current speech segment further includes: If the start segment of the non-silent segment has been detected before the current speech segment, the second label of the current speech segment is non-silent, and the second labels of the Y consecutive speech segments after the current speech segment are all silent, then the current speech segment is determined to be the end segment of the non-silent segment, where Y is a preset value and Y is a positive integer greater than or equal to 1.

16. The method according to claim 14, further comprising determining the first tag of the current speech segment based on the second tag of the current speech segment: If the start segment of the non-silent segment has been detected before the current speech segment and the end segment of the non-silent segment has not been detected, the first label of the current speech segment is determined to be the middle segment of the non-silent segment; The method further includes: If the first label of the current speech segment is the middle segment of the non-silent segment, the confidence threshold used to determine the second label of the current speech segment is lowered.

17. The method of claim 15, further comprising: The start time of the starting segment is shifted forward by a first preset time period, and the shifted start time of the starting segment is taken as the start time of the non-silent segment. The end time of the ending segment is shifted forward by a second preset time period, and the end time of the shifted ending segment is taken as the end time of the non-silent segment.

18. The method according to any one of claims 10-13, wherein, Obtain the current speech segment in the speech stream to be inspected, including: If the sampling rate of the speech stream to be tested is greater than the first preset sampling rate, the speech stream to be tested is downsampled so that the sampling rate of the speech stream to be tested becomes the first preset sampling rate. Based on the first preset sampling rate, the current speech segment in the speech stream to be detected is obtained; The speech features are input into a trained speech detection model to obtain the confidence score of the current speech segment, including: When the sampling rate of the speech stream to be detected is a first preset sampling rate, the speech features are input into a first speech detection model to obtain the confidence level of the current speech segment. The first speech detection model is a trained speech detection model, and the applicable sampling rate of the first speech detection model is the first preset sampling rate.

19. The method according to claim 18, wherein, The process of inputting the speech features into a trained speech detection model to obtain the confidence score of the current speech segment also includes: When the sampling rate of the speech stream to be detected is lower than the first preset sampling rate, the speech features are input into the second speech detection model to obtain the confidence level of the current speech segment. The second speech detection model is a trained speech detection model, and the applicable sampling rate of the second speech detection model is the second preset sampling rate.

20. A training device for a speech detection model, comprising: An acquisition unit is used to acquire a speech sample set, the speech sample set including the sample features of the speech samples and the annotation information of the speech samples; The encoding unit is used to input the sample features into the encoder of the speech detection model to obtain intermediate features, and the dimension of the intermediate features is smaller than the dimension of the sample features; the encoder includes a plurality of first convolutional blocks; each first convolutional block includes: a first separable convolutional layer, a first residual connection layer and a first normalization layer; The first separable convolutional layer is used to perform a first separable convolutional process on the input features of the first convolutional block to obtain a first feature; The first residual connection layer is used to process the first feature to obtain the second feature, and then fuse the second feature with the first feature to obtain the third feature; The first normalization layer is used to normalize the third feature; The decoding unit is used to input the intermediate features into the decoder of the speech detection model to obtain the prediction confidence of the speech sample, wherein the prediction confidence is used to characterize the probability that the speech sample is non-silent. The training unit is used to train the speech detection model based on the annotation information and the prediction confidence to obtain a trained speech detection model.

21. The apparatus according to claim 20, wherein, The encoder includes N first convolutional blocks connected in sequence, where N is a positive integer greater than or equal to 2; and among the N first convolutional blocks, the remaining first convolutional blocks except for the Nth first convolutional block are used to reduce the dimensionality of the sample features.

22. The apparatus according to claim 21, wherein, The first normalization layer is also used to perform a first convolution process on the third feature; Furthermore, the stride of the first convolution process in the remaining first convolution blocks is a positive integer greater than 1.

23. The apparatus according to any one of claims 20-22, wherein, The encoder further includes a second convolutional block connected between the (N-1)th first convolutional block and the Nth first convolutional block; The second convolutional block includes a second separable convolutional layer, a second residual connection layer, and a second normalization layer; The second separable convolutional layer is used to perform a second separable convolution process on the input features of the second convolutional block to obtain a fourth feature; The second residual connection layer is used to process the input features of the second convolutional block to obtain the fifth feature, and to fuse the fourth feature with the fifth feature to obtain the sixth feature; The second normalization layer is used to normalize the sixth feature.

24. The apparatus according to any one of claims 20-22, wherein, The decoding unit is also used for: The intermediate features are input into the recurrent neural network of the decoder to obtain multiple probability values ​​of the speech sample, wherein the speech sample includes multiple speech frames, and the multiple probability values ​​are the probabilities of the multiple speech frames respectively; The prediction confidence level is obtained by averaging the multiple probability values.

25. The apparatus according to any one of claims 20-22, further comprising: A preprocessing unit is used to extract the spectral features of the speech samples; The spectral features are normalized to obtain the first input features; The spectral features and the first input features are concatenated to obtain the sample features.

26. The apparatus according to any one of claims 20-22, further comprising: The sampling unit is used to acquire multiple speech samples for constructing the speech sample set according to a preset sampling rate and multiple preset sampling durations.

27. The apparatus according to any one of claims 20-22, further comprising: Generalization unit, used to acquire sample audio; By adding background noise, the sample audio is generalized to obtain the speech sample set.

28. The apparatus according to any one of claims 20-22, further comprising: A compression unit is used to compress the trained speech detection model to reduce the storage space of the trained speech detection model. The pruning unit is used to prune the library of the trained speech detection model to remove some operators in the model inference library that were not involved in the training process of the speech detection model.

29. A voice detection device, comprising: The segmentation unit is used to obtain the current speech segment in the speech stream to be inspected; A determining unit is used to determine the speech features of the current speech segment; The prediction unit is used to input the speech features into the trained speech detection model to obtain the confidence of the current speech segment, wherein the trained speech detection model is trained by the method according to any one of claims 1-9; The processing unit determines a first tag for the current speech segment based on the confidence level of the current speech segment. The first tag is used to determine the location of non-silent segments in the speech stream to be inspected.

30. The apparatus according to claim 29, wherein, The processing unit is also used for: Based on the confidence level of the current speech segment, a second label is determined for the current speech segment, which is used to indicate whether the speech segment is silent or not silent; The first tag of the current speech segment is determined based on the second tag of the current speech segment.

31. The apparatus according to claim 30, wherein, The processing unit is also used for: If the confidence level of the current speech segment is greater than the first confidence threshold, the second label of the current speech segment is determined to be non-silent; If the confidence level of the current speech segment is less than or equal to the first confidence level, the second label of the current speech segment is determined to be silent.

32. The apparatus according to claim 30, wherein, The processing unit is also used for: If the confidence level of the current speech segment is greater than the second confidence level threshold and the confidence level of the next speech segment of the current speech segment is greater than the third confidence level threshold, then the second label of the current speech segment is determined to be non-silent, wherein the second confidence level threshold is less than the third confidence level threshold. If the confidence level of the current speech segment is less than or equal to the second confidence threshold, the second label of the current speech segment is determined to be silent; If the confidence level of the current speech segment is greater than the second confidence level threshold and the confidence level of the next speech segment is less than or equal to the third confidence level threshold, the second label of the current speech segment is determined to be silent.

33. The apparatus according to any one of claims 30-32, wherein, The processing unit is also used for: If the second label of the previous speech segment is silent, the second label of the current speech segment is non-silent, and the second labels of the X consecutive speech segments following the current speech segment are all non-silent, then the first label of the current speech segment is determined to be the starting segment of the non-silent segment, where X is a preset value and X is a positive integer greater than or equal to 1.

34. The apparatus according to claim 33, wherein, The processing unit is also used for: If the start segment of the non-silent segment has been detected before the current speech segment, the second label of the current speech segment is non-silent, and the second labels of the Y consecutive speech segments after the current speech segment are all silent, then the current speech segment is determined to be the end segment of the non-silent segment, where Y is a preset value and Y is a positive integer greater than or equal to 1.

35. The apparatus of claim 33, wherein the processing unit is further configured to: If the start segment of the non-silent segment has been detected before the current speech segment and the end segment of the non-silent segment has not been detected, the first label of the current speech segment is determined to be the middle segment of the non-silent segment; The device further includes: An adjustment unit is configured to lower the confidence threshold for determining the second label of the current speech segment when the first label of the current speech segment is the middle segment of the non-silent segment.

36. The apparatus of claim 34, further comprising: The adjustment unit is used to shift the start time of the starting segment forward by a first preset time period and use the shifted start time of the starting segment as the start time of the non-silent segment; shift the end time of the ending segment forward by a second preset time period and use the shifted end time of the ending segment as the end time of the non-silent segment.

37. The apparatus according to any one of claims 29-32, wherein the segmentation unit is further configured to: perform downsampling processing on the speech stream to be inspected when the sampling rate of the speech stream to be inspected is greater than a first preset sampling rate, so that the sampling rate of the speech stream to be inspected becomes the first preset sampling rate; and obtain the current speech segment in the speech stream to be inspected according to the first preset sampling rate; The prediction unit is also used for: When the sampling rate of the speech stream to be detected is a first preset sampling rate, the speech features are input into a first speech detection model to obtain the confidence level of the current speech segment. The first speech detection model is a trained speech detection model, and the applicable sampling rate of the first speech detection model is the first preset sampling rate.

38. The apparatus according to claim 37, wherein, The prediction unit is also used for: When the sampling rate of the speech stream to be detected is lower than the first preset sampling rate, the speech features are input into the second speech detection model to obtain the confidence level of the current speech segment. The second speech detection model is a trained speech detection model, and the applicable sampling rate of the second speech detection model is the second preset sampling rate.

39. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-19.

40. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-19.

41. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-19.

Images