Efficient speech detection method and device based on feature enhancement pre-trained model

Abstract

This application relates to an effective speech detection method and apparatus based on a feature-enhanced pre-trained model. The method includes: acquiring speech to be detected containing different types of noise; inputting the speech to be detected into a first pre-trained model, and extracting effective speech features from the speech to be detected through the first pre-trained model; the first training data used by the first pre-trained model is obtained by enhancing the data features of unlabeled sample speech; inputting the effective speech features into a second pre-trained model, and performing effective speech classification through the second pre-trained model to obtain a classification result sequence; and outputting effective speech segments of the speech to be detected based on the classification result sequence; the effective speech segments are speech segments from which noise has been removed. This method can adapt to more application scenarios and noise types, effectively improving the effective speech detection effect and performance, thereby enhancing the performance of the speech recognition system.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to an effective speech detection method, apparatus, computer device, computer-readable storage medium, and computer program product based on a feature-enhanced pre-trained model. Background Technology

[0002] With the development of speech recognition technology, its application in power production activities is becoming increasingly widespread, such as speech analysis and processing based on intelligent power customer service platforms. However, the complexity of actual application environments also poses a severe challenge to speech recognition technology.

[0003] In related technologies, traditional speech recognition typically relies on VAD (Voice Activity Detection) technology to remove environmental noise from speech. However, due to the complexity of noise types and the variety of application scenarios in real-world environments, traditional speech recognition methods struggle to completely remove certain types of noise, and the unremoved noise significantly impacts the performance of the speech recognition system. Summary of the Invention

[0004] Therefore, it is necessary to provide an effective speech detection method, apparatus, computer device, computer-readable storage medium, and computer program product based on a feature-enhanced pre-trained model that can improve the effectiveness of speech detection, addressing the aforementioned technical problems.

[0005] Firstly, this application provides an effective speech detection method based on a feature-enhanced pre-trained model, including:

[0006] Acquire the speech to be detected, which contains different types of noise;

[0007] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0008] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0009] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0010] In one embodiment, the step of outputting the valid speech segments of the speech to be detected based on the classification result sequence includes:

[0011] In the classification result sequence, the start and end times of the valid speech frames are determined;

[0012] The effective speech segment is obtained based on the sequence segments corresponding to the start and end times of the effective speech frame.

[0013] In one embodiment, the method further includes:

[0014] Acquire unlabeled sample speech based on a speech recognition task;

[0015] The Mel spectrum matrix is ​​obtained by converting the unlabeled sample speech. The enhanced data features of the unlabeled sample speech are obtained by processing the time and frequency dimensions of the Mel spectrum matrix.

[0016] The enhanced data features of the unlabeled sample speech are used as the first training data.

[0017] In one embodiment, the method further includes:

[0018] Obtain the first trainable model based on the encoder and decoder structure;

[0019] By combining the first training data and the first loss function, the first model to be trained is subjected to self-supervised model training to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0020] In one embodiment, the method further includes:

[0021] Obtain labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data;

[0022] By inputting the second training data into the first pre-trained model for feature extraction, effective speech features of the sample are obtained.

[0023] In one embodiment, the method further includes:

[0024] Obtain a second training model based on a neural network; the second training model includes an effective speech classification model.

[0025] The effective speech features of the sample are used as input, and the effective speech classification model is trained according to the second loss function to obtain the classification result output model; the second loss function includes the cross-entropy function.

[0026] The second pre-trained model is obtained by combining the classification result output model and the effective speech segment output module.

[0027] Secondly, this application also provides an efficient speech detection device based on a feature-enhanced pre-trained model, comprising:

[0028] The speech acquisition module is used to acquire speech containing different types of noise.

[0029] An effective speech feature extraction module is used to input the speech to be detected into a first pre-trained model, and extract the effective speech features of the speech to be detected through the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech;

[0030] An effective speech classification module is used to input the effective speech features into a second pre-trained model, and perform effective speech classification through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0031] The effective speech segment output module is used to output effective speech segments of the speech to be detected based on the classification result sequence; the effective speech segments are speech segments with noise removed from the speech to be detected.

[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0033] Acquire the speech to be detected, which contains different types of noise;

[0034] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0035] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0036] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0038] Acquire the speech to be detected, which contains different types of noise;

[0039] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0040] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0041] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0043] Acquire the speech to be detected, which contains different types of noise;

[0044] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0045] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0046] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0047] The aforementioned effective speech detection method, apparatus, computer device, computer-readable storage medium, and computer program product based on a feature-enhanced pre-trained model acquires speech to be detected containing different types of noise, then inputs the speech to be detected into a first pre-trained model, extracts effective speech features from the speech to be detected, and the first training data used by the first pre-trained model is obtained by enhancing the data features of unlabeled sample speech. The effective speech features are then input into a second pre-trained model, which performs effective speech classification to obtain a classification result sequence. The second training data used by the second pre-trained model is obtained by enhancing the data features of labeled sample speech. This classification result sequence characterizes the probability of whether each frame of speech in the speech to be detected is effective speech. Based on the classification result sequence, effective speech segments of the speech to be detected are output. These effective speech segments are speech segments with noise removed from the speech to be detected. This optimizes the detection of effective speech. Based on feature enhancement and pre-training, the robustness of the effective speech detection model can be enhanced, adapting to more application scenarios and noise types, effectively improving the performance of effective speech detection and thus improving the performance of the speech recognition system. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating an effective speech detection method based on a feature-enhanced pre-trained model in one embodiment.

[0050] Figure 2 This is a schematic diagram of an efficient speech detection processing flow based on a feature-enhanced pre-trained model in one embodiment;

[0051] Figure 3a This is a schematic diagram of the training process of a feature-enhanced pre-trained model in one embodiment;

[0052] Figure 3b This is a schematic diagram of the model structure in one embodiment;

[0053] Figure 4 This is a flowchart illustrating an efficient speech detection method based on a feature-enhanced pre-trained model, as described in another embodiment.

[0054] Figure 5This is a block diagram of an efficient speech detection device based on a feature-enhanced pre-trained model in one embodiment.

[0055] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In one exemplary embodiment, such as Figure 1 As shown, an effective speech detection method based on a feature-enhanced pre-trained model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 101 to 104. Wherein:

[0058] Step 101: Obtain the speech to be detected, which contains different types of noise.

[0059] Among them, the speech to be detected can be obtained based on a speech recognition system. The speech processing of the speech recognition system includes the effective speech detection process. For example, the speech recognition system can be applied to fields such as intelligent customer service voice quality inspection and analysis, intelligent voice conferencing system and multimedia audio analysis.

[0060] As an example, different types of noise can be various noises in a real-world application environment, such as ambient music, ambient human voices, channel noise, etc.

[0061] In practical applications, such as Figure 2 As shown, taking the testing phase as an example, the input test speech can be used as the speech to be detected, so as to further perform effective speech detection processing on the pre-trained model based on feature enhancement for the speech to be detected.

[0062] Step 102: Input the speech to be detected into the first pre-trained model, and extract the effective speech features of the speech to be detected through the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0063] The first pre-trained model can be a pre-trained model obtained by using a specific algorithm for data augmentation and self-supervised pre-training. The first training data is obtained by performing data augmentation on unlabeled sample speech, which can transform the original features into enhanced features. Therefore, model training is performed based on feature enhancement, which can help enhance the robustness of the pre-trained model.

[0064] In practical implementation, the pre-trained model (i.e., the first pre-trained model) can be used as a feature extractor, such as... Figure 2 The robust VAD feature extraction module based on the pre-trained model can extract effective speech features of the speech to be detected by using the first pre-trained model as input.

[0065] Step 103: Input the effective speech features into the second pre-trained model, and perform effective speech classification through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech.

[0066] The classification result sequence can be used to characterize the probability of whether each frame of the speech to be detected is valid speech. For example, the speech to be detected can be processed frame by frame to determine whether each frame is valid speech.

[0067] After obtaining effective speech features, a trained classifier model (i.e., the second pre-trained model) can be used for effective speech classification. This is achieved by inputting the extracted effective speech features into the second pre-trained model, such as... Figure 2 The robust VAD feature extraction module based on a neural network classifier can classify and output a probability sequence (i.e., a classification result sequence) that represents whether each frame is valid speech.

[0068] In one example, second training data is obtained by performing data augmentation on the labeled sample speech. This second training data can then be used to train a second pre-trained model. Specifically, since the trained pre-trained model can serve as a robust effective speech detection feature extractor, i.e., the first pre-trained model, the second training data is used to perform feature extraction processing to train the model, thereby training a non-linear neural network classifier for determining effective speech.

[0069] Step 104: Based on the classification result sequence, output the effective speech segments of the speech to be detected; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0070] After obtaining the classification result sequence, by inputting this sequence into the effective speech segment output module of the second pre-trained model, the start and end times of the effective speech segments in the speech to be detected can be determined based on the classification result sequence. This allows for the extraction of noise-free effective speech segments from the speech to be detected. Therefore, effective speech detection based on the feature-enhanced pre-trained model can effectively improve the performance of the speech recognition system and contribute to improved speech recognition results.

[0071] In the aforementioned effective speech detection method based on a feature-enhanced pre-trained model, the method acquires speech samples containing different types of noise, inputs the speech samples into a first pre-trained model, extracts effective speech features from the speech samples, and then inputs these effective speech features into a second pre-trained model. The second pre-trained model performs effective speech classification to obtain a classification result sequence. Based on the classification result sequence, the effective speech segments of the speech samples are output. This method optimizes effective speech detection. Based on feature enhancement and pre-training, the robustness of the effective speech detection model can be enhanced, adapting to more application scenarios and noise types, effectively improving the performance of effective speech detection and thus improving the performance of the speech recognition system.

[0072] In an exemplary embodiment, outputting the valid speech segments of the speech to be detected based on the classification result sequence may include the following steps:

[0073] In the classification result sequence, the start time point and end time point of the valid speech frame are determined; based on the sequence segments corresponding to the start time point and end time point of the valid speech frame, the valid speech segment is obtained.

[0074] In practical applications, start-end point finding algorithms can be used to determine the start and end points of valid speech segments in the classification result sequence. For example, by using the start-end point finding algorithm, when a valid speech frame is detected to be more than a threshold, the valid speech frame can be identified as the start point of a valid speech segment (i.e., the start time point of the valid speech frame). By using the end point finding algorithm, when a noise frame is detected to be more than a threshold, the noise frame can be identified as the end point of a valid speech segment (i.e., the end time point of the valid speech frame).

[0075] In this embodiment, by determining the start and end times of valid speech frames in the classification result sequence, and then obtaining the valid speech segments based on the sequence segments corresponding to the start and end times of the valid speech frames, the valid speech segments can be effectively determined.

[0076] In one exemplary embodiment, the following steps may also be included:

[0077] Obtain unlabeled sample speech based on a speech recognition task; convert the unlabeled sample speech to obtain a Mel spectrum matrix, and process the time and frequency dimensions of the Mel spectrum matrix to obtain the enhanced data features of the unlabeled sample speech; use the enhanced data features of the unlabeled sample speech as the first training data.

[0078] In specific implementations, such as Figure 3a As shown, for the training phase, the overall process of an effective speech detection system based on a feature-enhanced pre-trained model can include the following modules: an unlabeled training data augmentation module based on a specific algorithm, a large model pre-training module based on unlabeled data, a robust VAD feature extraction module based on the pre-trained model, an labeled data augmentation module based on a specific algorithm, an effective speech classifier fine-tuning module based on a neural network, and an effective speech segment output module based on a neural network classifier.

[0079] For example, based on historical speech recognition tasks, unlabeled sample speech can be collected through a speech recognition system. Then, based on an unlabeled training data augmentation module with a specific algorithm, the unlabeled sample speech can be augmented to further input the augmented unlabeled data (i.e., the first training data) into a large model pre-training module for model training.

[0080] In one example, a log-mel spectrum-level data augmentation method can be used, which converts audio segments (i.e., unlabeled sample speech) into a mel spectrum matrix. v represents the frequency dimension. To represent the time dimension, the following steps can be taken:

[0081] 1. The Mel spectrum can be normalized to zero mean using xx.mean(), so that when masking is performed later, the mask position can be directly set to 0, which is also equivalent to filling the mean of the matrix.

[0082] 2. For time-dimension translation, the spectrum can be horizontally twisted left and right;

[0083] 3. Regarding time-dimensional masks, if the maximum range of a continuous time-dimensional mask is T, a uniform sample of t can be performed within the range [0, T]. A point t0 is randomly determined within the range, and then t masks are continuously applied along the time axis starting from position t0 (e.g., setting the matrix value to 0).

[0084] 4. For frequency dimension masking, if the maximum range of continuous time dimension masking is F, a uniform sample f can be performed in the range [0, F], and a point f0 can be randomly determined in the range [0, vf]. Then, the masking can be performed continuously f times along the time axis starting from the position f0 (e.g., setting the matrix value to 0).

[0085] Therefore, after data augmentation, the original features can be transformed into enhanced features, which helps to improve the robustness of subsequent model training.

[0086] In this embodiment, unlabeled sample speech based on a speech recognition task is acquired, and then a Mel spectrum matrix is ​​obtained by converting the unlabeled sample speech. By processing the time and frequency dimensions of the Mel spectrum matrix, the enhanced data features of the unlabeled sample speech are obtained. The enhanced data features of the unlabeled sample speech are then used as the first training data, which can provide data support for further model training.

[0087] In one exemplary embodiment, the following steps may also be included:

[0088] Obtain a first training model based on the encoder and decoder structure; combine the first training data and the first loss function to perform self-supervised model training on the first training model to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0089] In one example, unsupervised pre-training can be performed using data-enhanced features (i.e., the first training data). By using the pre-trained model for self-supervised pre-training, a pre-trained large model with enhanced features can be obtained, i.e., the first pre-trained model.

[0090] Optionally, for the pre-training process of large models based on unlabeled data feature enhancement, the network structure of the pre-trained large model (i.e., the first model to be trained) is as follows: Figure 3b As shown, the context network uses a transformer (encoder-decoder) structure. The feature vectors extracted by the encoder network can be directly input into the context transformer network, or the feature vectors can be quantized by the quantization module and used for subsequent loss function calculations (such as continuous inputs and quantized targets).

[0091] For example, the output vector Z of the encoder network can be discretized through product quantization. This can be achieved by splitting the vector Z into G subspaces (these subspaces are codebooks). If each codebook has V entries, and each entry has a length of d / G, the most similar entry to the input vector can be found within each codebook using GUMBEL-softmax or clustering methods. The discretized vectors output from each codebook can then be concatenated to obtain a d-dimensional Z-quantized vector. The main function of this quantization process is to compress and remove redundancy from the feature vector. Furthermore, clustering within each subspace enhances the robustness of the features, making them less susceptible to minor perturbations.

[0092] In another example, the first loss function may include two parts: contrast loss and diversity loss, and the final loss value can be obtained by weighting the two parts of the loss.

[0093] In an alternative embodiment, when applied to downstream tasks (such as effective speech detection), a linear layer can be added after the pre-trained model for fine-tuning. During fine-tuning, the parameters of the linear layer and the parameters of the transformer part can be updated, while the parameters of the encoder part can be frozen and remain unchanged.

[0094] In this embodiment, by obtaining a first training model based on an encoder and decoder structure, and then combining the first training data and the first loss function, the first training model is trained under self-supervised supervision to obtain the first pre-trained model for extracting effective speech features. This can enhance the robustness of the effective speech detection model by pre-training the model based on feature enhancement.

[0095] In one exemplary embodiment, the following steps may also be included:

[0096] Obtain labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data; by inputting the second training data into the first pre-trained model for feature extraction processing, obtain the effective speech features of the sample.

[0097] In practical applications, such as Figure 3aAs shown, for the robust and effective speech detection feature extraction process based on a pre-trained model, the labeled sample speech can be augmented to obtain second training data. The pre-trained model can then be used as a feature extractor to extract a robust representation vector from the second training data. By using the pre-trained model as a robust feature extractor, the data-augmented acoustic features can be input into the pre-trained model, and the resulting output vector (i.e., the effective speech features of the sample) can represent robust and effective speech detection.

[0098] In this embodiment, labeled sample speech based on a speech recognition task is obtained, and the enhanced data features of the labeled sample speech are used as second training data. Then, the second training data is input into the first pre-trained model for feature extraction processing to obtain effective speech features of the sample, which provides data support for further classifier model training.

[0099] In one exemplary embodiment, the following steps may also be included:

[0100] Obtain a second training model based on a neural network; the second training model includes an effective speech classification model; using the effective speech features of the sample as input, train the effective speech classification model according to a second loss function to obtain a classification result output model; the second loss function includes a cross-entropy function; combine the classification result output model and the effective speech segment output module to obtain the second pre-trained model.

[0101] In practical implementation, the extracted effective speech features of the samples can be used to train a neural network effective speech classifier. By using labeled data with effective speech segment annotations, a neural network-based effective speech binary classification model can be trained as the classification result output model. Based on the result sequence output by the classifier, the effective speech segments can be judged, and the start time and end time of the effective speech can be output.

[0102] In one example, such as Figure 3a As shown, for the fine-tuning process of an effective speech classifier based on a neural network, by inputting precisely labeled data (i.e., labeled sample speech) into a robust effective speech detection feature extraction module (i.e., the first pre-trained model) based on a pre-trained model, effective speech detection features can be extracted as input to the neural network. This neural network can be a fully connected neural network, a time-delay neural network, or a convolutional neural network, and the cross-entropy function can be selected as the loss function (i.e., the second loss function) to fine-tune the neural network. This allows the neural network to determine whether the input speech is effective frame by frame.

[0103] In another example, regarding the output process of effective speech segments based on a neural network classifier, after obtaining the label (such as effective speech, invalid speech) sequence calculated by the neural network classifier, glitch (such as short speech in a silent segment or short silence in a speech segment) in the sequence can be located and removed according to a set threshold, thereby ensuring the rationality of effective speech detection segmentation.

[0104] The technical solution of this embodiment is specifically improved based on the problem that effective speech detection has many application scenarios and complex noise in practical applications. By using data augmentation and self-supervised pre-training models with specific algorithms, a pre-trained model with robust effective speech detection feature extraction capabilities can be obtained. Then, through training a nonlinear classifier, the performance of effective speech detection can be effectively improved.

[0105] In this embodiment, a second training model based on a neural network is obtained, and then the effective speech features of the sample are used as input. The effective speech classification model is trained according to the second loss function to obtain the classification result output model. Then, the classification result output model and the effective speech segment output module are combined to obtain the second pre-trained model, which can improve the performance of effective speech detection and thus improve the performance of the speech recognition system.

[0106] In one exemplary embodiment, such as Figure 4 The diagram illustrates another efficient speech detection method based on a feature-enhanced pre-trained model. In this embodiment, the method includes the following steps:

[0107] In step 401, unlabeled sample speech based on the speech recognition task is acquired. A Mel spectrum matrix is ​​obtained from the unlabeled sample speech. The enhanced data features of the unlabeled sample speech are obtained by processing the time and frequency dimensions of the Mel spectrum matrix. These enhanced data features are used as the first training data. In step 402, a first model to be trained based on an encoder and decoder structure is acquired. The first training model is trained using the first training data and a first loss function under self-supervised conditions to obtain a first pre-trained model for extracting effective speech features. In step 403, labeled sample speech based on the speech recognition task is acquired. The enhanced data features of the labeled sample speech are used as the second training data. The second training data is input into the first pre-trained model for feature extraction to obtain effective speech features. In step 404, a second model to be trained based on a neural network is acquired. The effective speech features are used as input. The effective speech classification model is trained using a second loss function to obtain a classification result output model. The classification result output model and the effective speech segment output module are combined to obtain the second pre-trained model. In step 405, the speech to be detected, containing different types of noise, is acquired and input into a first pre-trained model. The first pre-trained model extracts effective speech features from the speech. In step 406, the effective speech features are input into a second pre-trained model, which performs effective speech classification to obtain a classification result sequence. In step 407, the start and end times of effective speech frames are determined within the classification result sequence. Based on the sequence segments corresponding to the start and end times of the effective speech frames, effective speech segments are obtained. It should be noted that the specific limitations of the above steps can be found in the above description of the specific limitations of an effective speech detection method based on a feature-enhanced pre-trained model, and will not be repeated here.

[0108] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0109] Based on the same inventive concept, this application also provides an effective speech detection device based on a feature-enhanced pre-trained model for implementing the effective speech detection method based on the feature-enhanced pre-trained model described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations in one or more embodiments of the effective speech detection device based on a feature-enhanced pre-trained model provided below can be found in the limitations of the effective speech detection method based on the feature-enhanced pre-trained model described above, and will not be repeated here.

[0110] In one exemplary embodiment, such as Figure 5 As shown, an effective speech detection device based on a feature-enhanced pre-trained model is provided, comprising:

[0111] The speech acquisition module 501 is used to acquire speech containing different types of noise.

[0112] The effective speech feature extraction module 502 is used to input the speech to be detected into a first pre-trained model, and extract the effective speech features of the speech to be detected through the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech;

[0113] The effective speech classification module 503 is used to input the effective speech features into the second pre-trained model, and perform effective speech classification through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0114] The effective speech segment output module 504 is used to output the effective speech segments of the speech to be detected based on the classification result sequence; the effective speech segments are speech segments with noise removed from the speech to be detected.

[0115] In one embodiment, the effective speech segment output module 504 includes:

[0116] The time point determination submodule is used to determine the start and end time points of valid speech frames in the classification result sequence.

[0117] The effective speech segment acquisition submodule is used to obtain the effective speech segment based on the sequence segments corresponding to the start time point and end time point of the effective speech frame.

[0118] In one embodiment, the apparatus further includes:

[0119] The unlabeled sample speech acquisition module is used to acquire unlabeled sample speech based on speech recognition tasks;

[0120] The data feature enhancement module is used to obtain a Mel spectrum matrix based on the unlabeled sample speech conversion, and to obtain the enhanced data features of the unlabeled sample speech by processing the time and frequency dimensions of the Mel spectrum matrix.

[0121] The first training data acquisition module is used to obtain the enhanced data features of the unlabeled sample speech as the first training data.

[0122] In one embodiment, the apparatus further includes:

[0123] The first trainable model acquisition module is used to acquire the first trainable model based on the encoder and decoder structure.

[0124] The first pre-trained model acquisition module is used to combine the first training data and the first loss function to perform self-supervised model training on the first model to be trained, so as to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0125] In one embodiment, the apparatus further includes:

[0126] The second training data acquisition module is used to acquire labeled sample speech based on the speech recognition task, and to use the enhanced data features of the labeled sample speech as the second training data.

[0127] The effective speech feature acquisition module is used to obtain effective speech features of the sample by inputting the second training data into the first pre-trained model for feature extraction processing.

[0128] In one embodiment, the apparatus further includes:

[0129] The second training model acquisition module is used to acquire a second training model based on a neural network; the second training model includes an effective speech classification model.

[0130] The classification model training module is used to train the effective speech classification model by taking the effective speech features of the sample as input and according to the second loss function to obtain the classification result output model; the second loss function includes the cross-entropy function;

[0131] The second pre-trained model acquisition module is used to combine the classification result output model and the effective speech segment output module to obtain the second pre-trained model.

[0132] Each module in the aforementioned effective speech detection device based on feature enhancement pre-trained models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0133] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an effective speech detection method based on a feature-enhanced pre-trained model.

[0134] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0136] Acquire the speech to be detected, which contains different types of noise;

[0137] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0138] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0139] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0140] In one embodiment, the processor, when executing the computer program, also implements the steps of the efficient speech detection method based on the feature-enhanced pre-trained model in the other embodiments described above.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0142] Acquire the speech to be detected, which contains different types of noise;

[0143] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0144] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0145] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0146] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the efficient speech detection method based on the feature-enhanced pre-trained model in the other embodiments described above.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0148] Acquire the speech to be detected, which contains different types of noise;

[0149] The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech.

[0150] The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech;

[0151] Based on the classification result sequence, the effective speech segments of the speech to be detected are output; the effective speech segments are speech segments from which noise has been removed from the speech to be detected.

[0152] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the efficient speech detection method based on the feature-enhanced pre-trained model in the other embodiments described above.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An effective speech detection method based on a feature-enhanced pre-trained model, characterized in that, The method includes: Acquire the speech to be detected, which contains different types of noise; The speech to be detected is input into a first pre-trained model, and the effective speech features of the speech to be detected are extracted by the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech using a log-Mel spectrogram level data augmentation method; The effective speech features are input into the second pre-trained model, and the effective speech is classified through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech; Based on the classification result sequence, output the effective speech segments of the speech to be detected; the effective speech segments are speech segments from which noise has been removed from the speech to be detected. The method further includes: Obtain a first trainable model based on an encoder and decoder structure; the first trainable model includes a quantization module for quantizing feature vectors; By combining the first training data and the first loss function, the first model to be trained is trained under self-supervised supervision to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss. The method further includes: Acquire unlabeled sample speech based on a speech recognition task; The Mel spectrum matrix is ​​obtained by converting the unlabeled sample speech. The enhanced data features of the unlabeled sample speech are obtained by shifting and masking the time dimension of the Mel spectrum matrix and by masking the frequency dimension of the Mel spectrum matrix. The enhanced data features of the unlabeled sample speech are used as the first training data.

2. The method according to claim 1, characterized in that, The step of outputting the effective speech segments of the speech to be detected based on the classification result sequence includes: In the classification result sequence, the start and end times of the valid speech frames are determined; The effective speech segment is obtained based on the sequence segments corresponding to the start and end times of the effective speech frame.

3. The method according to claim 1, characterized in that, The method further includes: Obtain labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data; By inputting the second training data into the first pre-trained model for feature extraction, effective speech features of the sample are obtained.

4. The method according to claim 3, characterized in that, The method further includes: Obtain a second training model based on a neural network; the second training model includes an effective speech classification model. The effective speech features of the sample are used as input, and the effective speech classification model is trained according to the second loss function to obtain the classification result output model; the second loss function includes the cross-entropy function. The second pre-trained model is obtained by combining the classification result output model and the effective speech segment output module.

5. An efficient speech detection device based on a feature-enhanced pre-trained model, characterized in that, The device includes: The speech acquisition module is used to acquire speech containing different types of noise. An effective speech feature extraction module is used to input the speech to be detected into a first pre-trained model, and extract the effective speech features of the speech to be detected through the first pre-trained model; the first training data used by the first pre-trained model is obtained by performing data feature enhancement on unlabeled sample speech using a log-Mel spectrogram level data augmentation method; An effective speech classification module is used to input the effective speech features into a second pre-trained model, and perform effective speech classification through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by performing data feature enhancement on labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech in each frame of the speech to be detected is effective speech; The effective speech segment output module is used to output effective speech segments of the speech to be detected based on the classification result sequence; the effective speech segment is a speech segment with noise removed from the speech to be detected. The device further includes: The first trainable model acquisition module is used to acquire a first trainable model based on the encoder and decoder structure; the first trainable model includes a quantization module for quantizing the feature vectors. The first pre-trained model acquisition module is used to combine the first training data and the first loss function to perform self-supervised model training on the first model to be trained, so as to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss. The device further includes: The unlabeled sample speech acquisition module is used to acquire unlabeled sample speech based on speech recognition tasks; The data feature enhancement module is used to obtain the Mel spectrum matrix based on the unlabeled sample speech conversion, and to obtain the enhanced data features of the unlabeled sample speech by performing translation and masking processing on the time dimension of the Mel spectrum matrix and masking processing on the frequency dimension of the Mel spectrum matrix. The first training data acquisition module is used to obtain the enhanced data features of the unlabeled sample speech as the first training data.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images