Effective voice detection method and device based on feature enhancement pre-training model

Abstract

The invention relates to an effective voice detection method and device based on a feature enhancement pre-training model. The method comprises the following steps: acquiring to-be-detected voice containing different types of noise; inputting the to-be-detected voice into a first pre-training model, and extracting effective voice features of the to-be-detected voice through the first pre-training model; first training data adopted by the first pre-training model is obtained by performing data feature enhancement on unlabeled sample voice; inputting the effective voice features into a second pre-training model, and performing effective voice classification through the second pre-training model to obtain a classification result sequence; according to the classification result sequence, effective voice segments of the to-be-detected voice are output; the effective voice segments are voice segments from which noise in the to-be-detected voice is removed. The method can adapt to more application scenes and noise types, effectively improves the effective voice detection effect, and improves the performance of effective voice detection so as to improve the performance of a voice recognition system.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to an effective speech detection method, apparatus, computer device, computer-readable storage medium and computer program product based on a feature-enhanced pre-training model. Background Art

[0002] With the development of speech recognition technology, its application in power production activities is becoming more and more extensive, such as speech analysis and processing based on the power intelligent customer service platform. However, the complexity of the actual application environment also brings severe challenges to speech recognition technology.

[0003] In related technologies, traditional speech recognition is usually based on VAD (Voice activity detection) technology to remove environmental noise contained in speech. Due to the complex types of noise in actual application environments and the numerous application scenarios, traditional speech recognition methods are difficult to completely remove certain noises, and the unremoved noise has a greater impact on the performance of the speech recognition system. Summary of the invention

[0004] Based on this, it is necessary to provide an effective speech detection method, device, computer equipment, computer-readable storage medium and computer program product based on a feature-enhanced pre-training model that can improve the effective speech detection effect in response to the above technical problems.

[0005] In a first aspect, the present application provides an effective speech detection method based on a feature-enhanced pre-trained model, comprising:

[0006] Acquire speech to be detected containing different types of noise;

[0007] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0008] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0009] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0010] In one embodiment, outputting the valid speech segment of the speech to be detected according to the classification result sequence includes:

[0011] In the classification result sequence, determining the starting time point and the ending time point of the valid speech frame;

[0012] The valid speech segment is obtained according to the sequence segments corresponding to the starting time point and the ending time point of the valid speech frame.

[0013] In one embodiment, the method further comprises:

[0014] Obtain unlabeled sample speech based on speech recognition tasks;

[0015] A mel spectrum matrix is ​​obtained according to the conversion of the unlabeled sample speech, and enhanced data features of the unlabeled sample speech are obtained by processing the time dimension and frequency dimension of the mel spectrum matrix;

[0016] The enhanced data features of the unlabeled sample speech are used as the first training data.

[0017] In one embodiment, the method further comprises:

[0018] Obtaining a first model to be trained based on an encoder and decoder structure;

[0019] In combination with the first training data and the first loss function, the first model to be trained is trained as a self-supervised model to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0020] In one embodiment, the method further comprises:

[0021] Acquire a labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data;

[0022] The second training data is input into the first pre-training model for feature extraction to obtain sample effective speech features.

[0023] In one embodiment, the method further comprises:

[0024] Acquire a second model to be trained based on a neural network; the second model to be trained includes a valid speech classification model;

[0025] Using the sample effective speech feature as input, training the effective speech classification model according to a second loss function to obtain a classification result output model; the second loss function includes a cross entropy function;

[0026] The second pre-training model is obtained by combining the classification result output model and the valid speech segment output module.

[0027] In a second aspect, the present application also provides an effective speech detection device based on a feature-enhanced pre-trained model, comprising:

[0028] A speech acquisition module to be detected, used to acquire the speech to be detected containing different types of noise;

[0029] An effective speech feature extraction module is used to input the speech to be detected into a first pre-training model, and extract effective speech features of the speech to be detected through the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0030] An effective speech classification module is used to input the effective speech features into a second pre-trained model, perform effective speech classification through the second pre-trained model, and obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0031] The effective speech segment output module is used to output the effective speech segment of the speech to be detected according to the classification result sequence; the effective speech segment is the speech segment with noise removed from the speech to be detected.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire speech to be detected containing different types of noise;

[0034] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0035] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0036] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Acquire speech to be detected containing different types of noise;

[0039] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0040] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0041] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Acquire speech to be detected containing different types of noise;

[0044] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0045] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0046] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0047] The above-mentioned effective speech detection method, device, computer equipment, computer-readable storage medium and computer program product based on feature enhancement pre-training model obtains the speech to be detected containing different types of noise, and then inputs the speech to be detected into the first pre-training model, extracts the effective speech features of the speech to be detected through the first pre-training model, the first training data used by the first pre-training model is obtained by data feature enhancement of unlabeled sample speech, and then inputs the effective speech features into the second pre-training model, and classifies the effective speech through the second pre-training model to obtain a classification result sequence, the second training data used by the second pre-training model is obtained by data feature enhancement of labeled sample speech, the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is effective speech, and outputs the effective speech segment of the speech to be detected according to the classification result sequence, the effective speech segment is the speech segment with the noise removed from the speech to be detected, and the optimization of effective speech detection is realized. Based on feature enhancement and pre-training model, the robustness of the effective speech detection model can be enhanced, adapting to more application scenarios and noise types, effectively improving the effective speech detection effect, and improving the performance of effective speech detection to improve the performance of speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of an effective speech detection method based on a feature-enhanced pre-trained model in one embodiment;

[0050] Figure 2 A schematic diagram of an effective speech detection process based on a feature-enhanced pre-trained model in one embodiment;

[0051] Figure 3a A schematic diagram of a training process based on a feature-enhanced pre-trained model in one embodiment;

[0052] Figure 3b is a schematic diagram of a model structure in one embodiment;

[0053] Figure 4 A schematic diagram of a flow chart of an effective speech detection method based on a feature-enhanced pre-trained model in another embodiment;

[0054] Figure 5A structural block diagram of an effective speech detection device based on a feature-enhanced pre-trained model in one embodiment;

[0055] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In an exemplary embodiment, Figure 1 As shown, an effective speech detection method based on a feature-enhanced pre-trained model is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 101 to 104. Among them:

[0058] Step 101: Acquire speech to be detected containing different types of noise.

[0059] Among them, the voice to be detected can be obtained based on the voice recognition system. The voice processing process of the voice recognition system includes an effective voice detection process. For example, the voice recognition system can be applied to the fields of intelligent customer service voice quality inspection and analysis, intelligent voice conferencing system and multimedia audio analysis.

[0060] As an example, different types of noise may be various noises in an actual application environment, such as ambient music, ambient human voice, channel noise, etc.

[0061] In practical applications, such as Figure 2 As shown, taking the test phase as an example, the input test speech can be used as the speech to be detected, so as to further perform effective speech detection processing on the speech to be detected based on the feature-enhanced pre-trained model.

[0062] Step 102: input the speech to be detected into a first pre-training model, and extract effective speech features of the speech to be detected through the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech.

[0063] Among them, the first pre-trained model can be a pre-trained model obtained by using a specific algorithm to perform data enhancement and self-supervised pre-training. The first training data is obtained by performing data enhancement processing on unlabeled sample speech, and the original features can be transformed into enhanced features, so that model training is performed based on feature enhancement, which can help enhance the robustness of the pre-trained model.

[0064] In a specific implementation, the pre-trained model (i.e., the first pre-trained model) that has been trained can be used as a feature extractor, such as Figure 2 The robust VAD feature extraction module based on the pre-trained model can use the first pre-trained model to extract the effective speech features of the speech to be detected by inputting the speech to be detected.

[0065] Step 103, input the effective speech features into a second pre-training model, perform effective speech classification through the second pre-training model, and obtain a classification result sequence; the second training data used by the second pre-training model is obtained by enhancing the data features of the labeled sample speech.

[0066] The classification result sequence can be used to characterize the probability of whether the speech in each frame of the speech to be detected is valid speech. For example, the speech to be detected can be processed frame by frame to determine whether each frame is valid speech.

[0067] After obtaining the effective speech features, the trained classifier model (i.e., the second pre-trained model) can be used to perform effective speech classification, by inputting the extracted effective speech features into the second pre-trained model, such as Figure 2 The robust VAD feature extraction module based on the neural network classifier can classify and output a probability sequence (i.e., a classification result sequence) representing whether each frame is valid speech.

[0068] In one example, second training data is obtained by performing data enhancement processing on labeled sample speech, and the second training data can be used to train a second pre-trained model; specifically, since the trained pre-trained model can be used as a robust effective sound detection feature extractor, that is, the first pre-trained model, the second training data is subjected to feature extraction processing by the first pre-trained model to train the model, and a nonlinear neural network classifier can be trained to determine effective speech.

[0069] Step 104: outputting a valid speech segment of the speech to be detected according to the classification result sequence; the valid speech segment is a speech segment in which noise in the speech to be detected is removed.

[0070] After obtaining the classification result sequence, by inputting the classification result sequence into the effective speech segment output module in the second pre-training model, the start time point and the end time point of the effective speech segment in the speech to be detected can be determined based on the classification result sequence, and then the effective speech segment with noise removed from the speech to be detected can be obtained. Therefore, effective speech detection based on the feature enhancement pre-training model can effectively improve the performance of the speech recognition system and help improve the speech recognition effect.

[0071] In the above-mentioned effective speech detection method based on feature enhancement pre-training model, by obtaining the speech to be detected containing different types of noise, and then inputting the speech to be detected into the first pre-training model, the effective speech features of the speech to be detected are extracted by the first pre-training model, and then the effective speech features are input into the second pre-training model, and the effective speech classification is performed by the second pre-training model to obtain a classification result sequence. According to the classification result sequence, the effective speech segment of the speech to be detected is output, thereby realizing the optimization of effective speech detection. Based on feature enhancement and pre-training model, the robustness of the effective speech detection model can be enhanced, adapting to more application scenarios and noise types, effectively improving the effective speech detection effect, and being able to improve the performance of effective speech detection to improve the performance of the speech recognition system.

[0072] In an exemplary embodiment, outputting the valid speech segment of the speech to be detected according to the classification result sequence may include the following steps:

[0073] In the classification result sequence, the starting time point and the ending time point of the valid speech frame are determined; and the valid speech segment is obtained according to the sequence segments corresponding to the starting time point and the ending time point of the valid speech frame.

[0074] In practical applications, a start and end point search algorithm can be used to determine the start and end points of a valid speech segment in a classification result sequence; illustratively, through a start point search algorithm, when it is detected that consecutive valid speech frames after a certain valid speech frame exceed a threshold, the certain valid speech frame can be confirmed as the start point of a valid speech segment (i.e., the starting time point of the valid speech frame); through an end point search algorithm, when consecutive noise frames after a certain noise frame exceed a threshold, the certain noise frame can be confirmed as the end point of a valid speech segment (i.e., the ending time point of the valid speech frame).

[0075] In this embodiment, by determining the starting time point and the ending time point of the valid speech frame in the classification result sequence, and then obtaining the valid speech segment according to the sequence segment corresponding to the starting time point and the ending time point of the valid speech frame, the valid speech segment can be effectively determined.

[0076] In an exemplary embodiment, the following steps may also be included:

[0077] Obtain unlabeled sample speech based on a speech recognition task; obtain a Mel spectrum matrix according to the conversion of the unlabeled sample speech, and obtain enhanced data features of the unlabeled sample speech by processing the time dimension and frequency dimension of the Mel spectrum matrix; and use the enhanced data features of the unlabeled sample speech as the first training data.

[0078] In the specific implementation, Figure 3a As shown, for the training stage, the overall process of the effective speech detection system based on the feature enhanced pre-trained model can include the following modules: an unlabeled training data enhancement module based on a specific algorithm, a large model pre-training module based on unlabeled data, a robust VAD feature extraction module based on the pre-trained model, a labeled data enhancement module based on a specific algorithm, an effective speech classifier finetune module based on a neural network, and an effective speech segment output module based on a neural network classifier.

[0079] For example, based on historical speech recognition tasks, unlabeled sample speech can be collected through a speech recognition system, and then the unlabeled sample speech can be enhanced based on an unlabeled training data enhancement module of a specific algorithm, so as to further input the enhanced unlabeled data (i.e., the first training data) into the large model pre-training module for model training.

[0080] In one example, a data augmentation method at the log mel spectrum level can be used. The matrix after converting the audio clip (i.e., the unlabeled sample speech) into the mel spectrum is , v represents the frequency dimension, Represents the time dimension, which can be processed in the following steps:

[0081] 1. The Mel spectrum can be normalized to zero mean xx.mean(), so that when masking is performed later, the mask position can be directly set to 0, which is equivalent to filling the mean of the matrix;

[0082] 2. For time dimension translation, the spectrum can be twisted horizontally left and right;

[0083] 3. For the time dimension mask, if the maximum range of the continuous mask in the time dimension is T, you can uniformly sample a t in the range [0, T], and then Randomly determine a point t within the range 0 , and then from t 0 The position starts to be masked t times continuously along the time axis (such as setting the matrix value to 0);

[0084] 4. For frequency dimension masking, if the maximum range of continuous masking in the time dimension is F, a uniform sample of f can be performed in the range [0, F], and a point f can be randomly determined in the range [0, vf] 0 , and then from f 0 The mask is performed f times continuously along the time axis starting from the position (such as setting the matrix value to 0).

[0085] Therefore, after data enhancement processing, the original features can be transformed into enhanced features, which is conducive to enhancing the robustness of subsequent model training.

[0086] In this embodiment, unlabeled sample speech based on a speech recognition task is obtained, and then a Mel spectrum matrix is ​​obtained based on the conversion of the unlabeled sample speech. The enhanced data features of the unlabeled sample speech are obtained by processing the time dimension and frequency dimension of the Mel spectrum matrix. The enhanced data features of the unlabeled sample speech are then used as the first training data, which can provide data support for further model training.

[0087] In an exemplary embodiment, the following steps may also be included:

[0088] Acquire a first model to be trained based on an encoder and decoder structure; perform self-supervised model training on the first model to be trained in combination with the first training data and a first loss function to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0089] In one example, the features after data enhancement (ie, the first training data) can be used for unsupervised pre-training, and a large pre-trained model with enhanced features, ie, the first pre-trained model, can be obtained by using the pre-trained model for self-supervised pre-training.

[0090] Optionally, for the large model pre-training process based on unlabeled data feature enhancement, the network structure of the pre-trained large model (i.e., the first model to be trained) is as follows: Figure 3b As shown in the figure, the context network part uses the transformer (encoder-decoder) structure. The feature vector extracted by the encoder network can be directly input into the context transformer network on the one hand, and on the other hand, the feature vector can be quantized by the quantization module and used for the subsequent loss function calculation (such as continuous inputs and quantized targets).

[0091] For example, the vector Z output by the encoder network can be discretized by product quantization. For example, the vector Z can be split into G subspaces (the series of subspaces is a codebook). If each codebook has V entries and the length of each entry is d / G, the entry most similar to the input vector can be found in each codebook by GUMBEL-softmax or clustering method, and then the discretized vectors output by each codebook can be spliced ​​to obtain a d-dimensional Z-quantized vector. The main function of the above quantization process is to compress and remove redundancy of the feature vector. At the same time, through clustering in each subspace, the feature can be made more robust and less susceptible to small disturbances.

[0092] In another example, the first loss function may include two parts: contrast loss and diversity loss, and the final loss value may be obtained by weighting the two parts of loss.

[0093] In an optional embodiment, when applied to downstream tasks (such as effective speech detection tasks), a linear layer can be added after the pre-trained model for fine-tuning. During the fine-tuning process, the parameters of the linear layer can be updated, and the parameters of the transformer part can be updated, and the parameters of the encoder part can be frozen and remain unchanged.

[0094] In this embodiment, by obtaining the first model to be trained based on the encoder and decoder structure, and then combining the first training data and the first loss function, the first model to be trained is self-supervised model training to obtain the first pre-trained model for extracting effective speech features. The pre-trained model can be enhanced based on the features, thereby effectively enhancing the robustness of the effective speech detection model.

[0095] In an exemplary embodiment, the following steps may also be included:

[0096] Acquire labeled sample speech based on speech recognition tasks, and use the enhanced data features of the labeled sample speech as the second training data; and obtain sample effective speech features by inputting the second training data into the first pre-training model for feature extraction processing.

[0097] In practical applications, such as Figure 3aAs shown, for the robust and effective speech detection feature extraction process based on the pre-trained model, the labeled sample speech can be data enhanced to obtain the second training data, and the trained pre-trained model is used as a feature extractor to extract a robust representation vector for the second training data. By using the trained pre-trained model as a robust feature extractor, the data-enhanced acoustic features can be input into the trained pre-trained model, and the obtained output vector (i.e., the sample effective speech feature) can represent the robust and effective speech detection.

[0098] In this embodiment, by obtaining labeled sample speech based on the speech recognition task, the enhanced data features of the labeled sample speech are used as the second training data, and then the second training data is input into the first pre-training model for feature extraction processing to obtain the effective speech features of the sample, thereby providing data support for further classifier model training.

[0099] In an exemplary embodiment, the following steps may also be included:

[0100] Acquire a second model to be trained based on a neural network; the second model to be trained includes a valid speech classification model; use the sample valid speech features as input, train the valid speech classification model according to a second loss function, and obtain a classification result output model; the second loss function includes a cross entropy function; combine the classification result output model and the valid speech segment output module to obtain the second pre-trained model.

[0101] In a specific implementation, the extracted sample effective speech features can be used to train a neural network effective speech classifier, and the neural network-based effective speech binary classification model can be trained by using labeled data with effective sound segments as the classification result output model. The effective speech segments can be judged based on the result sequence output by the classifier, and the start time and end time of the effective speech can be output.

[0102] In one example, if Figure 3a As shown, for the fine-tuning process of the effective speech classifier based on the neural network, by inputting the data with precise annotations (i.e., the labeled sample speech) into the robust effective speech detection feature extraction module based on the pre-training model (i.e., the first pre-training model), the effective speech detection features can be extracted as input to input into the neural network. The neural network can select a fully connected neural network, a time delay neural network, or a convolutional neural network, and can select a cross entropy function as a loss function (i.e., the second loss function) to fine-tune the neural network, so that the neural network can judge whether the input speech is effective speech frame by frame.

[0103] In another example, for the effective speech segment output process based on the neural network classifier, after obtaining the label sequence (such as effective speech, invalid speech) calculated by the neural network classifier, the burrs in the sequence (such as shorter speech in a silent segment or shorter silence in a speech segment) can be located and removed according to the set threshold, thereby ensuring the rationality of the effective speech detection segmentation.

[0104] The technical solution of this embodiment has made targeted improvements based on the problems of multiple usage scenarios and complex noise in effective speech detection in practical applications. By utilizing data enhancement and self-supervised pre-training models of specific algorithms, a pre-training model with robust effective speech detection feature extraction capabilities can be obtained. Furthermore, through the training of nonlinear classifiers, the performance of effective speech detection can be effectively improved.

[0105] In this embodiment, by obtaining a second model to be trained based on a neural network, and then using sample effective speech features as input, the effective speech classification model is trained according to a second loss function to obtain a classification result output model, and then combining the classification result output model and the effective speech segment output module to obtain a second pre-trained model, which can improve the performance of effective speech detection to enhance the performance of the speech recognition system.

[0106] In an exemplary embodiment, Figure 4 As shown, a flow chart of another effective speech detection method based on feature enhanced pre-training model is provided. In this embodiment, the method includes the following steps:

[0107] In step 401, an unlabeled sample speech based on a speech recognition task is obtained, and a Mel spectrum matrix is ​​obtained according to the conversion of the unlabeled sample speech. The enhanced data features of the unlabeled sample speech are obtained by processing the time dimension and frequency dimension of the Mel spectrum matrix, and the enhanced data features of the unlabeled sample speech are used as the first training data. In step 402, a first model to be trained based on an encoder and decoder structure is obtained, and the first training data and the first loss function are combined to perform self-supervisory model training on the first model to be trained to obtain a first pre-trained model for extracting effective speech features. In step 403, annotated sample speech based on a speech recognition task is obtained, and the enhanced data features of the annotated sample speech are used as the second training data. The second training data is input into the first pre-trained model for feature extraction processing to obtain sample effective speech features. In step 404, a second model to be trained based on a neural network is obtained, and the sample effective speech features are used as input. The effective speech classification model is trained according to the second loss function to obtain a classification result output model, and the classification result output model and the effective speech segment output module are combined to obtain a second pre-trained model. In step 405, a speech to be detected containing different types of noise is obtained, and the speech to be detected is input into the first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model. In step 406, the effective speech features are input into the second pre-training model, and effective speech classification is performed by the second pre-training model to obtain a classification result sequence. In step 407, in the classification result sequence, the starting time point and the ending time point of the effective speech frame are determined, and the effective speech segment is obtained according to the sequence segment corresponding to the starting time point and the ending time point of the effective speech frame. It should be noted that the specific definition of the above steps can be referred to the specific definition of an effective speech detection method based on a feature enhancement pre-training model above, which will not be repeated here.

[0108] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides an effective speech detection device based on a feature-enhanced pre-trained model for implementing the effective speech detection method based on a feature-enhanced pre-trained model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the effective speech detection device based on a feature-enhanced pre-trained model provided below can refer to the limitations of the effective speech detection method based on a feature-enhanced pre-trained model above, and will not be repeated here.

[0110] In an exemplary embodiment, Figure 5 As shown, an effective speech detection device based on a feature-enhanced pre-trained model is provided, comprising:

[0111] The to-be-detected speech acquisition module 501 is used to acquire the to-be-detected speech containing different types of noise;

[0112] The effective speech feature extraction module 502 is used to input the speech to be detected into a first pre-training model, and extract the effective speech features of the speech to be detected through the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing the data features of the unlabeled sample speech;

[0113] The effective speech classification module 503 is used to input the effective speech features into the second pre-training model, perform effective speech classification through the second pre-training model, and obtain a classification result sequence; the second training data used by the second pre-training model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0114] The effective speech segment output module 504 is used to output the effective speech segment of the speech to be detected according to the classification result sequence; the effective speech segment is the speech segment with noise removed from the speech to be detected.

[0115] In one embodiment, the effective speech segment output module 504 includes:

[0116] A time point determination submodule, used to determine the start time point and the end time point of a valid speech frame in the classification result sequence;

[0117] The effective speech segment obtaining submodule is used to obtain the effective speech segment according to the sequence segments corresponding to the starting time point and the ending time point of the effective speech frame.

[0118] In one embodiment, the apparatus further comprises:

[0119] An unlabeled sample speech acquisition module is used to acquire unlabeled sample speech based on speech recognition tasks;

[0120] A data feature enhancement module, configured to obtain a mel spectrum matrix according to the unlabeled sample speech conversion, and obtain enhanced data features of the unlabeled sample speech by processing the time dimension and frequency dimension of the mel spectrum matrix;

[0121] The first training data obtaining module is used to use the enhanced data features of the unlabeled sample speech as the first training data.

[0122] In one embodiment, the apparatus further comprises:

[0123] A first model to be trained acquisition module, used to acquire a first model to be trained based on an encoder and decoder structure;

[0124] The first pre-trained model obtaining module is used to combine the first training data and the first loss function to perform self-supervised model training on the first model to be trained, so as to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

[0125] In one embodiment, the apparatus further comprises:

[0126] A second training data obtaining module is used to obtain labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data;

[0127] The sample effective speech feature obtaining module is used to obtain the sample effective speech feature by inputting the second training data into the first pre-training model for feature extraction processing.

[0128] In one embodiment, the apparatus further comprises:

[0129] A second model to be trained acquisition module, used to acquire a second model to be trained based on a neural network; the second model to be trained includes a valid speech classification model;

[0130] A classification model training module, used to use the sample effective speech features as input, train the effective speech classification model according to a second loss function, and obtain a classification result output model; the second loss function includes a cross entropy function;

[0131] The second pre-training model obtaining module is used to combine the classification result output model and the effective speech segment output module to obtain the second pre-training model.

[0132] Each module in the above-mentioned effective speech detection device based on feature enhancement pre-training model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an effective speech detection method based on a feature-enhanced pre-trained model is implemented.

[0134] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0136] Acquire speech to be detected containing different types of noise;

[0137] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0138] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0139] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0140] In one embodiment, when the processor executes the computer program, it also implements the steps of the effective speech detection method based on the feature enhanced pre-training model in the other embodiments mentioned above.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0142] Acquire speech to be detected containing different types of noise;

[0143] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0144] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0145] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0146] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the effective speech detection method based on the feature enhanced pre-training model in the other embodiments described above.

[0147] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0148] Acquire speech to be detected containing different types of noise;

[0149] The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech;

[0150] The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech;

[0151] According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

[0152] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the effective speech detection method based on the feature enhanced pre-training model in the other embodiments described above.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0154] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0155] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An effective speech detection method based on feature-enhanced pre-trained model, characterized in that: The method comprises: Acquire speech to be detected containing different types of noise; The speech to be detected is input into a first pre-training model, and effective speech features of the speech to be detected are extracted by the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing data features of unlabeled sample speech; The effective speech features are input into a second pre-trained model, and effective speech classification is performed through the second pre-trained model to obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech; According to the classification result sequence, a valid speech segment of the speech to be detected is output; the valid speech segment is a speech segment with noise removed from the speech to be detected.

2. The method according to claim 1, characterized in that Outputting the valid speech segment of the speech to be detected according to the classification result sequence includes: In the classification result sequence, determining the starting time point and the ending time point of the valid speech frame; The valid speech segment is obtained according to the sequence segments corresponding to the starting time point and the ending time point of the valid speech frame.

3. The method according to claim 1, characterized in that The method further comprises: Obtain unlabeled sample speech based on speech recognition tasks; A mel spectrum matrix is ​​obtained according to the conversion of the unlabeled sample speech, and enhanced data features of the unlabeled sample speech are obtained by processing the time dimension and frequency dimension of the mel spectrum matrix; The enhanced data features of the unlabeled sample speech are used as the first training data.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a first model to be trained based on an encoder and decoder structure; In combination with the first training data and the first loss function, the first model to be trained is trained as a self-supervised model to obtain the first pre-trained model for extracting effective speech features; the first loss function includes contrast loss and diversity loss.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a labeled sample speech based on a speech recognition task, and use the enhanced data features of the labeled sample speech as the second training data; The second training data is input into the first pre-training model for feature extraction to obtain sample effective speech features.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a second model to be trained based on a neural network; the second model to be trained includes a valid speech classification model; Using the sample effective speech feature as input, training the effective speech classification model according to a second loss function to obtain a classification result output model; the second loss function includes a cross entropy function; The second pre-training model is obtained by combining the classification result output model and the valid speech segment output module.

7. An effective speech detection device based on a feature-enhanced pre-trained model, characterized in that: The device comprises: A speech acquisition module to be detected, used to acquire the speech to be detected containing different types of noise; An effective speech feature extraction module is used to input the speech to be detected into a first pre-training model, and extract effective speech features of the speech to be detected through the first pre-training model; the first training data used by the first pre-training model is obtained by enhancing the data features of unlabeled sample speech; An effective speech classification module is used to input the effective speech features into a second pre-trained model, perform effective speech classification through the second pre-trained model, and obtain a classification result sequence; the second training data used by the second pre-trained model is obtained by enhancing the data features of the labeled sample speech; the classification result sequence is used to characterize the probability of whether the speech of each frame in the speech to be detected is a valid speech; The effective speech segment output module is used to output the effective speech segment of the speech to be detected according to the classification result sequence; the effective speech segment is the speech segment with noise removed from the speech to be detected.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images