Effective speech detection method and apparatus

Abstract

The application provides an effective speech detection method and device, the method comprises the following steps: extracting audio features of an audio signal to be detected based on a feature extraction model; determining effective speech signals in the audio signal based on a first effective speech recognition model and applying the audio features; the feature extraction model and the first effective speech recognition model constitute a first detection model, the first detection model is jointly trained with a speech understanding model in a training stage, the speech understanding model takes the audio features extracted by the feature extraction model as input and is used for predicting speech content, and a total loss value of the joint training includes an effective speech detection loss value of the first detection model and a speech understanding loss value of the speech understanding model. The first detection model can be trained with the aid of the speech understanding task, so that the first detection model can avoid missing the effective speech, that is, the ability of the first detection model to detect the effective speech is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signal processing, and in particular to an effective speech detection method and device. BACKGROUND

[0002] In the field of audio signal processing, it is often necessary to use VAD (Voice Activity Detection) technology to detect speech segments in audio signals, and to perform speech recognition, semantic recognition and other processing on the detected speech segments according to different audio signal processing requirements.

[0003] At present, the audio features are often represented by the nonlinear transformation capability of a neural network, and a classifier uses the audio features to distinguish between speech and non-speech. However, the audio representation capability of the neural network is limited, and the feature generalization performance is poor. In the face of the constantly emerging new complex network environment and noise types, as well as the environment with high noise such as short wave and ultrashort wave, the accuracy, anti-interference and generalization ability of effective speech detection are not ideal. SUMMARY

[0004] The present application provides an effective speech detection method and device to solve the defects in the prior art.

[0005] The present application provides an effective speech detection method, comprising the following steps:

[0006] Based on a feature extraction model, audio features of a to-be-detected audio signal are extracted;

[0007] Based on a first effective speech recognition model, the audio features are applied to determine the effective speech signal in the audio signal;

[0008] The feature extraction model and the first effective speech recognition model constitute a first detection model, the first detection model is jointly trained with a speech understanding model in a training stage, the speech understanding model takes the audio features extracted by the feature extraction model as input, and is used for predicting speech content, and the total loss value of joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0009] According to the effective speech detection method provided by the present application, the joint training samples include sample audio signals, effective speech labels of the sample audio signals, corresponding transcription labels of the sample audio signals, and punctuation labels of speech texts;

[0010] The joint training process is as follows:

[0011] Based on the feature extraction model, sample audio features of the sample audio signal are extracted;

[0012] based on the speech understanding model, applying the sample audio feature, determining a transcription text of the sample audio feature and recognizing punctuation marks in the transcription text;

[0013] based on the first effective speech recognition model, applying the sample audio feature, determining a predicted speech signal;

[0014] based on the difference between the predicted speech signal and the effective speech label, the difference between the transcription text and the corresponding transcription label of the sample audio signal, and the difference between the punctuation mark and the punctuation label, updating the network parameters of the first detection model and the speech understanding model.

[0015] According to the effective speech detection method provided by the application, the initial model of the first detection model is updated based on the difference between the predicted speech signal and the effective speech label, the difference between the transcription text and the corresponding transcription label of the sample audio signal, and the difference between the punctuation mark and the punctuation label, to obtain the first detection model, comprising:

[0016] based on the difference between the predicted speech signal and the effective speech label, determining the effective speech detection loss value;

[0017] based on the difference between the transcription text and the corresponding transcription label of the sample audio signal, determining the speech transcription loss value in the speech understanding loss value;

[0018] based on the difference between the punctuation mark and the punctuation label, determining the punctuation prediction loss value in the speech understanding loss value;

[0019] based on the preset weight, the effective speech detection loss value, the speech transcription loss value and the punctuation prediction loss value are added by weight, to determine the joint training loss value;

[0020] based on the joint training loss value, updating the network parameters of the first detection model and the speech understanding model.

[0021] According to the effective speech detection method provided by the application, the effective speech signal in the audio signal is determined based on the first effective speech recognition model and the audio feature, comprising:

[0022] based on the second effective speech recognition model, applying the audio feature, determining the probability of each frame signal in the audio signal as an effective speech signal;

[0023] If a probability of any frame signal being an effective speech signal is within a preset range, an effective speech signal in the audio signal is determined based on the first effective speech recognition model and the audio feature;

[0024] The second effective speech recognition model is trained based on the sample audio signal and the effective speech label of the sample audio signal.

[0025] According to the effective speech detection method provided by the application, after the effective speech signal in the audio signal is determined, the method further comprises:

[0026] Based on the recognition task type, the time granularity of the effective speech signal is determined.

[0027] Based on the time granularity, the effective speech signal is smoothed to obtain a smoothed signal.

[0028] Based on the smoothed signal, a recognition result corresponding to the recognition task is determined.

[0029] According to the effective speech detection method provided by the application, the effective speech signal is a frame-level effective speech signal.

[0030] The smoothing processing of the effective speech signal to obtain a smoothed signal comprises:

[0031] If the recognition task type is speech content recognition, each frame signal is smoothed to obtain the smoothed signal.

[0032] If the recognition task type is attribute recognition, the frame-level effective speech signal is converted into a sample point-level effective speech signal, and each sample point signal is smoothed to obtain the smoothed signal.

[0033] According to the effective speech detection method provided by the application, the smoothing processing of each frame signal comprises:

[0034] A preset number of frame signals are extended before and after each frame signal.

[0035] A non-sound segment is filled between each adjacent two speech segments.

[0036] If the total number of frames of any speech segment is less than a first preset number of frames, the any speech segment is discarded.

[0037] If the total number of frames of any speech segment is greater than or equal to the first preset number of frames and less than or equal to a second preset number of frames, the any speech segment is combined with the nearest speech segment.

[0038] According to the effective speech detection method provided by the application, the smoothing processing of each sample point signal comprises:

[0039] If the total number of sampling points of any voice segment is less than the preset number of points, the any voice segment is discarded.

[0040] The application further provides an effective voice detection device, comprising the following modules:

[0041] An extraction unit is configured to extract audio features of an audio signal to be detected based on a feature extraction model.

[0042] A detection unit is configured to determine effective voice signals in the audio signal based on a first effective voice recognition model and the audio features.

[0043] The feature extraction model and the first effective voice recognition model constitute a first detection model, the first detection model is jointly trained in a training stage with a voice understanding model, the voice understanding model takes the audio features extracted by the feature extraction model as input and is configured to predict voice content, and a total loss value of the joint training includes an effective voice detection loss value of the first detection model and a voice understanding loss value of the voice understanding model.

[0044] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the effective voice detection method according to any one of the above when executing the program.

[0045] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the effective voice detection method according to any one of the above.

[0046] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the effective voice detection method according to any one of the above.

[0047] The effective voice detection method and device provided by the application, the total loss value of the joint training considers the effective voice detection loss value and the voice understanding loss value, that is, the joint training of the effective voice detection task and the voice understanding task is performed. By training the first detection model using the effective voice detection loss value, the first detection model has the detection capability of the effective voice signals. Further, the first detection model is trained by the voice understanding task, which can avoid missing the effective voice of the first detection model, that is, the detection capability of the first detection model for the effective voice is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0049] Figure 1 is a flowchart of an effective speech detection method provided by the present application.

[0050] Figure 2 is a schematic diagram of automatic annotation based on a phoneme state level speech recognition model provided by the present application.

[0051] Figure 3 is a flowchart of a joint training method provided by the present application.

[0052] Figure 4 is a flowchart of an embodiment of step 120 in the effective speech detection method provided by the present application.

[0053] Figure 5 is one of the flowcharts of the smoothing process provided by the present application.

[0054] Figure 6 is another flowchart of the smoothing process provided by the present application.

[0055] Figure 7 is a flowchart of another effective speech detection method provided by the present application.

[0056] Figure 8 is a structural schematic diagram of an effective speech detection device provided by the present application.

[0057] Figure 9 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0059] Currently, DNN (Deep Neural Network), CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory) and hybrid networks (such as LSTM-DNN hybrid network) are mostly used to construct a neural network for voice detection. The audio features are represented by the nonlinear transformation capability of the neural network, and the classifier uses the audio features to distinguish between voice and non-voice. However, the audio representation capability of the neural network is limited, and the feature generalization performance is poor. In the face of new complex network environments and noise types that are constantly emerging, as well as short-wave, ultra-short-wave and other noisy environments, the accuracy, anti-interference and generalization ability of effective voice detection are not ideal.

[0060] In addition, there are also effective voice detection based on time domain features, based on transform domain, based on template matching and the like, but the above-mentioned methods have poor anti-interference ability. In a high signal-to-noise ratio or single noise environment, they have good detection effect, but in a low signal-to-noise ratio and complex noise environment, the detection sensitivity will be greatly reduced.

[0061] Figure 1 The present application provides a flowchart of an effective voice detection method, as shown in Figure 1 The method comprises steps 110 and 120.

[0062] Step 110, based on a feature extraction model, extracting audio features of the audio signal to be detected;

[0063] Step 120, based on a first effective voice recognition model, applying the audio features to determine the effective voice signal in the audio signal.

[0064] The feature extraction model and the first effective voice recognition model constitute a first detection model, and the first detection model is jointly trained with a speech understanding model in the training stage. The speech understanding model takes the audio features extracted by the feature extraction model as input, and is used to predict the speech content. The total loss value of joint training includes the effective voice detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0065] Specifically, the audio signal to be detected can be an audio signal in any scene, such as an audio signal issued by a user in a human-computer dialogue (such as a dialogue scene between a virtual intelligent machine customer service and a real person in an intelligent customer service), or an audio signal issued by a user in a voice input scene, etc. For this purpose, the embodiments of the present application are not limited. In addition, the audio signal to be detected can be of any length and any language, and for this purpose, the embodiments of the present application are not limited.

[0066] The audio features of an audio signal to be detected refer to the features that can uniquely and accurately represent the audio signal. Through these audio features, different audio signals, or even identical audio signals, can be accurately distinguished. Specifically, these audio features can be of any type, such as audio data spectrum, filterbank features, bottleneck features, PLP (Perceptual Linear Prediction) features, EL features, log-Mel features, etc.

[0067] For example, the audio features of the aforementioned audio signal can be represented as feature vectors. The audio features can be obtained by directly performing feature extraction processing on the audio signal.

[0068] It should be noted that the processing methods for acquiring the aforementioned audio features should be consistent to ensure that the dimensions of each acquired audio feature are the same, thus ensuring that the audio signals of each audio feature are directly comparable. Furthermore, before extracting audio features from the audio signal, the audio signal can be standardized to the same format. This reduces the complexity of data processing during feature extraction and improves efficiency. For example, the audio signal can be standardized to an 8000Hz or 16000Hz, 16-bit, single-channel, PCM encoding format.

[0069] After obtaining the audio features, the audio features are input into the first effective speech recognition model. The first effective speech recognition model performs effective speech recognition on the audio signal based on the audio features to determine the effective speech signals in the audio signal. The first effective speech recognition model can output the start time, end time, and duration of the speech segment corresponding to the effective speech signal.

[0070] The first detection model comprises a feature extraction model and a first effective speech recognition model. During the training phase, the first detection model is jointly trained with a speech understanding model. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model. The effective speech detection loss value characterizes the ability of the first detection model to detect effective speech signals from the audio signal, while the speech understanding loss value characterizes the ability of the speech understanding model to recognize speech content, which may include semantic information, syntactic information, punctuation information, etc. Optionally, the speech understanding loss value can be determined based on the difference between the predicted speech content and the speech content label.

[0071] Semantic information provides the meaning of the speech content, thus aiding in understanding the intent behind the statement. By understanding the context, it's possible to more accurately determine which parts of the speech are valid, thereby reducing missed detections. Syntactic structure provides the organization of the sentence, helping to identify components such as subject, predicate, and object, thus effectively guiding the identification of valid speech and reducing the possibility of missed detections. In written language, punctuation marks indicate the structure and tone of a sentence; their presence can be inferred through pauses and intonation changes, helping to identify the completeness of the sentence and thus avoiding missed detections.

[0072] Therefore, the effective speech detection method provided in this embodiment of the invention considers both the effective speech detection loss and the speech understanding loss in the total loss value of joint training, that is, it jointly trains the effective speech detection task and the speech understanding task. By training the first detection model using the effective speech detection loss value, the first detection model acquires the ability to detect effective speech signals. Furthermore, by assisting the training of the first detection model with the speech understanding task, the first detection model can avoid missing effective speech, thus improving its ability to detect effective speech.

[0073] In this process, the speech understanding task is used as an auxiliary task to enrich semantic and syntactic information during joint training, but it is not used during the inference phase, thus ensuring the lightweight and independence of the first detection model during the inference phase.

[0074] Based on the above embodiments, the samples for joint training include sample audio signals, valid speech labels of the sample audio signals, transcription labels corresponding to the sample audio signals, and punctuation labels of the speech text. Among them, the sample audio signals can be obtained from the open-source VAD dataset, the valid speech labels of the sample audio signals can be understood as the annotations of valid speech in the sample audio signals, the transcription labels corresponding to the sample audio signals can be understood as the real speech text corresponding to the sample audio signals, and the punctuation labels of the speech text can be understood as the annotations of punctuation marks in the speech text.

[0075] While manual annotation can be used to label valid speech signals, such as manually marking the start and segment times of valid speech signals in sample audio signals, manual annotation is costly and often requires frame-level labeling, which is difficult to perform manually. Therefore, a phoneme-state-level speech recognition model can be used for automatic annotation. For example, a phoneme-state-level speech recognition model can be used for Force Alignment (FA) to obtain the alignment between frames and states. Then, by parsing the FA results, the phoneme states of valid and invalid speech (nospeech) in each frame can be obtained, and the start and end times of the speech and nospeech can be determined based on the phoneme states. Phoneme-state-level speech recognition models can include GMM-HMM, DNN-HMM, and BiLSTM-HMM, among others. Figure 2 This is a schematic diagram of the automatic annotation based on the phoneme state-level speech recognition model provided by the present invention, as shown below. Figure 2 As shown, a 1-second audio sampled at 16000Hz is divided into 1-100 frames. Each frame is subjected to FA (figurine analysis) and labeled accordingly. Frames 12-55 and 63-94 are valid speech (sp), while frames 0-12, 55-63, and 94-100 are invalid speech (nosp).

[0076] Optionally, after tagging valid speech, the start time, end time, and duration of the valid speech can be saved to a JSON file, as shown in the example below:

[0077]

[0078] In this context, "end" indicates the end time of the valid speech, "start" indicates the start time of the valid speech, and "duration" indicates the duration of the valid speech.

[0079] The following are examples of samples from the joint training mentioned above:

[0080]

[0081] Here, “fea_path” represents the sample audio signal, “voice_activity_path” represents the valid speech tag of the sample audio signal, and “text” represents the transcription tag of the sample audio signal and the punctuation tag of the speech text.

[0082] Figure 3 This is a flowchart illustrating the joint training method provided by the present invention, as shown below. Figure 3 As shown, the joint training process is as follows:

[0083] Step 310: Based on the feature extraction model, extract the sample audio features of the sample audio signal.

[0084] Specifically, the feature extraction model can be an audio encoder built based on a locally enhanced multi-head attention mechanism (CNN-MHSA). The CNN consists of C blocks composed of convolutions and activation functions. The embeddings obtained by the CNN, combined with positional encoding information, are input to a module consisting of multi-head attention, residual connections, Add&Layer Norm&Dropout, feedforward neural networks, and Add&Layer Norm&Dropout to extract audio features. By leveraging the ability of CNN to capture local time-frequency variation features and MHSA to capture global dependency features, the model enhances the representation of effective speech and noise in the extracted audio features.

[0085] Step 320: Based on the speech understanding model, apply the sample audio features to determine the transcribed text of the sample audio features and identify the punctuation marks in the transcribed text.

[0086] Specifically, the speech understanding model can include a semantic recognition module and a punctuation recognition module. The semantic recognition module is used to determine the transcribed text based on the sample audio features. The punctuation recognition module can directly determine the transcribed text based on the sample audio features and identify punctuation marks in the transcribed text, or it can take the transcribed text output by the semantic recognition module as input and identify punctuation marks in the transcribed text. Here, the transcribed text based on the sample audio features refers to the text content extracted and converted from the recorded audio. If the transcribed text matches the transcription label, it indicates that the speech understanding module has good semantic understanding capabilities; if the identified punctuation marks match the punctuation labels, it indicates that the punctuation recognition module has good punctuation prediction capabilities.

[0087] Optionally, the semantic recognition module can be built based on the Decoder in Transformer or based on unidirectional LSTM; this embodiment of the invention does not specifically limit this. The punctuation recognition module can be built based on Linear+Softmax.

[0088] Step 330: Based on the first effective speech recognition model, apply the sample audio features to determine the predicted speech signal.

[0089] For example, the first effective speech recognition model is used to classify each frame of the sample audio signal based on the sample audio features to determine whether each frame is a valid speech signal. If so, the corresponding frame signal is marked as a valid speech signal. Typically, a valid speech segment consists of multiple consecutive frame signals. In this case, the start time of the first frame signal can also be output as the start time of the valid speech segment, and the end time of the last frame signal can be output as the end time of the valid speech segment.

[0090] The aforementioned speech understanding model and the first effective speech recognition model share the feature extraction model, thus eliminating the need to repeatedly train the feature extraction model for both the speech understanding model and the first effective speech recognition model, thereby saving training resources.

[0091] Step 340: Based on the differences between the predicted speech signal and the effective speech label, the differences between the transcribed text and the corresponding transcribed label of the sample audio signal, and the differences between punctuation marks and punctuation labels, update the network parameters of the first detection model and the speech understanding model.

[0092] Specifically, joint training aims to minimize the differences between the predicted speech signal and the valid speech label, the differences between the transcribed text and the corresponding transcribed label of the sample audio signal, and the differences between punctuation marks and punctuation labels. In other words, convergence is achieved when these differences meet the joint training requirements. The difference between the transcribed text and the corresponding transcribed label of the sample audio signal is used to assist the first detection model in understanding semantic information, and the difference between punctuation marks and punctuation labels is used to assist the first detection model in understanding syntactic information. Thus, the trained first detection model can correct missed valid speech signals based on semantic and syntactic information, further improving the detection accuracy of valid speech signals.

[0093] Based on any of the above embodiments, the initial model of the first detection model is updated with parameters based on the differences between the predicted speech signal and the effective speech tag, the differences between the transcribed text and the corresponding transcribed tags of the sample audio signal, and the differences between punctuation marks and punctuation tags, to obtain the first detection model, including:

[0094] The effective speech detection loss value is determined based on the difference between the predicted speech signal and the effective speech label;

[0095] Based on the difference between the transcribed text and the corresponding transcribed tags of the sample audio signal, the speech transcription loss value in the speech understanding loss value is determined.

[0096] Based on the difference between punctuation marks and punctuation labels, the punctuation prediction loss value in the speech understanding loss value is determined;

[0097] Based on preset weights, the effective speech detection loss value, speech transcription loss value, and punctuation prediction loss value are weighted and summed to determine the joint training loss value;

[0098] Based on the joint training loss value, update the network parameters of the first detection model and the speech understanding model.

[0099] Specifically, the difference between the predicted speech signal and the effective speech label is used to characterize the effective speech detection capability; the smaller the difference, the stronger the effective speech detection capability. The difference between the transcribed text and the corresponding transcribed labels of the sample audio signal is used to characterize the semantic understanding capability; the smaller the difference, the stronger the semantic understanding capability. The difference between punctuation marks and punctuation labels is used to characterize the syntactic analysis capability; the smaller the difference, the stronger the syntactic analysis capability.

[0100] During joint training, based on preset weights, the effective speech detection loss value, speech transcription loss value, and punctuation prediction loss value are weighted and summed to determine the joint training loss value. This joint training loss value takes into account the model's effective speech detection capability, semantic understanding capability, and syntactic analysis capability. As a result, the first detection model trained can combine semantic and syntactic information to correct missed effective speech signals online, further ensuring the detection accuracy of effective speech signals.

[0101] Alternatively, the joint training loss value can be determined based on the following formula:

[0102]

[0103] in, This represents the joint training loss value. Indicates punctuation marks, Indicates transcription tag, Indicates a valid voice tag. Punctuation marks indicating prediction This represents the predicted transcribed text. This indicates the effective speech signal predicted. This represents the predicted loss value for punctuation. Indicates the speech transcription loss value. This represents the effective speech detection loss value. , and These represent the weights of the corresponding loss values.

[0104] Based on any of the above embodiments Figure 4 This is a flowchart illustrating an implementation of step 120 in the effective speech detection method provided by the present invention, as shown below. Figure 4As shown, step 120, based on the first effective speech recognition model, applies audio features to determine the effective speech signals in the audio signal, including:

[0105] Step 121: Based on the second effective speech recognition model, apply audio features to determine the probability that each frame of the audio signal is a valid speech signal;

[0106] Step 122: If the probability of any frame signal being a valid speech signal is within a preset range, then based on the first valid speech recognition model, apply audio features to determine the valid speech signal in the audio signal.

[0107] The second effective speech recognition model is trained based on sample audio signals and effective speech labels for the sample audio signals.

[0108] Specifically, the second effective speech recognition model is trained based on the sample audio signal and the effective speech labels of the sample audio signal. In other words, the second effective speech recognition model can be understood as being trained based on the effective speech detection task, without being jointly trained with the speech understanding task.

[0109] Compared to the first effective speech recognition model, the second effective speech recognition model has a higher detection efficiency, but it cannot correct missed effective speech signals online based on speech content.

[0110] To address this, this embodiment of the invention first performs preliminary detection based on a second effective speech recognition model to determine the probability that each frame of the audio signal is a valid speech signal. If the probability of any frame being a valid speech signal is within a preset range (e.g., the preset range could be 0.45 to 0.55), it indicates that the accuracy of the second effective speech recognition model has a problem. In this case, a correction is then performed based on the first effective speech recognition model, that is, audio features are applied to determine the valid speech signals in the audio signal to ensure the detection accuracy of valid speech signals.

[0111] Optionally, users can also manually configure whether to use the first effective speech recognition model or the second effective speech recognition model for detection, as needed. For example, manually configuring the state=1 indicates that the first effective speech recognition model should be used for detection to achieve higher detection accuracy; manually configuring the state=0 indicates that the second effective speech recognition model should be used for detection to achieve higher detection efficiency.

[0112] For example, the second effective speech recognition model can be a BoostedVAD frame-by-frame classifier built with a Linear+Softmax structure. This classifier can use cross-entropy as the training loss function, that is, the loss value of the second effective speech recognition model can be determined based on the following formula:

[0113]

[0114] in, This represents the loss value of the second effective speech recognition model, which is compared with the effective speech detection loss value of the first detection model. same, This indicates the total number of speech frames in the audio signal. , This indicates that the corresponding frame signal is a valid speech frame signal (sp). This indicates that the corresponding frame signal is an invalid speech frame signal (nosp).

[0115] Based on any of the above embodiments, after determining the valid speech signal in the audio signal, the method further includes:

[0116] Based on the task type, the temporal granularity of the valid speech signal is determined.

[0117] Based on time granularity, the effective speech signal is smoothed to obtain a smoothed signal;

[0118] Based on the smoothed signal, the recognition result for the corresponding recognition task is determined.

[0119] Specifically, considering that different types of recognition tasks have different requirements for the temporal granularity of effective speech signals, for example, for speech content recognition tasks, the goal is to transcribe spoken content into text, usually focusing on the overall meaning and semantics of sentences. Content recognition relies more on the contextual information of the language, so it can be based on speech signals with a larger temporal granularity. For attribute recognition tasks (such as language recognition, voiceprint recognition, etc.), the goal is to identify the speaker's identity or language type, usually requiring the capture of more detailed acoustic features. These attributes often change within a short period of time, thus requiring a more detailed analysis of the temporal granularity of the speech signal. The temporal granularity of the effective speech signal can be understood as the size of the time unit or time window used when analyzing or processing the speech signal.

[0120] To address this, embodiments of the present invention determine the temporal granularity of the valid speech signal based on the type of recognition task. Then, based on this temporal granularity, the valid speech signal is smoothed to reduce noise and unnecessary fluctuations, highlighting effective features and obtaining a smoothed signal. For example, during smoothing, the window size can be determined based on the temporal granularity, and the average value of the signal within the window can be used for smoothing to obtain a smoothed signal. For recognition tasks requiring a larger temporal granularity, a larger window can be used for smoothing; for recognition tasks requiring a smaller temporal granularity, a smaller window can be used.

[0121] After obtaining the smoothed signal, the recognition results can be extracted according to the specific recognition task. For example, the required features (MFCC, pitch, energy, etc.) can be extracted from the smoothed signal, and a classifier for the recognition task can be used to perform recognition based on the extracted features to obtain the final recognition result.

[0122] Based on any of the above embodiments, the effective speech signal is a frame-level effective speech signal, which can be understood as the first effective speech recognition model performing effective speech detection on the audio signal in units of frames.

[0123] The effective speech signal is smoothed to obtain a smoothed signal, including:

[0124] If the recognition task type is speech content recognition, then each frame signal is smoothed to obtain a smoothed signal;

[0125] If the recognition task type is attribute recognition, the frame-level valid speech signal is converted into a sample point-level valid speech signal, and the signal of each sample point is smoothed to obtain a smooth signal.

[0126] Specifically, if the recognition task type is speech content recognition, it means that the recognition task requires a large temporal granularity of the speech signal. In this case, each frame of the signal can be directly smoothed to preserve the temporal information and frequency characteristics of the speech, thereby accurately recognizing the speech content.

[0127] If the recognition task type is attribute recognition, it means that the recognition task requires a small temporal granularity of the speech signal. In this case, the frame-level effective speech signal is converted into a sampling point-level effective speech signal, and the signal of each sampling point is smoothed to extract fine-grained attribute features (such as pitch, speech rate, intonation changes, etc.) and then accurately perform attribute recognition.

[0128] Based on any of the above embodiments, smoothing processing is performed on each frame signal, including:

[0129] A preset number of frame signals are extended before and after each frame signal;

[0130] Fill in the gaps between each pair of adjacent speech segments with no sound segments;

[0131] If the total number of frames for any audio segment is less than the first preset number of frames, then that audio segment is discarded.

[0132] If the total number of frames for any audio segment is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames, then that audio segment will be merged with the nearest audio segment.

[0133] Specifically, a preset number of frame signals are extended before and after each frame signal to achieve smoothing, i.e., the frame signals are extended to the left. Frame, expand to the right Frame, in which .

[0134] If a gap is detected between two speech segments, a silent segment can be inserted between each pair of adjacent speech segments, such as by inserting a silence frame. This silent segment insertion is achieved through non-SP smoothing. If the number of non-SP frames between consecutive speech segments does not exceed `d` frames (e.g., `d=50`), then the latter SP segment is concatenated with the former SP segment. This is done by ensuring that the difference between the start time of the latter SP segment and the end time of the former SP segment does not exceed `d` frames. The two segments are then concatenated into one segment, with the start time of the concatenated segment being the start time of the former segment and the end time being the end time of the latter segment. Here, a speech segment can be understood as a segment composed of multiple consecutive frames of signal.

[0135] If the total number of frames for any audio segment is less than the first preset number of frames, then that audio segment is discarded. This smoothing process involves discarding excessively short segments; for example, discarding segments that are no more than [a certain number of frames]. The sp segment of a frame, such as =5.

[0136] If any speech segment has a total frame count greater than or equal to a first preset frame count and less than or equal to a second preset frame count, then that speech segment is merged with the nearest speech segment. This is achieved by smoothing the data through splicing shorter segments. For example, if the total frame count f of any speech segment sp is short... ,like If the total number of frames for the nearest speech segment exceeds a certain threshold, then the next nearest speech segment is concatenated with it. Frame, such as If the condition is not met, then no splicing operation will be performed. Indicates the first preset frame number. This indicates the second preset frame number.

[0137] Figure 5 This is one of the flowcharts illustrating the smoothing process provided by the present invention, such as... Figure 5As shown, during effective speech detection, the probability (sp probability) of each frame in the audio signal being an effective speech signal is obtained. When the probability exceeds a threshold 'a', the corresponding frame signal is considered an effective speech (sp) frame signal. Then, the endpoints of sp are calculated. The calculation method is as follows: if multiple consecutive frames are sp, the first consecutive frame is the VAD start time, and the last consecutive frame is the VAD end time. Then, smoothing strategies such as left and right frame expansion, non-sp smoothing, discarding too-short segments, and splicing shorter segments are applied. Finally, the sp endpoints are redefined, and the smoothed signal is output as the final VAD recognition result.

[0138] Based on any of the above embodiments, smoothing processing is performed on the signals at each sampling point, including:

[0139] If the total number of sampling points for any speech segment is less than the preset number, then that speech segment is discarded.

[0140] If the total number of sampling points for any audio segment is less than the preset number, then that audio segment is discarded. This is done by discarding segments that are too short for smoothing purposes. For example, segments that are too short (sp) with fewer than g sampling points are discarded. For example, if g=800 (taking the above 5 frames as an example, the segments of 5 frames of 16kHz audio are 5×16000 / 100=800).

[0141] Figure 6 This is the second schematic diagram of the smoothing process provided by the present invention, as shown below. Figure 6 As shown, during effective speech detection, the frame-level audio signal is converted into a sampling-point-level audio signal to obtain the probability (sp probability) that each sampling point signal in the audio signal is a valid speech signal. When the probability exceeds a threshold b, the corresponding sampling point signal is considered a valid speech (sp) frame signal. Then, the endpoints of sp are calculated. The calculation method is as follows: if multiple consecutive sampling point signals are sp, the first consecutive sampling point is the VAD start time, and the last consecutive sampling point is the VAD end time. Then, a short segment discarding strategy is used for smoothing, and the smoothed signal is output as the final VAD recognition result.

[0142] Based on any of the above embodiments Figure 7 This is a flowchart illustrating another effective speech detection method provided by the present invention, as shown below. Figure 7 As shown, the method includes:

[0143] An audio encoder is used to extract audio features from the audio signal to be detected. These features are then input into a first detection model to obtain the probability that each frame of the audio signal output by the first effective speech recognition model is a valid speech signal. Based on this probability, it is determined whether online error correction should be performed. If so, the audio features are input into a second effective speech recognition model for online error correction, resulting in the valid speech signal in the final audio signal. If not, the output of the first effective speech recognition model is used as the final valid speech signal.

[0144] The valid speech signal is input to the post-processing smoothing module, which performs smoothing to obtain a smoothed signal. The smoothing strategy of the post-processing smoothing module can be as follows: Figure 5 and Figure 6 conduct.

[0145] The effective speech detection device provided by the present invention is described below. The effective speech detection device described below can be referred to in correspondence with the effective speech detection method described above.

[0146] Based on any of the above embodiments Figure 8 This is a schematic diagram of the structure of the effective speech detection device provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0147] Extraction unit 810 is used to extract audio features of the audio signal to be detected based on a feature extraction model;

[0148] The detection unit 820 is used to determine the valid speech signal in the audio signal by applying audio features based on the first valid speech recognition model.

[0149] The feature extraction model and the first effective speech recognition model constitute the first detection model. The first detection model is jointly trained with the speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0150] Based on any of the above embodiments, the samples for joint training include sample audio signals, valid speech tags for the sample audio signals, transcription tags corresponding to the sample audio signals, and punctuation tags for the speech text.

[0151] The joint training process is as follows:

[0152] Based on the feature extraction model, sample audio features of the sample audio signal are extracted;

[0153] Based on a speech understanding model, the transcribed text of the sample audio features is determined and punctuation marks in the transcribed text are identified.

[0154] Based on the first effective speech recognition model, the sample audio features are applied to determine the predicted speech signal;

[0155] Based on the differences between the predicted speech signal and the effective speech label, the differences between the transcribed text and the corresponding transcribed labels of the sample audio signal, and the differences between punctuation marks and punctuation labels, the network parameters of the first detection model and the speech understanding model are updated.

[0156] Based on any of the above embodiments, the initial model of the first detection model is updated with parameters based on the differences between the predicted speech signal and the effective speech tag, the differences between the transcribed text and the corresponding transcribed tags of the sample audio signal, and the differences between punctuation marks and punctuation tags, to obtain the first detection model, including:

[0157] The effective speech detection loss value is determined based on the difference between the predicted speech signal and the effective speech label;

[0158] Based on the difference between the transcribed text and the corresponding transcribed tags of the sample audio signal, the speech transcription loss value in the speech understanding loss value is determined.

[0159] Based on the difference between punctuation marks and punctuation labels, the punctuation prediction loss value in the speech understanding loss value is determined;

[0160] Based on preset weights, the effective speech detection loss value, speech transcription loss value, and punctuation prediction loss value are weighted and summed to determine the joint training loss value;

[0161] Based on the joint training loss value, update the network parameters of the first detection model and the speech understanding model.

[0162] Based on any of the above embodiments, based on the first effective speech recognition model, audio features are applied to determine the effective speech signal in the audio signal, including:

[0163] Based on the second effective speech recognition model, audio features are applied to determine the probability that each frame of the audio signal is a valid speech signal.

[0164] If the probability of any frame signal being a valid speech signal is within a preset range, then based on the first valid speech recognition model, audio features are applied to determine the valid speech signal in the audio signal.

[0165] The second effective speech recognition model is trained based on sample audio signals and effective speech labels for the sample audio signals.

[0166] Based on any of the above embodiments, after determining the valid speech signal in the audio signal, the method further includes:

[0167] The temporal granularity of the valid speech signal is determined based on the type of recognition task.

[0168] Based on time granularity, the effective speech signal is smoothed to obtain a smoothed signal;

[0169] Based on the smoothed signal, the recognition result for the corresponding recognition task is determined.

[0170] Based on any of the above embodiments, the effective speech signal is a frame-level effective speech signal;

[0171] The effective speech signal is smoothed to obtain a smoothed signal, including:

[0172] If the recognition task type is speech content recognition, then each frame signal is smoothed to obtain a smoothed signal;

[0173] If the recognition task type is attribute recognition, the frame-level valid speech signal is converted into a sample point-level valid speech signal, and the signal of each sample point is smoothed to obtain a smooth signal.

[0174] Based on any of the above embodiments, smoothing processing is performed on each frame signal, including:

[0175] A preset number of frame signals are extended before and after each frame signal;

[0176] Fill in the gaps between each pair of adjacent speech segments with no sound segments;

[0177] If the total number of frames for any audio segment is less than the first preset number of frames, then that audio segment is discarded.

[0178] If the total number of frames for any audio segment is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames, then that audio segment will be merged with the nearest audio segment.

[0179] Based on any of the above embodiments, smoothing processing is performed on the signals at each sampling point, including:

[0180] If the total number of sampling points for any speech segment is less than the preset number, then that speech segment is discarded.

[0181] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an effective speech detection method. This method includes: extracting audio features of the audio signal to be detected based on a feature extraction model; applying the audio features based on a first effective speech recognition model to determine the effective speech signal in the audio signal; the feature extraction model and the first effective speech recognition model constitute a first detection model, which is jointly trained with a speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0182] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the effective speech detection method provided by the above methods. The method includes: extracting audio features of an audio signal to be detected based on a feature extraction model; applying the audio features based on a first effective speech recognition model to determine the effective speech signal in the audio signal; the feature extraction model and the first effective speech recognition model constitute a first detection model, which is jointly trained with a speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0184] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an effective speech detection method provided by the methods described above. The method includes: extracting audio features of an audio signal to be detected based on a feature extraction model; and applying the audio features to determine a valid speech signal in the audio signal based on a first effective speech recognition model. The feature extraction model and the first effective speech recognition model constitute a first detection model. The first detection model is jointly trained with a speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An effective speech detection method, characterized in that, include: Based on the feature extraction model, audio features of the audio signal to be detected are extracted; Based on the first effective speech recognition model, the audio features are applied to determine the effective speech signals in the audio signal; The feature extraction model and the first effective speech recognition model constitute the first detection model. The first detection model is jointly trained with the speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model. The samples for joint training include sample audio signals, valid speech tags for the sample audio signals, transcription tags corresponding to the sample audio signals, and punctuation tags for the speech text. The joint training process is as follows: Based on the feature extraction model, the sample audio features of the sample audio signal are extracted; Based on the speech understanding model, the sample audio features are applied to determine the transcribed text of the sample audio features and to identify punctuation marks in the transcribed text; Based on the first effective speech recognition model, the sample audio features are applied to determine the predicted speech signal; Based on the differences between the predicted speech signal and the effective speech tag, the differences between the transcribed text and the corresponding transcription tag of the sample audio signal, and the differences between the punctuation mark and the punctuation tag, the network parameters of the first detection model and the speech understanding model are updated.

2. The effective speech detection method according to claim 1, characterized in that, The method of updating the parameters of the initial model of the first detection model based on the differences between the predicted speech signal and the effective speech tags, the differences between the transcribed text and the corresponding transcribed tags of the sample audio signal, and the differences between the punctuation marks and the punctuation tags, to obtain the first detection model, includes: The effective speech detection loss value is determined based on the difference between the predicted speech signal and the effective speech label; Based on the difference between the transcribed text and the corresponding transcription tags of the sample audio signal, the speech transcription loss value in the speech understanding loss value is determined; Based on the difference between the punctuation marks and the punctuation labels, the punctuation prediction loss value in the speech understanding loss value is determined; Based on preset weights, the effective speech detection loss value, the speech transcription loss value, and the punctuation prediction loss value are weighted and summed to determine the joint training loss value; Based on the joint training loss value, update the network parameters of the first detection model and the speech understanding model.

3. The effective speech detection method according to any one of claims 1 to 2, characterized in that, The step of determining the valid speech signal in the audio signal by applying the audio features based on the first effective speech recognition model includes: Based on the second effective speech recognition model, the probability of each frame signal in the audio signal being a valid speech signal is determined by applying the audio features. If the probability of any frame signal being a valid speech signal is within a preset range, then based on the first valid speech recognition model, the audio features are applied to determine the valid speech signal in the audio signal. The second effective speech recognition model is trained based on the sample audio signal and the effective speech labels of the sample audio signal.

4. The effective speech detection method according to any one of claims 1 to 2, characterized in that, After determining the valid speech signal in the audio signal, the method further includes: The temporal granularity of the effective speech signal is determined based on the type of recognition task. Based on the time granularity, the effective speech signal is smoothed to obtain a smoothed signal; Based on the smoothed signal, the recognition result for the corresponding recognition task is determined.

5. The effective speech detection method according to claim 4, characterized in that, The effective speech signal is a frame-level effective speech signal; The smoothing process of the effective speech signal to obtain a smoothed signal includes: If the recognition task type is speech content recognition, then each frame signal is smoothed to obtain the smoothed signal; If the recognition task type is attribute recognition, then the frame-level valid speech signal is converted into a sampling point-level valid speech signal, and the signal of each sampling point is smoothed to obtain the smoothed signal.

6. The effective speech detection method according to claim 5, characterized in that, The smoothing process for each frame signal includes: A preset number of frame signals are extended before and after each frame signal; Fill in the gaps between each pair of adjacent speech segments with no sound segments; If the total number of frames for any audio segment is less than the first preset number of frames, then that audio segment is discarded. If the total number of frames for any audio segment is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames, then that audio segment is merged with the nearest audio segment.

7. The effective speech detection method according to claim 5, characterized in that, The smoothing process for the signals at each sampling point includes: If the total number of sampling points for any speech segment is less than the preset number, then that speech segment is discarded.

8. An effective speech detection device, characterized in that, include: The extraction unit is used to extract audio features of the audio signal to be detected based on the feature extraction model; The detection unit is used to determine the valid speech signal in the audio signal based on the first effective speech recognition model and by applying the audio features. The feature extraction model and the first effective speech recognition model constitute the first detection model. The first detection model is jointly trained with the speech understanding model during the training phase. The speech understanding model uses the audio features extracted by the feature extraction model as input to predict speech content. The total loss value of the joint training includes the effective speech detection loss value of the first detection model and the speech understanding loss value of the speech understanding model. The samples for joint training include sample audio signals, valid speech tags for the sample audio signals, transcription tags corresponding to the sample audio signals, and punctuation tags for the speech text. The joint training process is as follows: Based on the feature extraction model, the sample audio features of the sample audio signal are extracted; Based on the speech understanding model, the sample audio features are applied to determine the transcribed text of the sample audio features and to identify punctuation marks in the transcribed text; Based on the first effective speech recognition model, the sample audio features are applied to determine the predicted speech signal; Based on the differences between the predicted speech signal and the effective speech tag, the differences between the transcribed text and the corresponding transcription tag of the sample audio signal, and the differences between the punctuation mark and the punctuation tag, the network parameters of the first detection model and the speech understanding model are updated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the effective speech detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the effective speech detection method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the effective speech detection method as described in any one of claims 1 to 7.

Images