A method, device, equipment and storage medium for voice endpoint detection

Abstract

The present application provides a method, apparatus, device and storage medium for voice endpoint detection, wherein the voice endpoint detection method can determine whether the audio frame contained in the audio data to be detected is a silence frame, a noise frame or a voice frame, that is, the present application can detect the relatively accurate attributes of the audio frame contained in the audio data, and perform detection of the voice front-end point and the voice back-end point on this basis, so as to obtain relatively accurate detection results. On the basis of realizing voice endpoint detection, the present application can obtain the recognition text of the voice segment, and can determine the semantic scene of the recognition text based on the semantics of the recognition text, and then can set a suitable post-silence timeout threshold based on the semantic scene of the recognition text, thereby triggering a post-silence timeout event based on the suitable post-silence timeout threshold to improve the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of voice endpoint detection, and in particular to a voice endpoint detection method, apparatus, device and storage medium. Background Art

[0002] Voice Activity Detection (VAD) is a crucial step in speech recognition. It's understandable that not all audio frames in audio data are speech frames. Voice Activity Detection can identify speech endpoints, and thus speech segments. Compared to recognizing the entire audio data, recognizing only the speech segments in the audio data significantly reduces the amount of data and computation required, thereby shortening recognition time.

[0003] Most of the current voice endpoint detection schemes are as follows: predict whether the audio frame contained in the audio data to be detected is a silent frame or a non-silent frame; if the audio frame is a silent frame, determine that the audio frame is a non-speech frame; if the audio frame is a non-silent frame, determine that the audio frame is a speech frame; after obtaining the detection result of whether the audio frame is a speech frame or a non-speech frame, detect the voice endpoint based on the detection result.

[0004] However, in actual application scenarios, non-silent frames are not necessarily speech frames. Determining non-silent frames as speech frames will ultimately make it difficult to obtain more accurate speech endpoint detection results. Summary of the Invention

[0005] In view of this, the present application provides a voice endpoint detection method, apparatus, device, and storage medium to address the problem of low detection accuracy in existing voice endpoint detection solutions. The technical solution is as follows:

[0006] A voice endpoint detection method, comprising:

[0007] After obtaining audio data to be detected, obtaining first information and second information corresponding to an audio frame contained in the audio data, wherein the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is pronunciation information of the corresponding audio frame;

[0008] Based on the first information and the second information corresponding to the audio frames included in the audio data, distinguishing the audio frames included in the audio data as silence frames, noise frames, and speech frames;

[0009] According to the discrimination result corresponding to the audio frame contained in the audio data, the speech front-end point and the speech back-end point are detected.

[0010] Optionally, the voice endpoint detection method further includes:

[0011] After detecting the speech front point, recognizing the speech segment starting from the speech front point as text to obtain the recognized text and the confidence level corresponding to the recognized text;

[0012] A post-silence timeout threshold is set according to the semantics of the recognized text and the confidence level corresponding to the recognized text, and a post-silence timeout event is triggered based on the set post-silence timeout threshold.

[0013] Optionally, obtaining the first information and the second information corresponding to the audio frame contained in the audio data includes:

[0014] Using a pre-established multi-task joint model, predicting acoustic scores of silence frames and non-silence frames of the audio frames included in the audio data, as well as full phoneme acoustic scores corresponding to the audio frames included in the audio data;

[0015] Among them, the acoustic scores of an audio frame, namely a silent frame and a non-silent frame, are used as the first information corresponding to the audio frame, and the full phoneme acoustic score corresponding to an audio frame is used as the second information corresponding to the audio frame. The full phoneme acoustic score includes the acoustic scores corresponding to all phonemes contained in the language to which the audio data belongs.

[0016] Optionally, the distinguishing, based on the first information and the second information corresponding to the audio frames contained in the audio data, between silence frames, noise frames, and speech frames includes:

[0017] For the audio frame to be determined, determining whether the audio frame is a silent frame or a non-silent frame according to the first information corresponding to the audio frame;

[0018] When it is determined that the audio frame is a non-silent frame, the audio frame is distinguished as a noise frame or a speech frame according to the second information corresponding to the audio frame.

[0019] Optionally, the distinguishing the audio frame as a noise frame or a speech frame according to the second information corresponding to the audio frame includes:

[0020] If the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is less than the preset acoustic score threshold, the audio frame is determined to be a noise frame;

[0021] If the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is greater than or equal to a preset acoustic score threshold, the audio frame is determined to be a speech frame.

[0022] Optionally, the step of recognizing a speech segment starting from the speech front-end point as text to obtain recognized text and a confidence level corresponding to the recognized text includes:

[0023] Starting from the voice front-end, decoding the second information corresponding to the voice frame through a pre-constructed phoneme-level network, and ending the decoding after detecting the voice back-end and completing the decoding of the second information corresponding to the voice back-end, wherein the decoding is performed synchronously with the voice back-end detection, and the phoneme-level network is constructed based on the first corpus in the first corpus and the second corpus in the second corpus, the first corpus is semantically incomplete and requires a long wait before triggering the post-silence timeout event, and the second corpus is semantically complete and does not require a long wait to trigger the post-silence timeout event;

[0024] By backtracking the optimal decoding path, an optimal decoding result and a confidence level corresponding to the optimal decoding result are obtained as the recognized text and the confidence level corresponding to the recognized text.

[0025] Optionally, constructing a phoneme-level network based on the first corpus in the first corpus and the second corpus in the second corpus includes:

[0026] Connecting the first corpus in the first corpus and the second corpus in the second corpus in parallel to obtain a sentence-level network, where each corpus is a node in the sentence-level network;

[0027] Expand each corpus in the sentence-level network into single words to obtain an initial word-level network, where each single word is a node in the initial word-level network;

[0028] Merging the nodes and paths in the initial word-level network to obtain a final word-level network;

[0029] Expanding each word in the final word-level network into a phoneme to obtain an initial phoneme-level network, wherein each phoneme is a node in the initial phoneme-level network;

[0030] The nodes and paths in the initial phoneme-level network are merged to obtain a final phoneme-level network.

[0031] Optionally, setting a post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text includes:

[0032] Determining a semantic scene of the recognized text from set semantic scenes according to the semantics of the recognized text and the confidence level corresponding to the recognized text;

[0033] Setting a post-silence timeout threshold according to the semantic scenario of the recognized text;

[0034] Among them, the set semantic scenes include a first scene, a second scene and a default scene, each scene has a corresponding post-silence timeout threshold, the post-silence timeout threshold corresponding to the first scene is greater than the default post-silence timeout threshold corresponding to the default scene, and the post-silence timeout threshold corresponding to the second scene is less than the default post-silence timeout threshold corresponding to the default scene.

[0035] Optionally, determining the semantic scene of the recognized text from a set semantic scene based on the semantics of the recognized text and the confidence level corresponding to the recognized text includes:

[0036] Determining whether the recognized text is credible based on the confidence level corresponding to the recognized text;

[0037] If the recognized text is not credible, determining the semantic scene of the recognized text as a default scene;

[0038] If the recognized text is credible, determining the semantic similarity between the recognized text and the first corpus in the first corpus and the semantic similarity between the recognized text and the second corpus in the second corpus;

[0039] According to the determined semantic similarity, the semantic scene of the recognized text is determined from the set semantic scenes.

[0040] Optionally, determining the semantic scene of the recognized text from the set semantic scenes according to the determined semantic similarity includes:

[0041] If a maximum semantic similarity among the determined semantic similarities is greater than or equal to a preset similarity threshold, and the maximum semantic similarity is a similarity between the recognized text and the first corpus, determining that the semantic scene of the recognized text is the first scene;

[0042] If the maximum semantic similarity is greater than or equal to the preset similarity threshold, and the maximum semantic similarity is the semantic similarity between the recognized text and the second corpus, determining that the semantic scene of the recognized text is the second scene;

[0043] If the maximum semantic similarity is less than the preset similarity threshold, the semantic scene of the recognized text result is determined to be the default scene.

[0044] Optionally, setting a post-silence timeout threshold according to the semantic scenario of the recognized text includes:

[0045] If the semantic scene of the recognized text is the first scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the first scene;

[0046] If the semantic scene of the recognized text is the second scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the second scene;

[0047] If the semantic scene of the recognized text is the default scene, the post-silence timeout threshold is set to the default post-silence timeout threshold corresponding to the default scene.

[0048] A speech endpoint detection device comprises: a discrimination information acquisition module, an audio frame discrimination module and a speech endpoint detection module;

[0049] The discrimination information acquisition module is configured to, after obtaining the audio data to be detected, obtain first information and second information corresponding to an audio frame contained in the audio data, wherein the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is pronunciation information of the corresponding audio frame;

[0050] The audio frame identification module is configured to identify the audio frames contained in the audio data as silence frames, noise frames, and speech frames based on the first information and the second information corresponding to the audio frames contained in the audio data;

[0051] The speech endpoint detection module is used to detect the speech front-end endpoint and the speech back-end endpoint according to the discrimination result corresponding to the audio frame contained in the audio data.

[0052] Optionally, the voice endpoint detection device further includes: a voice segment recognition module, a post-silence timeout threshold setting module, and a post-silence timeout event triggering module;

[0053] The speech segment recognition module is used to recognize the speech segment starting from the speech front end point as text after the speech endpoint detection module detects the speech front end point, so as to obtain the recognized text and the confidence level corresponding to the recognized text;

[0054] The post-silence timeout threshold setting module is used to set the post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text;

[0055] The post-silence timeout event triggering module is configured to trigger a post-silence timeout event based on the post-silence timeout threshold set by the post-silence timeout threshold setting module.

[0056] A voice endpoint detection device, comprising: a memory and a processor;

[0057] The memory is used to store programs;

[0058] The processor is used to execute the program to implement each step of the voice endpoint detection method described in any one of the above items.

[0059] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the various steps of any of the above-mentioned voice endpoint detection methods.

[0060] The voice endpoint detection method, apparatus, equipment and storage medium provided in the present application, after obtaining the audio data to be detected, first obtains the first information (the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame) and the second information (the second information is the pronunciation information of the corresponding audio frame) corresponding to the audio frame contained in the audio data, and then uses the first information and the second information corresponding to the audio frame contained in the audio data as a basis to distinguish the audio frames contained in the audio data as silent frames, noise frames and speech frames, and finally detects the voice front-end point and the voice back-end point according to the discrimination results corresponding to the audio frames contained in the audio data. The voice endpoint detection method provided in the present application can distinguish whether the audio frame contained in the audio data is a silent frame, a noise frame or a speech frame, that is, the present application can detect the more accurate attributes of the audio frames contained in the audio data, and perform detection of the voice front-end point and the voice back-end point on this basis, so as to obtain more accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0062] Figure 1 A flowchart of a voice endpoint detection method provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of another voice endpoint detection method provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of an example of a sentence-level network provided in an embodiment of the present application;

[0065] Figure 4 The embodiments of this application provide Figure 3 Schematic diagram of the initial word-level network obtained by expanding each corpus in the sentence-level network into single words;

[0066] Figure 5 The embodiment of this application provides Figure 4 The result of merging nodes and paths of the initial word-level network shown;

[0067] Figure 6A schematic diagram of the structure of a voice endpoint detection device provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of the voice endpoint detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0070] In the process of implementing this application, the applicant discovered that most of the existing speech endpoint detection schemes are feature-based detection methods and model-based detection methods. Among them, the starting point of the feature-based detection method is to find features that can characterize the differences between silent frames and non-silent frames to achieve the distinction between silent frames and non-silent frames. The model-based detection method distinguishes whether the audio frame is a silent frame or a non-silent frame by modeling silent frames and non-silent frames. It can be seen that whether it is a feature-based detection method or a model-based detection method, for the audio frame in the audio data, it can only detect whether it is a silent frame or a non-silent frame. However, a non-silent frame is not necessarily a speech frame. Using a non-silent frame as a speech frame for speech endpoint detection will result in inaccurate detected speech endpoints. That is, the existing speech endpoint detection scheme cannot accurately detect the attributes of the audio frame, which ultimately makes it difficult to obtain a more accurate speech endpoint detection result.

[0071] Given that the existing voice endpoint detection accuracy is not high, the applicant attempted to propose a voice endpoint detection method with higher detection accuracy and conducted research for this purpose. Through continuous research, a voice endpoint detection method was finally proposed. This voice endpoint detection method can determine whether the audio frame contained in the audio data to be detected is a silence frame, a noise frame, or a voice frame. In other words, it can determine the more accurate attributes of the audio frame, and perform voice endpoint detection on this basis to obtain more accurate detection results. On the basis of realizing voice endpoint detection, the applicant also proposed to recognize the voice segment from the voice front-end point to the voice back-end point as text, set a suitable post-silence timeout threshold according to the semantics of the recognized text, and then trigger a post-silence timeout event based on the suitable post-silence timeout threshold to improve the user experience.

[0072] The voice endpoint detection method proposed in this application can be applied to electronic devices with data processing capabilities. The electronic device can be a terminal used on the user side, such as a smartphone, PC, notebook, PAD, smart home appliance, car terminal, etc. The electronic device can also be a server on the network side (which can be a single server, multiple servers or server clusters). The electronic device can detect relatively accurate voice endpoints according to the voice endpoint detection method provided in this application. The voice endpoint detection method provided in this application is introduced through the following examples.

[0073] First embodiment

[0074] See also Figure 1 , shows a flow chart of a voice endpoint detection method provided by an embodiment of the present application, which may include:

[0075] Step S101: After obtaining audio data to be detected, first information and second information corresponding to audio frames contained in the audio data are obtained.

[0076] The first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is the pronunciation information of the corresponding audio frame.

[0077] Optionally, the first information corresponding to an audio frame may include scores of the audio frame as a silent frame and a non-silent frame (ie, an acoustic score of the audio frame as a silent frame and an acoustic score of the audio frame as a non-silent frame).

[0078] Optionally, the second information may be the full phoneme acoustic score corresponding to the corresponding audio frame. The full phoneme acoustic score refers to the acoustic scores corresponding to all phonemes contained in the language to which the audio data belongs. Assuming that the language to which the audio data belongs contains all N phonemes, the second information includes the acoustic scores corresponding to the N phonemes.

[0079] Optionally, the process of obtaining the first information and the second information corresponding to the audio frames contained in the audio data may include: using a pre-established multi-task joint model to predict the acoustic scores of the audio frames contained in the audio data as silent frames and non-silent frames, respectively, and the full phoneme acoustic scores corresponding to the audio frames contained in the audio data.

[0080] Specifically, using a pre-established multi-task joint model, the process of predicting the acoustic scores of silence frames and non-silence frames of the audio frames included in the audio data, as well as the full phoneme acoustic scores corresponding to the audio frames included in the audio data may include:

[0081] Step a1: Obtain audio features of audio frames contained in audio data.

[0082] The audio features may include but are not limited to filterbank features, mfcc features, etc.

[0083] Step a2: Using a pre-established multi-task joint model, based on the audio features of the audio frames contained in the audio data, predict the acoustic scores of the audio frames contained in the audio data as silent frames and non-silent frames, as well as the full phoneme acoustic scores corresponding to the audio frames contained in the audio data.

[0084] Specifically, the audio features of the audio frames contained in the audio data are input into the multi-task joint model. The multi-task joint model predicts the acoustic scores of the corresponding audio frames, namely, silent frames and non-silent frames, as well as the full phoneme acoustic scores of the corresponding audio frames based on the input audio features.

[0085] Among them, the multi-task joint model includes an input layer and a hidden layer, and a first output layer and a second output layer that share the input layer and the hidden layer. The input layer is used to input the audio features of the audio frames contained in the audio data, the hidden layer is used to process the audio features input by the input layer, the first output layer is used to predict the acoustic scores of the corresponding audio frames, namely, silent frames and non-silent frames, according to the output of the hidden layer, and the second output layer is used to predict the full phoneme acoustic scores corresponding to the corresponding audio frames according to the output of the hidden layer.

[0086] It should be noted that the first output layer of the multi-task joint model first predicts the probability that the corresponding audio frames are silent frames and non-silent frames according to the output of the hidden layer, and then determines the acoustic scores of the corresponding audio frames as silent frames and non-silent frames according to the probability that the corresponding audio frames are silent frames and non-silent frames. Similarly, the second output layer first predicts the probability that the phonemes corresponding to the corresponding audio frames are the various phonemes in the full phonemes according to the output of the hidden layer, and then determines the acoustic scores of the phonemes corresponding to the corresponding audio frames as the various phonemes in the full phonemes according to the probability that the phonemes corresponding to the corresponding audio frames are the various phonemes in the full phonemes, that is, the full phoneme acoustic scores.

[0087] The multi-task joint model is trained using the training audio data and the first category label and the second category label corresponding to each audio frame contained in the training audio data, wherein the first category label corresponding to an audio frame is used to indicate whether the audio frame is a silent frame or a non-silent frame, and the second category label is used to indicate which phoneme the phoneme corresponding to the audio frame is among all phonemes. During training, the multi-task joint model has two tasks, one of which is to learn the classification of silent and non-silent, and the other is to learn the classification of all phonemes. For the two tasks, two loss functions are set, and the model parameters are updated according to the two loss functions, wherein one of the two loss functions is determined according to the probability of the audio frame being a silent frame and the probability of being a non-silent frame predicted by the multi-task joint model, as well as the first category label corresponding to the audio frame, and the other of the two loss functions is determined according to the probability of the phoneme corresponding to the audio frame being each phoneme among all phonemes predicted by the multi-task joint model, as well as the second category label corresponding to the audio frame.

[0088] Step S102: Based on the first information and the second information corresponding to the audio frames contained in the audio data, the audio frames contained in the audio data are distinguished as silence frames, noise frames, and speech frames.

[0089] Specifically, based on the first information and the second information corresponding to the audio frames included in the audio data, the process of distinguishing the audio frames included in the audio data as silence frames, noise frames, and speech frames may include:

[0090] Step b1: for each audio frame to be identified, identify the audio frame as a silent frame or a non-silent frame according to the first information corresponding to the audio frame.

[0091] The above content mentioned that the first information may include the acoustic scores of the corresponding audio frames being silent frames and non-silent frames respectively. In view of this, for the audio frame to be judged, it can be judged whether the audio frame is a silent frame or a non-silent frame based on the acoustic scores of the audio frame being a silent frame and a non-silent frame respectively. Specifically, if the acoustic score of the audio frame being a silent frame is greater than the acoustic score of the audio frame being a non-silent frame, the audio frame is judged to be a silent frame; otherwise, the audio frame is judged to be a non-silent frame.

[0092] Optionally, the first information may also include the frame energy of the corresponding audio frame. In view of this, for the audio frame to be judged, whether the audio frame is a silent frame or a non-silent frame can be judged based on the acoustic scores of the audio frame as a silent frame and a non-silent frame respectively and the frame energy of the audio frame. Specifically, if the frame energy of the audio frame is less than a preset frame energy threshold, and the acoustic score of the audio frame as a silent frame is greater than the acoustic score of the audio frame as a non-silent frame, the audio frame is judged to be a silent frame; otherwise, the audio frame is judged to be a non-silent frame.

[0093] Step b2: When the audio frame is determined to be a non-silent frame, the audio frame is distinguished as a noise frame or a speech frame according to the second information corresponding to the audio frame.

[0094] The above content mentioned that the first information may be the full phoneme score corresponding to the corresponding audio frame. In view of this, for the audio frame to be judged, it can be judged whether the audio frame is a noise frame or a speech frame based on the full phoneme score corresponding to the audio frame. Specifically, if the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is less than the preset acoustic score threshold, the audio frame is judged to be a noise frame; if the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is greater than or equal to the preset acoustic score threshold, the audio frame is judged to be a speech frame.

[0095] Step S103: Detecting the front-end point of speech and the back-end point of speech according to the discrimination result corresponding to the audio frame included in the audio data.

[0096] Specifically, when a first preset number of consecutive speech frames appear in the audio data, it is determined that a speech front end point has been detected, and the first speech frame among the first preset number of consecutive speech frames is determined as the speech front end point. After the speech front end point is detected, if a second preset number of consecutive non-speech frames appear in the audio data, it is determined that a speech back end point has been detected, and the forward adjacent speech frame of the first non-speech frame among the second preset number of consecutive non-speech frames is determined as the speech back end point. Optionally, the first preset number of frames can be, but is not limited to, an integer within the interval [10, 20], and the second preset number of frames can be, but is not limited to, an integer within the interval [30, 40].

[0097] Exemplarily, the first preset frame number is 20, and the second preset frame number is 40. Assuming that the 11th to 30th audio frames of the audio data are determined to be speech frames, it can be determined that the speech front end point is detected, and the 11th audio frame is determined as the speech front end point. After detecting the speech front end point, assuming that the 40 consecutive audio frames after the 60th audio frame are all non-speech frames, it can be determined that the speech back end point is detected, and the 60th audio frame is determined as the speech back end point.

[0098] It should be noted that if no voice front point is detected, it is determined whether the preceding silence duration exceeds a preset preceding silence timeout threshold. If it exceeds the preset preceding silence timeout threshold, the detection is terminated.

[0099] The voice endpoint detection method provided in the embodiment of the present application, after obtaining the audio data to be detected, first obtains the first information corresponding to the audio frame contained in the audio data (the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame) and the second information (the second information is the pronunciation information of the corresponding audio frame), and then uses the first information and the second information corresponding to the audio frame contained in the audio data as a basis to distinguish the audio frame contained in the audio data as a silent frame, a noise frame and a speech frame, and finally detects the voice front-end point and the voice back-end point according to the discrimination result corresponding to the audio frame contained in the audio data. The voice endpoint detection method provided in the embodiment of the present application can distinguish whether the audio frame contained in the audio data is a silent frame, a noise frame or a speech frame, that is, the embodiment of the present application can detect the more accurate attributes of the audio frame contained in the audio data, and perform the detection of the voice front-end point and the voice back-end point on this basis, so as to obtain a more accurate voice endpoint detection result. Compared with the existing voice endpoint detection method, the voice endpoint detection method provided in the embodiment of the present application improves the detection accuracy of the voice endpoint, and lays a solid foundation for the subsequent use of the voice segment.

[0100] Second embodiment

[0101] In some application scenarios, after detecting a voice endpoint, it is necessary to detect the post-silence duration. When the post-silence duration is greater than the set post-silence timeout threshold, a post-silence timeout event is triggered so that an application can perform some operations after receiving the post-silence timeout event.

[0102] For example, in recording scenarios, voice assistants on electronic devices typically need to automatically stop recording after the user finishes a complete, semantically sound sentence in order to recognize the user's speech and then perform semantic recognition and subsequent operations on the recognized content. Currently, this method automatically stops recording by setting a fixed post-silence timeout threshold, such as 800ms. After detecting the end point of speech, the post-silence duration is detected. If the post-silence duration exceeds the post-silence timeout threshold, such as 800ms, a post-silence timeout event is triggered. When the voice assistant receives the post-silence timeout event, recording stops. Currently, statistics on live user behavior show that based on the 800ms post-silence timeout event, approximately 95% of users' utterances are semantically complete, while approximately 4% of users' utterances are semantically incomplete. If the post-silence timeout event can be triggered more quickly after most users have finished speaking a complete, semantically sound sentence, and a longer post-silence timeout threshold can be set for users who have not expressed a complete, semantically sound sentence, giving them time to think, this can improve the user experience.

[0103] In view of this, this application provides another voice endpoint detection method, please refer to Figure 2 , which shows a flow chart of the voice endpoint detection method, which may include:

[0104] Step S201: After obtaining audio data to be detected, first information and second information corresponding to audio frames contained in the audio data are obtained.

[0105] The first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information can be the pronunciation information of the corresponding audio frame. Optionally, the first information can be the scores of the corresponding audio frame as a silent frame and a non-silent frame, respectively, and the second information can be the full phoneme acoustic score corresponding to the corresponding audio frame.

[0106] Step S202: Based on the first information and the second information corresponding to the audio frames included in the audio data, the audio frames included in the audio data are distinguished as silence frames, noise frames, and speech frames.

[0107] Step S203: Detecting the speech front-end point according to the discrimination result corresponding to the audio frame included in the audio data.

[0108] It should be noted that if no voice front end is detected, it is determined whether the previous silence duration exceeds the preset front silence timeout threshold. If it exceeds the preset front silence timeout threshold, the detection is terminated. If a voice front end is detected, the subsequent steps are executed.

[0109] Step S204: After the front-end point of the speech is detected, the speech segment starting from the front-end point of the speech is recognized as text, and during the recognition process, the back-end point of the speech is detected synchronously according to the discrimination result corresponding to the audio frame contained in the audio data. After the recognition from the front-end point of the speech to the back-end point of the speech is completed, the recognition is ended, and the recognized text and the confidence level corresponding to the recognized text are obtained.

[0110] It should be noted that the specific implementation process and related explanations of "detecting the rear endpoint of the voice based on the discrimination results corresponding to the audio frames contained in the audio data" in steps S201 to S203 and step S204 can be found in the specific implementation process and related explanations of steps S101 to S103 in the above embodiment, and will not be repeated here in this embodiment.

[0111] The process of recognizing a speech segment starting from a speech front-end point as text to obtain the recognized text and the confidence level corresponding to the recognized text may include:

[0112] Step c1: Starting from the speech front-end point, the second information corresponding to the speech frame (such as the full phoneme score corresponding to the speech frame) is decoded through a pre-built phoneme-level network.

[0113] Optionally, decoding may be performed using any decoding algorithm such as Viterbi, DTW, etc.

[0114] It should be noted that during the decoding process, the voice backend endpoint is synchronously detected. After detecting the voice backend endpoint and completing the decoding of the second information corresponding to the voice backend endpoint (such as the full phoneme score corresponding to the voice frame), the decoding ends.

[0115] The phoneme-level network in step c1 is constructed based on the first corpus in the first corpus set and the second corpus in the second corpus set. Among them, the first corpus set includes multiple first corpora, and the first corpus is a corpus with incomplete semantics that requires long-term waiting and then triggers a post-silence timeout event, such as "Let me think", "I want to find a", etc. The second corpus set includes multiple second corpora, and the second corpus is a corpus with complete semantics that does not require long-term waiting to trigger a post-silence timeout event, such as "Call the police", "Shut down immediately", etc.

[0116] Specifically, the process of constructing a phoneme-level network based on the first corpus in the first corpus set and the second corpus in the second corpus set may include:

[0117] Step d1: Parallelize the first corpus in the first corpus set and the second corpus in the second corpus set to obtain a sentence-level network.

[0118] Among them, each corpus is a node in the sentence-level network.

[0119] It should be noted that in step d1, all the corpora in the two corpus sets are parallelized together, and there is no need to distinguish and process the first corpus and the second corpus.

[0120] Please refer to Figure 3 , which shows a schematic diagram of an example of the sentence-level network. Figure 3 The shown sentence-level network is obtained by parallelizing the corpus "Call Zhang San" and the corpus "Call Li Si". The start of the sentence-level network is a silence node, and each corpus in the sentence-level network ends with a silence node.

[0121] Step d2: Expand each corpus in the sentence-level network into single characters to obtain an initial character-level network.

[0122] Among them, each single character is a node in the initial character-level network.

[0123] Please refer to Figure 4 , which shows Figure 3 a schematic diagram of the initial character-level network obtained by expanding each corpus in the shown sentence-level network into single characters. As Figure 4 shown, expanding "Call Zhang San" into single characters gives "Da", "Dian", "Hua", "Gei", "Zhang", "San", and the same is true for "Call Li Si". Each node in the initial character-level network except the silence node is a character in the corpus.

[0124] Step d3: Merge the nodes and paths in the initial character-level network to obtain the final character-level network.

[0125] Since there are identical nodes in some paths of the initial character-level network, in this embodiment, the identical nodes in different paths are merged, and path merging is inevitably performed when merging nodes. Optionally, a directed graph merging algorithm can be used to merge the nodes and paths in the initial character-level network.

[0126] As Figure 4 shown, Figure 4 there are identical nodes 'da', 'dian', 'hua','sil' on two paths in Figure 5 It shows the result of merging the nodes and paths of the initial character-level network shown in Figure 4 As shown in Figure 5 by merging the nodes and paths, redundant nodes and paths are removed, and the network structure of the character-level network is simplified.

[0127] Step d3: Expand each single character in the final character-level network into phonemes to obtain the initial phoneme-level network.

[0128] Each phoneme is a node in the initial phoneme-level network.

[0129] Optionally, each single character in the final character-level network can be expanded into single-phone phonemes. Taking the node 'da' in Figure 5 as an example, expanding 'da' into single-phone phonemes gives 'd' and 'a'. This embodiment is not limited to this, and each single character in the final character-level network can also be expanded into multi-phone phonemes, such as double-phone phonemes or triple-phone phonemes. Taking the node 'da' in Figure 5 as an example, after expanding 'da' into double-phone phonemes, it becomes'sil-d', 'd+a', and after expanding 'da' into triple-phone phonemes, it becomes'sil-d+a', 'd-a+d'.

[0130] Step d4: Merge the nodes and paths in the initial phoneme-level network to obtain the final phoneme-level network.

[0131] Similar to the initial character-level network, there may also be identical nodes on different paths in the initial phoneme-level network. Therefore, in this embodiment, the identical nodes on different paths in the initial phoneme-level network are merged, and path merging is inevitably performed when merging nodes. Optionally, a directed graph merging algorithm can be used to merge the nodes and paths in the initial phoneme-level network.

[0132] Step c2: by backtracking the optimal decoding path, obtain the optimal decoding result and the confidence level corresponding to the optimal decoding result, use the optimal decoding result as the recognized text, and use the confidence level corresponding to the optimal decoding result as the confidence level corresponding to the recognized text.

[0133] Step S205: setting a post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text, and triggering a post-silence timeout event based on the set post-silence timeout threshold.

[0134] Specifically, according to the semantics of the recognized text and the confidence level corresponding to the recognized text, the process of setting the post-silence timeout threshold may include:

[0135] Step e1: Determine the semantic scene of the recognized text from the set semantic scenes according to the semantics of the recognized text and the confidence level corresponding to the recognized text.

[0136] Among them, the set semantic scenes may include a first scene, a second scene and a default scene, each scene has a corresponding post-silence timeout threshold, the post-silence timeout threshold corresponding to the first scene is greater than the default post-silence timeout threshold corresponding to the default scene, and the post-silence timeout threshold corresponding to the second scene is less than the default post-silence timeout threshold corresponding to the default scene. It should be noted that the default post-silence timeout threshold corresponding to the default scene can be set according to the specific business scenario. For example, in the above-mentioned recording scenario, the default post-silence timeout threshold corresponding to the default scene can be set to 800ms. Based on this, the post-silence timeout threshold corresponding to the first scene is set to be greater than 800ms, and the post-silence timeout threshold corresponding to the second scene is set to be less than 800ms.

[0137] Specifically, the process of determining the semantic scene of the recognized text from the set semantic scenes according to the semantics of the recognized text and the confidence level corresponding to the recognized text may include:

[0138] Step e1-1: Determine whether the recognized text is credible based on the confidence level corresponding to the recognized text.

[0139] Specifically, if the confidence level corresponding to the recognized text is greater than or equal to a preset confidence threshold, the recognized text is determined to be credible; if the confidence level corresponding to the recognized text is less than the preset confidence threshold, the recognized text is determined to be uncredible.

[0140] Step e1-2a: If the recognized text is not credible, the semantic scene of the recognized text is determined to be a default scene.

[0141] Step e1-2b-1: If the recognized text is credible, determine the semantic similarity between the recognized text and the first corpus in the first corpus and the semantic similarity between the recognized text and the second corpus in the second corpus.

[0142] Specifically, when determining the similarity between a recognized text and a corpus, the representation vector of each word in the recognized text and the representation vector of each word in the corpus can be obtained first, and then the representation vector of each word in the recognized text is encoded to obtain the semantic representation vector of the recognized text. Similarly, the representation vector of each word in the corpus is encoded to obtain the semantic representation vector of the corpus. Finally, the semantic similarity between the recognized text and the corpus is determined based on the semantic representation vector of the recognized text and the semantic representation vector of the corpus. Optionally, any one of MLP, CNN, RNN, Self-attention, Transformer encoder, BERT, etc. can be used to encode the representation vector of each word in the recognized text and to encode the representation vector of each word in the corpus. When determining the similarity between the recognized text and the corpus based on the semantic representation vector of the recognized text and the semantic representation vector of the corpus, the cosine similarity, Gaussian distance, etc. of the semantic representation vector of the recognized text and the semantic representation vector of the corpus can be calculated.

[0143] Step e1-2b-2: Determine the semantic scene of the recognized text from the set semantic scenes according to the determined semantic similarity.

[0144] Specifically, based on the determined semantic similarity, the process of determining the semantic scene of the recognized text from the set semantic scene can include: if the maximum semantic similarity among the determined semantic similarities is greater than or equal to the preset similarity threshold, the semantic scene of the recognized text can be determined to be one of the first scene and the second scene. Further, if the maximum semantic similarity is the similarity between the recognized text and the first corpus, the semantic scene of the recognized text is determined to be the first scene; if the maximum semantic similarity is the semantic similarity between the recognized text and the second corpus, the semantic scene of the recognized text is determined to be the second scene; if the maximum semantic similarity is less than the preset similarity threshold, the semantic scene of the recognized text result is determined to be the default scene.

[0145] Step e2: Set a post-silence timeout threshold based on the semantic context of the recognized text.

[0146] Specifically, according to the semantic scene of the recognized text, the process of setting the post-silence timeout threshold may include: if the semantic scene of the recognized text is the first scene, the post-silence timeout threshold is set to the post-silence timeout threshold corresponding to the first scene; if the semantic scene of the recognized text is the second scene, the post-silence timeout threshold is set to the post-silence timeout threshold corresponding to the second scene; if the semantic scene of the recognized text is the default scene, the post-silence timeout threshold is set to the default post-silence timeout threshold corresponding to the default scene. It should be noted that when the semantic scene of the recognized text is the first scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the first scene can trigger the post-silence timeout event more quickly. When the semantic scene of the recognized text is the second scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the second scene can leave time for the user to think and ensure that the user's semantically complete words are obtained.

[0147] After setting the post-silence timeout threshold according to the semantic scenario of the recognized text, the post-silence timeout event can be triggered based on the set post-silence timeout threshold. Specifically, it is determined whether the post-silence duration exceeds the set post-silence timeout threshold. If so, the detection is ended and the post-silence timeout event is triggered. It should be noted that if the post-silence duration does not exceed the set post-silence timeout threshold, the voice front-end point continues to be detected. If the voice front-end point is detected, step S204 is continued to be executed. If no voice front-end point is detected, "determining whether the post-silence duration exceeds the set post-silence timeout threshold" is continued until it is detected that the post-silence duration exceeds the set post-silence timeout threshold or there is no detectable audio data.

[0148] The voice endpoint detection method provided in the embodiment of the present application can determine whether the audio frame contained in the audio data is a silence frame, a noise frame, or a voice frame. That is, the embodiment of the present application can detect the more accurate attributes of the audio frame contained in the audio data, and perform detection of the voice front-end point and the voice back-end point on this basis, so as to obtain more accurate detection results. On the basis of realizing voice endpoint detection, the present application can obtain the recognition text of the voice segment, and can determine the semantic scene of the recognition text based on the semantics of the recognition text, and then can set a suitable post-silence timeout threshold based on the semantic scene of the recognition text, thereby triggering a post-silence timeout event based on the suitable post-silence timeout threshold to improve the user experience.

[0149] Third embodiment

[0150] An embodiment of the present application also provides a speech endpoint detection device. The speech endpoint detection device provided in the embodiment of the present application is described below. The speech endpoint detection device described below and the speech endpoint detection method described above can be referenced to each other.

[0151] See also Figure 6, shows a structural schematic diagram of the voice endpoint detection device provided in an embodiment of the present application, which may include: a discrimination information acquisition module 601, an audio frame discrimination module 602 and a voice endpoint detection module 603.

[0152] The discrimination information acquisition module 601 is configured to obtain first information and second information corresponding to an audio frame contained in the audio data after obtaining the audio data to be detected.

[0153] The first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is the pronunciation information of the corresponding audio frame.

[0154] The audio frame identification module 602 is configured to identify the audio frames contained in the audio data as silence frames, noise frames, and speech frames based on the first information and the second information corresponding to the audio frames contained in the audio data.

[0155] The speech endpoint detection module 603 is used to detect the speech front-end endpoint and the speech back-end endpoint according to the discrimination result corresponding to the audio frame contained in the audio data.

[0156] Optionally, the voice endpoint detection device provided in the embodiment of the present application may further include: a voice segment recognition module, a post-silence timeout threshold setting module and a post-silence timeout event triggering module.

[0157] The speech segment recognition module is used to recognize the speech segment starting from the speech front end point as text after the speech endpoint detection module detects the speech front end point, so as to obtain the recognized text and the confidence level corresponding to the recognized text.

[0158] The post-silence timeout threshold setting module is used to set the post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text.

[0159] The post-silence timeout event triggering module is configured to trigger a post-silence timeout event based on the post-silence timeout threshold set by the post-silence timeout threshold setting module.

[0160] Optionally, when obtaining the first information and the second information corresponding to the audio frame included in the audio data, the discrimination information acquisition module 601 is specifically configured to:

[0161] Using a pre-established multi-task joint model, the acoustic scores of the audio frames contained in the audio data, which are silent frames and non-silent frames, and the full phoneme acoustic scores corresponding to the audio frames contained in the audio data are predicted.

[0162] Among them, the acoustic scores of an audio frame, namely a silent frame and a non-silent frame, are used as the first information corresponding to the audio frame, and the full phoneme acoustic score corresponding to an audio frame is used as the second information corresponding to the audio frame. The full phoneme acoustic score includes the acoustic scores corresponding to all phonemes contained in the language to which the audio data belongs.

[0163] Optionally, the audio frame identification module 602 includes a first identification submodule and a second identification submodule.

[0164] The first distinguishing submodule is configured to distinguish, for an audio frame to be distinguished, whether the audio frame is a silent frame or a non-silent frame according to first information corresponding to the audio frame.

[0165] The second discrimination submodule is configured to discriminate between a noise frame and a speech frame based on the second information corresponding to the audio frame when the audio frame is determined to be a non-silent frame.

[0166] Optionally, when the second distinguishing submodule distinguishes the audio frame as a noise frame or a speech frame according to the second information corresponding to the audio frame, the second distinguishing submodule is specifically configured to:

[0167] If the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is less than the preset acoustic score threshold, the audio frame is determined to be a noise frame; if the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is greater than or equal to the preset acoustic score threshold, the audio frame is determined to be a speech frame.

[0168] Optionally, the speech segment recognition module includes: a decoding module and a decoding result acquisition module.

[0169] The decoding module is used to decode the second information corresponding to the speech frame through a pre-constructed phoneme-level network starting from the speech front-end point, and end decoding after detecting the speech back-end point and completing the decoding of the second information corresponding to the speech back-end point, wherein the decoding is performed synchronously with the speech back-end point detection, and the phoneme-level network is constructed based on the first corpus in the first corpus and the second corpus in the second corpus, the first corpus is semantically incomplete and requires a long wait before triggering the post-silence timeout event, and the second corpus is semantically complete and does not require a long wait to trigger the post-silence timeout event.

[0170] The decoding result acquisition module is used to obtain the optimal decoding result and the confidence level corresponding to the optimal decoding result by backtracing the optimal decoding path, as the recognized text and the confidence level corresponding to the recognized text.

[0171] Optionally, the speech endpoint detection device provided in the embodiment of the present application may further include: a phoneme-level network construction module.

[0172] The phoneme-level network construction module is used to construct a phoneme-level network according to the first corpus in the first corpus and the second corpus in the second corpus.

[0173] Optionally, when constructing a phoneme-level network based on the first corpus in the first corpus and the second corpus in the second corpus, the phoneme-level network construction module is specifically configured to:

[0174] The first corpus in the first corpus and the second corpus in the second corpus are connected in parallel to obtain a sentence-level network, where each corpus is a node in the sentence-level network; each corpus in the sentence-level network is expanded into a single word to obtain an initial word-level network, where each single word is a node in the initial word-level network; the nodes and paths in the initial word-level network are merged to obtain a final word-level network; each single word in the final word-level network is expanded into a phoneme to obtain an initial phoneme-level network, where each phoneme is a node in the initial phoneme-level network; the nodes and paths in the initial phoneme-level network are merged to obtain a final phoneme-level network.

[0175] Optionally, the post-silence timeout threshold setting module includes: a semantic scene determination submodule and a post-silence timeout threshold setting submodule.

[0176] The semantic scene determination submodule is used to determine the semantic scene of the recognized text from set semantic scenes according to the semantics of the recognized text and the confidence level corresponding to the recognized text.

[0177] Among them, the set semantic scenes include a first scene, a second scene and a default scene, each scene has a corresponding post-silence timeout threshold, the post-silence timeout threshold corresponding to the first scene is greater than the default post-silence timeout threshold corresponding to the default scene, and the post-silence timeout threshold corresponding to the second scene is less than the default post-silence timeout threshold corresponding to the default scene.

[0178] The post-silence timeout threshold setting submodule is used to set the post-silence timeout threshold according to the semantic scenario of the recognized text.

[0179] Optionally, when determining the semantic scene of the recognized text from set semantic scenes based on the semantics of the recognized text and the confidence level corresponding to the recognized text, the semantic scene determination submodule is specifically configured to:

[0180] Based on the confidence level corresponding to the recognized text, determine whether the recognized text is credible; if the recognized text is not credible, determine the semantic scene of the recognized text as a default scene; if the recognized text is credible, determine the semantic similarity between the recognized text and the first corpus in the first corpus set, and determine the semantic similarity between the recognized text and the second corpus in the second corpus set; based on the determined semantic similarity, determine the semantic scene of the recognized text from the set semantic scene.

[0181] Optionally, when determining the semantic scene of the recognized text from the set semantic scenes according to the determined semantic similarity, the semantic scene determination submodule is specifically configured to:

[0182] If the maximum semantic similarity among the determined semantic similarities is greater than or equal to a preset similarity threshold, and the maximum semantic similarity is the similarity between the recognized text and the first corpus, then the semantic scene of the recognized text is determined to be the first scene; if the maximum semantic similarity is greater than or equal to the preset similarity threshold, and the maximum semantic similarity is the semantic similarity between the recognized text and the second corpus, then the semantic scene of the recognized text is determined to be the second scene; if the maximum semantic similarity is less than the preset similarity threshold, then the semantic scene of the recognized text result is determined to be the default scene.

[0183] Optionally, when the post-silence timeout threshold setting submodule sets the post-silence timeout threshold according to the semantic scenario of the recognized text, it is specifically configured to:

[0184] If the semantic scene of the recognized text is the first scene, the post-silence timeout threshold is set to the post-silence timeout threshold corresponding to the first scene; if the semantic scene of the recognized text is the second scene, the post-silence timeout threshold is set to the post-silence timeout threshold corresponding to the second scene; if the semantic scene of the recognized text is the default scene, the post-silence timeout threshold is set to the default post-silence timeout threshold corresponding to the default scene.

[0185] The speech endpoint detection device provided in the embodiment of the present application can determine whether the audio frame contained in the audio data is a silence frame, a noise frame, or a speech frame based on the first information corresponding to the audio frame contained in the audio data to be detected (the first information can indicate whether the corresponding audio frame is a silence frame or a non-silent frame) and the second information (the second information is the pronunciation information of the corresponding audio frame). That is, the embodiment of the present application can detect the more accurate attributes of the audio frame contained in the audio data, and perform detection of the speech front-end point and the speech back-end point on this basis, so as to obtain more accurate detection results. On the basis of realizing speech endpoint detection, the speech endpoint detection device provided in the embodiment of the present application can obtain the recognition text of the speech segment, and can determine the semantic scene of the recognition text based on the semantics of the recognition text, and then can set a suitable post-silence timeout threshold based on the semantic scene of the recognition text, thereby triggering a post-silence timeout event based on the suitable post-silence timeout threshold to improve the user experience.

[0186] Fourth embodiment

[0187] This application embodiment also provides a voice endpoint detection device, see Figure 7 , shows a schematic structural diagram of the voice endpoint detection device, the voice endpoint detection device may include: at least one processor 701, at least one communication interface 702, at least one memory 703 and at least one communication bus 704;

[0188] In the embodiment of the present application, the number of the processor 701, the communication interface 702, the memory 703, and the communication bus 704 is at least one, and the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704;

[0189] The processor 701 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0190] The memory 703 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0191] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0192] After obtaining audio data to be detected, obtaining first information and second information corresponding to an audio frame contained in the audio data, wherein the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is pronunciation information of the corresponding audio frame;

[0193] Based on the first information and the second information corresponding to the audio frames included in the audio data, distinguishing the audio frames included in the audio data as silence frames, noise frames, and speech frames;

[0194] According to the discrimination result corresponding to the audio frame contained in the audio data, the speech front-end point and the speech back-end point are detected.

[0195] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0196] Fifth embodiment

[0197] An embodiment of the present application further provides a computer-readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0198] After obtaining audio data to be detected, obtaining first information and second information corresponding to an audio frame contained in the audio data, wherein the first information can indicate whether the corresponding audio frame is a silent frame or a non-silent frame, and the second information is pronunciation information of the corresponding audio frame;

[0199] Based on the first information and the second information corresponding to the audio frames included in the audio data, distinguishing the audio frames included in the audio data as silence frames, noise frames, and speech frames;

[0200] According to the discrimination result corresponding to the audio frame contained in the audio data, the speech front-end point and the speech back-end point are detected.

[0201] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0202] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0203] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0204] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting speech endpoints, characterized in that: include: After obtaining the audio data to be detected, using a pre-established multi-task joint model, predicting the acoustic scores of the audio frames contained in the audio data, which are respectively silent frames and non-silent frames, and the full-phoneme acoustic scores corresponding to the audio frames contained in the audio data; wherein the acoustic scores of an audio frame, which are respectively silent frames and non-silent frames, serve as the first information corresponding to the audio frame, and the full-phoneme acoustic score corresponding to the audio frame serves as the second information corresponding to the audio frame, and the full-phoneme acoustic score includes the acoustic scores corresponding to all phonemes contained in the language to which the audio data belongs; Based on the first information and the second information corresponding to the audio frames included in the audio data, distinguishing the audio frames included in the audio data as silence frames, noise frames, and speech frames; According to the discrimination result corresponding to the audio frame contained in the audio data, the speech front-end point and the speech back-end point are detected.

2. The speech endpoint detection method according to claim 1, wherein: Also includes: After detecting the speech front point, recognizing the speech segment starting from the speech front point as text to obtain the recognized text and the confidence level corresponding to the recognized text; A post-silence timeout threshold is set according to the semantics of the recognized text and the confidence level corresponding to the recognized text, and a post-silence timeout event is triggered based on the set post-silence timeout threshold.

3. The speech endpoint detection method according to claim 1, characterized in that The step of distinguishing between a silence frame, a noise frame, and a speech frame for the audio frame included in the audio data based on the first information and the second information corresponding to the audio frame included in the audio data includes: For the audio frame to be determined, determining whether the audio frame is a silent frame or a non-silent frame according to the first information corresponding to the audio frame; When it is determined that the audio frame is a non-silent frame, the audio frame is distinguished as a noise frame or a speech frame according to the second information corresponding to the audio frame.

4. The speech endpoint detection method according to claim 3, characterized in that: The distinguishing the audio frame as a noise frame or a speech frame according to the second information corresponding to the audio frame includes: If the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is less than a preset acoustic score threshold, the audio frame is determined to be a noise frame; If the maximum acoustic score among the full phoneme acoustic scores corresponding to the audio frame is greater than or equal to a preset acoustic score threshold, the audio frame is determined to be a speech frame.

5. The speech endpoint detection method according to claim 2, wherein: The step of recognizing a speech segment starting from the speech front-end point as text to obtain a recognized text and a confidence level corresponding to the recognized text includes: Starting from the voice front-end, decoding the second information corresponding to the voice frame through a pre-constructed phoneme-level network, and ending the decoding after detecting the voice back-end and completing the decoding of the second information corresponding to the voice back-end, wherein the decoding is performed synchronously with the voice back-end detection, and the phoneme-level network is constructed based on the first corpus in the first corpus and the second corpus in the second corpus, the first corpus is semantically incomplete and requires a long wait before triggering the post-silence timeout event, and the second corpus is semantically complete and does not require a long wait to trigger the post-silence timeout event; By backtracking the optimal decoding path, an optimal decoding result and a confidence level corresponding to the optimal decoding result are obtained as the recognized text and the confidence level corresponding to the recognized text.

6. The method for voice endpoint detection according to claim 5, wherein: Constructing a phoneme-level network based on the first corpus in the first corpus and the second corpus in the second corpus includes: Connecting the first corpus in the first corpus and the second corpus in the second corpus in parallel to obtain a sentence-level network, where each corpus is a node in the sentence-level network; Expand each corpus in the sentence-level network into single words to obtain an initial word-level network, where each single word is a node in the initial word-level network; Merging the nodes and paths in the initial word-level network to obtain a final word-level network; Expanding each word in the final word-level network into a phoneme to obtain an initial phoneme-level network, wherein each phoneme is a node in the initial phoneme-level network; The nodes and paths in the initial phoneme-level network are merged to obtain a final phoneme-level network.

7. The method for voice endpoint detection according to claim 2, wherein: The step of setting a post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text includes: Determining a semantic scene of the recognized text from set semantic scenes according to the semantics of the recognized text and the confidence level corresponding to the recognized text; Setting a post-silence timeout threshold according to the semantic scenario of the recognized text; Among them, the set semantic scenes include a first scene, a second scene and a default scene, each scene has a corresponding post-silence timeout threshold, the post-silence timeout threshold corresponding to the first scene is greater than the default post-silence timeout threshold corresponding to the default scene, and the post-silence timeout threshold corresponding to the second scene is less than the default post-silence timeout threshold corresponding to the default scene.

8. The method for voice endpoint detection according to claim 7, wherein: Determining the semantic scene of the recognized text from a set semantic scene based on the semantics of the recognized text and the confidence level corresponding to the recognized text includes: Determining whether the recognized text is credible based on the confidence level corresponding to the recognized text; If the recognized text is not credible, determining the semantic scene of the recognized text as a default scene; If the recognized text is credible, determining the semantic similarity between the recognized text and the first corpus in the first corpus and the semantic similarity between the recognized text and the second corpus in the second corpus; According to the determined semantic similarity, the semantic scene of the recognized text is determined from the set semantic scenes.

9. The method for voice endpoint detection according to claim 8, wherein: Determining the semantic scene of the recognized text from the set semantic scenes based on the determined semantic similarity includes: If a maximum semantic similarity among the determined semantic similarities is greater than or equal to a preset similarity threshold, and the maximum semantic similarity is a similarity between the recognized text and the first corpus, determining that the semantic scene of the recognized text is the first scene; If the maximum semantic similarity is greater than or equal to the preset similarity threshold, and the maximum semantic similarity is the semantic similarity between the recognized text and the second corpus, determining that the semantic scene of the recognized text is the second scene; If the maximum semantic similarity is less than the preset similarity threshold, the semantic scene of the recognized text result is determined to be the default scene.

10. The method for voice endpoint detection according to claim 7, wherein: The setting of the post-silence timeout threshold according to the semantic scenario of the recognized text includes: If the semantic scene of the recognized text is the first scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the first scene; If the semantic scene of the recognized text is the second scene, setting the post-silence timeout threshold to the post-silence timeout threshold corresponding to the second scene; If the semantic scene of the recognized text is the default scene, the post-silence timeout threshold is set to the default post-silence timeout threshold corresponding to the default scene.

11. A speech endpoint detection device, characterized in that: include: Discrimination information acquisition module, audio frame discrimination module and voice endpoint detection module; The discriminant information acquisition module is configured to, after obtaining the audio data to be detected, use a pre-established multi-task joint model to predict the acoustic scores of the audio frames contained in the audio data, which are respectively silent frames and non-silent frames, and the full-phoneme acoustic scores corresponding to the audio frames contained in the audio data; wherein the acoustic scores of an audio frame, which are respectively silent frames and non-silent frames, serve as the first information corresponding to the audio frame, and the full-phoneme acoustic score corresponding to the audio frame serves as the second information corresponding to the audio frame, and the full-phoneme acoustic score includes the acoustic scores corresponding to all phonemes contained in the language to which the audio data belongs; The audio frame identification module is configured to identify the audio frames contained in the audio data as silence frames, noise frames, and speech frames based on the first information and the second information corresponding to the audio frames contained in the audio data; The speech endpoint detection module is used to detect the speech front-end endpoint and the speech back-end endpoint according to the discrimination result corresponding to the audio frame contained in the audio data.

12. The speech endpoint detection device according to claim 11, characterized in that: Also includes: Voice segment recognition module, post-silence timeout threshold setting module and post-silence timeout event triggering module; The speech segment recognition module is used to recognize the speech segment starting from the speech front end point as text after the speech endpoint detection module detects the speech front end point, so as to obtain the recognized text and the confidence level corresponding to the recognized text; The post-silence timeout threshold setting module is used to set the post-silence timeout threshold according to the semantics of the recognized text and the confidence level corresponding to the recognized text; The post-silence timeout event triggering module is configured to trigger a post-silence timeout event based on the post-silence timeout threshold set by the post-silence timeout threshold setting module.

13. A voice endpoint detection device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the voice endpoint detection method according to any one of claims 1 to 10.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the voice endpoint detection method according to any one of claims 1 to 10 is implemented.

Images