Voice endpoint detection method, related device, equipment and medium

Abstract

The invention discloses a voice endpoint detection method, a related device, equipment and a medium, and the method comprises the steps: carrying out the continuous prediction based on a streaming audio, and obtaining a probability value that an audio frame in the streaming audio belongs to a plurality of frame types; wherein the plurality of frame types comprise human voice, head noise, middle noise and tail noise; determining a target type to which the audio frame belongs in the plurality of frame types based on the probability values of the audio frame belonging to the plurality of frame types and the energy value of the audio frame; continuously monitoring the target type of each audio frame in the streaming audio based on a four-classification state machine to obtain a judgment result of the current frame; wherein the judgment result comprises at least one of whether the whole sentence is ended or not at the current frame and whether the clause is ended or not at the current frame, and adjacent states in the four-classification state machine correspond to any two frame types. According to the scheme, the semantic coherence after voice endpoint detection is executed can be improved.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech endpoint detection method and related apparatus, equipment and medium. Background Technology

[0002] Speech endpoint detection is a key preprocessing module in speech processing systems, designed to distinguish between human and non-human voice segments in audio data in order to better serve downstream tasks such as speech recognition.

[0003] Currently, existing speech endpoint detection technologies often lack the ability to model the overall semantic level. For example, when short pauses occur in audio data, even if the semantics are still within the same sentence, they may be misjudged as the end of a sentence. This misjudgment will be amplified during subsequent audio processing, causing what should be a coherent sentence to be excessively segmented into multiple incomplete fragments, resulting in serious semantic fragmentation and affecting downstream tasks. Therefore, improving the semantic coherence after speech endpoint detection has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a voice endpoint detection method and related apparatus, equipment and medium, which can improve the semantic coherence after performing voice endpoint detection.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a speech endpoint detection method, comprising: performing continuous prediction based on streaming audio to obtain probability values ​​of audio frames belonging to several frame types in the streaming audio; wherein, the several frame types include human voice, first noise, middle noise, and last noise; determining the target type of the audio frame among the several frame types based on the probability values ​​of the audio frame belonging to the several frame types and the energy value of the audio frame; continuously monitoring the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame; wherein, the judgment result includes at least one of whether the current frame is the end of a sentence or the end of a clause, and adjacent states in the four-class state machine correspond to any two frame types.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a speech endpoint detection device, comprising: a type prediction module, a type determination module, and a continuous monitoring module. The type prediction module is used to continuously predict based on streaming audio to obtain probability values ​​of audio frames in the streaming audio belonging to several frame types; wherein, the several frame types include human voice, first noise, middle noise, and last noise; the type determination module is used to determine the target type of an audio frame among the several frame types based on the probability values ​​of the audio frames belonging to the several frame types and the energy value of the audio frames; the continuous monitoring module is used to continuously monitor the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame; wherein, the judgment result includes at least one of whether the current frame is the end of a sentence or the end of a clause, and adjacent states in the four-class state machine correspond to any two frame types.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the voice endpoint detection method of the first aspect described above.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the voice endpoint detection method of the first aspect described above.

[0009] The above scheme continuously monitors streaming audio to obtain the probability values ​​of audio frames belonging to several frame types, including human voice, initial noise, middle noise, and final noise. Based on the probability values ​​and energy values ​​of the audio frames, the target type of each audio frame is determined. Then, a four-class classification state machine is used to continuously monitor the target type of each audio frame in the streaming audio, obtaining the judgment result for the current frame. The judgment result includes at least one of the following: whether the current frame is the end of a sentence or the end of a clause. The adjacent state pairs in the four-class classification state machine are also considered. Applicable to any two frame types, this approach, on the one hand, further refines the noise into initial noise, middle noise, and final noise. This allows for the determination of the semantic position of an audio frame within the streaming audio—whether it's the beginning, end, or middle clause of a sentence—based on whether the audio frame belongs to the initial, middle, or final noise category. This imbues speech endpoint detection with semantic capabilities. On the other hand, using a four-class state machine for continuous monitoring, compared to a simple state machine that only contains state transitions between human and non-human voices, further refines the state transitions, enabling better differentiation between sentences and clauses and helping to avoid excessive segmentation. Therefore, it improves the semantic coherence after performing speech endpoint detection. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the voice endpoint detection method of this application; Figure 2a This is a schematic diagram illustrating an embodiment of the process of obtaining sample audio in this application; Figure 2b This is a schematic diagram of a process of an embodiment of the voice endpoint detection method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the voice endpoint detection device of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0013] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0014] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the voice endpoint detection method of this application. Specifically, it may include the following steps: Step S11: Perform continuous prediction based on streaming audio to obtain the probability values ​​of audio frames in the streaming audio belonging to several frame types.

[0015] In this embodiment, several frame types include human voice, initial noise, intermediate noise, and final noise. It should be noted that initial noise refers to noise before the start of speech; that is, all non-human voice frames before the first human voice frame can be considered initial noise. In practical applications, initial noise usually has different audio characteristics from subsequent speech to help identify the starting boundary of speech activity. Unlike initial noise, final noise refers to noise after the end of speech; that is, all non-human voice frames after the last human voice frame can be considered final noise. In practical applications, introducing final noise can help identify the ending boundary of speech activity. Noise frames between initial noise and final noise can be considered intermediate noise. In practical applications, this noise usually appears in the middle of a speech segment; introducing intermediate noise can help better distinguish between speech and noise regions. Furthermore, the core value of streaming audio lies in breaking the limitation of "processing after complete input," that is, processing the audio stream during audio acquisition / input. For example, in this embodiment of the disclosure, when performing voice endpoint detection on streaming audio, if the end of a sentence is detected in the current frame, the audio data up to the current frame can be sent to the downstream task (e.g., speech recognition). At the same time, voice endpoint detection on streaming audio continues from the current frame. When the end of a sentence is detected again in a new current frame, the audio data up to this new current frame can be sent to the downstream task (e.g., speech recognition). This process is repeated until the streaming audio ends.

[0016] In one implementation scenario, as a possible approach, the probability value can be obtained by a speech activity detection model predicting audio frames in streaming audio. This model can be trained based on sample audio, and each sample audio frame can be labeled with its frame type, such as belonging to human voice, initial noise, middle noise, or final noise. This labeling allows the speech activity detection model to learn the start and end of human voice segments during training. It should be noted that the speech activity detection model can be constructed based on convolutional neural networks and long short-term memory networks; the network structure of the speech activity detection model is not limited here.

[0017] In a specific implementation scenario, the frame type of a sample audio can be further subdivided based on the binary-classified labeled sample audio. Specifically, binary-classified labeled sample audio can be obtained, and after binary classification labeling, each sample audio frame in the sample audio can be labeled with a preset type, which is either human voice or non-human voice. It should be noted that the binary-classified labeled sample audio can be directly taken from relevant open-source datasets, or it can be obtained by further filtering based on relevant open-source datasets (as one possible filtering method, please refer to the relevant description below). Of course, binary-classified labeled sample audio can be custom-collected, or it can be further filtered based on custom collection (as one possible filtering method, please refer to the relevant description below). Based on this, in response to the preset type of the sample audio frame being non-human voice, it can be determined whether to refine the frame type of the sample audio frame to first noise based on whether there are sample audio frames of the preset type being human voice before the sample audio frame, and whether to refine the frame type of the sample audio frame to tail noise based on whether there are sample audio frames of the preset type being human voice after the sample audio frame. Furthermore, if the sample audio frame is located between sample audio frames of the first noise type and sample audio frames of the tail noise type, the frame type of the sample audio frame can be refined to intermediate noise. For example, when the preset type of the sample audio frame is non-human voice, if there are no sample audio frames of the preset type being human voice before the sample audio frame, the frame type of the sample audio frame can be refined to first noise; if there are no sample audio frames of the preset type being human voice after the sample audio frame, the frame type of the sample audio frame can be refined to tail noise. After determining all first noise and tail noise, all sample audio frames of the non-human voice type between the first noise and tail noise can be refined to intermediate noise. The above method, based on the sample audio that has been binary-classified and labeled, further refines the sample audio frame into first noise, last noise, or middle noise when the preset type is non-human voice. This forces the speech activity detection model to learn the start and end of the human voice segment through the labels of human voice, first noise, last noise, or middle noise during the training process.

[0018] In a specific implementation scenario, please refer to the relevant documents. Figure 2a , Figure 2a This is a schematic diagram illustrating an embodiment of the process for obtaining sample audio in this application. Figure 2aAs shown, candidate audio files can first be obtained after binary classification annotation. It should be noted that the annotation results after binary classification include: the preset type to which each candidate audio frame belongs, where the preset type is either human voice or non-human voice. Furthermore, candidate audio files can come from relevant open-source datasets or can be custom-collected; the specific source of the candidate audio files is not limited here. Based on this, the candidate audio files can be predicted using different endpoint detection models, yielding prediction results for each candidate audio file after passing through different endpoint detection models. These prediction results can include the preset type to which each candidate audio frame belongs. For example, a convolutional neural network-based endpoint detection model can be used to predict the candidate audio files, yielding the prediction results; a recurrent neural network-based endpoint detection model can be used to predict the candidate audio files, yielding the prediction results; and a long short-term memory (LSTM)-based endpoint detection model can be used to predict the candidate audio files, yielding the prediction results. Of course, the above example is merely one possible illustration of candidate audio being predicted by different endpoint detection models; other possible scenarios will not be listed here. Furthermore, for each candidate audio: the prediction accuracy of the corresponding endpoint detection model for the candidate audio can be obtained based on the difference between the annotation result of the candidate audio and the prediction result of any endpoint detection model. Based on the prediction accuracy of each endpoint detection model for the candidate audio, it can be determined whether to select the candidate audio as sample audio. It should be noted that for each candidate audio, it can be determined whether the endpoint detection model has made an incorrect prediction for that audio frame based on whether the preset type annotated for the same audio frame is the same as the preset type predicted by the endpoint detection model. Finally, it is possible to statistically analyze whether all audio frames in the candidate audio have made incorrect predictions, thus obtaining the prediction accuracy of the endpoint detection model for the candidate audio. Furthermore, after obtaining the prediction accuracy of each endpoint detection model for the candidate audio, if the prediction accuracy of each endpoint detection model for the candidate audio exceeds a preset proportion (e.g., 80%, 90%, 100%), and the prediction accuracy of each model for the candidate audio is higher than the accuracy threshold (e.g., 0.9, 0.95, 0.99), then the candidate audio can be retained as sample audio; otherwise, it can be excluded from selection as sample audio.For ease of understanding, let's continue with the example of candidate audio being predicted using endpoint detection models based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory (LSTM) networks, respectively. We can determine the prediction accuracy of the CNN-based endpoint detection model by comparing the difference between the labeled results and the prediction results from the RNN-based model, and vice versa. Furthermore, if the prediction accuracy of each of these models is higher than 0.99, the candidate audio can be retained as a sample audio; otherwise, it should not be selected. The above method uses different endpoint detection models to predict candidate audio, obtaining prediction results for each candidate audio through different endpoint detection models. For each candidate audio, the prediction accuracy of the corresponding endpoint detection model is obtained based on the difference between the candidate audio's annotation result and the prediction result of any endpoint detection model. Based on the prediction accuracy of each endpoint detection model for the candidate audio, it is determined whether to select the candidate audio as sample audio. This method, with the assistance of multiple endpoint detection models, can screen for high-quality annotated sample audio as much as possible. Furthermore, as a possible implementation, for each candidate audio not selected as sample audio, samples can be taken and sent to manual binary classification annotation to obtain sample audio. In this way, through a series of operations including sampling and manual annotation, sample audio can be supplemented while ensuring the annotation quality of the supplementary sample audio as much as possible.

[0019] In a specific implementation scenario, after obtaining sample audio frames labeled with their respective frame types, the speech activity detection model can be trained accordingly. Specifically, the speech activity detection model can predict the sample audio, obtaining the predicted probability value of each sample audio frame belonging to its respective frame type. Then, the cross-entropy loss function is used to measure the loss based on the frame type labeled in the sample audio frame and the predicted probability value of each frame type, resulting in the training loss. Based on the training loss, the network parameters of the speech activity detection model can be adjusted. Through training optimization and backpropagation, and iterative loops, the speech activity detection model can be forced to have semantic awareness capabilities, that is, not limited to simple speech activity detection, but able to identify and understand different types of noise and their positions in speech segments, which helps to optimize the accuracy of subsequent downstream tasks such as speech recognition. As a possible implementation example, sample audio frames are labeled with their respective frame types. The labeling granularity can be a large frame (one large frame equals four small frames), while the prediction granularity of the speech activity detection model can be a small frame. Before performing loss measurement, the frame type labeled by the large frame can be "extended" to each of its subordinate small frames. That is, the frame type labeled by the large frame can be copied to each of its subordinate small frames, and then loss measurement can be performed based on the cross-entropy loss function.

[0020] In another implementation scenario, unlike the aforementioned implementation, as another possible approach, audio frames in the streaming audio can be predicted first based on a conventional VAD (Voice Activity Detection) model to obtain a first probability value indicating whether the audio frame belongs to human voice or not. It should be noted that the conventional VAD model can include, but is not limited to, convolutional neural networks, recurrent neural networks, long short-term memory networks, etc., and the network structure of the conventional VAD model is not limited here. Furthermore, unlike the aforementioned speech activity detection model in this disclosure which can perform four-class classification (human voice, initial noise, middle noise, and tail noise), the conventional VAD model only supports binary classification (human voice and non-human voice). Based on this, for each audio frame, we can combine it with several audio frames preceding it to continue prediction, obtaining the second probability values ​​of the audio frame belonging to the first noise, middle noise, and tail noise respectively. Then, based on the second probability values ​​of the audio frame belonging to the first noise, middle noise, and tail noise respectively, we can segment the audio frame according to the first probability value of the audio frame belonging to non-human voice (e.g., probability segmentation can be achieved through conditional probability), obtaining the probability values ​​of the audio frame belonging to the first noise, middle noise, and tail noise respectively. At the same time, the first probability value of the audio frame belonging to human voice can be used as the probability value of the audio frame belonging to human voice. For example, based on the conventional VAD model, predicting audio frames in a convective audio stream yields first probability values ​​of 0.3 and 0.7 for the audio frame belonging to human voice or non-human voice, respectively. Combining this with predictions of several preceding audio frames, second probability values ​​of 0.2, 0.5, and 0.3 for the audio frame belonging to the initial noise, middle noise, and final noise, respectively, can be used to segment the audio frame based on these second probability values ​​(0.2, 0.5, 0.3). This results in probability values ​​of 0.14, 0.35, and 0.21 for the audio frame belonging to the initial noise, middle noise, and final noise, respectively. Therefore, the final probability values ​​for the audio frame belonging to human voice, initial noise, middle noise, and final noise are 0.3, 0.14, 0.35, and 0.21, respectively. Of course, the above example is only one possible scenario in practical applications; other possible scenarios will not be listed here.

[0021] Step S12: Based on the probability values ​​of the audio frames belonging to several frame types and the energy values ​​of the audio frames, determine the target type to which the audio frames belong among the several frame types.

[0022] In one implementation scenario, as one possible implementation method, please refer to [the relevant documentation / reference]. Figure 2b , Figure 2b This is a schematic diagram illustrating the process of an embodiment of the voice endpoint detection method of this application. Figure 2bAs shown, after obtaining the probability values ​​of audio frames belonging to several frame types in the streaming audio, post-processing decisions can be performed based on the energy values ​​of the audio frames to determine the target type of the audio frames among several frame types, and then the data is sent to a four-class classification state machine for continuous monitoring. Specifically, each audio frame can be selected sequentially as a frame to be decided. If a frame to be decided meets the human voice decision criteria, the target type of the frame to be decided can be determined to be human voice. It should be noted that the human voice decision criteria include: the energy value of the frame to be decided is not lower than the energy threshold, the probability value of the frame to be decided belonging to human voice is not lower than the probability threshold, and the frame to be decided is not penalized. Furthermore, if a frame to be decided does not meet the human voice decision criteria, a penalty is imposed on the frame to be decided and several subsequent audio frames if the energy value of the frame to be decided is lower than the energy threshold or the probability value of the frame to be decided belonging to human voice is lower than the probability threshold. It should be noted that if the target type of the frame to be decided is not determined to be human voice, the target type of the frame to be decided can be determined based on the probability values ​​of the frame to be decided belonging to the first noise, middle noise, and tail noise, respectively. By repeating this process, the target type of each audio frame in the process audio can be determined sequentially. This method, by combining the energy value of the audio frame, the probability value predicted by the speech activity detection model to belong to each frame type, and whether it has been penalized by previous non-human voice frames, helps improve the accuracy of the target type determination during post-processing.

[0023] In a specific implementation scenario, as a possible example, the energy value of an audio frame can be calculated using the following formula:

[0024] In the above formula, curenergy represents the energy value of the audio frame, N represents the total number of sampling points in the audio frame, and x i The value represents the i-th sample point in the audio frame, μ represents the average value of all sample points in the audio frame, and frame_length represents the frame length (e.g., usually 400). Of course, the above example is only one possible way to calculate the energy value of an audio frame. Other possible ways to calculate the energy value of an audio frame are not limited here, nor will they be listed one by one.

[0025] In a specific implementation scenario, the specific values ​​of the energy threshold and probability threshold can be set according to the actual application scenario. For example, in a practical application scenario where the voice signal is generally strong, the energy threshold can be set appropriately higher; conversely, in a practical application scenario where the voice signal is generally relatively weak, the energy threshold can be set appropriately lower. Furthermore, when the accuracy of voice recognition is relatively high, the probability threshold can be set appropriately higher; conversely, when the accuracy requirements for voice recognition are relatively relaxed, the probability threshold can be set moderately. Of course, the above examples are merely possible settings for the energy threshold and probability threshold, and the specific values ​​of the energy threshold and probability threshold are not limited here.

[0026] In a specific implementation scenario, when the energy value of the frame to be decided is lower than the energy threshold, or when the probability value of the frame to be decided being a human voice is lower than the probability threshold, a penalty can be applied to the frame to be decided and the two audio frames that follow. Of course, in practical applications, this is not the only possibility. For example, a penalty can be applied to the frame to be decided and the audio frame that follows, or even to the frame to be decided and three or more audio frames that follow. These examples will not be listed here.

[0027] In a specific implementation scenario, as mentioned earlier, if the target type of the frame to be decided is not determined to be human voice, the target type of the frame to be decided can be determined based on the probability values ​​of the frame belonging to the first noise, middle noise, and last noise, respectively. Specifically, the frame to be decided can be selected as the target type of the frame with the highest probability value among the first noise, middle noise, and last noise.

[0028] In a specific implementation scenario, for ease of understanding, a concrete example is provided below. For the first audio frame in the streaming audio, when it is used as the frame to be decided, if it does not meet the human voice decision criteria, then if its energy value is below the energy threshold or its probability of belonging to human voice is below the probability threshold, a penalty can be imposed on it and the following two audio frames. Based on the largest probability value among first noise, middle noise, and tail noise, it is determined to be first noise, and its target type can be identified as first noise. When the second audio frame is used as the frame to be decided, because it has been penalized, it will inevitably not meet the human voice decision criteria. Furthermore, if its probability of belonging to human voice is below the probability threshold or its energy value is below the energy threshold, a penalty can be imposed on it and the following two audio frames. Based on the largest probability value among first noise, middle noise, and tail noise, it is determined to be first noise, and its target type can be identified as first noise. When the third audio frame is the frame to be decided, it will inevitably fail to meet the human voice decision condition due to the penalty it receives. Furthermore, if its probability of belonging to human voice is not lower than the probability threshold and its energy value is not lower than the energy threshold, its target type can be determined as intermediate noise based on the largest probability value among the first noise, middle noise, and tail noise. Similarly, when the fourth audio frame is the frame to be decided, it will inevitably fail to meet the human voice decision condition due to the penalty it receives. Furthermore, if its probability of belonging to human voice is not lower than the probability threshold and its energy value is not lower than the energy threshold, its target type can be determined as intermediate noise based on the largest probability value among the first noise, middle noise, and tail noise. This process is repeated to determine the target type of each audio frame. For example, when "0" represents human voice, "1" represents initial noise, "2" represents intermediate noise, and "3" represents final noise, the above decision process yields the following label sequence for each audio frame in the streaming audio: 112211003223000232000123. It should be noted that, although according to the aforementioned definitions of initial noise, intermediate noise, and final noise, frames 3 and 4 should be initial noise, frames 9, 12, and 17 should be intermediate noise, and frames 22 and 23 should both be final noise, the latter two should not be initial noise and the former intermediate noise respectively. However, in practical applications, decisions are inevitably made due to factors such as energy deviation and model interference. Therefore, using a subsequent four-class state machine to continuously monitor state transitions to determine the end of a clause or sentence can mitigate these errors as much as possible. For details, please refer to the subsequent descriptions; they will not be elaborated upon here.

[0029] In another implementation scenario, unlike the aforementioned implementation, as another possible approach, each audio frame can be selected sequentially as a frame to be decided. If the energy value of the frame to be decided is not lower than an energy threshold and the probability value of the frame belonging to human voice is not lower than a probability threshold, the target type of the frame to be decided is determined to be human voice. Otherwise (i.e., the energy value of the frame to be decided is lower than the energy threshold, or the probability value of the frame belonging to human voice is lower than the probability threshold), the target type of the frame to be decided can be determined based on the probability values ​​of the frame belonging to the first noise, middle noise, and last noise, respectively. For example, the target type of the frame to be decided can be determined based on the largest probability value among the first noise, middle noise, and last noise. This process can be repeated to sequentially determine the target type of each audio frame in the audio stream.

[0030] Step S13: Based on a four-class state machine, continuously monitor the target type of each audio frame in the streaming audio to obtain the judgment result of the current frame.

[0031] In this embodiment of the disclosure, the determination result includes at least one of: whether the current frame is the end of a complete sentence or whether the current frame is the end of a clause. For example, the determination result may include whether the current frame is the end of a complete sentence; or, for example, the determination result may include whether the current frame is the end of a clause; or, for example, the determination result may include both whether the current frame is the end of a clause and whether the current frame is the end of a complete sentence. Furthermore, in this embodiment of the disclosure, adjacent states in the four-class state machine correspond to any two frame types. For example, when the number "0" represents human voice, the number "1" represents initial noise, the number "2" represents intermediate noise, and the number "3" represents tail noise, theoretically, there are a total of eight states (corresponding to different transition scenarios between adjacent frames): 11 (i.e., initial noise transitions to initial noise), 10 (i.e., initial noise transitions to human voice), 00 (i.e., human voice transitions to human voice), 02 (i.e., human voice transitions to intermediate noise), 22 (i.e., intermediate noise transitions to intermediate noise), 20 (i.e., intermediate noise transitions to human voice), 03 (i.e., human voice transitions to tail noise), and 33 (i.e., tail noise transitions to tail noise). Of course, in practical applications, as mentioned above, due to various disturbances, 4 will actually occur. 2 =16 states.

[0032] In one implementation scenario, when the judgment result includes whether the current frame is the end of a complete sentence, a four-class state machine can be used to continuously monitor whether the target type in the streaming audio up to the current frame meets the sentence end detection condition. It should be noted that the sentence end detection condition can include any of the following: the target type of the first consecutive number of audio frames up to the current frame is tail noise; or the target type of the second consecutive number of audio frames up to the current frame is characterized as noise. Furthermore, when the target type of an audio frame is characterized as first noise, middle noise, or tail noise, it can be considered that the audio frame belongs to noise. Based on this, in response to the target type in the streaming audio up to the current frame meeting the sentence end detection condition, it can be determined that the judgment result of the current frame includes that the current frame is the end of a complete sentence; in response to the target type in the streaming audio up to the current frame not meeting the sentence end detection condition, it can be determined that the judgment result of the current frame includes that the current frame is not yet the end of a complete sentence. The above method, by continuously monitoring whether the target type in the streaming audio up to the current frame meets the sentence end detection condition using a four-class state machine, can detect whether the current frame is the end of a complete sentence as promptly and accurately as possible.

[0033] In a specific implementation scenario, the first number can be less than the second number. For example, the first number can be set to 5, and the second number can be set to 10. Of course, the above example is only one possible example of the first and second numbers, and the specific values ​​of the first and second numbers are not limited here. For ease of description, taking the first number as 3 and the second number as 4 as an example, the four-class state machine contains a path 3-3-3, and the judgment result pointed to by this path is the end of the sentence. The four-class state machine can also contain paths such as: 1-2-3-3, 2-2-2-3, 2-2-3-3, 2-3-3-3, etc. (that is, as long as four consecutive states represent noise), and the judgment result pointed to by these paths can be the end of the sentence.

[0034] In a specific implementation scenario, the second number can be equal to the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine (i.e., containing only four state transitions consisting of human voice and non-human voice) for endpoint detection. In the binary classification state machine, adjacent states correspond to any two preset types, where the preset type is either human voice or non-human voice. For example, adjacent states could be human voice to human voice, human voice to non-human voice, non-human voice to human voice, or non-human voice to non-human voice. For instance, if the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine for endpoint detection is 10, the second number can also be set to 10. Of course, the above example is merely one possible example in practical applications, and the specific value of the second number is not limited here.

[0035] In a specific implementation scenario, the first number can be less than the second number, and the second number can be equal to the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine (i.e., containing only four state transitions consisting of human voice and non-human voice) for endpoint detection. For ease of description, the first number can be denoted as n1, and the second number as n2. Then, when the target type of the n1 consecutive audio frames up to the current frame is tail noise, or the target type of the n2 consecutive audio frames up to the current frame is characterized as noise, it can be determined that the sentence has ended at the current frame. That is, as long as one of the aforementioned two judgment sub-conditions is met, it can be determined that the sentence has ended at the current frame. Since the first number n1 is less than the second number n2, the judgment delay for determining the end of the sentence at the current frame when the sentence is determined to end by triggering the former judgment sub-condition is less than the judgment delay for determining whether the sentence has ended at the current frame when triggered by the latter judgment sub-condition, that is, less than the judgment delay for determining whether the sentence has ended at the current frame using a binary classification state machine, which helps to improve the smoothness of interaction.

[0036] In one implementation scenario, when the judgment result includes whether a clause has ended at the current frame, a four-class state machine can be used to continuously monitor whether the target type in the streaming audio up to the current frame meets the clause end detection condition. It should be noted that the clause end detection condition can include any of the following: the target type of the third consecutive audio frame up to the current frame is intermediate noise; or the target type of the fourth consecutive audio frame up to the current frame is characterized as noise. Furthermore, when the target type of an audio frame is characterized as first noise, intermediate noise, or last noise, it can be considered that the audio frame belongs to noise. Based on this, in response to the target type in the streaming audio up to the current frame meeting the clause end detection condition, it can be determined that the judgment result of the current frame includes that a clause has ended at the current frame; in response to the target type in the streaming audio up to the current frame not meeting the clause end detection condition, it can be determined that the judgment result of the current frame includes that a clause has not yet ended at the current frame. The above method, by continuously monitoring whether the target type in the streaming audio up to the current frame meets the clause end detection condition using a four-class state machine, can detect whether a clause has ended at the current frame as promptly and accurately as possible.

[0037] In a specific implementation scenario, the third number can be less than the fourth number. For example, the third number can be set to 8, and the fourth number can be set to 9. Of course, the above example is only one possible example of the third and fourth numbers, and the specific values ​​of the third and fourth numbers are not limited here. For ease of description, taking the third number as 3 and the fourth number as 2 as an example, the four-class state machine contains a path 2-2-2, and the judgment result pointed to by this path is the end of the clause. The four-class state machine can also contain paths such as 2-3, 3-2, etc. (that is, as long as two consecutive states represent noise), and the judgment result pointed to by these paths can be the end of the clause.

[0038] In a specific implementation scenario, the third number can be equal to the number of silent frames used to determine whether a clause has ended when using a binary classification state machine for endpoint detection. In the binary classification state machine, adjacent states correspond to any two preset types, which are either human voice or non-human voice. For the specific meaning of the binary classification state machine, please refer to the aforementioned description, which will not be repeated here. For example, when the number of silent frames used to determine whether a clause has ended when using a binary classification state machine for endpoint detection is 8, the third number can also be set to 8. Of course, the above example is merely one possible example in practical application, and the specific value of the third number is not limited here. Furthermore, when the second number is equal to the number of silent frames used to determine whether a complete sentence has ended when using a binary classification state machine for endpoint detection, the second number can be greater than the third number.

[0039] In a specific implementation scenario, the first number can be less than the second number, and the second number can be equal to the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine for endpoint detection. For ease of description, the third number can be denoted as m1, and the fourth number as m2. When the target type of m1 consecutive audio frames up to the current frame is intermediate noise, or the target type of m2 consecutive audio frames up to the current frame is noise, it can be determined that the clause at the current frame has ended. That is, as long as one of the aforementioned two judgment sub-conditions is met, it can be determined that the clause at the current frame has ended. Since the third number m1 is less than the fourth number m2, when the clause at the current frame is determined to have ended due to the latter judgment sub-condition, the number of frames following the end of the clause can be greater than the number of frames following the end of the clause when the clause at the current frame is determined to have ended due to the former judgment sub-condition. That is, it is greater than the number of frames following the end of the clause when the clause at the current frame is determined using a binary classification state machine. This helps to connect adjacent clauses with a certain semantic correlation more easily, so that the audio semantics can be more complete when going through subsequent downstream tasks such as speech recognition.

[0040] In one implementation scenario, when the judgment result includes whether the whole sentence ends and whether the clause ends in the current frame, the specific process of continuous monitoring using the four-class state can be referred to the two methods mentioned above, and will not be repeated here. In this way, the delay of judging the end of the whole sentence can be reduced, the smoothness of interaction can be improved, and it is easier to connect adjacent clauses with a certain semantic relevance.

[0041] The above scheme continuously monitors streaming audio to obtain the probability values ​​of audio frames belonging to several frame types, including human voice, initial noise, middle noise, and final noise. Based on the probability values ​​and energy values ​​of the audio frames, the target type of each audio frame is determined. Then, a four-class classification state machine is used to continuously monitor the target type of each audio frame in the streaming audio, obtaining the judgment result for the current frame. The judgment result includes at least one of the following: whether the current frame is the end of a sentence or the end of a clause. The adjacent state pairs in the four-class classification state machine are also considered. Applicable to any two frame types, this approach, on the one hand, further refines the noise into initial noise, middle noise, and final noise. This allows for the determination of the semantic position of an audio frame within the streaming audio—whether it's the beginning, end, or middle clause of a sentence—based on whether the audio frame belongs to the initial, middle, or final noise category. This imbues speech endpoint detection with semantic capabilities. On the other hand, using a four-class state machine for continuous monitoring, compared to a simple state machine that only contains state transitions between human and non-human voices, further refines the state transitions, enabling better differentiation between sentences and clauses and helping to avoid excessive segmentation. Therefore, it improves the semantic coherence after performing speech endpoint detection.

[0042] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the speech endpoint detection device of this application. The speech endpoint detection device 30 includes: a type prediction module 31, a type determination module 32, and a continuous monitoring module 33. The type prediction module 31 is used to continuously predict based on streaming audio to obtain the probability values ​​of audio frames in the streaming audio belonging to several frame types; wherein, the several frame types include human voice, first noise, middle noise, and last noise; the type determination module 32 is used to determine the target type of the audio frame among the several frame types based on the probability values ​​of the audio frame belonging to several frame types and the energy value of the audio frame; the continuous monitoring module 33 is used to continuously monitor the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame; wherein, the judgment result includes at least one of whether the current frame is the end of a whole sentence or whether the current frame is the end of a clause, and adjacent states in the four-class state machine correspond to any two frame types.

[0043] In the above scheme, the speech endpoint detection device 30 continuously monitors the streaming audio to obtain the probability values ​​of audio frames belonging to several frame types, including human voice, initial noise, middle noise, and final noise. Based on the probability values ​​and energy values ​​of the audio frames, the target type of each audio frame is determined. Then, a four-class classification state machine is used to continuously monitor the target type of each audio frame in the streaming audio, obtaining the judgment result for the current frame. The judgment result includes at least one of the following: whether the current frame is the end of a sentence or the end of a clause. The four-class classification state machine... Adjacent states correspond to any two frame types. Therefore, on the one hand, by further refining the noise into first noise, middle noise, and last noise, the semantic position of an audio frame in the streaming audio—whether it belongs to the first, middle, or last noise—can be determined to help identify whether it is the beginning, end, or middle clause of a sentence. This gives speech endpoint detection semantic capabilities. On the other hand, using a four-class state machine for continuous monitoring, compared to a simple state machine that only contains state transitions between human and non-human voices, further refines the state transitions, allowing it to better distinguish between sentences and clauses, thus helping to avoid excessive segmentation. Therefore, it can improve the semantic coherence after performing speech endpoint detection.

[0044] In some disclosed embodiments, the continuous monitoring module 33 includes a sentence-ending judgment submodule, used to continuously monitor whether the target type in the streaming audio up to the current frame meets the sentence-ending detection condition based on a four-classification state machine, provided that the judgment result includes whether the sentence ends at the current frame. The sentence-ending detection condition includes any one of the following: the target type of a first consecutive number of audio frames up to the current frame is tail noise; or the target type of a second consecutive number of audio frames up to the current frame is characterized as audio frames belonging to noise. The continuous monitoring module 33 includes a first sentence-ending determination submodule, used to determine that the judgment result of the current frame includes that the sentence has ended at the current frame, in response to the target type in the streaming audio up to the current frame meeting the sentence-ending detection condition. The continuous monitoring module 33 includes a second sentence-ending determination submodule, used to determine that the judgment result of the current frame includes that the sentence has not yet ended at the current frame, in response to the target type in the streaming audio up to the current frame not meeting the sentence-ending detection condition.

[0045] In some disclosed embodiments, the first number is less than the second number; and / or, the second number is equal to the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine for endpoint detection, wherein adjacent states in the binary classification state machine correspond to any two preset types, and the preset types are either human voice or non-human voice.

[0046] In some disclosed embodiments, the continuous monitoring module 33 includes a clause end judgment submodule, used to continuously monitor whether the target type in the streaming audio up to the current frame meets the clause end detection condition based on a four-class state machine, provided that the judgment result includes whether the clause ends at the current frame. The clause end detection condition includes any one of the following: the target type of the third consecutive audio frames up to the current frame is intermediate noise; the target type of the fourth consecutive audio frames up to the current frame is characterized as audio frames belonging to noise. The continuous monitoring module 33 includes a first clause determination submodule, used to determine that the judgment result of the current frame includes that the clause has ended at the current frame, in response to the target type in the streaming audio up to the current frame meeting the clause end detection condition. The continuous monitoring module 33 includes a second clause determination submodule, used to determine that the judgment result of the current frame includes that the clause has not yet ended at the current frame, in response to the target type in the streaming audio up to the current frame not meeting the clause end detection condition.

[0047] In some disclosed embodiments, the third number is less than the fourth number; and / or, the third number is equal to the number of silent frames used to determine whether a clause has ended when using a binary classification state machine for endpoint detection, wherein adjacent states in the binary classification state machine correspond to any two preset types, and the preset types are either human voice or non-human voice.

[0048] In some disclosed embodiments, the probability value is obtained by the speech activity detection model predicting audio frames in the streaming audio. The speech activity detection model is trained based on sample audio, and the sample audio frames in the sample audio are labeled with the frame type to which the sample audio frames belong.

[0049] In some disclosed embodiments, the voice endpoint detection device 30 includes a sample acquisition module for acquiring sample audio that has been binary-classified and labeled. After binary classification and labeling, each sample audio frame in the sample audio is labeled with a preset type, which is either human voice or non-human voice. The voice endpoint detection device 30 includes a label refinement module for, in response to the preset type of a sample audio frame being non-human voice, determining whether to refine the frame type of the sample audio frame to first noise based on whether there are sample audio frames of the preset type being human voice before the sample audio frame, and determining whether to refine the frame type of the sample audio frame to tail noise based on whether there are sample audio frames of the preset type being human voice after the sample audio frame, and refining the frame type of the sample audio frame to intermediate noise when the sample audio frame is located between sample audio frames of the first noise type and sample audio frames of the tail noise type.

[0050] In some disclosed embodiments, the speech endpoint detection device 30 includes a candidate acquisition module for acquiring candidate audio after binary classification labeling; wherein, the labeling result of the candidate audio after binary classification labeling includes: the preset type to which each candidate audio frame in the candidate audio belongs, the preset type being either human voice or non-human voice; the speech endpoint detection device 30 includes an endpoint prediction module for predicting the candidate audio based on different endpoint detection models, obtaining the prediction results of the candidate audio after passing through different endpoint detection models; wherein, the prediction results include the preset type to which each candidate audio frame in the candidate audio belongs; the speech endpoint detection device 30 includes an audio filtering module for, for each candidate audio: based on the difference between the labeling result of the candidate audio and the prediction result after passing through any endpoint detection model, obtaining the prediction accuracy of the corresponding endpoint detection model for the candidate audio, and based on the prediction accuracy of each endpoint detection model for the candidate audio, determining whether to select the candidate audio as sample audio.

[0051] In some disclosed embodiments, the voice endpoint detection device 30 includes an audio sampling module for sampling from each candidate audio among the unselected sample audio to obtain sampled audio; the voice endpoint detection device 30 includes a manual annotation module for sending the sampled audio to a manual binary classification annotation to obtain sample audio.

[0052] In some disclosed embodiments, the type determination module 32 includes an audio selection submodule for sequentially selecting each audio frame as a frame to be decided; the type determination module 32 includes a first response submodule for determining the target type of the frame to be decided as human voice in response to the frame to be decided meeting the human voice decision conditions; wherein, the human voice decision conditions include: the energy value of the frame to be decided is not lower than an energy threshold, the probability value of the frame to be decided belonging to human voice is not lower than a probability threshold, and the frame to be decided is not penalized; the type determination module 32 includes a second response submodule for penalizing the frame to be decided and several subsequent audio frames in response to the frame to be decided not meeting the human voice decision conditions, when the energy value of the frame to be decided is lower than an energy threshold, or the probability value of the frame to be decided belonging to human voice is lower than a probability threshold; wherein, when the target type of the frame to be decided is not determined to be human voice, the target type of the frame to be decided is determined based on the probability values ​​of the frame to be decided belonging to the first noise, the middle noise, and the tail noise, respectively.

[0053] Please see Figure 4 , Figure 4This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other. The memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the voice endpoint detection method. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. The electronic device 40 may include, but is not limited to, smartphones, tablets, learning machines, translators, notebooks, in-vehicle systems, smart screens, servers, etc. The specific type of the electronic device 40 is not limited here.

[0054] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the above-described voice endpoint detection method embodiments. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0055] In the above scheme, the electronic device 40 continuously monitors the streaming audio to obtain the probability values ​​of audio frames belonging to several frame types, including human voice, initial noise, middle noise, and final noise. Based on the probability values ​​of the audio frames belonging to several frame types and the energy values ​​of the audio frames, the target type of the audio frame is determined among the several frame types. Then, based on a four-class classification state machine, the target type of each audio frame in the streaming audio is continuously monitored to obtain the judgment result of the current frame. The judgment result includes at least one of the following: whether the current frame is the end of a sentence or the end of a clause. The adjacent values ​​in the four-class classification state machine are also considered. The state corresponds to either of the two frame types. Therefore, on the one hand, by further refining the noise into first noise, middle noise, and last noise, the semantic position of an audio frame in the streaming audio—whether it belongs to the first, middle, or last noise—can be determined to help identify whether it is the beginning, end, or middle clause of a sentence. This gives speech endpoint detection semantic capabilities. On the other hand, using a four-class state machine for continuous monitoring, compared to a simple state machine that only contains state transitions between human and non-human voices, further refines the state transitions, allowing it to better distinguish between sentences and clauses, and helping to avoid excessive segmentation. Therefore, it can improve the semantic coherence after performing speech endpoint detection.

[0056] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described embodiments of the voice endpoint detection method.

[0057] In the above scheme, the computer-readable storage medium 50 continuously monitors the streaming audio to obtain the probability values ​​of audio frames belonging to several frame types, including human voice, initial noise, middle noise, and final noise. Based on the probability values ​​of the audio frames belonging to several frame types and the energy values ​​of the audio frames, the target type of the audio frame is determined among the several frame types. Then, a four-class classification state machine is used to continuously monitor the target type of each audio frame in the streaming audio to obtain the judgment result of the current frame. The judgment result includes at least one of the following: whether the current frame is the end of a sentence or the end of a clause. The adjacent states correspond to any two frame types. Therefore, on the one hand, by further refining the noise into first noise, middle noise, and last noise, the semantic position of an audio frame in the streaming audio—whether it belongs to the first, middle, or last noise—can be determined by whether it is the beginning, end, or middle clause of a sentence. This gives speech endpoint detection semantic capabilities. On the other hand, using a four-class state machine for continuous monitoring, compared to a simple state machine that only contains state transitions between human and non-human voices, can further refine the state transitions of the state machine, making it better able to distinguish between sentences and clauses, and helping to avoid excessive segmentation as much as possible. Therefore, it can improve the semantic coherence after performing speech endpoint detection.

[0058] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0059] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0060] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0062] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0063] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A method for detecting speech endpoints, characterized in that, include: Based on the streaming audio, continuous prediction is performed to obtain the probability values ​​of audio frames in the streaming audio belonging to several frame types; wherein, the several frame types include human voice, first noise, middle noise, and last noise; Based on the probability values ​​of the audio frames belonging to the plurality of frame types and the energy values ​​of the audio frames, the target type to which the audio frame belongs is determined among the plurality of frame types; The target type of each audio frame in the streaming audio is continuously monitored based on a four-class state machine to obtain the judgment result of the current frame; wherein, the judgment result includes at least one of whether the current frame is the end of a whole sentence or whether the current frame is the end of a clause, and adjacent states in the four-class state machine correspond to any two frame types.

2. The method according to claim 1, characterized in that, If the judgment result includes whether the current frame is the end of a complete sentence, the continuous monitoring of the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame includes: The four-class state machine continuously monitors whether the target type in the streaming audio up to the current frame satisfies the sentence end detection condition; wherein, the sentence end detection condition includes any one of the following: the target type of the first consecutive number of audio frames up to the current frame is the tail noise, or the target type of the second consecutive number of audio frames up to the current frame is characterized as the audio frame belonging to noise; In response to the target type in the streaming audio up to the current frame satisfying the sentence end detection condition, the judgment result of the current frame is determined to include that the sentence has ended at the current frame; In response to the fact that the target type in the streaming audio up to the current frame does not meet the sentence end detection condition, the judgment result of the current frame is determined to include that the sentence has not yet ended at the current frame.

3. The method according to claim 2, characterized in that, The first number is less than the second number; and / or, the second number is equal to the number of silent frames used to determine whether a sentence has ended when using a binary classification state machine for endpoint detection, wherein adjacent states in the binary classification state machine correspond to any two preset types, and the preset type is either human voice or non-human voice.

4. The method according to claim 1, characterized in that, If the judgment result includes whether a clause ends at the current frame, the continuous monitoring of the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame includes: The four-class state machine continuously monitors whether the target type in the streaming audio up to the current frame satisfies the clause end detection condition; wherein, the clause end detection condition includes any one of the following: the target type of the third consecutive number of audio frames up to the current frame is the intermediate noise, or the target type of the fourth consecutive number of audio frames up to the current frame is characterized as the audio frame belonging to noise; In response to the target type in the streaming audio up to the current frame satisfying the clause end detection condition, the determination result of the current frame includes that the clause has ended at the current frame; In response to the fact that the target type in the streaming audio up to the current frame does not meet the clause end detection condition, the determination result of the current frame includes that there is no clause end at the current frame.

5. The method according to claim 4, characterized in that, The third number is less than the fourth number; and / or, the third number is equal to the number of silent frames used to determine whether a clause has ended when using a binary classification state machine for endpoint detection, wherein adjacent states in the binary classification state machine correspond to any two preset types, and the preset type is either human voice or non-human voice.

6. The method according to claim 1, characterized in that, The probability value is obtained by the speech activity detection model predicting the audio frames in the streaming audio. The speech activity detection model is trained based on sample audio, and the sample audio frames are labeled with the frame type to which the sample audio frames belong.

7. The method according to claim 6, characterized in that, The annotation steps for the frame type to which the sample audio belongs include: Obtain sample audio that has been binary-classified and labeled; wherein, after the sample audio has been binary-classified and labeled, the sample audio frames in the sample audio are labeled with a preset type to which the sample audio frames belong, and the preset type is either human voice or non-human voice; In response to the fact that the preset type to which the sample audio frame belongs is non-human voice, based on whether there is a sample audio frame of the preset type belonging to human voice before the sample audio frame, it is determined whether to refine the frame type to which the sample audio frame belongs to the first noise; based on whether there is a sample audio frame of the preset type belonging to human voice after the sample audio frame, it is determined whether to refine the frame type to which the sample audio frame belongs to the tail noise; and if the sample audio frame is located between a sample audio frame of the frame type belonging to the first noise and a sample audio frame of the frame type belonging to the tail noise, the frame type to which the sample audio frame belongs to the intermediate noise is refined.

8. The method according to claim 6 or 7, characterized in that, The steps for obtaining the sample audio include: Obtain candidate audio after binary classification labeling; wherein, the labeling result of the candidate audio after binary classification labeling includes: the preset type to which each candidate audio frame in the candidate audio belongs, and the preset type is either human voice or non-human voice; The candidate audio is predicted based on different endpoint detection models to obtain the prediction results of the candidate audio after passing through the different endpoint detection models; wherein, the prediction result includes the preset type to which each candidate audio frame in the candidate audio belongs; For each candidate audio: based on the difference between the annotation result of the candidate audio and the prediction result of any endpoint detection model, the prediction accuracy of the corresponding endpoint detection model for the candidate audio is obtained, and based on the prediction accuracy of each endpoint detection model for the candidate audio, it is determined whether to select the candidate audio as the sample audio.

9. The method according to claim 8, characterized in that, The method further includes: Sample audio is obtained by sampling from each of the candidate audios that were not selected as sample audio. The sampled audio is sent to a human to perform the binary classification labeling, and the sample audio is obtained.

10. The method according to claim 1, characterized in that, The step of determining the target type of the audio frame among the several frame types based on the probability values ​​of the audio frame belonging to the several frame types and the energy value of the audio frame includes: Each of the aforementioned audio frames is selected sequentially as the frame to be decided; In response to the fact that the frame to be judged meets the human voice judgment conditions, the target type of the frame to be judged is determined to be the human voice; wherein, the human voice judgment conditions include: the energy value of the frame to be judged is not lower than the energy threshold, the probability value of the frame to be judged belonging to the human voice is not lower than the probability threshold, and the frame to be judged has not been penalized; In response to the fact that the frame to be decided does not meet the human voice decision condition, if the energy value of the frame to be decided is lower than the energy threshold, or if the probability value of the frame to be decided belonging to the human voice is lower than the probability threshold, a penalty is imposed on the frame to be decided and several subsequent audio frames; wherein, if the target type to which the frame to be decided belongs is not determined to be the human voice, the target type to which the frame to be decided belongs is determined based on the probability values ​​of the frame to be decided belonging to the first noise, the middle noise, and the tail noise, respectively.

11. A voice endpoint detection device, characterized in that, include: The type prediction module is used to perform continuous prediction based on streaming audio to obtain the probability values ​​of audio frames in the streaming audio belonging to several frame types; wherein, the several frame types include human voice, first noise, middle noise, and last noise; A type determination module is used to determine the target type of the audio frame among the several frame types based on the probability value of the audio frame belonging to the several frame types and the energy value of the audio frame; The continuous monitoring module is used to continuously monitor the target type of each audio frame in the streaming audio based on a four-class state machine to obtain the judgment result of the current frame; wherein, the judgment result includes at least one of whether the current frame is the end of a whole sentence or whether the current frame is the end of a clause, and adjacent states in the four-class state machine correspond to any two frame types.

12. An electronic device, characterized in that, The method includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the voice endpoint detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the voice endpoint detection method according to any one of claims 1 to 10.

Images