A voice endpoint detection method and related device

Abstract

The application discloses a voice endpoint detection method and related equipment, and relates to the technical field of voice processing. The method comprises the following steps: determining a voice starting endpoint and a candidate voice ending endpoint by performing frame-level voice endpoint detection on streaming audio frames; when the voice starting endpoint is determined, obtaining streaming recognition text starting from the voice starting endpoint, and performing phoneme spike detection to obtain the time sequence position of the phoneme spike; when the candidate voice ending endpoint is determined, performing acoustic correction on the candidate voice ending endpoint according to the time sequence position of each phoneme spike; performing semantic integrity evaluation on the obtained streaming recognition text to obtain a semantic integrity score; and determining a final voice ending endpoint according to the continuous silence duration after the corrected voice ending endpoint and the semantic integrity score. The application ensures the semantic integrity of the voice segment while reducing the endpoint detection delay through the synergistic effect of phoneme spike detection and semantic integrity evaluation.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech endpoint detection method and related equipment. Background Technology

[0002] Voice Activity Detection (VAD) is a fundamental front-end technology in voice interaction systems. Its core task is to accurately detect the start and end points of a user's speech from a continuous audio stream, providing precise speech segment boundaries for backend modules such as Automatic Speech Recognition (ASR) and semantic understanding. The accuracy and latency of VAD directly affect the response speed, recognition accuracy, and user experience of the entire voice interaction chain.

[0003] Current speech endpoint detection schemes work by classifying audio streams frame by frame as either spoken or silent based on a speech endpoint detection model. The speech start endpoint is determined when multiple consecutive frames are identified as spoken, and the speech end endpoint is determined when multiple consecutive frames are identified as silent. However, these schemes suffer from large decision delays and inaccurate endpoint localization. Summary of the Invention

[0004] In view of this, the present invention provides a voice endpoint detection method and related equipment to solve the problems of large decision delay and inaccurate endpoint localization in existing voice endpoint detection schemes. The technical solution is as follows:

[0005] The first aspect of the present invention provides a voice endpoint detection method, comprising:

[0006] By performing frame-level speech endpoint detection on streaming audio, the speech start endpoint and candidate speech end endpoint are determined.

[0007] When the speech start endpoint is determined, the streaming recognition text is obtained starting from the speech start endpoint, and phoneme spike detection is performed to obtain the temporal position of the phoneme spike, wherein the temporal position of the phoneme spike is the temporal position of an effective vocalization action within the speech segment starting from the speech start endpoint.

[0008] When the candidate speech endpoint is determined, the candidate speech endpoint is acoustically corrected according to the temporal position of each phoneme peak to obtain the corrected speech endpoint.

[0009] The semantic integrity of the obtained streaming recognized text is evaluated to obtain a semantic integrity score.

[0010] The final speech termination point is determined based on the duration of continuous silence following the corrected speech termination point and the semantic integrity score.

[0011] In one possible implementation, the step of determining the speech start endpoint and candidate speech end endpoint by performing frame-level speech endpoint detection on the streaming audio includes:

[0012] Real-time acquisition of ambient noise intensity; determination of target noise level based on ambient noise intensity.

[0013] Determine the back field of view length, the number of initial decision frames, and the number of final decision frames that are adapted to the target noise level, wherein the back field of view length is the number of future audio frames required for the speech endpoint detection model to make a single-frame decision;

[0014] Based on the aforementioned voice endpoint detection model, the streaming audio is judged frame by frame for whether it contains sound or not.

[0015] When the number of consecutive audio frames reaches the initial decision frame number, the speech start endpoint is determined;

[0016] When the number of consecutive silent frames reaches the number of frames for the end decision, the candidate speech end endpoint is determined.

[0017] In one possible implementation, the voice endpoint detection method further includes:

[0018] When the target noise level changes, if the ambient noise intensity remains within the range corresponding to the new noise level for M consecutive audio frames, the new noise level is determined to be effective, and the back field of view length is switched to the back field of view length adapted to the new noise level, the starting decision frame number is switched to the starting decision frame number adapted to the new noise level, and the ending decision frame number is switched to the ending decision frame number adapted to the new noise level, where M is a preset positive integer;

[0019] During the switching process between old and new noise level parameters, the accumulated number of sound frames or silent frames will not be cleared.

[0020] In one possible implementation, determining the number of starting decision frames and the number of ending decision frames adapted to the target noise level includes:

[0021] Obtain the basic start decision frame number and basic end decision frame number configured for the target noise level;

[0022] The noise weighting factor is determined based on the environmental noise intensity.

[0023] Based on the noise weighting factor, the basic starting decision frame number and the basic ending decision frame number are corrected to obtain the starting decision frame number and the ending decision frame number that are adapted to the target noise level.

[0024] In one possible implementation, the voice endpoint detection method further includes:

[0025] Once the speech start endpoint is determined, a dedicated buffer is created for the speech segment starting from the speech start endpoint;

[0026] The phoneme spike detection results and streaming text, starting from the speech start endpoint, are stored in the cache area;

[0027] Once the final end point of the voice recording is determined, the buffer is released;

[0028] The acoustic correction of the candidate speech termination endpoints based on the temporal position of each phoneme peak includes:

[0029] Based on the temporal position of each phoneme spike stored in the buffer, the candidate speech end point is acoustically corrected;

[0030] The semantic integrity assessment of the obtained streaming text includes:

[0031] The semantic integrity of the streaming recognized text stored in the cache is evaluated.

[0032] In one possible implementation, phoneme spike detection is performed starting from the speech start endpoint, including:

[0033] Starting from the speech start point, every n audio frames obtained are concatenated to obtain a large audio frame, where n is a preset positive integer;

[0034] The audio large frame is encoded to obtain the embedding vector of the audio large frame;

[0035] Based on the embedding vector of the large audio frame, predict the probability distribution of the large audio frame on each phoneme tag in the phoneme tag set, and obtain the phoneme probability distribution of the large audio frame.

[0036] Based on the phoneme probability distribution of each audio frame, phoneme spikes are identified.

[0037] In one possible implementation, identifying phoneme spikes based on the phoneme probability distribution of each of the audio frames includes:

[0038] For each of the aforementioned audio frames, the phoneme tag corresponding to the highest probability in the phoneme probability distribution of the audio frame is obtained, and the phoneme tag corresponding to the audio frame is obtained.

[0039] If the phoneme tag corresponding to the large audio frame is a non-blank tag, and the phoneme tag corresponding to the large audio frame satisfies the first condition or the second condition, then it is determined that the large audio frame generates a phoneme spike.

[0040] The first condition is that the phoneme tag is different from that of the audio frame that generates the previous phoneme peak. The second condition is that the phoneme tag is the same as that of the audio frame that generates the previous phoneme peak, but the time interval between the audio frame and the audio frame that generates the previous phoneme peak exceeds a preset threshold for the duration of the same phoneme.

[0041] The timing position at which a phoneme spike is generated is the timing position of the audio frame in which the phoneme spike is generated.

[0042] In one possible implementation, the step of acoustically correcting the candidate speech termination endpoints based on the temporal position of each phoneme spike to obtain corrected speech termination endpoints includes:

[0043] Retrieve the temporal position of each phoneme spike and locate the temporal position of the last phoneme spike;

[0044] The timing position of the last phoneme peak is superimposed with a preset human voice attenuation compensation duration to obtain the corrected speech end point.

[0045] In one possible implementation, the step of evaluating the semantic integrity of the obtained streaming recognized text to obtain a semantic integrity score includes:

[0046] Sentence-level semantic integrity is scored on the obtained streaming recognition text to obtain a sentence-level semantic integrity score.

[0047] The completeness of the response is verified based on the current dialogue context of the obtained streaming recognition text, and a response completeness score is obtained.

[0048] The obtained streaming recognition text is subjected to intent integrity determination to obtain the intent integrity probability.

[0049] The sentence-level semantic integrity score, the response completeness score, and the intent integrity probability are fused together, and the fusion result is used as the semantic integrity score of the obtained streaming recognition text.

[0050] In one possible implementation, the step of performing a response completeness check on the obtained streaming recognition text based on the current dialogue context to obtain a response completeness score includes:

[0051] Obtain the current dialogue context, which includes at least the question information of the current interaction round and the semantic slot information expressed by the user;

[0052] Based on the semantic slot information already expressed by the user, determine the currently unfilled mandatory semantic slots;

[0053] Detect whether the obtained streaming recognition text contains necessary response elements for filling the required semantic slots and for the question information;

[0054] If the obtained streaming recognition text contains the necessary response elements, the response completeness score will be set to a score that is not lower than a preset completeness threshold.

[0055] If the obtained streaming text does not contain the necessary response elements, the response completeness score will be set to a score lower than the completeness threshold.

[0056] In one possible implementation, determining the final speech termination point based on the duration of continuous silence following the corrected speech termination point and the semantic integrity score includes:

[0057] If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, and the semantic integrity score is greater than or equal to the semantic integrity threshold, then the corrected speech end point is determined as the final speech end point.

[0058] In one possible implementation, the voice endpoint detection method further includes:

[0059] If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, but the semantic integrity score is less than the semantic integrity threshold, it is determined to be an intermediate pause, the silent segment audio after the corrected speech end point is removed and the detection continues.

[0060] If the semantic integrity score is less than the semantic integrity threshold, and the duration of continuous silence after the corrected speech end point is greater than or equal to the maximum waiting threshold, then the corrected speech end point is determined as the final speech end point.

[0061] In one possible implementation, the process of determining the mute threshold and the maximum waiting threshold includes:

[0062] Extract non-ending internal silence segments from the current speaker's complete historical speech segments, and calculate the duration of each internal silence segment;

[0063] Calculate the mean and standard deviation of the duration of each of the aforementioned internal silent segments;

[0064] The silence threshold is determined based on the mean and the standard deviation.

[0065] The maximum waiting threshold is determined based on the mute threshold.

[0066] A second aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0067] The memory is used to store computer programs;

[0068] The processor is used to execute the computer program to enable the electronic device to implement any of the above-described voice endpoint detection methods.

[0069] A third aspect of this application provides a computer-readable storage medium storing computer instructions for causing an electronic device to implement any of the above-described voice endpoint detection methods.

[0070] Using the above technical solution, the speech endpoint detection method provided by this invention, after determining the candidate speech end endpoint through frame-level speech endpoint detection, does not directly output it as the final endpoint. Instead, it performs acoustic correction on the candidate speech end endpoint based on the temporal position of each phoneme peak to obtain the corrected speech end endpoint. Since the temporal position of each phoneme peak is the temporal position of the effective vocalization action within the speech segment, the correction of the candidate speech end endpoint based on this temporal position directly relies on the temporal information of the vocalization action rather than simply the cumulative silence judgment, effectively reducing the endpoint detection latency. At the same time, the semantic integrity of the streaming recognition text obtained from the speech start endpoint is evaluated to obtain a semantic integrity score. In the final endpoint judgment, the final speech end endpoint is determined by the duration of continuous silence after the corrected speech end endpoint and the semantic integrity score. This makes the endpoint judgment subject to both acoustic and semantic conditions, thereby obtaining a more accurate speech endpoint. The speech endpoint detection method provided by this invention ensures the accuracy of endpoint positioning through acoustic correction and ensures that the speech segment defined by the endpoint has complete semantics through semantic integrity assessment. This avoids endpoint positioning deviation caused by acoustic interference such as breathing pauses and also prevents premature truncation in the case of incomplete semantics. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0072] Figure 1 This is a flowchart illustrating the voice endpoint detection method provided in an embodiment of the present invention.

[0073] Figure 2 This is a schematic diagram of the process for determining the start endpoint and candidate end endpoint of speech by performing frame-level speech endpoint detection on streaming audio, as provided in an embodiment of the present invention.

[0074] Figure 3 A schematic diagram illustrating the current dialogue context maintenance strategy provided in an embodiment of the present invention;

[0075] Figure 4 This is a flowchart illustrating the process of determining the final speech termination endpoint based on the duration of continuous silence after the corrected speech termination endpoint and the semantic integrity score, as provided in an embodiment of the present invention.

[0076] Figure 5 This is a schematic diagram illustrating the speech recognition system that uses the speech endpoint detection method provided in this embodiment of the invention to perform speech recognition.

[0077] Figure 6 This is a schematic diagram illustrating the method for determining the silence threshold and the maximum waiting threshold provided in an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram of the structure of the voice endpoint detection device provided in an embodiment of the present invention. Detailed Implementation

[0079] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0080] The embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0081] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0082] Currently, mainstream streaming speech recognition systems adopt a three-tiered hybrid architecture: "CTC streaming incremental decoding + ED offline final recognition + large-scale model post-correction." The complete workflow is as follows: the front end performs 10ms frame-by-frame processing on the audio stream. The resulting audio frames are input to a lightweight speech endpoint detection model for frame-by-frame audible / silent discrimination. After satisfying the continuous frame judgment condition, the start and end endpoints are output. After outputting the speech start endpoint, the ED model is triggered to initiate the streaming recognition process, connecting to the temporal classification CTC branch to synchronously output real-time text. After outputting the speech end endpoint, the ED model performs global decoding to output the final text. Finally, entity database retrieval and large-scale model error correction are performed based on the final text, and the front end's real-time text is refreshed. In this architecture, the quality of the front-end speech endpoint detection directly affects the overall response latency, recognition accuracy, and large-scale model error correction effect.

[0083] However, most existing speech endpoint detection solutions employ fixed threshold-based detection strategies, typically configuring fixed back-field lengths (e.g., 110ms) and fixed start and end decision frames (e.g., 20 start decision frames and 30 end decision frames, meaning 20 consecutive audible frames determine the speech start endpoint, and 30 consecutive silent frames determine the speech end endpoint). This results in an overall decision latency of over 300ms for a single endpoint, and a fixed wait of 700ms in scenarios where the voice assistant is silent at the end, causing severe delays in interactive response. Furthermore, existing speech endpoint detection solutions usually determine whether there is sound or silence based solely on audio energy and lightweight speech endpoint detection models, failing to distinguish between natural breathing pauses and actual clause terminations during user speech. They often misjudge long pauses of over 300ms as the end of speech, outputting incomplete semantic fragments. When these truncated, incomplete texts are directly fed into the backend decoding and large-scale model error correction stages, it not only leads to entity retrieval bias and context loss, significantly reducing the effectiveness of large-scale model error correction, but can even trigger reverse error correction.

[0084] To address the aforementioned problems, this invention provides a voice endpoint detection method, which will be described in the following embodiments.

[0085] Please see Figure 1 The diagram illustrates a flowchart of a voice endpoint detection method provided by an embodiment of the present invention. The voice endpoint detection method may include:

[0086] Step S101: Frame-level speech endpoint detection is performed on the streaming audio to determine the speech start endpoint and candidate speech end endpoint.

[0087] Frame-level speech endpoint detection for streaming audio refers to using a lightweight speech endpoint detection model to determine whether each frame of the real-time input audio stream, divided into segments with a preset time granularity (e.g., 10ms), contains sound or is silent. This step aims to quickly locate the coarse boundaries of speech segments, providing a foundation for subsequent fine-grained processing.

[0088] Step S102: When the speech start endpoint is determined, start from the speech start endpoint, acquire the streaming recognition text, and perform phoneme spike detection to obtain the temporal position of the generated phoneme spike.

[0089] This step processes the speech data in real time, starting from the speech start point. On one hand, it obtains the streaming text for recognition; on the other hand, it detects phoneme spikes and records their temporal positions. The temporal position where a phoneme spike is generated is the temporal position of a valid vocalization action within a speech segment, which here refers to the speech segment starting from the speech start point.

[0090] By introducing phoneme spike detection, the interference of audio energy fluctuations and short-term silences on endpoint localization can be reduced, accurately anchoring the timing position of the actual sound-producing action, and providing a fine acoustic localization basis for subsequent endpoint correction.

[0091] In one possible implementation, when the speech start endpoint is determined, a dedicated buffer (i.e., a dedicated data storage space in memory) can be created for the speech segment starting from the speech start endpoint. Then, the phoneme spike detection results and streaming text from the speech start endpoint are stored in the buffer. Preferably, the buffer can employ a circular buffer to achieve cyclic read / write, avoiding memory overflow caused by prolonged voice interaction. The logical structure uses a sequential memory that only increments and never decrements, ensuring that all timing data from the start endpoint to the final end endpoint is completely retained without being cleared by intermediate pauses.

[0092] Step S103: When the candidate speech endpoint is determined, the candidate speech endpoint is acoustically corrected according to the temporal position of each phoneme peak to obtain the corrected speech endpoint.

[0093] When a candidate speech endpoint is identified, an acoustic correction process is triggered. Specifically, based on the temporal position of each phoneme peak, the candidate speech endpoint is acoustically corrected to obtain the corrected speech endpoint.

[0094] In this embodiment, after the start endpoint of the speech is determined, phoneme spike detection is continuously performed, and relevant information of each phoneme spike (such as phoneme spike label, timing position, etc.) is recorded in the buffer in real time. When the candidate end endpoint of the speech is determined, the timing position of each phoneme spike can be read directly from the buffer without additional waiting.

[0095] Step S104: Perform semantic integrity assessment on the obtained streaming recognition text to obtain a semantic integrity score.

[0096] The semantic integrity of the streaming recognition text stored in the cache can be evaluated to obtain a semantic integrity score. Preferably, when evaluating the semantic integrity of the streaming recognition text, the semantic integrity of the streaming recognition text can be evaluated from multiple dimensions.

[0097] Step S105: Determine the final speech termination point based on the duration of continuous silence after the corrected speech termination point and the semantic integrity score.

[0098] The duration of continuous silence after the corrected speech end point is continuously detected. If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, and the semantic integrity score is greater than or equal to the preset semantic integrity threshold, then the corrected speech end point is determined as the final speech end point.

[0099] It should be noted that after determining the final end point of the voice segment, the buffer is destroyed and the memory resources it occupies are reclaimed. This action marks the end of the current voice segment's lifecycle. When the next voice segment start point is detected, a new buffer is rebuilt. Through this dynamic resource management mechanism, memory leaks can be effectively prevented, while ensuring that data processing for each voice segment is completed in an independent space, guaranteeing long-term stable operation.

[0100] Through the synergistic effect of the above steps, this invention sequentially completes the rapid localization of coarse-grained frame-level endpoints, the acoustic refinement of endpoints based on phoneme spikes, and the endpoint output verification based on semantic integrity scores, forming a progressive detection architecture with hierarchical progression and acoustic-semantic co-constraints.

[0101] The speech endpoint detection method provided in this invention, after determining candidate speech end endpoints through frame-level speech endpoint detection, does not directly output them as the final endpoints. Instead, it performs acoustic correction on the candidate speech end endpoints based on the temporal position of each phoneme peak. Since the temporal position of each phoneme peak is the temporal position of the effective vocalization action within the speech segment, the correction process directly relies on the temporal information of the vocalization action, effectively reducing endpoint detection latency. Simultaneously, the speech endpoint detection method provided in this invention performs semantic integrity evaluation on the streaming recognition text acquired from the speech start endpoint, obtaining a semantic integrity score. During endpoint determination, it comprehensively considers the duration of continuous silence after the corrected speech end endpoint with the semantic integrity score to determine the final speech end endpoint, ensuring that endpoint determination is subject to both acoustic and semantic constraints.

[0102] The speech endpoint detection method provided in this invention, while ensuring endpoint positioning accuracy through acoustic correction, ensures that the speech segment defined by the endpoint has complete semantics through semantic integrity assessment. This avoids endpoint positioning deviations caused by acoustic interference such as breath pauses and also prevents premature truncation in cases of incomplete semantics. The speech endpoint detection method provided in this invention shortens the waiting time between the user stopping speaking and triggering downstream processing, improving interaction fluency. Furthermore, because the speech segment defined by the endpoint has complete semantics, the downstream processing effect is improved.

[0103] In some embodiments of this application, the implementation process of "step S101: determining the speech start endpoint and candidate speech end endpoint by performing frame-level speech endpoint detection on the streaming audio" in the above embodiment is described.

[0104] In one possible implementation, such as Figure 2 As shown, the process of determining the speech start endpoint and candidate speech end endpoint by performing frame-level speech endpoint detection on streaming audio can include:

[0105] Step S201: Acquire the ambient noise intensity in real time and determine the target noise level based on the ambient noise intensity.

[0106] The ambient noise intensity is the background sound energy level detected by the audio acquisition device during real-time sound recording, typically quantified in decibels (dB). Because the clarity and intelligibility of speech signals vary significantly under different noise environments—for example, a speech endpoint detection model can easily distinguish between sound and silence in a quiet environment, while more stringent decision conditions are needed in noisy environments to avoid noise being misidentified as speech—this step provides an environmental basis for the adaptive adjustment of subsequent decision parameters through real-time perception and classification of noise intensity.

[0107] In one possible implementation, multiple noise levels can be preset, and a corresponding noise intensity range can be configured for each noise level. After obtaining the ambient noise intensity, the noise intensity range to which the ambient noise intensity belongs can be determined, and then the noise level corresponding to the noise intensity range to which the ambient noise intensity belongs can be determined as the target noise level.

[0108] For example, three noise levels are preset: low noise, medium noise, and high noise. Corresponding noise intensity ranges are configured for each of the three noise levels. If the noise intensity range to which the real-time acquired ambient noise intensity belongs is the noise intensity range corresponding to low noise, then the target noise level is determined to be low noise.

[0109] Step S202: Determine the back field of view length, the number of initial decision frames, and the number of final decision frames that are adapted to the target noise level.

[0110] Here, the back view length refers to the number of future audio frames required for the speech endpoint detection model to make a single-frame decision. During the frame-level inference process of the speech endpoint detection model, to improve decision accuracy, the model typically needs to refer to audio information from several frames following the current frame for joint decision-making; these future frames constitute the back view. This step dynamically adjusts the back view length based on the noise level; the back view length increases with increasing noise level to provide the model with more comprehensive future frame information in noisy environments.

[0111] The starting decision frame number is the minimum number of consecutive audible frames required to determine the start of speech, and the ending decision frame number is the minimum number of consecutive silent frames required to determine the end of speech.

[0112] In low-noise environments, to reduce unnecessary waiting for future frames and thus lower latency, a relatively small back field of view length (e.g., 30ms) can be used, along with a relatively small number of start and end decision frames. In high-noise environments, to enhance anti-interference capabilities and achieve better detection results, a relatively large back field of view length (e.g., 40ms) can be used, along with a relatively large number of start and end decision frames.

[0113] By employing a dynamic adaptation mechanism based on noise levels, optimal detection performance is maintained under different signal-to-noise ratio conditions, avoiding false triggering in high-noise environments or response lag in low-noise environments.

[0114] In one possible implementation, a rear field of view length can be configured for each of the preset noise levels. After the target noise level is determined, the rear field of view length configured for the target noise level can be obtained as the rear field of view length adapted to the target noise level.

[0115] In addition to configuring the back field of view length for each of the preset noise levels, you can also configure the base start decision frame number and base end decision frame number for each of the preset noise levels. It should be noted that the higher the noise level, the larger the base decision frame number, to ensure that noise fluctuations will not be misinterpreted as speech in noisy environments.

[0116] For example, the preset noise levels include low noise, medium noise, and high noise, and the base starting decision frame number and base ending decision frame number configured for low noise, medium noise, and high noise are as follows:

[0117] Low noise: The basic starting decision frame count is 8, and the basic ending decision frame count is 15;

[0118] Medium noise: The basic starting decision frame number is 10, and the basic ending decision frame number is 20;

[0119] High noise: The basic start decision frame number is 12, and the basic end decision frame number is 25.

[0120] It should be noted that the above values ​​are merely illustrative examples. In practical applications, they can be calibrated and adjusted according to the performance of the voice endpoint detection model and the specific application scenario. This embodiment does not impose any specific limitations on this.

[0121] In one possible implementation, the process of determining the starting decision frame number and the ending decision frame number adapted to the target noise level may include:

[0122] Step a1: Obtain the basic starting decision frame number and basic ending decision frame number configured for the target noise level.

[0123] For example, if the target noise level is low noise, then the basic start decision frame number and basic end decision frame number for the low noise configuration are obtained, that is, the basic start decision frame number is 8 and the basic end decision frame number is 15.

[0124] Step a2: Determine the noise weighting factor based on the real-time acquired environmental noise intensity.

[0125] Among them, the noise weighting factor is a dynamic coefficient that reflects the degree of influence of environmental noise intensity on the number of decision frames. Its function is to quantify the environmental noise intensity into a computable correction weight. The noise weighting factor is usually negatively correlated with the environmental noise intensity or has a specific functional relationship, aiming to achieve the transition from step-type noise classification to continuous smooth parameter adjustment.

[0126] Specifically, the noise weighting factor α can be calculated using the following formula:

[0127] α = 1.0 / (1.0 + noise decibel value / 60) (1).

[0128] Among them, the noise decibel value is the environmental noise intensity.

[0129] The noise weighting factor can change continuously with the noise intensity, providing a precise basis for subsequent decision frame number correction.

[0130] For example, when the ambient noise intensity is 30 dB, the calculated noise weighting factor is relatively large ( This indicates that the environment is relatively quiet, requiring a smaller correction margin. However, when the ambient noise rises to 90 dB, the noise weighting factor decreases significantly. This indicates a need to significantly relax decision criteria to address strong noise interference. This example demonstrates that the introduction of a noise weighting factor allows for the perception of subtle changes in noise intensity, rather than relying solely on coarse classifications, thereby improving the sensitivity of parameter adjustments.

[0131] Step a3: Based on the noise weighting factor, correct the basic starting decision frame number and the basic ending decision frame number to obtain the starting decision frame number and the ending decision frame number that are adapted to the target noise level.

[0132] The basic decision frame number obtained in step a1 is calculated with the noise weighting factor calculated in step a2 to obtain the corrected actual decision frame number, which is the starting decision frame number and the ending decision frame number that are adapted to the target noise level.

[0133] The purpose of this step is to achieve continuous and smooth adjustment of the number of decision frames based on noise classification, using a noise weighting factor. This ensures that the number of decision frames changes both stepwise with macroscopic changes in noise level and continuously with microscopic changes in noise decibel value. The base decision frame number provides a reference value corresponding to the noise level, and the correction process adds the influence of the noise weighting factor to this reference value, thereby achieving a dual adaptation effect of "macroscopic classification + microscopic smoothing," ensuring that the number of decision frames remains optimally configured under different noise environments.

[0134] Preferably, the following formula can be used for correction:

[0135] (2).

[0136] in, N is the base decision frame number, N is the actual decision frame number, i.e., the number of decision frames adapted to the target noise level, and S is the noise level.

[0137] Taking a medium-noise environment (level value = 2) as an example, the base end-of-frame count is 20 frames. If the current noise level is 40 dB, then... Actual number of frames at the end of the judgment This correction increases the actual number of decision frames from the base value based on the finer intensity of the current noise, better adapting to the actual noise level of the current environment.

[0138] Through the above corrections, the final number of starting and ending decision frames not only reflects the basic configuration corresponding to the noise level, but also incorporates the subtle influence of the current environmental noise intensity, thus achieving adaptive voice endpoint detection parameters.

[0139] It should be noted that this embodiment is not limited to using the process of steps a1 to a3 to determine the number of starting decision frames and the number of ending decision frames adapted to the target noise level. In another possible implementation, the basic number of starting decision frames and the basic number of ending decision frames configured for the target noise level can be used as the number of starting decision frames and the number of ending decision frames adapted to the target noise level.

[0140] Step S203: Based on the speech endpoint detection model, determine the sound frames and silent frames of the streaming audio frame by frame.

[0141] Under the speech endpoint detection parameters (i.e., back field of view length, number of starting decision frames, and number of ending decision frames) determined in step S202, a lightweight speech endpoint detection model is used to perform binary classification decision of audible / silent audio frames frame by frame on the audio stream divided according to a preset time granularity.

[0142] Specifically, the speech endpoint detection model extracts acoustic features for each audio frame and, based on model inference, outputs the probability that the frame is a spoken or silent frame. Then, it classifies the frame as a spoken or silent frame according to a preset probability decision threshold. In this step, the current frame serves as the basic input for model inference. Simultaneously, the back-view length determined in step S202 specifies the range of future audio frames that the model additionally references when inferring the current frame. That is, when making a decision about the current frame, the model simultaneously inputs the acoustic features of several frames following the current frame as context to improve the accuracy of the decision.

[0143] During frame-level speech endpoint detection, two counters can be maintained in real time: a continuous spoken frame counter and a continuous silent frame counter. The corresponding counter is updated according to the decision result of each frame, and the corresponding counter is cleared to zero when a contrary decision result is detected.

[0144] Step S204: When the number of consecutive audio frames reaches the starting decision frame number, determine the speech start endpoint.

[0145] Based on the frame-by-frame decision in step S203, the number of consecutive audio frames judged as having sound is counted in real time, i.e., the number of consecutive sound frames. When the number of consecutive sound frames reaches the starting decision frame number determined in step S202, it is determined that the user has started speaking. The speech start endpoint is determined from the temporal position of the consecutive sound frames. For example, the temporal position of the first sound frame in the consecutive sound frames is determined as the speech start endpoint. This consecutive frame decision mechanism effectively avoids misjudgments caused by single-frame noise or short-term disturbances.

[0146] Once the speech start endpoint is determined, it serves as a trigger signal to initiate subsequent phoneme spike detection and streaming text acquisition processes. The speech start endpoint also defines the starting boundary of a speech segment, which, together with the subsequently determined speech end endpoint, constitutes the complete time range of the speech segment.

[0147] It should be noted that the determination of the speech start endpoint only triggers the start of the above-mentioned phoneme spike detection process and streaming text acquisition process, and does not affect the continued operation of the frame-level speech endpoint detection process that has already been performed. That is, after the speech start endpoint is determined, the subsequent audio frames are still distinguished as having sound or not until the candidate speech end endpoint is determined.

[0148] Step S205: When the number of consecutive silent frames reaches the end decision frame number, determine the candidate speech end endpoint.

[0149] Based on the frame-by-frame decision in step S203, the number of audio frames that are consecutively decided as silent frames is counted in real time, i.e. the number of consecutive silent frames. When the number of consecutive silent frames reaches the number of end decision frames determined in step S202, it is determined that the user may have stopped speaking, and the candidate speech end endpoint is determined from the temporal position of the consecutive silent frames. For example, the temporal position of the first silent frame in the consecutive silent frames is determined as the candidate speech end endpoint.

[0150] It's important to note that this step identifies candidate speech termination points, not the final speech termination point. A candidate speech termination point is an unconfirmed end position. Subsequent steps will acoustically correct it using the temporal positions of detected phoneme peaks, and combine this with a semantic integrity score for a comprehensive decision to determine the final speech termination point. This design allows for rapid response with fewer decision frames, while delegating endpoint refinement and semantic verification to subsequent steps, balancing low latency and high accuracy.

[0151] This invention dynamically determines the backfield length, starting decision frame number, and ending decision frame number to match the target level by acquiring ambient noise intensity in real time and classifying noise levels. Then, it performs frame-by-frame judgment on streaming audio based on adaptive parameters. Through the synergistic effect of the above steps, the speech endpoint detection parameters achieve real-time response to ambient noise. Specifically, in low-noise scenarios, the backfield length is shortened and the number of decision frames is reduced, significantly compressing end-to-end detection latency. In high-noise scenarios, the backfield length and frame number requirements are appropriately relaxed, enhancing the robustness of the judgment. This dynamic adjustment mechanism effectively solves the problems of large latency or high false positive rate caused by fixed parameters in existing technologies, providing an accurate and timely acoustic boundary basis for subsequent phoneme spike detection and semantic integrity assessment.

[0152] During the frame-level speech endpoint detection process through steps S201 to S205, the ambient noise may change over time (e.g., a user moves from a quiet indoor environment to a noisy street). To address this change, this embodiment also provides a noise level switching mechanism. When a change in the target noise level is detected, the speech endpoint detection parameters (back field of view length, start decision frame number, and end decision frame number) are adaptively switched, and the decision information accumulated during the switching process is not lost.

[0153] Specifically, when the target noise level changes, the voice endpoint detection parameters (back field of view length, start decision frame number, and end decision frame number) are not switched immediately. Instead, a stability check is performed first, determining whether the ambient noise intensity remains within the noise intensity range corresponding to the new noise level for M consecutive audio frames. If the ambient noise intensity remains within the range corresponding to the new noise level for M consecutive audio frames, the new noise level is deemed effective. This stability check mechanism is used to avoid frequent switching of voice endpoint detection parameters due to instantaneous noise fluctuations (such as sudden door closing sounds, vehicle horns, etc.), thereby ensuring the stability of endpoint detection.

[0154] After the new noise level is determined to be in effect, the speech endpoint detection parameters are switched. Specifically, the currently used backfield length is switched to a backfield length adapted to the new noise level, the currently used starting decision frame number is switched to a starting decision frame number adapted to the new noise level, and the currently used ending decision frame number is switched to an ending decision frame number adapted to the new noise level. After the switch, frame-level speech endpoint detection continues to be performed according to the new speech endpoint detection parameters.

[0155] It should be noted that during the switching process between old and new noise level parameters, the accumulated number of audible or silent frames is not reset to zero. Specifically, if the current process involves accumulating audible frames (i.e., some consecutive audible frames have been detected, but the initial decision frame count has not yet been reached), the accumulated number of audible frames remains unchanged after the switch, and subsequent accumulation continues based on the initial decision frame count after the switch. Similarly, if the current process involves accumulating silent frames (i.e., some consecutive silent frames have been detected, but the final decision frame count has not yet been reached), the accumulated number of silent frames remains unchanged after the switch, and subsequent accumulation continues based on the final decision frame count after the switch. This design ensures that the valid information accumulated during the switching process is not discarded, avoids detection interruptions or resets caused by parameter switching, and guarantees the continuity and smoothness of endpoint detection.

[0156] This application achieves smooth migration in noise-adaptive speech endpoint detection by setting a noise stability verification mechanism for consecutive M frames and a count-preserving strategy during parameter switching. By constraining the duration of M consecutive audio frames, it can accurately distinguish between persistent environmental noise changes and instantaneous interference pulses. The post-view length and decision frame number are only updated when a substantial change in the noise environment is confirmed, thus avoiding the waste of computational resources and decision instability caused by frequent oscillations in speech endpoint detection parameters. Furthermore, by combining this with the constraint that the accumulated frame count is not reset during parameter switching, the state memory of speech endpoint detection is fully preserved, ensuring that the ongoing start or end endpoint detection process is not interrupted during noise level transitions. This smooth migration logic, working in conjunction with the aforementioned dynamic noise grading mechanism, not only significantly improves the robustness of speech endpoint detection in dynamic noise environments but also effectively solves problems such as endpoint jitter, missed start points, or premature truncation caused by noise jumps, ultimately achieving a balance between low latency and high stability.

[0157] As mentioned in the above embodiments, when the speech start endpoint is determined, phoneme spike detection is performed starting from the speech start endpoint to obtain the temporal position where the phoneme spike is generated. This process is described in some embodiments of this application.

[0158] In one possible implementation, the process of phoneme spike detection, starting from the speech start point, includes:

[0159] Step b1: Starting from the beginning of the speech, obtain n audio frames and then concatenate the n audio frames to obtain a large audio frame.

[0160] An audio frame is a small frame obtained by dividing the real-time acquired audio stream into smaller frames according to a preset time granularity (e.g., 10ms). n is a preset positive integer (e.g., 4), and its value is determined by balancing latency requirements and feature stability needs. Concatenating n audio frames means fusing the acoustic features (such as filter bank energy features or Mel-frequency cepstral coefficients, MFCCs) of n consecutive small frames in the time domain in a concatenated manner to form a larger audio frame with a longer time span. For example, when n is 4, if each small frame is 10ms long, the concatenated audio frame will be 40ms long. This concatenation method enhances the stability of short-term acoustic features, smooths energy fluctuations within a single frame, and reduces the frequency of subsequent encoder calls, thereby reducing the computational load.

[0161] Step b2: Encode the large audio frame to obtain the embedding vector of the large audio frame.

[0162] In this approach, a shallow encoder in a streaming speech recognition model can be used to encode large audio frames. Specifically, the shallow encoder can be the first few layers of an end-to-end (ED) speech recognition model (e.g., the first 8 Transformer encoder layers), which has fewer layers than the total number of layers required for complete decoding. Its purpose is to extract intermediate features containing rich phoneme information with minimal inference latency.

[0163] An embedding vector is a high-dimensional, dense vector output after processing by a shallow encoder. Its dimension is determined by the size of the encoder's hidden layers. This vector encapsulates the acoustic context information within a large audio frame. For example, inputting the aforementioned 40ms audio frame into the first 8 layers of the ED model skips subsequent deep networks and autoregressive decoders, directly outputting the embedding vector. By using only shallow networks for encoding, the additional inference latency introduced by deep networks is avoided, achieving low-latency feature extraction while preserving discriminative information sufficient to distinguish different phonemes.

[0164] Step b3: Based on the embedding vector of the large audio frame, predict the probability distribution of each phoneme tag in the phoneme tag set of the large audio frame, and obtain the phoneme probability distribution of the large audio frame.

[0165] The embedding vector of a large audio frame can be input into the linear projection layer of the CTC branch and normalized using the Softmax activation function to obtain the phoneme probability distribution of the large audio frame. The phoneme probability distribution is a vector with a dimension equal to the size of the phoneme label set, where each element represents the probability that the large audio frame belongs to the corresponding phoneme label. It should be noted that the embedding vector and phoneme probability distribution of the large audio frame can also be stored in a cache.

[0166] This step constructs a probability mapping channel from continuous acoustic features to discrete phoneme labels, enabling vocalization actions to be quantified into specific probability values, providing a data foundation for subsequent determination of valid vocalization moments.

[0167] Step b4: Identify phoneme spikes based on the phoneme probability distribution of each audio frame.

[0168] In one possible implementation, the process of identifying phoneme spikes based on the phoneme probability distribution of each large audio frame may include:

[0169] Step b41: For each audio frame, obtain the phoneme tag corresponding to the highest probability in the phoneme probability distribution of the audio frame, and obtain the phoneme tag corresponding to the audio frame.

[0170] The phoneme label corresponding to the highest probability in the phoneme probability distribution of the audio frame is taken as the representative phoneme label of the audio frame.

[0171] Step b42: If the phoneme tag corresponding to the audio frame is a non-blank tag, and the phoneme tag corresponding to the audio frame satisfies the first condition or the second condition, then it is determined that the audio frame generates a phoneme spike.

[0172] The first condition is that the phoneme tag is different from that of the audio frame that generated the previous phoneme peak. The second condition is that the phoneme tag is the same as that of the audio frame that generated the previous phoneme peak, but the time interval between the audio frame and the audio frame that generated the previous phoneme peak exceeds a preset threshold for the duration of the same phoneme.

[0173] The first condition focuses on detecting phoneme switching. When the phoneme label corresponding to the current audio frame is different from the phoneme label corresponding to the audio frame that produced the previous phoneme spike (e.g., changing from / b / to / a / ), it is directly determined that a new phoneme spike has been generated. This corresponds to the new sound wave starting point generated by the change in the position of the vocal organs. The second condition focuses on detecting the persistence and repetition of the same phoneme. A phoneme persistence duration threshold is introduced (e.g., set to 150ms). When the phoneme label corresponding to the current audio frame is the same as the phoneme label corresponding to the audio frame that produced the previous phoneme spike, the time interval between the two audio frames is calculated. If the time interval does not exceed the phoneme persistence duration threshold, it is considered a continuation of the same vocal action, and the phoneme spike is not recorded repeatedly. If the time interval exceeds the phoneme persistence duration threshold, it is determined that the user has exerted force twice or uttered a reduplicated word (e.g., "try it" or "wait"), thus determining that a new phoneme spike has been generated. By introducing the above conditions, redundant spikes caused by prolonged pronunciation can be effectively filtered out, while ensuring that key pronunciation boundary information is not lost in scenarios such as reduplicated words and repeated interjections.

[0174] When a phoneme spike is detected in the audio frame, the temporal position of the audio frame is determined as the temporal position of the phoneme spike it generated. Then, the label and temporal position of the phoneme spike generated by the audio frame can be stored in the buffer for subsequent use in correcting the candidate speech end points.

[0175] This application achieves high-precision, low-latency phoneme peak detection through the synergistic effect of maximum probability label extraction, differentiated peak determination, and homophone duration threshold control in the aforementioned steps. By distinguishing between phoneme switching and homophone repetition, it not only avoids endpoint misjudgment caused by long syllable continuation but also significantly improves peak recall in reduplicated word scenarios. The above process can output unambiguous phoneme peak sequences in real time with low computational consumption, providing a highly reliable data foundation for subsequent acoustic endpoint correction based on phoneme timing.

[0176] As mentioned in the above embodiments, after determining the speech start endpoint, in addition to detecting phoneme spikes, it is also necessary to obtain streaming recognition text. Specifically, after obtaining the phoneme probability distribution of each audio frame, the corresponding phoneme sequence can be decoded from the phoneme probability distribution of each audio frame through greedy decoding or beam search decoding, and converted into real-time incremental recognition text as streaming recognition text output.

[0177] It should be noted that the shallow encoder branch and CTC decoding branch mentioned above only begin computation after the speech start endpoint is determined. Before detecting the speech start endpoint, the shallow encoder branch and CTC decoding branch are in a dormant state to reduce power consumption and computational power consumption. Furthermore, the acquisition of streaming text is incremental; that is, as large audio frames are continuously input, the recognized text is accumulated and updated frame by frame, thereby achieving a streaming effect that synchronously outputs the recognition results while the user is speaking.

[0178] As mentioned in the above embodiments, when a candidate speech endpoint is determined, the candidate speech endpoint is acoustically corrected according to the temporal position of each phoneme peak to obtain the corrected speech endpoint. This process is described in some embodiments of this application.

[0179] In one possible implementation, the process of acoustically correcting the candidate speech termination endpoints based on the temporal position of each phoneme peak to obtain the corrected speech termination endpoints may include:

[0180] Step c1: Retrieve the temporal position of each phoneme peak and locate the temporal position of the last phoneme peak.

[0181] Once the frame-level speech endpoint detection identifies the candidate speech end endpoint, the temporal positions of each recorded phoneme peak are read from the buffer, and the phoneme peak with the largest temporal position is located, i.e., the last phoneme peak.

[0182] Throughout the entire speech segment starting from the speech start point, phoneme spike detection is continuously performed. Whenever a phoneme spike is detected, its corresponding temporal position is recorded in real-time in the buffer. When the candidate speech end point arrives, the temporal position information of these phoneme spikes is already ready in the buffer and can be read directly without waiting for any additional processing. During retrieval, all phoneme spikes recorded in the buffer are traversed to locate the temporal position of the last phoneme spike, which represents the moment of the last valid phoneme's phonation. Locating the temporal position of the last phoneme spike is equivalent to finding the moment of the last valid phonation action within the speech segment, providing a precise acoustic basis for subsequent endpoint correction.

[0183] Step c2: Add the preset human voice attenuation compensation duration to the timing position of the last phoneme peak to obtain the corrected speech end point.

[0184] This step uses the timing position of the last phoneme peak as a reference and shifts it backward by a preset human voice attenuation compensation duration to obtain the corrected speech end point.

[0185] Specifically, the temporal position of the last phoneme peak corresponds to the moment when the last effective vocalization occurs within a speech segment. However, the human voice does not disappear completely immediately after the vocalization action ends. The attenuation of vocal cord vibration and the tailing of the vocal tract impulse response cause the speech energy to gradually weaken rather than terminate instantaneously. The actual time of speech termination perceived by the user is usually slightly later than the peak time of the last effective phoneme. To compensate for this physical attenuation process, a duration needs to be added to the temporal position of the last phoneme peak so that the corrected endpoint is closer to the actual speech termination position perceived by the human ear.

[0186] The preset voice attenuation compensation duration is a pre-calibrated fixed value used to compensate for the natural attenuation of acoustic energy after the pronunciation ends. Its typical range is 20ms to 50ms, preferably 30ms. However, this embodiment is not limited to this; the voice attenuation compensation duration can be calibrated and adjusted according to different speech endpoint detection models, front-end microphone characteristics, and specific application scenarios. For example, in far-field pickup scenarios, due to sound propagation delay and reverberation effects, the compensation duration can be appropriately increased; in near-field pickup scenarios, the compensation duration can be appropriately decreased. By superimposing the voice attenuation compensation duration, the corrected speech endpoint can more accurately approximate the actual speech end position, eliminating the trailing delay caused by waiting for multiple consecutive frames of silent judgment, while simultaneously compensating for the natural attenuation of acoustic energy after the pronunciation ends.

[0187] This invention locates the moment of the last valid vocalization by retrieving the timing position of the phoneme peak and performs boundary correction by combining the preset human voice attenuation compensation duration, thereby achieving fine calibration of the acoustic endpoint.

[0188] In some embodiments of this application, the process of evaluating the semantic integrity of streaming recognized text and obtaining a semantic integrity score is described.

[0189] In one possible implementation, the process of evaluating the semantic integrity of streaming recognized text to obtain a semantic integrity score may include:

[0190] Step d1-a: Sentence-level semantic integrity is scored on the obtained streaming recognition text to obtain the sentence-level semantic integrity score.

[0191] Sentence-level semantic integrity scoring of the obtained streaming text refers to evaluating its semantic integrity at the sentence level based on the syntactic structure and linguistic features of the streaming text itself, thereby obtaining a quantitative integrity score.

[0192] Specifically, syntactic boundary features are extracted from the streaming text. These features include at least one or more of the following: pause filler words (such as “uh”, “um”, “that”, etc.), sentence-final modal particles (such as “ba”, “ne”, “ma”, etc.), subject-verb-object structure completeness, etc. Then, the above syntactic boundary features are input into a pre-trained lightweight semantic scoring model (such as a neural network model based on a small Transformer structure). After the model infers, it outputs a numerical score, namely the sentence-level semantic completeness score. This score reflects the completeness of the streaming text at the syntactic level. The higher the score, the more complete the syntactic structure.

[0193] The main purpose of this step is to provide a basic score for subsequent semantic integrity fusion through rapid syntactic judgment. For example, for the text "I want to book a ticket", the text has a complete subject-verb-object structure and a high sentence-level semantic integrity score. For the text "I want to book", the text lacks an object and has a low sentence-level semantic integrity score.

[0194] Step d1-b: Perform response completeness verification on the obtained streaming recognition text based on the current dialogue context to obtain the response completeness score.

[0195] Validating the completeness of the response based on the current dialogue context means determining whether the streamed text has adequately responded to the question in the current interaction round, and outputting a score reflecting the completeness of the response.

[0196] The core function of this step is to identify semantic information that is omitted in multi-turn dialogues but is implicit in the context, thus avoiding misjudgment of semantic incompleteness due to syntactic incompleteness.

[0197] In multi-turn dialogue scenarios, user responses are often closely related to the question, and may even be elliptical sentences (e.g., if the question is "Which day would you like to book your flight?", and the user only answers "tomorrow"). Scoring solely based on sentence-level semantic completeness might result in such elliptical sentences being penalized as syntactic incompleteness, even though they have actually fully addressed the question within the context of the dialogue. Therefore, it is necessary to incorporate contextual information from the dialogue for supplementary judgment.

[0198] Specifically, the current dialogue context is maintained, which includes at least the question information of the current interaction round and the semantic slot information expressed by the user. Based on the current dialogue context, it is detected whether the streaming recognition text contains necessary response elements for the question information. Specifically, the current dialogue context is first obtained, and then the required semantic slots that have not yet been filled are determined based on the semantic slot information expressed by the user. Then, it is detected whether the streaming recognition text contains necessary response elements for filling the required semantic slots and for the question information. If the obtained streaming recognition text contains necessary response elements (e.g., in the ticket booking scenario, the question information is "Which day do you want to book a flight?", and the user answers "tomorrow", the answer contains a date element, and this date element is a necessary response element), then the response completeness score is set to a score not lower than a preset completeness threshold (e.g., 85 points or a score higher than 85 points). If the obtained streaming recognition text does not contain necessary response elements, then the response completeness score is set to a score lower than the completeness threshold (e.g., a score lower than 85 points).

[0199] In one possible implementation, the maintenance of the current dialogue context adopts a "large model initialization + small model relay matching" approach to balance the depth of understanding of complex intentions in the first round of interaction with the low latency and low cost requirements in subsequent rounds of interaction.

[0200] Specifically, such as Figure 3 As shown, in the first round of interaction or key transition rounds (such as when user intent changes or the dialogue topic changes significantly), the large language model is invoked for processing. The large language model's processing includes: performing intent recognition and cross-domain slot extraction on the recognized text of the user's speech, outputting a structured dialogue state (e.g., JSON format), generating the question information for the current round (e.g., which necessary slots are currently missing), and outputting feature vectors or rule information needed for matching by the smaller models in subsequent rounds. After the large model is invoked, the complete structured state of the current dialogue is obtained, providing a contextual basis for subsequent rounds.

[0201] like Figure 3 As shown, in subsequent rounds, the large model is no longer invoked; instead, lightweight models or rule-based matching are used entirely. Specifically, slot filling is performed using lightweight models such as BERT-mini or DeBERTa-tiny, referential resolution and omission completion are performed based on the previous round's dialogue state combined with rules or small models, and response completeness is scored. Through this approach, subsequent rounds avoid the high latency and high cost associated with invoking the large model for every interaction, achieving a balance between understanding depth and response speed in multi-turn dialogue context management.

[0202] Steps d1-c: Determine the intent integrity of the obtained streaming recognition text to obtain the intent integrity probability.

[0203] Determining the completeness of intent in the obtained streaming text means judging whether the core intent expressed by the user has reached an executable state based on the semantic content of the current text and outputting a probability value that reflects the completeness of the intent.

[0204] Specifically, the current streaming text is input into a lightweight intent completeness classifier (e.g., with fewer than 3M parameters, based on a 3-layer Transformer structure). This intent completeness classifier outputs in real time the intent type (e.g., booking tickets, navigation, query) and the completeness of the core intent elements of the currently recognized text. The determination dimensions of the intent completeness classifier include: intent type identification, i.e., determining what type the user's current intent belongs to (booking tickets, navigation, query, etc.); completeness of core intent elements, i.e., determining whether the core elements necessary to complete the intent are complete (e.g., booking tickets requires departure point, destination, and date; navigation requires destination, etc.); intent expression termination tendency, i.e., determining whether the sentence structure and expression tend to await completion (e.g., "I want..." usually tends to be incomplete, while "I want to book a ticket to Beijing" is usually fully expressed). Based on the comprehensive judgment of the above dimensions, the intent completeness classifier outputs an intent completeness probability value.

[0205] For example, if a user says "Help me book a ticket," although the sentence-level semantic score may be high (the subject-verb-object structure is complete), the core elements necessary for the intention to book a ticket (departure location, destination, date, etc.) have not yet been provided. Therefore, the probability of the intention being complete is low, indicating that the user has not finished speaking and needs to supplement key information.

[0206] The core function of this step is to assess whether the semantics are truly complete from the perspective of task completion, and to avoid premature truncation when the intent elements are not complete.

[0207] Step d2: The sentence-level semantic integrity score, response completeness score, and intent integrity probability are fused to obtain the semantic integrity score of the streaming text.

[0208] In one possible implementation, the sentence-level semantic integrity score, response completeness score, and intent integrity probability can be weighted and summed to obtain the semantic integrity score of the streaming text. :

[0209] (3).

[0210] in, This represents the sentence-level semantic integrity score. This indicates the completeness score of the response. The probability value of representing complete intent. , , The weights can be pre-calibrated according to specific application scenarios or learned from training data.

[0211] This invention achieves comprehensive semantic perception of streaming text by constructing a three-level fusion semantic verification system encompassing sentence-level syntactic and semantic integrity, dialogue context response completeness, and intent executability. Sentence-level semantic integrity scores ensure basic language standardization, response completeness scores address the challenge of identifying elliptical sentences in multi-turn dialogues, and intent completeness probability ensures information sufficiency from a task execution perspective. The final semantic integrity score integrates semantic information from three levels: sentence-level syntactic structural integrity, multi-turn dialogue context response completeness, and intent task executability, providing a comprehensive and accurate semantic basis for subsequent endpoint determination.

[0212] After obtaining the corrected speech termination endpoint and the semantic integrity score of the streaming recognized text, the final speech termination endpoint can be determined based on the duration of continuous silence after the corrected speech termination endpoint and the semantic integrity score. This process is described in some embodiments of this application.

[0213] like Figure 4 As shown, the process of determining the final speech termination point based on the duration of continuous silence after the corrected speech termination point and the semantic integrity score may include:

[0214] Step S401: Determine whether the duration of continuous silence after the end of the corrected speech is greater than or equal to the silence threshold.

[0215] If the duration of continuous silence after the corrected speech endpoint is greater than or equal to the silence threshold, proceed to step S402a. If the duration of continuous silence after the corrected speech endpoint is less than the silence threshold, proceed to step S402b.

[0216] Step S402a: Determine whether the semantic integrity score of the streaming text is greater than or equal to the semantic integrity threshold.

[0217] If the semantic integrity score of the streaming text is greater than or equal to the semantic integrity threshold, then proceed to step S403a; if the semantic integrity score of the streaming text is less than the semantic integrity threshold, then proceed to step S403b.

[0218] Step S402b: Continue to perform frame-level speech endpoint detection and continuously monitor the cumulative silence duration.

[0219] Step S403a: Determine the corrected speech end endpoint as the final speech end endpoint.

[0220] When the duration of continuous silence after the corrected speech termination endpoint is greater than or equal to the silence threshold, and the semantic integrity score is greater than or equal to the semantic integrity threshold, it indicates that the user has stopped speaking acoustically and the currently recognized text is complete semantically. Since both conditions are met, the corrected speech termination endpoint is determined as the final speech termination endpoint, which can then trigger subsequent downstream processing flows (e.g., ...). Figure 5 (Including global decoding of the ED model, entity retrieval, and large model error correction, etc.).

[0221] Step S403b: Determine whether the duration of continuous silence after the end of the corrected speech is greater than or equal to the maximum waiting threshold.

[0222] If the duration of continuous silence after the corrected voice end point is greater than or equal to the maximum waiting threshold, then step S403a is executed, that is, the corrected voice end point is determined as the final voice end point. This branch is used to prevent infinite waiting caused by long user pauses (such as thinking, being interrupted, etc.) and to ensure the smoothness of the interaction. The maximum waiting threshold is greater than the silence threshold.

[0223] If the duration of continuous silence after the end of the corrected speech is less than the maximum waiting threshold, then proceed to step S404.

[0224] Step S404: If the pause is determined to be in the middle, remove the silent segment audio after the end point of the corrected speech and continue the detection.

[0225] If the silence duration has not exceeded the maximum waiting threshold, it indicates that the user may be pausing to breathe or thinking, and will continue speaking later. In this case, it is determined as an intermediate pause. The audio segment after the corrected speech end point is removed (i.e., the data of this silent segment is discarded to avoid interference in subsequent processing), and the audio monitoring state is maintained. The cached phoneme spike sequence and streaming recognition text are not cleared, and the user is waited to continue inputting subsequent speech to supplement semantic information.

[0226] Through the above steps, the endpoint output is subject to both acoustic and semantic constraints: when the silence condition is met, the endpoint can be output if the semantics are complete; when the semantics are incomplete, the endpoint is blocked and continues to wait for the user to supplement the semantics; in the event of a timeout, forced output is performed to ensure smooth interaction. This strategy avoids endpoint positioning deviations caused by acoustic interference such as pauses in breathing, and also prevents premature truncation in the case of incomplete semantics, ensuring that the speech segment defined by the endpoint has complete semantics.

[0227] It should be noted that the voice endpoint detection method provided by this invention does not require reconstruction of the ASR link or retraining of the model. It is fully compatible with the existing CTC-ED hybrid architecture and large model error correction process, does not require modification of the original model weights and business code, supports plug-and-play rapid deployment, and has extremely low engineering deployment costs.

[0228] In one possible implementation, the aforementioned silence threshold and maximum waiting threshold can be determined through a personalized adaptive approach to accommodate the differences in speaking habits and speech rates among different users.

[0229] Specifically, the process of determining the silence threshold and the maximum waiting threshold may include:

[0230] Step e1: Extract non-ending internal silence segments from the current speaker's historical complete speech segments and count the duration of each internal silence segment.

[0231] Historical complete speech segments refer to speech segments that have been identified and whose final endpoints have been output. Internal silence segments refer to silent fragments that occur within a speech segment (i.e., silent segments located between the start and end endpoints of the speech), rather than the final silent segment at the end of the speech segment. By extracting these internal silence segments, we can obtain the duration distribution data of the speaker's natural pauses during the current speech process.

[0232] For example, if the current speaker's complete speech is "I want to book a flight from Beijing to Shanghai (pause 0.4 seconds)," then the 0.4-second silence segment corresponding to the pause is extracted as an internal silence segment sample. As the speaker interacts multiple times, such internal silence segment duration data is continuously accumulated, providing a data foundation for statistical analysis.

[0233] Step e2: Calculate the average duration of each internal silent segment. and standard deviation .

[0234] Statistical analysis was performed on the duration of all internal silence segments extracted in step e1, and their mean was calculated. (i.e., average pause duration) and standard deviation (Reflecting the degree of fluctuation in pause duration). Among them, the mean... Reflects the typical duration and standard deviation of the speaker's habitual pauses. This reflects the degree of dispersion of the speaker's pause duration relative to the mean. If the standard deviation... A smaller standard deviation indicates that the speaker's pause duration is relatively stable. A larger value indicates that the speaker's pause duration fluctuates significantly.

[0235] Step e3: Based on the mean and standard deviation Determine the silence threshold.

[0236] The mean calculated based on step e2 and standard deviation Determine the silence threshold, which is equal to the mean. There is a positive correlation, meaning that the longer the speaker's average pause, the higher the silence threshold. At the same time, this silence threshold is limited to a preset upper and lower limit range to avoid abnormal behavior caused by thresholds that are too high or too low.

[0237] For example, the silence threshold It can be determined by the following formula:

[0238] (4).

[0239] Step e4: Determine the maximum waiting threshold based on the mute threshold.

[0240] Based on the silence threshold determined in step e3 Determine the maximum waiting threshold The maximum waiting threshold With silence threshold Positive correlation, i.e., silence threshold The larger the value, the higher the maximum waiting threshold. The maximum waiting threshold will also increase accordingly. It is also limited to the preset upper and lower limits.

[0241] For example, the maximum waiting threshold It can be determined by the following formula:

[0242] (5).

[0243] Maximum waiting threshold With silence threshold Maintain a reasonable ratio to ensure sufficient waiting time when a pause is misjudged, while preventing infinite waiting.

[0244] The specific values ​​in the above steps (such as 150ms, 700ms, 2.0 times, 1200ms, etc.) are for illustrative purposes only. In practical applications, they can be calibrated and adjusted according to specific scenarios and needs. This application does not impose any specific limitations on them.

[0245] It should be noted that a personalized pause parameter file can be maintained for each registered user (the initial personalized pause parameter file contains preset mute thresholds and maximum wait thresholds), such as... Figure 6 As shown, when a new user interacts for the first time, a preset mute threshold and a maximum waiting threshold can be used. After the interaction is completed, the mute threshold and the maximum waiting threshold can be determined based on the above steps. Then, the mute threshold and the maximum waiting threshold in the personalized pause parameter file can be updated based on the determined mute threshold and the maximum waiting threshold. The parameters in the personalized pause parameter file can be updated after each interaction, or at regular intervals.

[0246] The silence threshold and maximum waiting threshold determined by the above personalized adaptive method can be dynamically adjusted according to the speaker's historical pausing habits. Compared with the fixed threshold scheme, it can better adapt to user groups with different speaking speeds and different expression habits, and avoid endpoint misjudgment caused by individual differences.

[0247] This invention also provides a voice endpoint detection device, such as... Figure 7 As shown, the speech endpoint detection device may include: a coarse-grained speech endpoint detection module 701, a phoneme spike detection module 702a, a streaming text acquisition module 702b, a speech end endpoint refinement module 703, a semantic integrity evaluation module 704, and a speech end endpoint determination module 705.

[0248] The coarse-grained speech endpoint detection module 701 is used to perform frame-level speech endpoint detection through streaming audio to determine the speech start endpoint and candidate speech end endpoint.

[0249] The phoneme spike detection module 702a is used to perform phoneme spike detection starting from the speech start endpoint when the speech start endpoint is determined, so as to obtain the timing position of the phoneme spike. The timing position of the phoneme spike is the timing position of a valid vocalization action within the speech segment starting from the speech start endpoint.

[0250] The streaming text acquisition module 702b is used to acquire streaming text starting from the speech start endpoint.

[0251] The speech termination endpoint refinement module 703 is used to perform acoustic correction on the candidate speech termination endpoints based on the temporal position of each phoneme peak when a candidate speech termination endpoint is determined, so as to obtain the corrected speech termination endpoint.

[0252] The semantic integrity assessment module 704 is used to assess the semantic integrity of the obtained streaming recognition text and obtain a semantic integrity score.

[0253] The speech termination endpoint determination module 705 is used to determine the final speech termination endpoint based on the duration of continuous silence after the corrected speech termination endpoint and the semantic integrity score.

[0254] In one possible implementation, the coarse-grained speech endpoint detection module 701, when performing frame-level speech endpoint detection through streaming audio to determine the speech start endpoint and candidate speech end endpoint, is specifically used for:

[0255] Real-time acquisition of ambient noise intensity; determination of target noise level based on ambient noise intensity.

[0256] Determine the back field of view length, the number of initial decision frames, and the number of final decision frames that are adapted to the target noise level. The back field of view length is the number of future audio frames required for the speech endpoint detection model to make a single-frame decision.

[0257] The audio stream is classified into frames with sound and frames without sound based on the speech endpoint detection model.

[0258] When the number of consecutive audio frames reaches the initial decision frame number, the speech start endpoint is determined;

[0259] When the number of consecutive silent frames reaches the end decision frame number, the candidate speech end endpoint is determined.

[0260] In one possible implementation, the coarse-grained speech endpoint detection module 701 is further used for:

[0261] When the target noise level changes, if the ambient noise intensity remains within the range corresponding to the new noise level for M consecutive audio frames, the new noise level is determined to be effective. The back field of view length is switched to the back field of view length adapted to the new noise level, the starting decision frame number is switched to the starting decision frame number adapted to the new noise level, and the ending decision frame number is switched to the ending decision frame number adapted to the new noise level. M is a preset positive integer.

[0262] During the switching process between old and new noise level parameters, the accumulated number of sound frames or silent frames will not be cleared.

[0263] In one possible implementation, the coarse-grained speech endpoint detection module 701, when determining the starting decision frame number and the ending decision frame number adapted to the target noise level, is specifically used for:

[0264] Obtain the base start decision frame number and base end decision frame number configured for the target noise level;

[0265] The noise weighting factor is determined based on the environmental noise intensity.

[0266] Based on the noise weighting factor, the basic starting decision frame number and the basic ending decision frame number are corrected to obtain the starting decision frame number and the ending decision frame number that are adapted to the target noise level.

[0267] In one possible implementation, the voice endpoint detection device may further include a storage resource management module. The storage resource management module is used for:

[0268] Once the speech start endpoint is determined, a dedicated buffer is created for the speech segment starting from the speech start endpoint;

[0269] The phoneme spike detection results starting from the speech start point and the streaming recognized text are stored in the cache area;

[0270] Once the final voice endpoint is determined, the buffer is released.

[0271] When the speech termination refinement module 703 performs acoustic correction on the candidate speech termination endpoints based on the temporal position of each phoneme peak, it is specifically used to perform acoustic correction on the candidate speech termination endpoints based on the temporal position of each phoneme peak stored in the buffer.

[0272] The semantic integrity assessment module 704 is specifically used to assess the semantic integrity of the obtained streaming recognition text in the cache.

[0273] In one possible implementation, the phoneme spike detection module 702a, when performing phoneme spike detection starting from the speech start point, is specifically used for:

[0274] Starting from the beginning of the speech, every n audio frames are obtained and then concatenated to obtain a large audio frame, where n is a preset positive integer;

[0275] Encode large audio frames to obtain the embedding vector of the large audio frame;

[0276] Based on the embedding vector of the audio frame, predict the probability distribution of the audio frame on each phoneme tag in the phoneme tag set, and obtain the phoneme probability distribution of the audio frame.

[0277] Based on the phoneme probability distribution of each large audio frame, phoneme spikes are identified.

[0278] In one possible implementation, the phoneme spike detection module 702a, when identifying phoneme spikes based on the phoneme probability distribution of each large audio frame, is specifically used for:

[0279] For each audio frame, obtain the phoneme label corresponding to the highest probability in the phoneme probability distribution of the audio frame, and obtain the phoneme label corresponding to the audio frame.

[0280] If the phoneme label corresponding to the large audio frame is a non-blank label, and the phoneme label corresponding to the large audio frame satisfies the first condition or the second condition, then it is determined that the large audio frame generates a phoneme spike.

[0281] The first condition is that the phoneme tag is different from that of the audio frame that produces the previous phoneme peak. The second condition is that the phoneme tag is the same as that of the audio frame that produces the previous phoneme peak, but the time interval between the audio frame and the audio frame that produces the previous phoneme peak exceeds the preset duration threshold of the same phoneme.

[0282] The timing position where a phoneme spike is generated is the timing position of the audio frame in which the phoneme spike is generated.

[0283] In one possible implementation, the speech termination refinement module 703, when performing acoustic correction on candidate speech termination endpoints based on the temporal position of each phoneme peak to obtain the corrected speech termination endpoints, specifically performs the following:

[0284] Retrieve the temporal position of each phoneme peak and locate the temporal position of the last phoneme peak;

[0285] The timing position of the last phoneme peak is superimposed with the preset human voice attenuation compensation duration to obtain the corrected speech end point.

[0286] In one possible implementation, the semantic integrity assessment module 704, when assessing the semantic integrity of the obtained streaming recognized text and obtaining a semantic integrity score, is specifically used for:

[0287] Sentence-level semantic integrity is scored on the obtained streaming recognition text to obtain a sentence-level semantic integrity score.

[0288] The completeness of the response is verified based on the current dialogue context of the obtained streaming recognition text, and a response completeness score is obtained.

[0289] The obtained streaming recognition text is subjected to intent integrity determination to obtain the intent integrity probability.

[0290] The sentence-level semantic integrity score, response completeness score, and intent integrity probability are fused together, and the fusion result is used as the semantic integrity score of the obtained streaming recognition text.

[0291] In one possible implementation, when the semantic integrity assessment module 704 performs a response completeness check on the obtained streaming recognized text based on the current dialogue context and obtains a response completeness score, it is specifically used for:

[0292] Obtain the current dialogue context, which includes at least the question information of the current interaction round and the semantic slot information expressed by the user;

[0293] Determine the required semantic slots that are not yet filled based on the semantic slot information already expressed by the user;

[0294] The detection function checks whether the obtained streaming text contains necessary response elements for filling the required semantic slots and addressing the question information.

[0295] If the obtained streaming recognition text contains the necessary response elements, the response completeness score will be set to a score that is not lower than the preset completeness threshold.

[0296] If the obtained streaming text does not contain the necessary response elements, the response completeness score will be set to a score below the completeness threshold.

[0297] In one possible implementation, the speech termination endpoint determination module 705, when determining the final speech termination endpoint based on the duration of continuous silence following the corrected speech termination endpoint and the semantic integrity score, specifically performs the following:

[0298] If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, and the semantic integrity score is greater than or equal to the semantic integrity threshold, then the corrected speech end point is determined as the final speech end point.

[0299] In one possible implementation, the voice end-point determination module 705 is further configured to:

[0300] If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, but the semantic integrity score is less than the semantic integrity threshold, it is judged as an intermediate pause, the audio segment of silence after the corrected speech end point is removed and the detection continues.

[0301] If the semantic integrity score is less than the semantic integrity threshold, and the duration of continuous silence after the corrected speech end endpoint is greater than or equal to the maximum waiting threshold, then the corrected speech end endpoint will be determined as the final speech end endpoint.

[0302] In one possible implementation, the voice endpoint detection device may further include a threshold determination module. The threshold determination module is used to determine a silence threshold and a maximum waiting threshold.

[0303] The threshold determination module is specifically used to determine the mute threshold and the maximum wait threshold, specifically for:

[0304] Extract non-ending internal silence segments from the current speaker's complete historical speech segments and calculate the duration of each internal silence segment;

[0305] Calculate the mean and standard deviation of the duration of each internal silent segment;

[0306] The silence threshold is determined based on the mean and standard deviation;

[0307] Determine the maximum waiting threshold based on the silence threshold.

[0308] The speech endpoint detection device provided in this embodiment of the invention ensures the accuracy of endpoint positioning through acoustic correction and ensures that the speech segment defined by the endpoint has complete semantics through semantic integrity assessment. This avoids endpoint positioning deviation caused by acoustic interference such as breathing pauses and also prevents premature truncation in the case of incomplete semantics.

[0309] This invention also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0310] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention; the memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage.

[0311] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the voice endpoint detection method provided in the above embodiments.

[0312] This invention also provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement the voice endpoint detection method provided in the above embodiments.

[0313] This invention also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the voice endpoint detection method provided in the above embodiments.

[0314] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by this invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0315] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for the present invention, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0316] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0317] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for detecting speech endpoints, characterized in that, include: By performing frame-level speech endpoint detection on streaming audio, the speech start endpoint and candidate speech end endpoint are determined. When the speech start endpoint is determined, the streaming recognition text is obtained starting from the speech start endpoint, and phoneme spike detection is performed to obtain the temporal position of the phoneme spike, wherein the temporal position of the phoneme spike is the temporal position of an effective vocalization action within the speech segment starting from the speech start endpoint. When the candidate speech endpoint is determined, the candidate speech endpoint is acoustically corrected according to the temporal position of each phoneme peak to obtain the corrected speech endpoint. The semantic integrity of the obtained streaming recognized text is evaluated to obtain a semantic integrity score. The final speech termination point is determined based on the duration of continuous silence following the corrected speech termination point and the semantic integrity score. The step of acoustically correcting the candidate speech termination endpoints based on the temporal position of each phoneme peak to obtain the corrected speech termination endpoints includes: Retrieve the temporal position of each phoneme spike and locate the temporal position of the last phoneme spike; The timing position of the last phoneme peak is superimposed with a preset human voice attenuation compensation duration to obtain the corrected speech end point. The step of determining the final speech termination point based on the duration of continuous silence following the corrected speech termination point and the semantic integrity score includes: If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, and the semantic integrity score is greater than or equal to the semantic integrity threshold, then the corrected speech end point is determined as the final speech end point.

2. The voice endpoint detection method according to claim 1, characterized in that, The step of performing frame-level speech endpoint detection on streaming audio to determine the speech start endpoint and candidate speech end endpoint includes: Real-time acquisition of ambient noise intensity; determination of target noise level based on ambient noise intensity. Determine the back field of view length, the number of initial decision frames, and the number of final decision frames that are adapted to the target noise level, wherein the back field of view length is the number of future audio frames required for the speech endpoint detection model to make a single-frame decision; Based on the aforementioned voice endpoint detection model, the streaming audio is judged frame by frame for whether it contains sound or not. When the number of consecutive audio frames reaches the initial decision frame number, the speech start endpoint is determined; When the number of consecutive silent frames reaches the number of frames for the end decision, the candidate speech end endpoint is determined.

3. The voice endpoint detection method according to claim 2, characterized in that, Also includes: When the target noise level changes, if the ambient noise intensity remains within the range corresponding to the new noise level for M consecutive audio frames, the new noise level is determined to be effective, and the back field of view length is switched to the back field of view length adapted to the new noise level, the starting decision frame number is switched to the starting decision frame number adapted to the new noise level, and the ending decision frame number is switched to the ending decision frame number adapted to the new noise level, where M is a preset positive integer; During the switching process between old and new noise level parameters, the accumulated number of sound frames or silent frames will not be cleared.

4. The voice endpoint detection method according to claim 2, characterized in that, Determining the starting and ending decision frames that are compatible with the target noise level includes: Obtain the basic start decision frame number and basic end decision frame number configured for the target noise level; The noise weighting factor is determined based on the environmental noise intensity. Based on the noise weighting factor, the basic starting decision frame number and the basic ending decision frame number are corrected to obtain the starting decision frame number and the ending decision frame number that are adapted to the target noise level.

5. The voice endpoint detection method according to claim 1, characterized in that, Also includes: Once the speech start endpoint is determined, a dedicated buffer is created for the speech segment starting from the speech start endpoint; The phoneme spike detection results and streaming text, starting from the speech start endpoint, are stored in the cache area; Once the final end point of the voice recording is determined, the buffer is released; The acoustic correction of the candidate speech termination endpoints based on the temporal position of each phoneme peak includes: Based on the temporal position of each phoneme spike stored in the buffer, the candidate speech end point is acoustically corrected; The semantic integrity assessment of the obtained streaming text includes: The semantic integrity of the streaming recognized text stored in the cache is evaluated.

6. The voice endpoint detection method according to claim 1, characterized in that, Starting from the speech start point, phoneme spike detection is performed, including: Starting from the speech start point, every n audio frames obtained are concatenated to obtain a large audio frame, where n is a preset positive integer; The audio large frame is encoded to obtain the embedding vector of the audio large frame; Based on the embedding vector of the large audio frame, predict the probability distribution of the large audio frame on each phoneme tag in the phoneme tag set, and obtain the phoneme probability distribution of the large audio frame. Based on the phoneme probability distribution of each audio frame, phoneme spikes are identified.

7. The voice endpoint detection method according to claim 6, characterized in that, The step of identifying phoneme spikes based on the phoneme probability distribution of each of the audio frames includes: For each of the aforementioned audio frames, the phoneme tag corresponding to the highest probability in the phoneme probability distribution of the audio frame is obtained, and the phoneme tag corresponding to the audio frame is obtained. If the phoneme tag corresponding to the large audio frame is a non-blank tag, and the phoneme tag corresponding to the large audio frame satisfies the first condition or the second condition, then it is determined that the large audio frame generates a phoneme spike. The first condition is that the phoneme tag is different from that of the audio frame that generates the previous phoneme peak. The second condition is that the phoneme tag is the same as that of the audio frame that generates the previous phoneme peak, but the time interval between the audio frame and the audio frame that generates the previous phoneme peak exceeds a preset threshold for the duration of the same phoneme. The timing position at which a phoneme spike is generated is the timing position of the audio frame in which the phoneme spike is generated.

8. The voice endpoint detection method according to claim 1, characterized in that, The semantic integrity assessment of the obtained streaming text, to obtain a semantic integrity score, includes: Sentence-level semantic integrity is scored on the obtained streaming recognition text to obtain a sentence-level semantic integrity score. The completeness of the response is verified based on the current dialogue context of the obtained streaming recognition text, and a response completeness score is obtained. The obtained streaming recognition text is subjected to intent integrity determination to obtain the intent integrity probability. The sentence-level semantic integrity score, the response completeness score, and the intent integrity probability are fused together, and the fusion result is used as the semantic integrity score of the obtained streaming recognition text.

9. The voice endpoint detection method according to claim 8, characterized in that, The step of performing a response completeness check on the obtained streaming recognition text based on the current dialogue context to obtain a response completeness score includes: Obtain the current dialogue context, which includes at least the question information of the current interaction round and the semantic slot information expressed by the user; Based on the semantic slot information already expressed by the user, determine the currently unfilled mandatory semantic slots; Detect whether the obtained streaming recognition text contains necessary response elements for filling the required semantic slots and for the question information; If the obtained streaming recognition text contains the necessary response elements, the response completeness score will be set to a score that is not lower than a preset completeness threshold. If the obtained streaming text does not contain the necessary response elements, the response completeness score will be set to a score lower than the completeness threshold.

10. The voice endpoint detection method according to claim 1, characterized in that, Also includes: If the duration of continuous silence after the corrected speech end point is greater than or equal to the silence threshold, but the semantic integrity score is less than the semantic integrity threshold, it is determined to be an intermediate pause, the silent segment audio after the corrected speech end point is removed and the detection continues. If the semantic integrity score is less than the semantic integrity threshold, and the duration of continuous silence after the corrected speech end point is greater than or equal to the maximum waiting threshold, then the corrected speech end point is determined as the final speech end point.

11. The voice endpoint detection method according to claim 10, characterized in that, The process of determining the mute threshold and the maximum waiting threshold includes: Extract non-ending internal silence segments from the current speaker's complete historical speech segments, and calculate the duration of each internal silence segment; Calculate the mean and standard deviation of the duration of each of the aforementioned internal silent segments; The silence threshold is determined based on the mean and the standard deviation. The maximum waiting threshold is determined based on the mute threshold.

12. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the voice endpoint detection method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing an electronic device to perform the voice endpoint detection method as described in any one of claims 1 to 11.

Images