Voice endpoint recognition method and device, storage medium and electronic device

Abstract

The application provides a speech endpoint recognition method and device, a storage medium and an electronic device, and relates to the field of speech recognition. The method comprises the following steps: in the process of collecting the current speech stream of a user in real time, if it is detected that there is a mute segment in the current speech stream that meets a preset trigger condition, the historical speech stream of the user is obtained; the prosodic memory vector of the current speech stream and the historical speech stream is determined; a speech stream sequence is constructed according to the speech stream features of the current speech stream, the prosodic memory vector is spliced at the front end to obtain a first spliced vector, the acoustic features of a hypothetical mute segment are spliced after the speech stream features to construct a hypothetical mute sequence, the prosodic memory vector is spliced at the front end to obtain a second spliced vector; the first end probability of the first spliced vector and the second end probability of the second spliced vector are predicted; and whether the current speech stream is ended is determined according to the difference between the first end probability and the second end probability. The application solves the problem of low speech endpoint recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of speech recognition, and more particularly to a speech endpoint recognition method, apparatus, storage medium, and electronic device. Background Technology

[0002] In existing technologies, Voice Activity Detection (VAD) is a key technology in real-time voice interaction systems, used to determine when a user begins and ends speaking.

[0003] In existing technologies, the common method to identify whether a user has finished speaking is to set a fixed threshold. If the silence duration exceeds the fixed threshold, the user is considered to have finished speaking. However, this method is not suitable for scenarios where the user is thinking in the middle of a conversation, as it is easy for the speaker to "interrupt" the conversation, that is, reply to the user before the user has actually finished speaking. Therefore, in existing technologies, voice endpoint recognition is inaccurate. Summary of the Invention

[0004] This application provides a voice endpoint recognition method, apparatus, storage medium, and electronic device to solve the problem of low accuracy in voice endpoint recognition.

[0005] In a first aspect, this application provides a voice endpoint recognition method, comprising: during the real-time acquisition of a user's current voice stream, if a silent segment satisfying a preset trigger condition is detected in the current voice stream, then acquiring the user's historical voice stream; determining the prosodic memory vectors of the current voice stream and the historical voice stream; constructing a voice stream sequence based on the voice stream features of the current voice stream, and concatenating the prosodic memory vectors at the beginning to obtain a first concatenated vector, and concatenating the acoustic features of a hypothetical silent segment after the voice stream features to construct a hypothetical silent sequence, and concatenating the prosodic memory vectors at the beginning to obtain a second concatenated vector; predicting a first end probability of the first concatenated vector and a second end probability of the second concatenated vector respectively; and determining whether the current voice stream has ended based on the difference between the first end probability and the second end probability.

[0006] Secondly, this application provides a voice endpoint recognition device, comprising: an acquisition module, configured to acquire the user's historical voice stream if a silent segment satisfying a preset trigger condition is detected in the current voice stream during real-time acquisition of the user's current voice stream; a determination module, configured to determine the prosodic memory vectors of the current voice stream and the historical voice stream; a construction module, configured to construct a voice stream sequence based on the voice stream features of the current voice stream, and concatenate the prosodic memory vectors at the beginning to obtain a first concatenation vector, and concatenate the acoustic features of a hypothetical silent segment after the voice stream features to construct a hypothetical silent sequence, and concatenate the prosodic memory vectors at the beginning to obtain a second concatenation vector; a prediction module, configured to predict a first end probability of the first concatenation vector and a second end probability of the second concatenation vector respectively; and a judgment module, configured to determine whether the current voice stream has ended based on the difference between the first end probability and the second end probability.

[0007] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of any of the above.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above.

[0009] Compared with the prior art, the technical solution provided in this application has the following advantages: In this technical solution, if a silent segment that meets a preset trigger condition is detected in the current voice stream during real-time acquisition of the user's current voice stream, the user's historical voice stream is obtained; the prosodic memory vectors of the current voice stream and the historical voice stream are determined; a voice stream sequence is constructed based on the voice stream features of the current voice stream, and the prosodic memory vector is concatenated at the beginning to obtain a first concatenated vector; and the acoustic features of the hypothetical silent segment are concatenated after the voice stream features to construct a hypothetical silent sequence, and the prosodic memory vector is concatenated at the beginning to obtain a first concatenated vector. The first concatenation vector is concatenated at the front end to obtain the second concatenation vector. The first end probability of the first concatenation vector and the second end probability of the second concatenation vector are predicted respectively. Based on the difference between the first end probability and the second end probability, the method of whether the current speech stream has ended is determined. Thus, in the process of detecting speech endpoints for users, by constructing a hypothetical silence sequence, the end probability is predicted together with the real speech stream sequence to further determine the difference between the two. Moreover, the prosodic features of the user's speech are enhanced by prosodic memory vectors. Finally, if the difference between the two is greater, it indicates that the user is more likely to stop speaking, thus achieving the effect of accurately identifying speech endpoints. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0013] Figure 1 A flowchart of the voice endpoint recognition method provided in the embodiments of this application; Figure 2 A flowchart illustrating another voice endpoint recognition method provided in this application embodiment; Figure 3 A flowchart illustrating another voice endpoint recognition method provided in this application embodiment; Figure 4 A flowchart illustrating another voice endpoint recognition method provided in this application embodiment; Figure 5 A flowchart illustrating another voice endpoint recognition method provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a voice endpoint recognition device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0016] The following will describe in detail a voice endpoint recognition method provided in this application embodiment, with reference to specific implementation methods. Figure 1 As shown, the method includes: S102, during the real-time acquisition of the user's current voice stream, if a silent segment that meets the preset triggering conditions is detected in the current voice stream, then the user's historical voice stream is acquired. S104, Determine the prosodic memory vectors of the current speech stream and the historical speech stream; S106. Construct a speech stream sequence based on the speech stream features of the current speech stream, and concatenate the prosodic memory vector at the front end to obtain the first concatenation vector. Concatenate the acoustic features of the hypothetical silence segment after the speech stream features to construct the hypothetical silence sequence, and concatenate the prosodic memory vector at the front end to obtain the second concatenation vector. S108, predict the first end probability of the first concatenation vector and the second end probability of the second concatenation vector respectively; S110, determine whether the current speech stream has ended based on the difference between the first end probability and the second end probability.

[0017] This application can be applied in various scenarios. For example, in intelligent customer service and call center scenarios, customer service robots often "intercept" (respond before the user has finished speaking), resulting in a very poor user experience. The solution adopted in this application concatenates a prosodic memory vector before the user's current speech stream sequence to obtain a first concatenated vector. It also constructs an imaginary silence sequence by combining the speech stream features of the current speech stream with the acoustic features of the imaginary silence segment, and concatenates the prosodic memory vector at the front end to obtain a second concatenated vector. It predicts the end probability of the two vectors and determines whether the current speech stream has ended based on the difference between the two end probabilities. This accurately identifies the endpoint of the speech stream, avoids intelligent customer service robots "intercepting" the user, and improves the user experience.

[0018] In scenarios like intelligent in-vehicle voice interaction systems, a driver might habitually pause for 400ms to breathe while speaking. Conventional solutions would typically interpret this as the user stopping speaking, when in reality, the user is still thinking. Therefore, the solution proposed in this application prepends a prosodic memory vector to the user's current speech stream sequence to obtain a first prepend vector. It also constructs an imaginary silence sequence by combining the speech stream features of the current stream with the acoustic features of an imaginary silence segment, and prepends this imaginary silence sequence with a prosodic memory vector to obtain a second prepend vector. The solution predicts the end probability of both probabilities and determines whether the current speech stream has ended based on the difference between the two probabilities. This accurately identifies the endpoint of the speech stream, avoids premature responses, and improves the user experience.

[0019] In this application, during user voice interaction, the system can capture the user's current voice stream in real time via a microphone. During the capture process, the system continuously monitors the current voice stream to determine whether there are any silent segments that meet preset trigger conditions.

[0020] The current speech stream is a continuous audio waveform data that the microphone is capturing in real time, representing what the user is currently speaking. This is the physical signal that the system is currently processing. The system needs to extract acoustic features from it to understand what the user is saying.

[0021] A silence segment is a time window detected within the real-time captured speech stream that contains no human speech. A silence segment triggers the entire complex prediction process. The system only performs subsequent endpoint prediction when a silence segment that meets the specified conditions is detected, thus avoiding wasting computing power while the user is speaking normally.

[0022] For example, if a user is speaking and generating the current audio stream, and then pauses for 300ms, there is no spoken audio during those 300ms; it may be silent. Therefore, it is necessary to determine whether the 300ms constitutes a silent segment. If it does, the process described in this application is initiated to determine whether the current audio stream has ended. The method for determining whether a silent segment has been formed is explained in detail below.

[0023] After detecting a silent segment that meets preset trigger conditions, the system retrieves the user's historical voice stream. The historical voice stream consists of the audio data that the user has spoken before the current sentence in the current dialogue turn.

[0024] In the current conversation turn, all or part of the audio data spoken by the user before the current sentence can be used as a historical speech stream. This historical data source serves as a foundation for building a personalized speaking profile for the user, allowing for the extraction of past prosodic habits to smooth out any potential pronunciation errors and ensure the continuity of memory.

[0025] Once the historical speech stream is acquired, a shared prosodic memory vector can be constructed for both the current and historical speech streams. This prosodic memory vector reflects the user's personalized speaking habits formed during multi-turn conversations and is used to provide global contextual information in subsequent prediction steps.

[0026] The prosodic memory vector can be a fixed-dimensional vector, formed by a weighted fusion of the prosodic features of the current speech stream and the prosodic features of the historical speech stream. This vector can incorporate user-specific pauses and speech rate habits when analyzing each frame of sound, reducing the misjudgment rate.

[0027] After obtaining the prosodic memory vector, a speech stream sequence is constructed based on the speech stream features of the current speech stream. The speech stream sequence is a two-dimensional matrix composed of the acoustic features of the current speech stream arranged in chronological order (e.g., [100 frames, 80 dimensions], representing 100 frames of the current speech stream, with each frame including 80 dimensions of features). The prosodic memory vector is concatenated to the beginning of the speech stream sequence to obtain the first concatenated vector. The first concatenated vector represents the user's actual speaking state, combined with the user's personalized speaking habits (prosodic memory vector) formed during multiple rounds of dialogue.

[0028] Furthermore, this application can also construct hypothetical silence sequences in parallel. A hypothetical silence sequence requires constructing hypothetical silence segments, which are the sound fragments of a hypothetical user ending their speech. Based on the current environment (quiet, noisy, or multi-person meeting), an adaptively constructed segment is generated representing the sound the microphone would record next if the user had finished speaking (e.g., zero fill, background noise, or historical silence). This hypothetical silence segment can serve as a hypothetical prop for controlling variable testing. The system constructs the illusion that the user might have stopped speaking by splicing this hypothetical silence segment onto the current speech stream. Specifically, during splicing, the acoustic features of the hypothetical silence segment are spliced ​​after the speech stream features to construct the hypothetical silence sequence. The hypothetical silence sequence then represents the complete state assuming the user has finished speaking. Adding the prosodic memory vector at the front end constitutes a second splicing vector for comparison. The second splicing vector represents the test state assuming the user has finished speaking, and differs from the first splicing vector only in the presence of a hypothetical silence segment at the end.

[0029] Next, predictions are made on the first and second concatenated vectors to obtain the first and second termination probabilities. The first termination probability represents the probability that the current speech stream corresponding to the predicted speech stream sequence has ended; the second termination probability represents the probability that the hypothetical silence segment corresponding to the hypothetical silence sequence has ended. In other words, these are the probabilities of whether the user's current speech stream has finished, and whether the current speech stream has finished after concatenating with the hypothetical silence segment.

[0030] The system calculates the difference between the first end probability and the second end probability, and determines whether the current speech stream has ended based on the difference.

[0031] When the difference meets the preset termination condition, the system determines that the current voice stream has ended and triggers the subsequent voice recognition or command execution process; when the difference does not meet the preset termination condition, the system determines that the user has not finished speaking and continues to maintain the listening state.

[0032] For example, in a smart customer service scenario, the system determines whether the user has finished speaking while they are speaking in real time.

[0033] When a user enters voice mode, this application begins to collect the user's current voice stream in real time. Suppose the user is asking, "I want to check what I bought last week..." At this point, the user pauses briefly while thinking about the specific product name. The system continuously monitors this current voice stream in the background. When it detects a silent segment in the current voice stream that meets preset trigger conditions (e.g., silence duration and silence probability reaching specific thresholds), the system determines that endpoint detection logic needs to be activated to determine whether the user has finished speaking or is merely taking a breath or thinking in the middle of a sentence.

[0034] At this point, the system immediately retrieves the user's previous speech history from the current conversation. For example, the user may have said short phrases like "hello" and "transfer to human operator" in the previous conversation. The system combines the current speech stream ("I want to check the one I bought last week...") with the previous speech history to analyze the user's pronunciation habits, speaking speed, and pause patterns, thereby determining the user's prosodic memory vector. This vector encapsulates the user's unique prosodic features, providing a personalized benchmark for subsequent judgments.

[0035] The system first extracts the speech stream features from the current speech stream to construct a speech stream sequence. Next, the system concatenates the determined prosodic memory vector to the beginning of the speech stream sequence, forming the first concatenated vector. This vector represents the user's current speaking state, which may not yet be finished.

[0036] Simultaneously, the system constructs a hypothetical silence segment (e.g., a standard silent data segment), concatenates its acoustic features to the speech stream features of the current speech stream, forming a hypothetical silence sequence. Subsequently, the prosodic memory vector is concatenated to the beginning of this hypothetical silence sequence to obtain a second concatenated vector. This vector simulates the hypothetical state where, if the user has indeed finished speaking here, a period of silence will immediately follow.

[0037] The system inputs the first concatenated vector into the prediction model and outputs the first end probability. This probability represents the likelihood that the model considers the current speech stream to be complete without considering additional silence assumptions. In parallel, the system inputs the second concatenated vector into the same prediction model and outputs the second end probability. This probability represents the likelihood that the model considers the user to have actually finished speaking after forcibly concatenating a hypothetical silence.

[0038] Finally, the system calculates the difference between the second termination probability and the first termination probability.

[0039] If the difference is large (e.g., exceeding the preset threshold), it means that after introducing the hypothetical silence segment, the model's confidence in the user's judgment that the user has finished speaking has significantly improved. The system then determines that the current voice stream has ended and immediately sends the voice message "I want to check the one I bought last week..." into the voice recognition engine for transcription, triggering the intelligent customer service's intent recognition and response process.

[0040] If the difference is small, it means that even with the addition of hypothetical silence, the model still thinks the user has not finished speaking (possibly because the user's usual speaking habit is to pause for a long time). The system then determines that the current speech stream has not ended, continues to record the sound, and waits for the user to say the following product name, thus avoiding interrupting the user's natural expression.

[0041] For example, if the difference is ≥ 0.37, it indicates a significant increase in the model's confidence in ending after the silence assumption, and is judged as a true termination, with the system immediately triggering a proxy response. If the difference is ≤ 0.11, it indicates that the model's confidence in ending after the silence assumption has not increased or has even decreased, and is judged as a pause in thinking, with the system directly continuing to listen and silently waiting for the user to continue. If 0.11 < difference < 0.37, it enters the uncertainty interval, and the system initiates a short (e.g., 180ms) traditional lag observation window, waits, and then retryes, or prepares to trigger the cloud-based final judgment process.

[0042] The cloud arbitration process of this application is as follows: if the judgment falls into the uncertainty range twice in a row, the speech flow features and prosodic memory vector of the current speech stream are automatically uploaded to a higher-precision large model in the cloud for final adjudication (results are returned in about 9ms).

[0043] If cloud arbitration is not required, the prediction model will merge the prosodic features of the current speech stream into the historical prosodic memory vector of the previous round after each successful response. If a change in voiceprint or a long period of silence (>30 seconds) is detected, the historical prosodic memory vector will be exponentially decayed or reset to prevent cross-user contamination.

[0044] In this technical solution, during the real-time acquisition of the user's current speech stream, if a silent segment meeting a preset trigger condition is detected in the current speech stream, the user's historical speech stream is acquired; the prosodic memory vectors of the current speech stream and the historical speech stream are determined; a speech stream sequence is constructed based on the speech stream features of the current speech stream, and the prosodic memory vector is concatenated at the front end to obtain a first concatenated vector; acoustic features of a hypothetical silent segment are concatenated after the speech stream features to construct a hypothetical silent sequence, and the prosodic memory vector is concatenated at the front end to obtain a second concatenated vector; the first end probability of the first concatenated vector and the second end probability of the second concatenated vector are predicted respectively; based on the difference between the first end probability and the second end probability, the manner in which the current speech stream ends is determined. Thus, during the process of detecting the user's speech endpoint, by constructing a hypothetical silent sequence and predicting the end probability together with the real speech stream sequence, the difference between the two is further determined. Moreover, the prosodic features of the user's speech are enhanced by the prosodic memory vector. Finally, if the difference between the two is greater, it indicates that the user is more likely to stop speaking, thus achieving the effect of accurately identifying speech endpoints.

[0045] As an optional implementation method, such as Figure 2 As shown, determining the prosodic memory vectors of the current speech stream and the historical speech stream includes: S202, extract multi-dimensional prosodic features from the current speech stream, calculate at least one statistical value among the mean, standard deviation, maximum value, and minimum value for each prosodic feature, and concatenate the multi-dimensional prosodic features with the corresponding statistical values ​​to obtain the prosodic vector of the current speech stream; S204, the prosodic memory vector of the previous interactive round of the historical speech stream is weighted and summed with the prosodic vector of the current speech stream to obtain the prosodic memory vector.

[0046] In this application, to obtain the prosodic memory vectors of the current speech stream and historical speech streams, multi-dimensional prosodic features can be extracted from the current speech stream. These prosodic features are used to characterize a user's speaking habits at a macroscopic level, and may specifically include at least one of the following: pitch entropy: used to characterize the disorder of pitch distribution in the current speech stream within a preset time window; speech rate variation coefficient: the ratio of the standard deviation to the mean of syllable duration, used to characterize the stability of speech rate; average pause duration: the arithmetic mean of the durations of non-speech segments in the current speech stream; energy envelope slope: used to characterize the energy change trend of the current speech stream within a short time frame; and pause distribution histogram: used to statistically analyze the frequency of occurrence of pause intervals of different durations in the current speech stream.

[0047] Next, for each extracted prosodic feature, the system calculates at least one statistical value among the mean, standard deviation, maximum value, and minimum value (the pause distribution histogram, as a graphical feature, can calculate at least one statistical value among the mean, standard deviation, maximum value, and minimum value of the values ​​in the graph).

[0048] Subsequently, the system concatenates the multidimensional prosodic features with their corresponding statistical values ​​to obtain the prosodic vector of the current speech stream. For example, if the system selects four prosodic features—pitch entropy, speech rate variation coefficient, average pause duration, and energy envelope slope—and calculates four statistical values ​​for each feature—mean, standard deviation, maximum value, and minimum value—then the four features are concatenated with the four statistical values ​​to obtain a 16-dimensional feature vector as the prosodic vector of the current speech stream.

[0049] The system reads the historical prosodic memory vector generated in the previous interaction round of the historical speech stream from memory. Using a preset smoothing coefficient, the system performs a weighted sum of the historical prosodic memory vector and the prosodic vector of the current speech stream to obtain the updated prosodic memory vector.

[0050] In this application, if it is the first round of speech, there is no historical speech stream. If it is the second round of speech, the first round of speech is used as the historical speech stream, and the prosodic vector of the first round of speech is used as the prosodic memory vector of the first round of speech. In the second round of speech, the prosodic memory vector of the first round of speech is used as the historical prosodic memory vector of the historical speech stream used in the second round of speech to determine the prosodic memory vector of the second round. In the third round of speech, the prosodic memory vector of the second round of speech is used as the historical prosodic memory vector of the historical speech stream to determine the prosodic memory vector of the third round. The historical prosodic memory vector can be stored in memory for later retrieval.

[0051] Specifically, the weighted summation can be implemented using the Exponential Moving Average (EMA) algorithm. For example, setting the smoothing coefficient α to 0.3, the updated prosodic memory vector = 0.3 × the prosodic vector of the current speech stream + 0.7 × the historical prosodic memory vector. Through this mechanism, the system can smoothly integrate the latest prosodic performance of the current round while preserving the user's long-term speaking habits, effectively filtering out transient abnormal features caused by coughing, throat clearing, or emotional fluctuations, so that the prosodic memory vector can adaptively approximate the user's true long-term speaking style.

[0052] The smoothing coefficient (i.e., the weight) is not a fixed constant, but a variable that is adaptively adjusted according to the dynamic characteristics of the current interaction scenario. The system dynamically determines the value of the smoothing coefficient by evaluating the stability of the current speech stream, the degree of environmental noise interference, and the user's emotional state, in order to achieve a balance between quickly adapting to user changes and maintaining the stability of historical memory.

[0053] In this application, the conditions and adjustment strategies affecting the smoothing coefficient include multiple dimensions. For example, adjustment is based on prosodic feature volatility. The system calculates the short-term volatility of prosodic features (such as speech rate and pitch) in the current speech stream in real time and adjusts the smoothing coefficient according to the magnitude of the volatility. When the prosodic feature volatility of the current speech stream exceeds a preset first volatility threshold (e.g., the user suddenly speeds up their speech or increases their volume), it is determined that the user's speaking habits have changed significantly. At this time, the system increases the smoothing coefficient. Increasing the weight of the prosodic vector of the current speech stream allows the prosodic memory vector to quickly forget old habits, rapidly capture the user's current new state, and improve the sensitivity to sudden changes in speech rate. When the prosodic feature volatility is lower than a preset second volatility threshold (e.g., the user is in a calm narration state), it is determined that the user's speaking habits are stable. At this time, the system decreases the smoothing coefficient.

[0054] Alternatively, adjustment can be based on the ambient signal-to-noise ratio (SNR). The system acquires the background noise level in decibels from the current audio acquisition device, calculates the SNR, and adjusts the smoothing coefficient accordingly. When the SNR is below a preset noise threshold (e.g., in a car with the windows open or in a noisy shopping mall), the prosodic features extracted from the current speech stream may contain significant noise interference, reducing reliability. In this case, the system forcibly limits the smoothing coefficient to an extremely low range. When the SNR is above the preset noise threshold (e.g., in a quiet indoor environment), the reliability of the current speech stream features is high. In this case, the system allows the smoothing coefficient to return to the standard range.

[0055] Alternatively, the smoothing coefficient can be adjusted based on the confidence score of the interaction rounds. The system adjusts the smoothing coefficient in reverse based on the confidence scores of speech recognition in historical interaction rounds. If the confidence score of the recognition result in the previous round is low (e.g., below 0.6), it indicates that the prosodic memory vector updated in the previous round may not be accurate enough. In this case, the system automatically decays the smoothing coefficient when calculating in the current round.

[0056] In this application, by determining the prosodic memory vectors of the current speech stream and the historical speech stream, the user's speech habits can be obtained, which can assist in subsequent speech endpoint detection.

[0057] As an optional implementation method, such as Figure 3 As shown, the acoustic features of splicing hypothetical silence segments after speech stream features include: S302, Based on the user's interaction scenario, construct an imaginary silence segment that matches the interaction scenario, wherein the sound included in the imaginary silence segment is consistent with the sound of the interaction scenario; S304, the extracted acoustic features of the hypothetical silence segment are concatenated after the speech stream features.

[0058] In this application, since a hypothetical silence segment needs to be constructed, different hypothetical silence segments can be constructed for different environments.

[0059] First, the system identifies the interaction scenario and constructs a matching hypothetical silence segment. While acquiring the current voice stream, the system monitors and identifies the user's current interaction scenario in real time. The identification of the interaction scenario can be based on sensor data from the terminal device (such as a positioning system or accelerometer), contextual information from the application (such as running navigation software or a music player), or environmental sound classification results. Based on the identified interaction scenario, the system retrieves and constructs a hypothetical silence segment that matches the scenario from a pre-set acoustic material library. Unlike conventional zero-silence or Gaussian white noise, the hypothetical silence segment constructed in this application includes a sound background that is highly consistent with the ambient sound of the interaction scenario (this also includes cases where the ambient sound is silent or Gaussian white noise, therefore the hypothetical silence segment is silent or Gaussian white noise; however, in this application, because the environment itself is silent or Gaussian white noise, the constructed silence segment is also silent or Gaussian white noise, emphasizing that the hypothetical silence segment is consistent with the ambient sound).

[0060] For example, if the system detects that the user is in a moving vehicle, the hypothetical quiet zone it constructs will include smooth tire noise, wind noise, and low-frequency engine hum. If the system detects that the user is in a home environment with a television playing, the hypothetical quiet zone it constructs will include faint background human voices or background noise from the television program. If the system detects that the user is on an outdoor street, the hypothetical quiet zone it constructs will include distant traffic sounds, wind sounds, or intermittent pedestrian noise.

[0061] The system extracts acoustic features from the constructed hypothetical silence segment. Specifically, it uses the same algorithms as those used to extract features from the current speech stream, such as Mel-Frequency Cepstral Coefficients (MFCCs) and Filter Bank (FBANK) features, to extract the acoustic feature vector of the hypothetical silence segment. Subsequently, the system concatenates the extracted acoustic features of the hypothetical silence segment after the speech stream features of the current speech stream.

[0062] In real-world voice interactions, the silence after a user finishes speaking is often not absolute, but rather a return to ambient background noise. Therefore, this application uses hypothetical silence segments consistent with the scene to essentially implant an acoustic fingerprint of ambient noise into the prediction model. When the model processes the concatenated vector, it learns that the current speech stream features plus this specific background noise feature equals the completion of the speech. This allows the model to better distinguish between background noise during the user's speech and background noise after the user's pause, thus achieving more accurate and faster speech endpoint detection in noisy environments.

[0063] As an optional implementation method, such as Figure 4 As shown, the prediction of the first end probability of the first concatenated vector and the second end probability of the second concatenated vector includes: S402, the first concatenation vector and the second concatenation vector are input in parallel into the bidirectional invertible Transformer encoder of the pre-trained prediction model to obtain their respective deep semantic feature matrices; S404 maps their respective deep semantic feature matrices into one-dimensional features through a fully connected layer; S406 compresses each one-dimensional feature to a range of 0-1 to obtain their respective termination probabilities.

[0064] This application provides a method for using a prediction model to predict the termination probability of the first concatenation vector and the second concatenation vector.

[0065] The model structure is a bidirectional reversible Transformer encoder architecture.

[0066] The encoder consists of M identical layers stacked together. For example, M can be 12, representing a 12-layer bidirectional reversible Transformer encoder architecture. In this application, each layer contains two main sublayers: a multi-head self-attention mechanism and a feed-forward network. To support bidirectional information flow, the model applies different attention masking strategies to the first and second concatenated vectors at the input layer, or uses a dual-stream processing mechanism to ensure that the model can simultaneously capture the forward temporal dependencies and backward global semantics of the sequence.

[0067] The system inputs the first concatenated vector (containing current speech stream features and historical prosodic memory features) and the second concatenated vector (containing current speech stream features, historical prosodic memory features and hypothetical silence features) in parallel into the bidirectional reversible Transformer encoder.

[0068] In each layer of the encoder, the input vector first passes through a multi-head self-attention mechanism. This mechanism captures long-distance dependencies between different time steps within the vector by calculating the dot product of the query, key, and value matrices.

[0069] Subsequently, the data stream undergoes a nonlinear transformation via a feedforward neural network. In this process, this embodiment introduces a reversible residual connection. The reversible residual block divides the input tensor into two parts, x_1 and x_2, and performs forward propagation using the following formula: y1 = x1 + Attention(x2)(1) y2 = x2 + FFN(y1)(2) Here, (y1, y2) is the output, and (x1, x2) can be recovered from (y1, y2) without loss.

[0070] After N layers of stacked processing, the model outputs its corresponding deep semantic feature matrix. This matrix contains highly abstracted temporal semantic information and prosodic context information.

[0071] The system obtains the aforementioned deep semantic feature matrix and inputs it into a fully connected layer. The fully connected layer maps the high-dimensional deep semantic feature matrix into a one-dimensional feature scalar. This one-dimensional feature represents the model's raw score for whether the current moment is the end of speech.

[0072] Finally, in order to convert the original score into a probabilistic value, the system uses the Sigmoid activation function to compress the one-dimensional feature to a range of 0 to 1, thus obtaining the final termination probability.

[0073] In this application, due to the modification of the bidirectional reversible Transformer encoder architecture, the model not only supports conventional left-to-right (forward) sequence processing to build an understanding of historical and current speech streams, but also simultaneously supports right-to-left (reverse) sequence processing paths. These two paths are not independent, but exchange information through reversible connections at specific layers. Reversible residual connections: A reversible residual network design replaces standard residual connections. The activation values ​​of each layer (or each group of layers) can be accurately reconstructed based on the output of the next layer, thus eliminating the need to store the intermediate activation values ​​of each layer during forward propagation for backward propagation; only the output of the last layer and the necessary parameters need to be stored. Through separate attention and feed-forward network (FFN) modules, in the reversible block, the multi-head self-attention module and the feed-forward network module are separated into two different computational paths and coupled through the aforementioned reversible functions (Equations 1 and 2).

[0074] Through the above steps, the system obtains the first termination probability corresponding to the first concatenation vector and the second termination probability corresponding to the second concatenation vector. These two probability values ​​quantify whether speech has ended from two dimensions: environmental acoustic matching degree and prosodic history consistency, providing a reliable basis for subsequent comprehensive judgment.

[0075] As an optional implementation method, such as Figure 5 As shown, after inputting the first concatenated vector and the second concatenated vector in parallel into the bidirectional invertible Transformer encoder of the pre-trained prediction model to obtain their respective deep semantic feature matrices, the process also includes: S502, if a new speech is generated after a silence segment is detected, the intermediate features of the Nth layer of the bidirectional reversible Transformer encoder are calculated in reverse according to the deep semantic feature matrix of the first splicing vector, where N is greater than or equal to 1 and less than the total number of layers of the bidirectional reversible Transformer encoder. S504, determine the speech stream sequence of the new speech by splicing the intermediate features with the latest prosodic memory vector; S506, determine the intermediate features, splice the latest prosodic memory vector, splice the speech stream sequence of the new speech, and splice the acoustic features of the hypothetical silence segment to form the hypothetical total vector of the new speech; S508: The new speech total vector and the new speech hypothetical total vector are processed through the N+1th to the last layer of the bidirectional invertible Transformer encoder to obtain their respective deep semantic feature matrices, which are then mapped to one dimension and compressed to the range of 0-1 to obtain their respective termination probabilities, in order to determine whether the new speech has ended.

[0076] In this application, if a silent segment that meets the preset triggering conditions is detected in the current voice stream and the judgment process of this application is performed, and the user generates new voice during the judgment process, such as when the user speaks and pauses for 300ms, the process of this application is performed, but the user starts speaking again and generates new voice.

[0077] At this point, the system triggers the incremental prediction mechanism. Instead of re-inputting the complete long sequence of the current speech stream, silence segments, and new speech into the model, the system directly calls the deep semantic feature matrix of the first concatenated vector that has already been calculated and cached in the previous process. Utilizing the reversible property of the bidirectional reversible Transformer encoder, starting from the output of layer N (where N is an integer greater than or equal to 1 and less than the total number of layers), the intermediate features of layer N are accurately reconstructed through the reverse propagation path.

[0078] As mentioned earlier, since the encoder layer uses reversible residual connections (see Equation 1), the system can infer the input without loss by using the output and Equation 1 above. This process enables the system to quickly obtain the context state (intermediate features) of the Nth layer by simply performing reverse reasoning without storing all intermediate activation values.

[0079] For new speech, the system extracts its acoustic features and obtains the latest prosodic memory vector (which has been updated based on previous interaction history). The system then concatenates the latest prosodic memory vector with intermediate features to obtain the new speech stream sequence, resulting in a new speech total vector. Conversely, it concatenates the latest prosodic memory vector with intermediate features, concatenates the latest prosodic memory vector with intermediate features, concatenates the new speech stream sequence with the acoustic features of hypothetical silence segments, and obtains a new speech hypothetical total vector. These two total vectors are processed through layers N+1 to the last layer of a bidirectional invertible Transformer encoder to obtain their respective deep semantic feature matrices. Subsequent mapping, compression, and probability calculation steps determine the termination probability of each feature, thus identifying whether the new speech has ended.

[0080] In this process, although new speech is generated, the model only needs to calculate the output of the remaining layers because the calculation results of the first N layers have been effectively utilized through reverse computation. This process greatly reduces the amount of computation and enables a rapid response to new speech.

[0081] As an optional implementation, the method further includes: during the real-time acquisition of the current speech stream, determining the silence probability of each frame in the current speech stream in real time; if the silence probability of multiple consecutive frames is detected to be greater than a preset probability threshold, and the total duration of multiple consecutive frames reaches a preset trigger threshold, then it is determined that there is a silence segment that meets the preset trigger conditions.

[0082] In this application, the probability of silence for each frame refers to the numerical value calculated for each frame of audio data after the system processes the real-time acquired audio signal into frames, representing the likelihood that the audio in that frame belongs to a silent state (i.e., a non-human voice state). This value is usually a floating-point number normalized to the interval [0, 1].

[0083] In this application, each frame of audio signal in the current speech stream is analyzed in real time to determine the probability that it belongs to a silent state, and a timing logic is used to determine whether a preset silence segment has been triggered. During the real-time acquisition of the current speech stream, the system performs frame-by-frame processing on the input audio signal. For example, the audio stream can be truncated according to a preset sampling rate (e.g., 16kHz), the length of each frame can be set to 20ms to 40ms (e.g., 25ms), and the frame shift (i.e., the time interval between the start points of two adjacent frames) can be set to 10ms. For each frame of audio data acquired, the system performs a real-time calculation of the silence probability. This step can be achieved through various technical means, such as using speech activity detection algorithms or trained lightweight acoustic models to analyze the acoustic features of the current frame (e.g., Mel frequency cepstral coefficients, zero-crossing rate, short-time energy, etc.) and output a value between 0 and 1 as the silence probability of that frame. This silence probability characterizes the likelihood that the current frame audio belongs to non-speech (i.e., silence or background noise).

[0084] Subsequently, the system will determine whether there is a silent segment that meets the preset trigger conditions based on the calculated silence probability and preset logical conditions. The specific determination logic is as follows: The system will monitor the silence probability of each frame in real time. If the silence probability of the current frame is detected to be greater than the preset probability threshold (for example, the threshold can be set to 0.8 or 0.9, and the specific value can be adjusted according to the noise level of the actual environment), then the frame is marked as a potential silent frame.

[0085] To eliminate false positives caused by transient noise or speech pauses, the system further introduces a duration-based determination mechanism. The system calculates the total duration of consecutive frames marked as potential silence. If the silence probability of multiple consecutive frames is greater than the preset probability threshold, and the total duration of these consecutive frames reaches a preset trigger threshold (for example, the trigger threshold can be set to 300ms, 500ms, 800ms, or 1s), then the system officially determines that a silence segment meeting the preset trigger conditions has appeared in the current speech stream.

[0086] The probability threshold plus continuous duration threshold determination mechanism in this application can effectively determine whether there are silent segments in the current speech stream, thus improving the detection accuracy of silent segments.

[0087] As an optional implementation, the method further includes: during the real-time acquisition of the current speech stream, using a speech activity detection model as a trigger to detect in parallel whether there are silent segments in the current speech stream, wherein the size of the speech activity detection model is smaller than a preset size.

[0088] In one embodiment of this application, a method for detecting silent segments is provided.

[0089] This method introduces a lightweight speech activity detection model as a trigger during the real-time acquisition of the current speech stream, and detects in parallel whether there are silent segments in the current speech stream that meet the conditions.

[0090] This speech activity detection model is a classification model specifically designed to distinguish between human voices and non-human voices (silence or background noise). To avoid impacting the real-time performance of the main chain (i.e., the inference process of the bidirectional reversible Transformer encoder), the size of this speech activity detection model is designed to be smaller than a preset size. Here, preset size refers to the parameter scale or computational complexity of a conventional high-precision speech recognition model. For example, this speech activity detection model could be a convolutional neural network with fewer layers (e.g., 2-3 layers) or a simplified recurrent neural network, with its parameter count controlled to be less than 1 / 10 or even 1 / 100 of the main chain model's parameter count. By limiting the model size, the system can run the trigger with extremely low latency on resource-constrained edge devices (such as smart speakers and in-vehicle terminals). In this application, the size of the speech activity detection model is less than 1MB.

[0091] For example, in this application, the current speech stream can be acquired by the data acquisition branch of the prediction model. This could involve acquiring audio at a fixed sampling rate (e.g., 16kHz), uniformly extracting 80-dimensional Log-Mel spectral features for each frame, and performing sliding normalization. Simultaneously, a lightweight Voice Activity Detection (VAD) model of less than 1MB is used as a trigger to work in parallel. When the duration of a detected continuous silence probability exceeding a threshold (e.g., >0.85) reaches a relatively short threshold (e.g., 300ms, which statistically covers most thought pauses), a potential endpoint event is immediately generated, notifying the prediction model to initiate the prediction process.

[0092] This trigger signal is used to activate subsequent computationally intensive steps, and after new speech is detected, to initiate the inverse processing flow of the bidirectional reversible Transformer encoder to obtain intermediate features and perform subsequent semantic prediction.

[0093] Taking a user interaction scenario with a smart speaker as an example, the user (with a habitual long pause) asks the smart speaker, "Remind me tomorrow afternoon..." (then begins thinking, lasting 450ms). The lightweight VAD triggers an event when silence lasts for 300ms. The speech stream features of "Remind me tomorrow afternoon" are extracted, and the user's prosodic memory vector is read (historical records show a relatively high average pause duration).

[0094] A dual-input configuration is established, with the hypothetical silent segment using background noise from a home environment.

[0095] Bidirectional Transformer inference yielded the following probabilities: the first termination probability is 0.48, the second termination probability is 0.43, and the difference is 0.05. Since 0.05 ≤ 0.11, the system momentarily determined this to be a pause in thought, maintained listening mode, and the speaker did not respond. The user continued, "...meeting at three o'clock." After pausing again, the VAD was triggered again after 300ms.

[0096] The current audio stream contains the complete sentence: "Remind me of the meeting at 3 PM tomorrow afternoon." The dual-path inference yielded a first termination probability of 0.52 and a second termination probability of 0.90, with a difference of 0.38 ≥ 0.37. Therefore, it was immediately determined to be a true termination, and a response was initiated approximately 420ms later: "Okay, a meeting reminder for tomorrow at 3 PM has been set for you." Upon successful response, the prosodic features calculated during this interaction (including a long pause for thought) will be updated to the user's personalized prosodic memory vector, making the user more tolerant of similar pauses in the future.

[0097] Another aspect of this application provides a voice endpoint recognition device, such as... Figure 6 As shown, it includes: The acquisition module 602 is used to acquire the user's historical voice stream if a silent segment that meets a preset trigger condition is detected in the current voice stream during the real-time acquisition of the user's current voice stream. The determination module 604 is used to determine the prosodic memory vectors of the current speech stream and the historical speech stream; The construction module 606 is used to construct a speech stream sequence based on the speech stream features of the current speech stream, and concatenate the prosodic memory vector at the front end to obtain the first concatenation vector. Furthermore, it concatenates the acoustic features of the hypothetical silence segment after the speech stream features to construct the hypothetical silence sequence, and concatenates the prosodic memory vector at the front end to obtain the second concatenation vector. The prediction module 608 is used to predict the first end probability of the first concatenated vector and the second end probability of the second concatenated vector, respectively. The judgment module 610 is used to determine whether the current voice stream has ended based on the difference between the first end probability and the second end probability.

[0098] This application can be applied in various scenarios. For example, in intelligent customer service and call center scenarios, customer service robots often "intercept" (respond before the user has finished speaking), resulting in a very poor user experience. The solution adopted in this application concatenates a prosodic memory vector before the user's current speech stream sequence to obtain a first concatenated vector. It also constructs an imaginary silence sequence by combining the speech stream features of the current speech stream with the acoustic features of the imaginary silence segment, and concatenates the prosodic memory vector at the front end to obtain a second concatenated vector. It predicts the end probability of the two vectors and determines whether the current speech stream has ended based on the difference between the two end probabilities. This accurately identifies the endpoint of the speech stream, avoids intelligent customer service robots "intercepting" the user, and improves the user experience.

[0099] In this technical solution, during the real-time acquisition of the user's current speech stream, if a silent segment meeting a preset trigger condition is detected in the current speech stream, the user's historical speech stream is acquired; the prosodic memory vectors of the current speech stream and the historical speech stream are determined; a speech stream sequence is constructed based on the speech stream features of the current speech stream, and the prosodic memory vector is concatenated at the front end to obtain a first concatenated vector; acoustic features of a hypothetical silent segment are concatenated after the speech stream features to construct a hypothetical silent sequence, and the prosodic memory vector is concatenated at the front end to obtain a second concatenated vector; the first end probability of the first concatenated vector and the second end probability of the second concatenated vector are predicted respectively; based on the difference between the first end probability and the second end probability, the manner in which the current speech stream ends is determined. Thus, during the process of detecting the user's speech endpoint, by constructing a hypothetical silent sequence and predicting the end probability together with the real speech stream sequence, the difference between the two is further determined. Moreover, the prosodic features of the user's speech are enhanced by the prosodic memory vector. Finally, if the difference between the two is greater, it indicates that the user is more likely to stop speaking, thus achieving the effect of accurately identifying speech endpoints.

[0100] Other examples of this application can be found in the examples above, and will not be repeated here.

[0101] like Figure 7 As shown, this application provides an electronic device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0102] Memory 113 is used to store computer programs.

[0103] In one embodiment of this application, the processor 111, when executing a program stored in the memory 113, implements the method provided in any of the foregoing method embodiments.

[0104] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in any of the foregoing method embodiments.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0107] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0108] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for recognizing voice endpoints, characterized in that, include: During the real-time acquisition of the user's current voice stream, if a silent segment that meets a preset trigger condition is detected in the current voice stream, then the user's historical voice stream is acquired. Determine the prosodic memory vectors of the current speech stream and the historical speech stream; A speech stream sequence is constructed based on the speech stream features of the current speech stream, and the prosodic memory vector is concatenated at the beginning to obtain a first concatenated vector. The acoustic features of the hypothetical silence segment are concatenated after the speech stream features to construct a hypothetical silence sequence, and the prosodic memory vector is concatenated at the beginning to obtain a second concatenated vector. Predict the first end probability of the first concatenated vector and the second end probability of the second concatenated vector, respectively; The difference between the first end probability and the second end probability is used to determine whether the current voice stream has ended.

2. The method according to claim 1, characterized in that, Determining the prosodic memory vectors of the current speech stream and the historical speech stream includes: From the current speech stream, multidimensional prosodic features are extracted, and at least one statistical value among the mean, standard deviation, maximum value, and minimum value is calculated for each prosodic feature. The multidimensional prosodic features are concatenated with the corresponding statistical values ​​to obtain the prosodic vector of the current speech stream. The prosodic memory vector is obtained by weighted summing of the historical prosodic memory vector of the previous interaction round of the historical speech stream and the prosodic vector of the current speech stream.

3. The method according to claim 1, characterized in that, The acoustic features of splicing an imaginary silence segment after the speech stream features include: Based on the user's interaction scenario, a hypothetical silence segment matching the interaction scenario is constructed, wherein the sound included in the hypothetical silence segment is consistent with the sound of the interaction scenario; The extracted acoustic features of the hypothetical silence segment are concatenated after the speech stream features.

4. The method according to claim 1, characterized in that, Predicting the first end probability of the first concatenated vector and the second end probability of the second concatenated vector respectively includes: The first concatenated vector and the second concatenated vector are input in parallel into the bidirectional invertible Transformer encoder of the pre-trained prediction model to obtain their respective deep semantic feature matrices. Each of the deep semantic feature matrices is mapped to one-dimensional features through a fully connected layer; Each one-dimensional feature is compressed to the range of 0-1 to obtain its respective termination probability.

5. The method according to claim 4, characterized in that, After inputting the first concatenated vector and the second concatenated vector in parallel into the bidirectional invertible Transformer encoder of the pre-trained prediction model to obtain their respective deep semantic feature matrices, the process further includes: If new speech is generated after the silence segment is detected, the intermediate features of the Nth layer of the bidirectional reversible Transformer encoder are calculated in reverse according to the deep semantic feature matrix of the first splicing vector, wherein N is greater than or equal to 1 and less than the total number of layers of the bidirectional reversible Transformer encoder. The speech stream sequence obtained by concatenating the intermediate features with the latest prosodic memory vector and then concatenating the new speech is determined as the new speech total vector; The intermediate features are combined with the latest prosodic memory vector to form the speech stream sequence of the new speech, and the acoustic features of the hypothetical silence segment are combined to form the hypothetical total vector of the new speech. The new speech vector and the new hypothetical speech vector are processed through the N+1th to the last layer of the bidirectional invertible Transformer encoder to obtain their respective deep semantic feature matrices, which are then mapped to one dimension and compressed to the range of 0-1 to obtain their respective termination probabilities, thereby determining whether the new speech has ended.

6. The method according to claim 1, characterized in that, The method further includes: During the real-time acquisition of the current speech stream, the probability of silence in each frame of the current speech stream is determined in real time. If the probability of silence in multiple consecutive frames is greater than a preset probability threshold, and the total duration of the multiple consecutive frames reaches a preset trigger threshold, then it is determined that there is a silence segment that meets the preset trigger conditions.

7. The method according to claim 1, characterized in that, The method further includes: During the real-time acquisition of the current speech stream, a speech activity detection model is used as a trigger to detect in parallel whether the silence segment exists in the current speech stream. The size of the speech activity detection model is smaller than a preset size.

8. A voice endpoint recognition device, characterized in that, include: The acquisition module is used to acquire the user's historical voice stream if a silent segment that meets a preset trigger condition is detected in the current voice stream during the real-time acquisition of the user's current voice stream. The determination module is used to determine the prosodic memory vectors of the current speech stream and the historical speech stream; The construction module is used to construct a speech stream sequence based on the speech stream features of the current speech stream, and concatenate the prosodic memory vector at the front end to obtain a first concatenation vector. It also concatenates the acoustic features of the hypothetical silence segment after the speech stream features to construct a hypothetical silence sequence, and concatenates the prosodic memory vector at the front end to obtain a second concatenation vector. The prediction module is used to predict the first end probability of the first concatenated vector and the second end probability of the second concatenated vector, respectively. The judgment module is used to determine whether the current voice stream has ended based on the difference between the first end probability and the second end probability.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as claimed in any one of claims 1 to 7.

Images