Interactive digital human voice interrupt defense method for echo cancellation

Abstract

The application discloses an interactive digital human voice interruption defense method for echo cancellation, and comprises the following steps: continuously collecting audio by a microphone, entering VAD acoustic energy gating judgment; judging whether the voice energy of the collected audio exceeds an interruption threshold; performing text fuzzy matching on the collected candidate audio, and performing cross-correlation calculation on the candidate audio and recent self-played reference audio; marking the candidate audio as suspected echo and increasing the text fuzzy matching parameter for the suspected echo; and sending an interruption signal to an interruption processing module. The application does not need to install a special hardware echo cancellation module on the device to eliminate echo, significantly improves the accuracy of echo rejection, balances the two mutually restrictive targets of interruption sensitivity and misjudgment suppression, and can realize zero false interruption in a long-time, large-volume self-echo broadcast scene.

Description

Technical Field

[0001] This invention relates to an interactive digital human speech interruption defense method for echo cancellation. Background Technology

[0002] Interactive digital human systems (such as large-screen digital human assistants and digital human guides in exhibition halls) typically need to support "interruption" capabilities: when a user starts speaking while the digital human is broadcasting / responding to a speech, the system should be able to quickly stop the current broadcast and switch to a new round of speech recognition and response generation to obtain a natural interactive experience similar to real-person dialogue.

[0003] To reduce costs and simplify hardware design, these devices often employ a speaker-based approach combined with near-field or array microphones for sound pickup, rather than featuring a dedicated echo reference channel and professional acoustic echo cancellation (AEC) hardware / firmware modules. Under these conditions of "no real echo reference signal at the software level," the audio signal picked up by the microphone inevitably contains the digital human's own voice played through the speaker (i.e., acoustic echo), presenting the following main technical problems:

[0004] Issue 1 (Self-Interruption): After the digital human's own voice is picked up by the microphone through room reverberation, desktop reflection, etc., the voice activity detection (VAD) module or speech recognition (ASR) module may misjudge it as "new user voice", thus erroneously triggering the interruption logic, causing the digital human's response to be interrupted unnecessarily, resulting in the abnormal phenomenon of "interrupting itself".

[0005] Problem 2 (True interruptions are missed): To avoid Problem 1, simply increasing the volume / confidence threshold for interruption detection, or directly blocking microphone input during the digital human's broadcast, will cause the user's true interruption intentions to be unresponsive in a timely manner, resulting in the opposite problem of "shouting but not being able to interrupt".

[0006] Issue 3 (Incomplete interruption in transit): When an interruption signal has been generated but there is no "playing" audio or video stream at that time (e.g., the large model / speech synthesis is still generating the next sentence that has not yet started playing), if the interruption logic only cancels the single stream that is "currently playing", it will miss the subsequent speech that is still in the generation chain (in-flight), resulting in the problem of "the interruption has been completed but the sound continues to play after a while";

[0007] Question 4 (Mistaken Initial Very Short Phrases): Existing echo text filtering schemes based on "comparing similarity with the most recent digital human's own text" may easily discard very short, genuine follow-up questions uttered by the user within a very short time window as echoes if the new round of speech recognition results are filtered indiscriminately within the time window.

[0008] In existing technologies, echo cancellation solutions all struggle to simultaneously achieve the two mutually restrictive goals of "accurately rejecting echoes" and "not missing any real interruptions" without a hardware echo reference signal, resulting in insufficient robustness. Summary of the Invention

[0009] The technical problem this invention aims to solve is to overcome the shortcomings of existing echo cancellation solutions, which struggle to simultaneously achieve the goals of accurately rejecting echoes and not missing any real interruptions, and lack robustness. This invention provides an interactive digital human voice interruption defense method for echo cancellation that eliminates the need for a dedicated hardware echo cancellation module, significantly improves the accuracy of echo rejection, balances the mutually constraining goals of interruption sensitivity and false alarm suppression, and achieves zero-error interruption in scenarios with long-duration, high-volume self-echoing broadcasts.

[0010] The present invention solves the above-mentioned technical problems through the following technical solution:

[0011] An interactive digital human speech interruption defense method for echo cancellation, characterized in that the interactive digital human speech interruption defense method includes:

[0012] S1. The microphone continuously collects audio and enters the VAD acoustic energy gating judgment;

[0013] S2. For audio that exceeds the initial threshold, determine whether the speech energy of the audio exceeds the interruption threshold. If yes, proceed to step S3; otherwise, proceed to step S4. The value of the interruption threshold is higher than the initial threshold.

[0014] S3. Perform fuzzy text matching on the collected candidate audio to determine whether the candidate audio is an echo. If it is, discard the candidate audio and execute step S1 again. Otherwise, execute step S7.

[0015] S4. Perform cross-correlation calculation between the candidate audio and the recently played reference audio, and determine whether the correlation coefficient is greater than the preset value. If yes, proceed to step S5; otherwise, proceed to step S6.

[0016] S5. Mark the candidate audio as a suspected echo and increase the text fuzzy matching parameters for the suspected echo, then proceed to step S3.

[0017] S6. Perform step S3 according to the standard parameters for text fuzzy matching;

[0018] S7. Send an interrupt signal to the interrupt handling module.

[0019] Preferably, the interactive digital human voice interruption defense method includes:

[0020] During the digital human's voice broadcast, it continuously retains its own audio waveform that was recently sent to the speaker for playback.

[0021] When the microphone picks up a candidate audio, it performs a cross-correlation calculation between the candidate audio and its own audio waveform in the time domain, and determines whether the correlation coefficient is greater than a preset value.

[0022] Preferably, step S3 includes:

[0023] Utilize speech recognition to output candidate text from candidate audio;

[0024] The candidate text is compared with its nearest text corresponding to its own audio waveform using a multi-algorithm combination fuzzy comparison. The multi-algorithm includes at least two of the following: the longest common subsequence length ratio, pinyin similarity comparison, and the calculation of the longest continuous matching segment length.

[0025] The matching results of multi-algorithm combined fuzzy comparison are judged according to preset weights or rules to determine whether the candidate audio is an echo.

[0026] Preferably, for the initial candidate audio within a preset time window after the digital human begins broadcasting, candidate text of the initial candidate audio is output using speech recognition.

[0027] The longest common subsequence length ratio is calculated by taking the length of the longest common subsequence between the candidate text and its nearest text, and normalizing it with the length of the candidate text itself to obtain the similarity ratio;

[0028] Pinyin similarity comparison involves converting candidate texts into pinyin sequences and then comparing them with the nearest texts to the candidate text.

[0029] The longest consecutive matching segment length is calculated by calculating the length of the longest consecutive character matching segment between the candidate text and its nearest text, and identifying echo segments that are repeated continuously but whose overall order is scrambled by the recognition engine.

[0030] Preferably, step S7 includes:

[0031] Send an interrupt signal to the centralized interrupt processing module;

[0032] The centralized interruption handling module executes the cleaning process;

[0033] Entering a new round of speech recognition and response generation process.

[0034] Preferably, step S7 includes:

[0035] Monitor interruption signals from multiple signal sources;

[0036] Send an interrupt signal to the centralized interrupt processing module;

[0037] The centralized interruption processing module executes a cleanup process, which includes: if the interruption signal carries a target data stream identifier, then tracing upwards along the dependency chain of the target data stream and canceling all associated generated streams level by level; and performing a unified scan and cancellation on all currently registered digital human response text streams, digital human synthesized speech streams, and client playback streams according to their types.

[0038] Preferably, the centralized interruption processing module, in performing the cleaning process, further includes:

[0039] Send a cancellation command to the speech synthesis module and the digital human video rendering module to discard the intermediate audio and video data currently being synthesized and / or inferred in the pipeline;

[0040] Perform a clear operation on the multi-level buffer queue of the digital human video rendering module, and add a global identifier for video rendering;

[0041] For each processing queue in the digital human video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

[0042] Preferably, the interactive digital human voice interruption defense method is obtained through a generation method, the generation method comprising:

[0043] The feasibility of four single schemes—pure energy threshold, pure cross-correlation, NLMS adaptive filtering AEC, and client-side native AEC—was verified respectively.

[0044] The feasibility study was used to obtain the layered and progressive architecture of the interactive digital human voice interruption defense method.

[0045] Preferably, the generation method includes:

[0046] The interruption threshold was optimized in multiple rounds to obtain verification results showing that the optimal value of the interruption threshold was strongly correlated with the actual acoustic deployment method of the device.

[0047] The range of values ​​for the interruption threshold and the runtime adjustable attributes are obtained using the verification results.

[0048] Preferably, the interactive digital human voice interruption defense method further includes:

[0049] A test client is constructed programmatically, and the client is used to simulate audio and video communication connections with an interactive digital human system.

[0050] Two test scenarios were established: one was to overlay the device's own echo audio during the digital human's own broadcast, and the other was to overlay real user interruption audio during the digital human's own broadcast.

[0051] The execution results of the interactive digital human system in multi-round test scenarios are determined by monitoring the changes in the energy envelope of the downlink audio and video streams.

[0052] Statistical analysis was conducted on the false interruption rate, missed interruption rate, and interruption delay for the test client to generate a reproducible verification report and confirm the final deployment version.

[0053] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0054] The positive and progressive effects of this invention are as follows:

[0055] Without the need to install a dedicated hardware echo cancellation module or provide a hardware echo reference channel, the device can effectively identify and reject its own external echo at the software level, reducing hardware costs and the complexity of peripheral circuits.

[0056] The combination of acoustic cross-correlation determination and text multi-algorithm fuzzy matching determination takes into account both the two types of echo misjudgment cases: "similar volume features but unrecognizable content" and "significantly different volume features but identical recognized content". This significantly improves the accuracy of echo rejection compared to single-level determination methods.

[0057] The graded energy threshold design ensures echo suppression while retaining a fast response path for high-confidence real interruptions, thus balancing the two mutually restrictive objectives of interruption sensitivity and misjudgment suppression.

[0058] The centralized interrupt signal clearing architecture decouples "decision-making" from "execution," supports access from multiple interrupt signal sources, and ensures that any interrupt triggered by any source can completely terminate all related data streams, including content generated in transit, thus avoiding the user experience problem of "incomplete interruption."

[0059] Verified through multiple rounds of automated end-to-end testing (including scenarios with multiple consecutive rapid interruptions) on a real interactive digital human system, the method described in this invention achieves zero-error interruption in scenarios with long-duration, high-volume self-echo playback while maintaining real-time interruption response (interruption delay can be as low as sub-second). Attached Figure Description

[0060] Figure 1 This is a flowchart of the interactive digital human voice interruption defense method according to Embodiment 1 of the present invention. Detailed Implementation

[0061] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0062] Example 1

[0063] This embodiment provides an interactive digital human voice interruption defense system for echo cancellation. The interactive digital human voice interruption defense system can be implemented through a server in conjunction with input devices and input / output devices.

[0064] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0065] See Figure 1 This embodiment provides a method for echo cancellation in preventing voice interruption in interactive digital humans, utilizing the aforementioned interactive digital human voice interruption defense system. The method includes:

[0066] Step S1: The microphone continuously collects audio and enters the VAD acoustic energy gating judgment.

[0067] Step S2: For audio that exceeds the initial threshold, determine whether the speech energy of the collected audio exceeds the interruption threshold. If yes, proceed to step S3; otherwise, proceed to step S4.

[0068] Step S3: Perform text fuzzy matching on the collected candidate audio to determine whether the candidate audio is an echo. If it is, discard the candidate audio and execute step S1 again; otherwise, execute step S7.

[0069] Step S4: Perform cross-correlation calculation between the candidate audio and the recently played reference audio, and determine whether the correlation coefficient is greater than the preset value. If yes, proceed to step S5; otherwise, proceed to step S6.

[0070] Step S5: Mark the candidate audio as a suspected echo and increase the text fuzzy matching parameters for the suspected echo, then proceed to step S3;

[0071] Step S6: Execute step S3 according to the regular parameters for text fuzzy matching;

[0072] Step S7: Send an interrupt signal to the interruption processing module.

[0073] Specifically, the interactive digital human voice interruption defense method further includes:

[0074] During the digital human's voice broadcast, it continuously retains its own audio waveform that was recently sent to the speaker for playback.

[0075] When the microphone picks up a candidate audio, it performs a cross-correlation calculation between the candidate audio and its own audio waveform in the time domain, and determines whether the correlation coefficient is greater than a preset value.

[0076] Specifically, step S3 includes:

[0077] Utilize speech recognition to output candidate text from candidate audio;

[0078] The candidate text is compared with its nearest text corresponding to its own audio waveform using a multi-algorithm combination fuzzy comparison. The multi-algorithm includes at least two of the following: the longest common subsequence length ratio, pinyin similarity comparison, and the calculation of the longest continuous matching segment length.

[0079] The matching results of multi-algorithm combined fuzzy comparison are judged according to preset weights or rules to determine whether the candidate audio is an echo.

[0080] For the initial candidate audio within a preset time window after the digital human begins broadcasting, the candidate text of the initial candidate audio is output using speech recognition.

[0081] The longest common subsequence length ratio is calculated by taking the length of the longest common subsequence between the candidate text and its nearest text, and normalizing it with the length of the candidate text itself to obtain the similarity ratio;

[0082] Pinyin similarity comparison involves converting candidate texts into pinyin sequences and then comparing them with the nearest texts to the candidate text.

[0083] The longest consecutive matching segment length is calculated by calculating the length of the longest consecutive character matching segment between the candidate text and its nearest text, and identifying echo segments that are repeated continuously but whose overall order is scrambled by the recognition engine.

[0084] Step S7 includes:

[0085] Send an interrupt signal to the centralized interrupt processing module;

[0086] The centralized interruption handling module executes the cleaning process;

[0087] Entering a new round of speech recognition and response generation process.

[0088] The interruption defense method provided in this embodiment includes a layered and progressive four-stage judgment / processing mechanism, which are as follows:

[0089] Acoustic energy gating layer;

[0090] Acoustic cross-correlation self-echo recognition layer;

[0091] Text multi-algorithm fuzzy matching layer;

[0092] Disrupt the centralized sweep-cancel execution layer.

[0093] The above four mechanisms work together to solve the technical problems in the background technology.

[0094] Regarding the acoustic energy gating layer, the system performs voice activity detection (VAD) on the audio signals collected by the microphone, setting two thresholds based on the signal energy (dB): the first is the "basic voice start threshold," used to determine whether voice activity exists; the second is the "fast interruption threshold" (denoted as barge_fast_db). When the detected voice energy is significantly higher than this threshold, and the voice probability output by the corresponding voice activity detection model exceeds the set threshold and remains there for several consecutive frames, while a certain grace period is added, it is determined as a "high confidence" candidate for interruption and enters the text multi-algorithm fuzzy matching layer for judgment. Voice activity with energy between the basic threshold and the fast threshold enters the more stringent text layer judgment and does not directly trigger interruption. This hierarchical threshold design allows for faster interruption response when the sound is significantly louder than the device's own external speaker (such as when the user raises their volume or shouts closer), while avoiding direct passage of signals with lower energy that are more likely to be external speaker echoes. For stricter text-level judgment, a multi-algorithm fuzzy matching process can be used to adjust parameters between candidate text and its own text. This includes matching consecutive characters that allow for misalignment, matching consecutive characters at the pinyin level, and matching the LCS ratio of pinyin sequences. If any matching condition is met, the text is judged as an echo and discarded. Any condition that is met for an echo to be discarded is: a matching ratio reaching a dynamic threshold, a consecutive matching length of at least N, and an LCS ratio. Candidates with fewer than 3 valid characters will not be used for interruption, even if they pass text filtering.

[0095] Regarding the acoustic cross-correlation self-echo recognition layer, during the digital human's speech playback, the system continuously retains its own audio waveforms that have been sent to the speaker in the recent period as an echo reference signal (without needing to obtain a hardware echo reference channel from the sound card / amplifier, but directly reusing the digital human's own synthesized speech data that the system already knows and is about to / is currently sending to play). When the microphone picks up a candidate speech segment, it performs cross-correlation (XCORR) calculation on the segment and the retained self-reference audio in the time domain to obtain the peak correlation coefficient. If the peak correlation coefficient exceeds a preset threshold, the candidate speech segment is determined to be highly correlated with the device's current playback content, marked as a suspected acoustic echo, and transferred to a more rigorous text layer review, rather than being directly determined as real user speech.

[0096] Regarding the text multi-algorithm fuzzy matching layer, the candidate text output by speech recognition (ASR) is compared with the most recently played avatar text from the digital human, using a multi-algorithm combined fuzzy comparison. Specifically, this includes:

[0097] Longest Common Subsequence (LCS) Length Ratio: Calculate the length of the longest common subsequence between a candidate text and its nearest text, and normalize it by the length of the candidate text itself to obtain the similarity ratio;

[0098] Pinyin similarity comparison: The candidate text is converted into a pinyin sequence and compared with its nearest text to address the recognition error caused by homophones / near-homophones in speech recognition and improve the recall ability in the case of "echo being recognized as homophones";

[0099] Longest consecutive matching segment (run) length: Calculates the length of the longest consecutive character matching segment between the candidate text and its nearest text. It is used to identify echo segments that are repeated continuously but whose overall order is scrambled by the recognition engine.

[0100] The above three matching results are determined by a combination of preset weights or rules. In this embodiment, when the matching degree of any algorithm exceeds the corresponding threshold (or meets the preset short text / long text differentiation judgment rules), the candidate text is determined to be a misidentification of the digital human's own voice echo and is discarded without triggering an interruption; otherwise, it is determined to be real user voice and enters the interruption execution layer.

[0101] In this embodiment, a special processing rule for "extremely short candidate text" is further set: when the length of the candidate text is less than a preset character threshold (e.g., 3 characters), even if it does not hit the aforementioned echo similarity judgment, it will not be filtered and discarded simply because it is "within the time window after the digital human has just started broadcasting", thereby solving the problem of mistakenly swallowing the initial extremely short sentence as described in the fourth background technical problem; this time window filtering rule has been proven to mistakenly affect real user short follow-up questions, so the present invention removes it in the specific implementation and only retains the judgment based on length and similarity.

[0102] Regarding the centralized sweep-cancel execution layer for interrupt signals, this embodiment decouples the decision logic of "when to interrupt" from the specific operational logic of "how to execute the interrupt": multiple signal sources in the system (including but not limited to real user voice detected by the speech recognition module, manual interrupt button on the client interface, semantic round detection module, etc.) send a standardized "interrupt" signal to a centralized interrupt processing module (InterruptHandler) when they determine that an interrupt should be made, instead of directly operating the playback / generation status of each functional module.

[0103] Upon receiving an interruption signal, the centralized interruption processing module executes the following cleanup process:

[0104] If the signal carries a clear target data stream identifier (such as the currently playing client playback stream), then trace back up along the dependency chain (cancel_stream_chain) of that data stream and cancel all associated generation streams (including text generation streams, speech synthesis audio streams, etc.) level by level.

[0105] Regardless of whether the aforementioned target stream exists, all three types of data streams currently registered in the system—"digital human response text stream," "digital human synthesized speech stream," and "client playback stream"—are uniformly scanned and canceled by type. This ensures that even if some response content is still in the "in-flight" stage of large model generation or speech synthesis and has not yet formed a playable audio or video stream, it will be terminated as well, thereby solving the problem of incomplete interruption in transit mentioned in Background Technical Problem 3.

[0106] The cancellation operation is simultaneously sent to the speech synthesis module and the digital human video rendering module, causing them to discard the intermediate audio and video data currently stuck in the synthesis / inference pipeline, thus preventing the playback of old content after interruption.

[0107] Regarding the centralized sweep-cancel execution layer for interrupting signals, specifically:

[0108] Step S7 includes:

[0109] Monitor interruption signals from multiple signal sources;

[0110] Send an interrupt signal to the centralized interrupt processing module;

[0111] The centralized interruption processing module executes a cleanup process, which includes: if the interruption signal carries a target data stream identifier, then tracing upwards along the dependency chain of the target data stream and canceling all associated generated streams level by level; and performing a unified scan and cancellation on all currently registered digital human response text streams, digital human synthesized speech streams, and client playback streams according to their types.

[0112] The centralized interruption processing module executes the cleaning process, which also includes:

[0113] Send a cancellation command to the speech synthesis module and the digital human video rendering module to discard the intermediate audio and video data currently being synthesized and / or inferred in the pipeline;

[0114] Perform a clear operation on the multi-level buffer queue of the digital human video rendering module, and add a global identifier for video rendering;

[0115] For each processing queue in the digital human video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

[0116] In a specific embodiment, it is deployed on an interactive digital human server system built on real-time audio and video communication (RTC) technology. The system includes a voice activity detection module, a voice recognition module, a large language model response generation module, a voice synthesis module, a digital human video rendering module, and a centralized interruption processing module. The modules transmit standardized signals such as "stream start / stream end / stream cancellation / interruption" through a unified signal bus.

[0117] The speech activity detection module sets a basic speech start threshold (e.g., -40dB) and a rapid interruption threshold (e.g., -17dB), and sets parameters such as the speech probability threshold (e.g., 0.35), duration (e.g., 3 frames), and grace period (e.g., 1.5 seconds). After the speech segment is detected, the speech recognition module uses an independent thread to asynchronously call the speech recognition engine to avoid blocking the real-time reception and processing of the digital human's own response text, thereby ensuring that the "recent self-text" buffer on which acoustic cross-correlation and text fuzzy matching determination depend can be updated in a timely and accurate manner. The text fuzzy matching determination is weighted by a comprehensive LCS ratio threshold (e.g., 0.6), pinyin similarity, and the longest continuous matching run length threshold (e.g., 3 characters).

[0118] The centralized interruption handling module is implemented as an independent processor (Handler). It only listens for interruption type signals. Upon receiving a signal, it sequentially performs the cancellation of the target stream dependency chain and the full sweep cancellation of the three types of data streams (digital human response text stream, digital human synthesized speech stream, and client playback stream). The cancellation results are recorded in the session history for subsequent statistics and problem tracing.

[0119] Furthermore, this embodiment provides the specific process and quantitative data of comparing and verifying multiple technical routes, iteratively optimizing parameters, and performing a complete regression localization in order to locate the technical problem of "false triggering interruption of self-external echo" during actual R&D and production deployment. This is used to prove that the hierarchical progressive scheme finally determined by the present invention is not the only intuitive choice, but the optimal solution determined after systematically exhaustively exploring and disproving multiple common solutions in existing technologies with measured data.

[0120] The interactive digital human voice interruption defense method is obtained through a generation method, which includes:

[0121] The feasibility of four single schemes—pure energy threshold, pure cross-correlation, NLMS adaptive filtering AEC, and client-side native AEC—was verified respectively.

[0122] The feasibility study was used to obtain the layered and progressive architecture of the interactive digital human voice interruption defense method.

[0123] The generation method includes:

[0124] The interruption threshold was optimized in multiple rounds to obtain verification results showing that the optimal value of the interruption threshold was strongly correlated with the actual acoustic deployment method of the device.

[0125] The range of values ​​for the interruption threshold and the runtime adjustable attributes are obtained using the verification results.

[0126] The interactive digital human voice interruption defense method also includes:

[0127] A test client is constructed programmatically, and the client is used to simulate audio and video communication connections with an interactive digital human system.

[0128] Two test scenarios were established: one was to overlay the device's own echo audio during the digital human's own broadcast, and the other was to overlay real user interruption audio during the digital human's own broadcast.

[0129] The execution results of the interactive digital human system in multi-round test scenarios are determined by monitoring the changes in the energy envelope of the downlink audio and video streams.

[0130] Statistical analysis was conducted on the false interruption rate, missed interruption rate, and interruption delay for the test client to generate a reproducible verification report and confirm the final deployment version.

[0131] During the R&D process, the following four common single-dimensional judgment schemes in existing technologies were implemented and tested in sequence, with all verifications based on data collected in real device external playback scenarios. The results are as follows:

[0132] Option 1 (Pure Volume Energy Threshold): Actual measurements of the energy distribution range of real user voice and device echo revealed a significant overlap between the two – the measured energy range of real user voice is approximately -50 to -10 dB, while the measured energy range of device echo is approximately -44 to -7 dB. The two completely overlap within the -44 to -10 dB range, and there is no single energy threshold that can physically separate them. This verifies that the pure volume solution has inherent limitations in external speaker scenarios.

[0133] Scheme 2 (Acoustic Energy Envelope Cross-Correlation): When used alone as a criterion, the peak cross-correlation value of the echo candidate is approximately 0.49 to 0.62, while the peak cross-correlation value of the real user speech is approximately 0.17 to 0.26. The distribution ranges of the two types of samples also partially overlap (especially in the range of 0.2 to 0.6), indicating that relying solely on this indicator cannot achieve reliable differentiation. In the end, this invention retains it as an auxiliary signal in the multi-layer judgment, rather than using it as the sole criterion.

[0134] Option 3 (Adaptive Filtering Acoustic Echo Cancellation, NLMS-AEC): This option attempts to introduce adaptive filtering echo cancellation based on the normalized least mean square algorithm. The audio played by the device itself is used as a reference signal to adaptively align and cancel the microphone signal. The measured results show that the echo return loss gain (ERLE) is entirely negative (approximately -0.5 to -23.5 dB), the cross-correlation coefficient between the reference signal and the microphone signal is extremely low (approximately 0.02 to 0.36), and the delay estimation results fluctuate significantly between adjacent calculation windows (no stable convergence value). This indicates that under conditions where the propagation delay of the "speaker output - spatial propagation - microphone pickup" link is uncertain and jitter exists, this type of adaptive filtering scheme cannot converge. It not only fails to eliminate echo but may also introduce additional signal distortion, verifying that this technical approach is not feasible in the "no dedicated hardware echo reference channel" scenario addressed by this invention.

[0135] Option 4 (Browser / Client Native Acoustic Echo Cancellation Interface): Verification confirms that this type of solution is only effective in scenarios where "echo reference signals can be directly obtained from the local playback link" (such as headphone playback or local playback on the same terminal). However, in the scenario targeted by this invention, where "reference signals such as external speaker playback or external speakers are not available," the client cannot take effect due to the lack of usable reference signals. Therefore, it cannot be used as an independent solution for the deployment scenario described in this invention.

[0136] After systematically verifying that the aforementioned four single-dimensional schemes all have limitations in principle or scenario, a hierarchical progressive judgment architecture (energy-gated initial screening + acoustic cross-correlation auxiliary judgment + multi-algorithm text fuzzy matching final judgment) was determined, so that each layer can undertake the more reliable judgment responsibility in its respective dimension, making up for the limitations of single-dimensional judgment methods.

[0137] The parameter "barge_fast_db" (also known as the interruption threshold or interruption threshold during broadcasting) underwent multiple rounds of optimization during actual R&D and multiple field deployments. Test data shows that the optimal value of this parameter is strongly correlated with the actual acoustic deployment method of the device (whether it is externally played and the power level of the external speaker). There is no universally applicable optimal value independent of the deployment scenario. The optimization process is recorded below:

[0138] In the case of external speaker, the measured peak echo of the device itself can reach about -8dB (even close to -7dB in some scenarios), which is significantly higher than the usual expectation; when the threshold is adjusted to about -15 to -17dB, it can block most echoes while retaining the real interruption capability.

[0139] In headphone / near-field microphone scenarios (without significant external speaker echo), the measured real user voice energy can be as low as approximately -48 to -40 dB. If the -15 to -17 dB threshold used in external speaker scenarios is applied, the real user voice will not be able to trigger interruption ("it is very difficult to interrupt"). Therefore, the threshold in this scenario needs to be adjusted to approximately -38 to -42 dB to ensure normal interruption sensitivity.

[0140] The repeated iterations during the aforementioned on-site optimization process (after more than ten value adjustments) ultimately led to the design of this invention in a specific implementation as a runtime parameter that can be adjusted in real time on the deployment site and takes effect without restarting the service (see the parameterized design concept accompanying the "Centralized Interruption Processing Module"), rather than a fixed constant hard-coded in the code. This design conclusion that "the parameter should be an on-site adjustable item rather than a fixed value" is directly driven by the strong scenario dependence of the aforementioned real deployment data.

[0141] The optimization process for the two parameters, the speech probability threshold (barge_speech_prob) and the number of sustained frames (barge_sustain_frames), is also representative: Initially, it was required that candidate speech had a probability exceeding 0.5 for several consecutive frames (approximately 8 frames, corresponding to about 0.25 seconds) to be considered a high-confidence candidate. However, actual testing revealed that the frame-by-frame speech probability of real user speech itself exhibits jitter within a normal range (e.g., fluctuating between 0.63 and 0.46). When this jitter causes the probability of a particular frame to fall slightly below the 0.5 threshold, the probability of the next frame... The frame counter was reset and started counting again, causing the user to repeat the message several times before an interruption was triggered. The diagnosis confirmed that the root cause was probability jitter rather than an overly strict requirement for consecutive frames. The final adjustment was to moderately lower the probability threshold for single-frame speech judgment (from 0.5 to about 0.35, so that speech within the normal jitter range could still be counted in the consecutive frame count), rather than simply reducing the required number of consecutive frames. This optimization verified the necessity of "accurately locating the root cause of the phenomenon (probability jitter or frame requirement) and then selecting the correct adjustment knob" for the optimization of such multi-parameter coupled systems.

[0142] The threshold for determining the length of the longest continuous matching segment (run) has undergone an iterative process during its development, evolving from "completely continuous and precise matching" to "fuzzy continuous matching that tolerates certain breakpoints," and then to multiple adjustments to tighten / loosen the values.

[0143] The initial implementation requires that the candidate text must find a completely continuous matching segment with its nearest text, without any character misalignment. However, actual testing revealed that this method is too sensitive to omissions, misspellings, and homophone distortions (common errors caused by the recognition engine after echoes propagate through space) generated by the speech recognition engine. A large number of candidates that were indeed echoes failed to be identified as echoes and were mistakenly allowed to proceed because the continuous matching segments were broken due to recognition distortions.

[0144] The improvement allows a preset number of misaligned characters to be tolerated during the continuous matching process (gap-tolerant run), and adopts the criterion that an echo is determined when either of the two conditions, i.e., "the matching proportion exceeds a preset ratio" or "the absolute continuous matching length exceeds a preset minimum value (ECHO_MIN_RUN)", is satisfied, which significantly improves the interception capability for identifying deformed echoes;

[0145] The minimum continuous matching length parameter (ECHO_MIN_RUN) has undergone multiple adjustments of 3→4→5→6 in subsequent on-site verification: reducing this value will increase the interception rate for short and severely deformed echo segments, but at the same time increase the risk that "a real short sentence of the user accidentally overlaps with several characters of the user's own recent text by coincidence" is misjudged as an echo, and then the real user input is discarded; increasing this value will produce the opposite effect. After multiple rounds of iterative adjustments driven by real misjudgment cases (including two opposite types of problems: real user short sentences being mistakenly filtered as echoes, and missed echoes triggering the digital human to respond to itself), the final value is stabilized at 6, that is, a high-confidence echo is determined only when there is a continuous matching segment of 6 or more characters that are exactly the same between the candidate text and its own latest text (a proportion threshold criterion is additionally superimposed as a supplement). After sufficient testing, this value is confirmed to achieve a better balance point under the current deployment scenario between "protecting real user input from being accidentally injured" and "reliably intercepting echoes".

[0146] The "recent own text" buffer (recent_avatar_text) parameters (buffer time window, maximum number of buffered characters) supporting fuzzy text matching have also undergone multiple rounds of adjustment: an excessively short buffer window will cause the echo segments near the end of the long-time broadcast of the digital human to fail to be compared and identified because the content in the buffer has expired, resulting in missed judgment; an excessively large buffer window or no upper limit on the number of characters will lead to an excessively large total amount of text accumulated in the buffer, and the fuzzy matching algorithm is more prone to false matches where "any short sentence coincidentally hits several consecutive characters" in the ultra-long text, which instead increases the accidental injury rate to real user input. After actual measurement and optimization, the two upper limits of the buffer time window and the total number of characters are finally limited at the same time (double restriction), which significantly reduces the false match misjudgment rate caused by an excessively large buffer compared with limiting only a single dimension (such as limiting only the time window).

[0147] During the aforementioned parameter iteration process, a technical regression once occurred: in the long broadcast scenario that had been previously verified as normal, self-interruption reappeared after a new round of optimization, and the phenomenon was manifested as the digital human starting to repeat the user's previous input content, forming a self-dialogue loop. The troubleshooting process is recorded below as an actual verification case of the supporting engineering methodology of the present invention:

[0148] Phenomenon localization: After the echo is confirmed to be misidentified as several irrelevant phrases by speech recognition, it fails to be intercepted by the text fuzzy matching layer because of its low literal overlap with the digital human's most recent text. As a result, it is sent to the response generation process as real user input. The digital human generates a new response voice based on this, which is then picked up by the microphone again and misidentified as user input again. This process repeats to form a self-dialogue loop.

[0149] Root cause investigation method: Instead of directly adding a new remedial mechanism to the existing judgment logic, the method of "diffing" was adopted to compare all parameter changes involved in this round of optimization. The only substantial difference was located in this round of optimization. In order to solve another problem of "multiple consecutive sentences being mistakenly segmented into multiple independent responses", a certain sample number threshold parameter that controls the "reconnection judgment triggered by speech recognition results" was adjusted from "disabled state (value 0, i.e., no reconnection judgment is triggered)" to "enabled state (non-zero value)".

[0150] Root cause confirmed: After this parameter was changed from disabled to enabled, echo candidates, which were originally only filtered by the text layer and then the processing flow ended, were additionally triggered by the reconnection logic of "judging as a possible new round of user input, canceling the current playback stream and re-establishing the processing flow". This caused echo segments to be repeatedly canceled and rebuilt. In this process, the recognition engine was processing repeatedly truncated audio segments, which made it more likely to produce the aforementioned serious recognition distortions such as "recognizing irrelevant phrases, low literal overlap, and bypassing text filtering". This is the root cause of this regression. Moreover, there is no intuitive causal relationship between this parameter change and the performance of the initial regression (an "anti-segmentation" parameter adjustment unexpectedly triggered a "self-dialogue loop"). If a systematic difference comparison method is not used and only intuitive investigation is used, it is difficult to locate it in time.

[0151] Handling method: After confirming the root cause, the parameter, along with other parameter changes made in this round of optimization, is reverted to a known, normally functioning historical state. Only other improvements that have been verified to be irrelevant to this regression (such as relaxing the response text length limit and browser native echo cancellation audio constraints) are retained. This case verifies and forms an engineering methodology conclusion in the supporting R&D process of this invention: When observing a regression phenomenon of "previously normal, now recurring," the root cause should be located first by comparing the systematic differences with the known normal version, rather than continuously adding new remedial logic on the existing basis—the latter is prone to repeatedly adjusting in the wrong direction and masking the true root cause.

[0152] To avoid relying solely on manual field experiments for parameter tuning of the aforementioned decision-making logic layers, which lacks reproducible objective verification methods, an automated end-to-end verification method was developed. This method simulates the audio and video communication connection of a real interactive digital human system through a programmatically constructed test client. Two test scenarios are injected: "the digital human's own broadcast is superimposed with the device's own echo audio" and "the digital human's own broadcast is superimposed with real user interruption voice". By monitoring the changes in the energy envelope of the downlink audio and video streams (start and end times, whether there is continuous energy that cannot be explained by a single subsequent voice), the system automatically determines whether the system has experienced "erroneous self-interruption", "correctly maintained continuous broadcast", or "correctly responded to real interruption" in each test scenario. This transforms the verification process, which originally relied on manual auditory judgment and was difficult to reproduce on a large scale, into an automated test that can be run repeatedly and whose success rate can be quantified. The final layering determination scheme described in this invention and the multi-queue linkage clearing mechanism in the related invention "Audio-visual frame-level alignment and seamless splicing rendering method for streaming voice-driven digital human video" were determined to be the final deployment version only after multiple rounds of repeated testing by the aforementioned automated verification method, covering various scenario combinations such as "single interruption", "continuous rapid multiple interruptions", and "long-term high-volume self-echo during broadcast". It has reproducible verification basis, rather than relying solely on a single manual trial to determine its effectiveness.

[0153] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method for preventing voice interruption in interactive digital humans using echo cancellation, characterized in that, The interactive digital human voice interruption defense method includes: S1. The microphone continuously collects audio and enters the VAD acoustic energy gating judgment; S2. For audio that exceeds the initial threshold, determine whether the speech energy of the audio exceeds the interruption threshold. If yes, proceed to step S3; otherwise, proceed to step S4. The value of the interruption threshold is higher than the initial threshold. S3. Perform fuzzy text matching on the candidate text of the collected candidate audio to determine whether the candidate audio is an echo. If it is, discard the candidate audio and execute step S1 again. Otherwise, execute step S7. S4. Perform cross-correlation calculation between the candidate audio and the recently played reference audio, and determine whether the correlation coefficient is greater than the preset value. If yes, proceed to step S5; otherwise, proceed to step S6. S5. Mark the candidate audio as a suspected echo and increase the text fuzzy matching parameters for the suspected echo, then proceed to step S3. S6. Perform step S3 according to the standard parameters for text fuzzy matching; S7. Send an interrupt signal to the interrupt handling module.

2. The interactive digital human speech interruption defense method for echo cancellation as described in claim 1, characterized in that, The interactive digital human voice interruption defense method includes: During the digital human's voice broadcast, it continuously retains its own audio waveform that was recently sent to the speaker for playback. When the microphone picks up a candidate audio, it performs a cross-correlation calculation between the candidate audio and its own audio waveform in the time domain, and determines whether the correlation coefficient is greater than a preset value.

3. The interactive digital human speech interruption defense method for echo cancellation as described in claim 1, characterized in that, Step S3 includes: Utilize speech recognition to output candidate text from candidate audio; The candidate text is compared with its nearest text corresponding to its own audio waveform using a multi-algorithm combination fuzzy comparison. The multi-algorithm includes at least two of the following: the longest common subsequence length ratio, pinyin similarity comparison, and the calculation of the longest continuous matching segment length. The matching results of multi-algorithm combined fuzzy comparison are judged according to preset weights or rules to determine whether the candidate audio is an echo.

4. The interactive digital human speech interruption defense method for echo cancellation as described in claim 3, characterized in that, The interactive digital human voice interruption defense method includes: For the initial candidate audio within a preset time window after the digital human begins broadcasting, candidate text of the initial candidate audio is output using speech recognition, wherein, The longest common subsequence length ratio is calculated by taking the length of the longest common subsequence between the candidate text and its nearest text, and normalizing it with the length of the candidate text itself to obtain the similarity ratio; The pinyin similarity comparison is performed by converting the candidate text and its nearest text into pinyin sequences and then comparing them. The longest consecutive matching segment length is calculated by calculating the length of the longest consecutive character matching segment between the candidate text and its nearest text, and identifying echo segments that are repeated continuously but whose overall order is scrambled by the recognition engine.

5. The interactive digital human speech interruption defense method for echo cancellation as described in claim 1, characterized in that, Step S7 includes: Send an interrupt signal to the centralized interrupt processing module; The centralized interruption handling module executes the cleaning process; Entering a new round of speech recognition and response generation process.

6. The interactive digital human speech interruption defense method for echo cancellation as described in claim 5, characterized in that, Step S7 also includes: The process of monitoring interruption signals from multiple signal sources includes: if the interruption signal carries a target data stream identifier, tracing upwards along the dependency chain of the target data stream and canceling all associated generated streams level by level; and performing a unified scan and cancellation on all currently registered digital human response text streams, digital human synthesized speech streams, and client playback streams according to their types.

7. The interactive digital human speech interruption defense method for echo cancellation as described in claim 5, characterized in that, The centralized interruption processing module executes the cleaning process, which also includes: Send a cancellation command to the speech synthesis module and the digital human video rendering module to discard the intermediate audio and video data currently being synthesized and / or inferred in the pipeline; Perform a clear operation on the multi-level buffer queue of the digital human video rendering module, and add a global identifier for video rendering; For each processing queue in the digital human video rendering module, determine whether the identifier of the output data of the processing queue is the latest identifier. If it is, the output data enters the next processing queue; otherwise, the output data is discarded.

8. The interactive digital human speech interruption defense method for echo cancellation as described in claim 1, characterized in that, The interactive digital human voice interruption defense method is obtained through a generation method, which includes: The feasibility of four single schemes—pure energy threshold, pure cross-correlation, NLMS adaptive filtering AEC, and client-side native AEC—was verified respectively. The layered and progressive architecture of the interactive digital human voice interruption defense method was obtained using the feasibility study results.

9. The interactive digital human speech interruption defense method for echo cancellation as described in claim 8, characterized in that, The generation method includes: The interruption threshold was optimized in multiple rounds to obtain verification results showing that the optimal value of the interruption threshold was strongly correlated with the actual acoustic deployment method of the device. The range of values ​​for the interruption threshold and the runtime adjustable attributes are obtained using the verification results.

10. The interactive digital human speech interruption defense method for echo cancellation as described in claim 1, characterized in that, The interactive digital human voice interruption defense method also includes: A test client is constructed programmatically, and the client is used to simulate audio and video communication connections with an interactive digital human system. Two test scenarios were established: one was to overlay the device's own echo audio during the digital human's own broadcast, and the other was to overlay real user interruption audio during the digital human's own broadcast. The execution results of the interactive digital human system in multi-round test scenarios are determined by monitoring the changes in the energy envelope of the downlink audio and video streams. Statistical analysis was conducted on the false interruption rate, missed interruption rate, and interruption delay for the test client to generate a reproducible verification report and confirm the final deployment version.

Images