A web-based intelligent voice interaction and mix control system and method

Abstract

The application discloses a kind of intelligent voice interaction and mix sound control system and method based on web end to the problems of existing transmission model being limited, acoustic coupling causing echo pollution and experience degradation caused by interactive timing mismatch in prior art.The application controls the gain of user voice, AI voice and background audio independently by three-track dynamic mixer, i.e., the three signals can be dynamically fused into a standard audio stream to be sent to the remote end.The local playback and echo eradication unit performs adaptive echo filtering on the microphone collection stream, and the AI audio is played through a dedicated channel independent of the microphone path, thereby cutting off the acoustic feedback loop and suppressing howling.The dialogue state machine controller makes multi-dimensional state decisions based on real-time voice activity, AI service signaling and timer events, realizes smooth switching of interactive states and dynamically adjusts the gain of each channel, thereby avoiding voice overlap and mechanical frustration caused by hard switching.

Description

Technical Field

[0001] This invention belongs to the technical field of computer application technology, real-time communication, audio signal processing and artificial intelligence, and specifically relates to a web-based intelligent voice interaction and mixing control system and method. Background Technology

[0002] Currently, WebRTC (Web Real-Time Communication, an open-source project designed to enable real-time communication between web browsers and mobile applications, allowing devices to connect and transmit audio, video, or other types of data without relying on proprietary equipment or cumbersome installation processes) is widely used for real-time audio and video communication on browsers, and AI voice technology is also maturing. However, existing technologies have the following key shortcomings when building a "human-AI-human" three-way voice interaction system: (1) Limited transmission model: the problem of multi-source fusion under single-stream constraints; the WebRTC protocol stipulates that a single RTCPeerConnection (an object in the WebRTC API that provides real-time communication between two browsers, using audio, video and data streams for peer-to-peer communication without the participation of an intermediary server) can only send one MediaStream (MediaStream is an intermediate layer connecting the WebRTC API and the underlying physical stream, consisting of one or more MediaStreamTrack objects, which represent video and audio from different input devices), and cannot directly carry the parallel transmission of multiple independent audio sources (such as user voice, AI synthesized voice, and background sound effects); while traditional solutions usually mix AI voice into microphone input or rely on the server for mixing, resulting in complex architecture, increased latency and difficulty in ensuring privacy.

[0003] (2) Acoustic coupling causes echo pollution; when AI voice is played through local speakers, its sound waves are easily picked up by microphones and transmitted back to the far end, forming a perceptible echo or even positive feedback howling. Especially in open office or mobile scenarios, the risk of acoustic leakage is significant. The existing browser AEC mechanism is mainly designed for remote calls and lacks effective means to suppress AI voice played locally.

[0004] (3) Interaction timing mismatch leads to experience degradation; human dialogue relies on fine timing control (such as the silence period after speaking, the AI ​​thinking gap, and the natural transition at the start of speech); among them, existing systems mostly adopt hard switching logic based on energy threshold, which is prone to speech overlap, mechanical switching or redundant noise, which seriously weakens the naturalness and immersion of interaction. Moreover, after the introduction of background audio, the complexity of multi-source collaborative control is further increased. Therefore, traditional static mixing strategies are difficult to adapt to dynamic semantic scenarios.

[0005] Therefore, given the aforementioned shortcomings, how to provide a low-latency solution for achieving intelligent fusion of multi-source audio, echo immunity, and state-aware switching in a pure browser environment has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a web-based intelligent voice interaction and mixing control system and method to solve the problems of limited transmission models, echo pollution caused by acoustic coupling, and experience degradation caused by interaction timing mismatch in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a web-based intelligent voice interaction and mixing control system is provided, including: The multi-source audio capture and preprocessing unit is used to acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server. The local playback and echo removal unit is used to perform adaptive echo filtering on the initial audio stream to obtain the user audio stream; The dialogue state machine controller maintains several voice interaction states, which are used to acquire system timer monitoring events and receive AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling and system timer monitoring events, the current voice interaction state is determined from several voice interaction states. Based on the current voice interaction state, the gain control parameters corresponding to the three audio streams are determined. The gain control parameters are different in different voice interaction states. The three-track dynamic mixer includes gain control nodes corresponding to the user audio stream, background audio stream, and AI audio stream, respectively. It is used to perform smooth gain processing on the three audio streams based on the gain control parameters and gain control nodes corresponding to the three audio streams, so as to obtain three processed audio streams. The three processed audio streams are then mixed to obtain a mixed audio stream that is transmitted to the WebRTC communication adapter unit. WebRTC communication adapter unit, used to send mixed audio streams to a remote location; The local playback and echo erasure unit is also used to acquire the AI ​​audio stream from the multi-source audio capture and preprocessing unit and play it locally through a playback channel independent of the local microphone acquisition path.

[0008] Based on the aforementioned disclosure, this invention utilizes a "three-track dynamic mixer" to independently control the gain of user voice, AI voice, and background audio on the browser side. Without relying on server-side mixing or modifying WebRTC single-stream constraints, it dynamically fuses the three signals into a single standard audio stream for transmission to a remote location. This reduces server-side computational load and bandwidth costs while preventing the external transmission of raw audio data, effectively protecting user privacy. Simultaneously, the local playback and echo cancellation unit adaptively filters the audio stream captured by the microphone, actively canceling out echoes after acquisition. This fundamentally suppresses feedback howling from AI voice, and the AI ​​audio is played through a dedicated channel independent of the microphone path, effectively cutting off acoustic feedback. The feeder loop ensures clear local hearing and does not affect the quality of remote calls. Finally, the dialogue state machine controller performs multi-dimensional state adjudication based on real-time voice activity, AI service signaling, and timer events, realizing smooth switching of voice interaction states and simultaneously determining the gain control parameters of each audio stream under different voice interaction states to achieve smooth gain processing of each audio stream. In this way, the voice overlap and mechanical stuttering caused by hard switching can be fundamentally avoided, making the "human-AI-human" interaction process closer to natural human dialogue. Therefore, this invention, with a lightweight pure browser architecture, realizes low-latency, highly robust, and privacy-strength "human-AI-human" three-way intelligent voice interaction, making it very suitable for large-scale application and promotion.

[0009] In one possible design, when acquiring an AI audio stream pushed by an AI server, the multi-source audio capture and preprocessing unit is configured as follows: The incremental PCM segments pushed by the AI ​​server are received through a streaming interface, wherein the incremental PCM segments represent audio data returned by the AI ​​server. The incremental PCM segment is semantically segmented to obtain several audio cache blocks, wherein each audio cache block is associated with a segmentation sequence number, a start and end timestamp, and the corresponding text; Several audio cache blocks are stored in an indexable buffer according to their segmentation numbers to construct the AI ​​audio stream that supports random positioning.

[0010] In one possible design, when the initial audio stream is subjected to adaptive echo filtering to obtain the user audio stream, the local playback and echo erasure unit is configured as follows: Based on the AI ​​audio stream, a filtered reference signal is generated; Obtain the filter weight vector at the k-th iteration, where the initial value of k is 1. When k is 1, the filter weight vector at the k-th iteration is the initial vector, and the length of the filter weight vector is the same as the length of the filtered reference signal. Based on the filtered reference signal and the filter weight vector at the kth iteration, echo estimation is performed to obtain the echo signal. Calculate the error signal between the initial audio stream and the echo signal, wherein the error signal is the filtered initial audio stream; Determine if the iteration stopping condition is met; If not, then use the error signal to update the filter weight vector at the kth iteration to obtain the filter weight vector at the (k+1)th iteration. Increment k by 1 and reacquire the filter weight vector at the kth iteration until the iteration stopping condition is met, so that the error signal at the time of the iteration stopping condition is used as the user audio stream.

[0011] In one possible design, when performing echo estimation based on the filtered reference signal and the filter weight vector at the k-th iteration, the local playback and echo erasure unit is configured as follows: The echo signal is generated according to the following formula; ; In the formula, Indicates an echo signal. This represents the filter weight vector at the k-th iteration. Indicates the filter reference signal. Indicates the transpose operation; Accordingly, when updating the filter weight vector at the k-th iteration using the error signal, the local playback and echo erasure unit is configured as follows: The filter weight vector for the (k+1)th iteration is generated according to the following formula; ; In the formula, This represents the filter weight vector at the (k+1)th iteration. Indicates the update step size. This represents the error signal. For regularization parameters, Represents the magnitude of a vector.

[0012] In one possible design, several voice interaction states include: The MIC_ACTIVE state indicates that the user is speaking. In the MIC_ACTIVE state, the gain control parameter of the user audio stream is 1, the gain control parameter of the background audio stream is 0.2 to 0.5, and the gain control parameter of the AI ​​audio stream is 0. The USER_END_SILENCE state represents the silence state after the user finishes speaking. In the USER_END_SILENCE state, the gain control parameters of the three audio streams are all 0, and the duration of the silence state is 50 milliseconds to 200 milliseconds. The AI_THINKING state indicates that the AI ​​server is in the request processing state. In the AI_THINKING state, the gain control parameters of the three audio streams are all 0. The AI_SPEAKING state indicates that the audio is in AI voice playback mode. In the AI_SPEAKING state, the gain control parameter of the user audio stream is 0, the gain control parameter of the background audio stream is 0.05 to 0.3, and the gain control parameter of the AI ​​audio stream is 1. The BACKGROUND_ONLY state indicates an idle state with no interactive activity. In the BACKGROUND_ONLY state, the gain control parameter for the background audio stream is 0.1 to 0.3, while the gain control parameter for the user audio stream and the AI ​​audio stream is 0.

[0013] In one possible design, the AI ​​service signaling includes AI_THINKING signaling and AI_RESPONSE_READY signaling, and the system timer monitors events including user voice interaction events and AI service signaling interaction events; When the current voice interaction state is determined from several voice interaction states based on events monitored by the user audio stream, AI service signaling, and system timers, the dialogue state machine controller is configured as follows: The user's audio stream is segmented into frames to obtain several audio frames; Calculate the short-time energy of each audio frame and determine the zero-crossing rate of each audio frame; When the short-term energy of several consecutive audio frames is higher than the energy threshold and the zero-crossing rate of several consecutive audio frames is within the preset range, it is determined that there is a voice start event in the user audio stream and the current voice interaction state is determined to be MIC_ACTIVE state; otherwise, the current voice interaction state is determined to be USER_END_SILENCE state. When in the USER_END_SILENCE state, if the AI_THINKING signaling is received, the system switches from the USER_END_SILENCE state to the AI_THINKING state, and uses the AI_THINKING state as the current voice interaction state. When in AI_THINKING state, if AI_RESPONSE_READY signaling is received and the first frame of audio data pushed by the AI ​​server is received, then switch from AI_THINKING state to AI_SPEAKING state, and use AI_SPEAKING state as the current voice interaction state. If no user voice interaction event or AI service signaling interaction event is detected within a preset time period, the system will directly switch to the BACKGROUND_ONLY state, making it the current voice interaction state; and In any state other than MIC_ACTIVE, if a voice start event is detected in the user's audio stream, the system switches from any state to MIC_ACTIVE, and uses MIC_ACTIVE as the current voice interaction state.

[0014] In one possible design, when determining the gain control parameters for each of the three audio streams, the smoothing gain time for each of the three audio streams is also determined, and the three-track dynamic mixer also includes: a sample-level mixing node. Specifically, when smoothing the gain of the three audio streams based on their respective gain control parameters and gain control nodes, the three-track dynamic mixer is configured as follows: For the user audio stream, obtain the smoothed gain time corresponding to the user audio stream; Based on the gain control node corresponding to the user audio stream, the amplitude of the user audio stream is adjusted to the target parameter using a Hanning window or linear interpolation within the smooth gain time to obtain the processed user audio stream. The target parameter is the gain control parameter corresponding to the user audio stream. Accordingly, when mixing the three processed audio streams, the sample-level mixing node is configured as follows: The three processed audio streams are weighted and summed to obtain the initial mixed audio stream. The amplitude of the initial mixed audio stream is adjusted to the range of [-1, 1] to obtain the mixed audio stream.

[0015] In one possible design, the local playback and echo cancellation unit is also used to encapsulate the AI ​​audio stream into RTC-encoded audio frames and inject them into the reference signal channel of the acoustic echo cancellation module in WebRTC for caching, thereby completing the backup of the AI ​​audio stream.

[0016] In one possible design, the local playback and echo removal unit is also used to perform echo removal verification on the mixed audio stream before the WebRTC communication adapter unit sends the mixed audio stream to the remote end, and if the verification fails, read the RTC encoded audio frame from the reference signal channel and transmit it to the WebRTC communication adapter unit so that the WebRTC communication adapter unit can directly send the RTC encoded audio frame to the remote end.

[0017] Secondly, a web-based intelligent voice interaction and mixing control method is provided, wherein the method is executed based on the web-based intelligent voice interaction and mixing control system described in the first aspect or any possible design of the first aspect, and the method includes: Acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server; Adaptive echo filtering is applied to the initial audio stream to obtain the user audio stream; The system acquires system timer monitoring events and receives AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling, and system timer monitoring events, it determines the current voice interaction state from several preset voice interaction states. Based on the current voice interaction state, it determines the gain control parameters corresponding to each of the three audio streams. The gain control parameters are different in different voice interaction states. Each of the three audio streams has a corresponding gain control node. Based on the gain control parameters and gain control nodes corresponding to the three audio streams, smooth gain processing is performed on the three audio streams to obtain three processed audio streams, which are then mixed to obtain a mixed audio stream that is transmitted to the WebRTC communication adapter unit. Send the mixed audio stream to a remote location; and The AI ​​audio stream is acquired from the multi-source audio capture and preprocessing unit and played locally through a playback channel independent of the local microphone acquisition path.

[0018] Thirdly, a web-based intelligent voice interaction and mixing control device is provided. Taking the device as an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the web-based intelligent voice interaction and mixing control method as described in the second aspect.

[0019] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the Web-based intelligent voice interaction and mixing control method as described in the second aspect.

[0020] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the Web-based intelligent voice interaction and mixing control method as described in the second aspect.

[0021] Beneficial effects: (1) Breaking through the limitations of single-stream transmission architecture, multi-source decoupling and low-latency fusion are realized. This invention completes independent gain control and real-time mixing of user voice, AI voice and background audio on the Web end through a "three-track dynamic mixer". It does not rely on the server for complex mixing processing, which avoids the stream constraints of a single RTCPeerConnection, significantly reduces transmission latency and server load, and can simultaneously ensure that the user's original audio and AI audio are mixed locally, thereby effectively protecting the privacy and security of voice data.

[0022] (2) It eliminates echo pollution caused by local AI voice. The present invention uses the "multi-source audio capture and preprocessing unit" to generate reference signals based on AI audio stream for active echo filtering, and combines the "local playback and echo elimination playback unit" with a dedicated playback channel independent of the microphone acquisition path to cut off the echo feedback loop from both signal processing and physical path levels. This completely solves the problem of the failure of the traditional browser AEC mechanism to suppress local AI voice playback and eliminates the risk of howling.

[0023] (3) The interaction timing has been optimized, giving the dialogue a natural rhythm. The dialogue state machine controller dynamically adjusts the gain of each audio stream based on real-time voice activity, AI service signaling, and silence timeout events. In different states such as "user speaking", "AI thinking", and "AI responding", it intelligently realizes silence attenuation, smooth transition or background noise avoidance (i.e., through gain control parameters in different states), thereby effectively avoiding voice overlap and mechanical cut-off, creating a smooth rhythm and immersive experience close to human dialogue. Attached Figure Description

[0024] Figure 1 A structural diagram of a web-based intelligent voice interaction and mixing control system provided in an embodiment of the present invention; Figure 2 A flowchart of a web-based intelligent voice interaction and mixing control method provided in an embodiment of the present invention. Detailed Implementation

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0026] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0027] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0028] Example: See Figure 1 As shown, the web-based intelligent voice interaction and mixing control system provided in this embodiment is deployed on a client browser. It autonomously completes the dynamic fusion and semantic alignment control of user voice, AI response voice, and background audio through the front end. Combined with an innovative echo cancellation mechanism and a WebRTC-compatible transmission architecture, it achieves a high-fidelity, low-latency, and echo-free "human-AI-human" three-way voice interaction closed loop. Specifically, the web-based intelligent voice interaction and mixing control system may include, but is not limited to, a multi-source audio capture and preprocessing unit, a local playback and echo removal unit, a dialogue state machine controller, a three-track dynamic mixer, and a WebRTC communication adaptation unit. The operating logic of each of the aforementioned units is as follows: First, the multi-source audio capture and preprocessing unit is used to acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server. In specific implementation, navigator.mediaDevices.getUserMedia({audio: true}) can be called to obtain the microphone input (which is the core API in the WebRTC standard used to capture audio and video media streams from user devices, and is the only standard entry point for browsers to access hardware devices such as microphones and cameras). Then, after front-end VAD (Voice Activity Detection) and spectral subtraction noise reduction, a normalized PCM stream (micStream) is output, thus obtaining the initial audio stream.

[0029] Meanwhile, when acquiring the background audio stream from the background audio source, this system loads it using one of the following two methods: Local files are decoded into floating-point PCM using AudioContext.decodeAudioData() to obtain the background audio stream; remote audio resources are... <audio>The element is connected to the Web Audio graph via MediaElementSourceNode, generating a background stream (bgStream) with controllable gain.

[0030] It should be noted that: <audio>The `<audio>` element is a native media tag in the HTML5 standard used to embed, play, and control audio content in web pages. It's a container for the browser's built-in audio player, capable of playing audio files in formats such as MP3, WAV, and OGG without any plugins. The `MediaElementSourceNode` is: The Web Audio API provides a method for converting HTML... <audio>The audio content being played / loaded by the element is imported into the Web Audio processing graph, enabling it to connect and be processed with other Web Audio nodes (such as GainNode, AudioWorkletNode, etc.), rather than being controlled independently by the browser's media engine outside of the Web Audio graph. Therefore, the aforementioned access to the Web Audio graph refers to: Use audioContext.createMediaElementSource(audioEl) (which is a Web Audio API used to create HTML files). <audio>The core method for integrating elements into the Web Audio processing graph, once called, transforms audio playback, originally controlled independently by the browser's media engine, into a source node within the Web Audio processing graph. This allows it to connect, process, and mix with other nodes in the graph. <audio>The element acts as a Web Audio source node, and its audio is not played out directly but enters the processing graph.

[0031] Meanwhile, the source node connects to a GainNode (the core interface in the Web Audio API used to control the volume (gain) of audio signals; it is essentially a programmable volume adjuster that controls the audio volume in real time by multiplying each sample point of the input audio signal by a gain value (amplitude scaling factor)). The source node adjusts the volume in real time and then feeds the audio into the mixer. In addition, in this embodiment, background audio refers to the audio material actively played by the system (BGM, prompts, background music, etc.), not the background noise of the physical environment where the microphone is located.

[0032] Thus, after acquiring the background audio stream, the AI ​​audio stream can be acquired. When acquiring the AI ​​audio stream pushed by the AI ​​server, the multi-source audio capture and preprocessing unit is configured to perform the following operations: First, incremental PCM segments pushed by the AI ​​server are received through a streaming interface (such as WebSocket or HTTP). Then, the incremental PCM segments are semantically segmented to obtain several audio cache blocks, and each audio cache block is associated with a segmentation number, start and end timestamps, and corresponding text. Finally, the several audio cache blocks are stored in an indexable buffer according to the segmentation number, thereby constructing the AI ​​audio stream that supports random positioning.

[0033] It should be noted that the incremental PCM segment refers to the audio data returned by the AI ​​server (which can be pushed via cloud-based TTS service). The AI ​​server synthesizes and pushes the data simultaneously, pushing a small segment (approximately 20-200ms) of 16-bit linear PCM (16k / 24kHz) each time, rather than synthesizing the entire sentence and returning it all at once. The purpose is to start playback as soon as the first segment is received, reducing the latency of the first frame. Thus, after obtaining the incremental PCM segment, it can be divided into cache blocks according to semantic boundaries (a sentence, a utterance unit). Each block is accompanied by a sequence number, start and end timestamps, and corresponding text, and a boundary marker (such as sentence_end) is sent at the end of the sentence. The front end encapsulates the data accordingly, so that fade-in and fade-out and state switching can be aligned to the "sentence" boundary, avoiding hard cutting in the middle of half a word.

[0034] Meanwhile, since each buffer block is stored in an indexable buffer according to its sequence number, and each buffer block has an index, the player can jump to any unit to read (seek), thus giving the AI ​​audio stream the characteristic of "random access" (such as interrupting and skipping to a sentence, or replaying a sentence).

[0035] Thus, after acquiring the three audio streams, the multi-source audio capture and preprocessing unit, in this embodiment, deploys an adaptive filter in the microphone input link to cancel the echo components collected by the microphone. This filter uses the aiStream (AI audio stream) as a reference signal to estimate and cancel the echo components in real time. The process is as follows: The local playback and echo removal unit is used to perform adaptive echo filtering on the initial audio stream to obtain the user audio stream; wherein, when performing adaptive echo filtering, the local playback and echo removal unit is configured to perform the following operations: Step 1: Generate a filter reference signal based on the AI ​​audio stream. In this embodiment, a filter reference signal can be formed by obtaining sampling points of length L from the AI ​​audio stream. Then, the initial audio stream can be iteratively filtered by iteratively updating the filter weights, as shown in steps 2 to 7 below.

[0036] Step 2: Obtain the filter weight vector at the k-th iteration, where the initial value of k is 1. When k is 1, the filter weight vector at the k-th iteration is the initial vector, and the length of the filter weight vector is the same as the length of the filtered reference signal. In this embodiment, each element in the initial vector is a preset value, and this embodiment does not impose specific limitations.

[0037] Thus, after obtaining the filter weight vector at the k-th iteration, echo estimation can be performed based on it, as shown in the third step below.

[0038] Step 3: Based on the filtered reference signal and the filter weight vector at the k-th iteration, perform echo estimation to obtain the echo signal; in specific implementation, the echo signal can be generated according to, but is not limited to, the following formula.

[0039] ; In the formula, Indicates an echo signal. This represents the filter weight vector at the k-th iteration. Indicates the filter reference signal. This indicates the transpose operation.

[0040] After estimating the echo signal at the current iteration, the filtered initial audio stream for the current iteration can be generated based on this, as shown in step four below.

[0041] Step 4: Calculate the error signal between the initial audio stream and the echo signal, wherein the error signal is the filtered initial audio stream; in specific implementation, the error signal can be obtained by subtracting the echo signal at the current iteration from the initial audio stream; then, determine whether the iteration stopping condition is met, as shown in Step 5 below.

[0042] Step 5: Determine if the iteration stopping condition is met. In practice, for example, the iteration stopping condition is: the ERLE (Echo Return Loss Enhancement) index is greater than 30dB. The calculation process for the ERLE index is as follows: first, calculate the power of the initial audio stream and the power of the error signal; then, calculate the ratio between the two to obtain the power ratio; next, based on the power ratio, calculate the ERLE index. Optionally, the calculation formula is: In the formula, This represents the power ratio.

[0043] If the aforementioned iteration stopping condition is not met, then a weight update is required, as shown in step six below.

[0044] Step 6: If not, use the error signal to update the filter weight vector at the k-th iteration to obtain the filter weight vector at the (k+1)-th iteration.

[0045] In practical implementation, for example, the local and echo eradication units can be configured, but are not limited to, to use the following formula to generate the filter weight vector for the (k+1)th iteration.

[0046] ; In the formula, This represents the filter weight vector at the (k+1)th iteration. Indicates the update step size ( (Controlling convergence speed and stability). This represents the error signal. This is a regularization parameter (to prevent the denominator from being 0; it is a constant). Represents the magnitude of a vector.

[0047] Thus, based on the aforementioned formula, after obtaining the filter weight vector at the (k+1)th iteration, echo estimation can be performed again, and filtering can be performed. This iterative filtering can be performed continuously until the iteration stopping condition is met, at which point the user audio stream can be obtained. The process is shown in step seven below.

[0048] Step 7: Increment k by 1 and reacquire the filter weight vector at the kth iteration until the iteration stopping condition is met. Use the error signal at the time the iteration stopping condition is met as the user audio stream.

[0049] Therefore, after the local playback and echo cancellation unit completes echo cancellation based on an adaptive filter, it can enter the state switching process. In this embodiment, the voice interaction state switching is performed through a dialogue state machine controller, and the gain control parameters for the corresponding state are determined. The specific operation process is as follows: In this embodiment, the dialogue state machine controller maintains several voice interaction states, which are used to acquire system timer monitoring events and receive AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling and system timer monitoring events, the controller determines the current voice interaction state from the several voice interaction states. Based on the current voice interaction state, the controller determines the gain control parameters corresponding to the three audio streams. The gain control parameters are different in different voice interaction states.

[0050] In practical implementation, the dialogue state machine controller maintains five finite states (i.e., five voice interaction states) to achieve fine-grained interactive control. Specifically, the five voice interaction states and their corresponding gain control parameters are as follows: The MIC_ACTIVE state indicates that the user is in a speaking state. In the MIC_ACTIVE state, the gain control parameter of the user audio stream is 1, the gain control parameter of the background audio stream is 0.2 to 0.5, and the gain control parameter of the AI ​​audio stream is 0. That is, in this state, the mixed output is mainly based on the microphone track, with background bass as a secondary feature.

[0051] The USER_END_SILENCE state represents the silence state after the user finishes speaking. In the USER_END_SILENCE state, the gain control parameters of all three audio streams are 0, and the duration of the silence state is 50 milliseconds to 200 milliseconds. That is, in this state, all tracks are muted to simulate a natural pause.

[0052] The AI_THINKING state indicates that the AI ​​server is in the request processing state. In the AI_THINKING state, the gain control parameters of the three audio streams are all 0; thus, the entire track is muted in this state, thereby suppressing uplink noise.

[0053] The AI_SPEAKING state indicates that the AI ​​voice playback state is in progress. In the AI_SPEAKING state, the gain control parameter of the user audio stream is 0, the gain control parameter of the background audio stream is 0.05 to 0.3, and the gain control parameter of the AI ​​audio stream is 1. In this embodiment, in the AI_SPEAKING state, the AI ​​audio stream is played, the microphone is muted, and the background is ducking (background noise is reduced).

[0054] The BACKGROUND_ONLY state represents an idle state with no interactive activity. In the BACKGROUND_ONLY state, the gain control parameter of the background audio stream is 0.1 to 0.3, while the gain control parameter of the user audio stream and the AI ​​audio stream is 0. Thus, in this state, which is a long period of no interaction, only low-gain background sound is retained to prevent the atmosphere from becoming dull.

[0055] It should be noted that the gain control parameter is used to adjust the amplitude of the audio stream. That is, the gain control parameter is equivalent to an amplitude scaling factor. Multiplying it by the original amplitude will complete the gain adjustment.

[0056] Therefore, by maintaining the aforementioned five voice interaction states, state transitions can be performed within these five states. These state transitions are driven by multiple time sources, specifically: user audio stream (i.e., energy jumps), AI service signaling, and system timer monitoring events. For example, AI service signaling may include, but is not limited to, AI_THINKING and AI_RESPONSE_READY signaling (which are configured and issued by the AI ​​server for different states). System timer monitoring events include user voice interaction events and AI service signaling interaction events, i.e., monitoring user voice input and AI service signaling. Based on this, the state machine simultaneously listens to the aforementioned three types of event sources to perform state transitions.

[0057] Specifically, when the current voice interaction state is determined from several voice interaction states based on user audio streams, AI service signaling, and system timer monitoring events, the dialogue state machine controller is configured to perform the following operations: First, the user audio stream is segmented into frames to obtain several audio frames. Then, the short-time energy (sum of squares of intra-frame samples, reflecting volume) of each audio frame is calculated, and the zero-crossing rate (number of times the signal crosses zero, distinguishing between voiced and unvoiced human voices) of each audio frame is determined. Then, when the short-time energy of several consecutive audio frames (the number of frames can be preset, such as 2 frames, 3 frames, etc.) is higher than the energy threshold, and the zero-crossing rate of several consecutive audio frames is within the preset range, it is determined that there is a voice start event in the user audio stream, and the current voice interaction state is determined to be MIC_ACTIVE. Otherwise, the current voice interaction state is determined to be USER_END_SILENCE.

[0058] Meanwhile, for example, the energy threshold is: obtain the short-time energy of the non-speech segment (background noise), and take one-third of the short-time energy of the non-speech segment as the energy threshold.

[0059] Thus, when the energy of several consecutive frames is higher than the threshold and the zero-crossing rate falls within the human voice range (the aforementioned preset range), "voice start" is determined, triggering entry into the MIC_ACTIVE state; otherwise, the voice is determined to end, entering the USER_END_SILENCE state.

[0060] Furthermore, when in the USER_END_SILENCE state, if an AI_THINKING signal is received, the system switches from the USER_END_SILENCE state to the AI_THINKING state, using the AI_THINKING state as the current voice interaction state. Conversely, when in the AI_THINKING state, if an AI_RESPONSE_READY signal is received, and the first frame of audio data pushed by the AI ​​server is received (i.e., the first frame is ready), the system switches from the AI_THINKING state to the AI_SPEAKING state, using the AI_SPEAKING state as the current voice interaction state.

[0061] Furthermore, if no user voice interaction event or AI service signaling interaction event is detected within a preset time period, the system will directly switch to the BACKGROUND_ONLY state, which will be used as the current voice interaction state. In any state other than the MIC_ACTIVE state, if a voice start event is detected in the user audio stream, the system will switch to the MIC_ACTIVE state, which will be used as the current voice interaction state.

[0062] Thus, in the aforementioned state transition process, the user interruption event has the highest priority. That is, in any state other than MIC_ACTIVE, if a voice start event is detected, the current state is immediately terminated and the transition to MIC_ACTIVE is forced. Next, the priority of AI service signaling is lower. Finally, the timer monitoring event has the lowest priority, which is used to indicate entering an idle state without interactive activity.

[0063] Thus, after the dialogue state machine controller completes the state transition, it can determine the gain control parameters for each of the three audio streams based on the current voice interaction state. Simultaneously, this embodiment also obtains the smooth gain time under the current voice interaction state, which is the duration required for the gain to smoothly change from the current value to the target value, measured in milliseconds (ms). It controls the speed of gain change, not the magnitude of the gain itself. The smooth gain time corresponding to the aforementioned five states can be, but is not limited to, 10ms, 80-150ms, 5ms, 5ms, and 20ms respectively. Of course, the above examples are merely illustrations, and this embodiment is not limited to them.

[0064] Based on this, after determining the audio gain parameters in the current voice interaction state, gain processing and audio mixing of the three audio streams can be performed. The process is as follows: The three-track dynamic mixer includes gain control nodes corresponding to the user audio stream, background audio stream, and AI audio stream, respectively. It is used to perform smooth gain processing on the three audio streams based on the gain control parameters and gain control nodes corresponding to the three audio streams, so as to obtain three processed audio streams. The three processed audio streams are then mixed to obtain a mixed audio stream that is transmitted to the WebRTC communication adapter unit.

[0065] In practical implementation, the three-track dynamic mixer constructs a low-latency audio processing graph based on the Web Audio API and creates three independent GainNode nodes (referring to the programmable volume adjustment node in the Web Audio API, i.e., the gain control node, used to perform gain (amplification / attenuation) processing on the audio signal, thus achieving track-level gain control). At the same time, the three-track dynamic mixer also includes a sample-level mixing node. Based on this, the three GainNodes are respectively connected to the microphone track, background track, and AI track, and after each independently adjusting its volume, they are merged into the sample-level mixing node for mixing processing.

[0066] Furthermore, when performing smooth gain processing on the three audio streams based on their respective gain control parameters and gain control nodes, the three-track dynamic mixer is configured to perform the following operations: First, for the user audio stream, obtain the smoothing gain time corresponding to the user audio stream; then, based on the gain control node corresponding to the user audio stream, within the smoothing gain time, use Hanning window or linear interpolation to adjust the amplitude of the user audio stream to the target parameter to obtain the processed user audio stream (the target parameter is the gain control parameter corresponding to the user audio stream); thus, by using Hanning window or linear interpolation to gradually adjust the current gain to the target parameter, a smooth transition can be achieved; of course, Hanning window and linear interpolation are commonly used smoothing methods, and their principles will not be elaborated here.

[0067] Thus, based on the aforementioned method, after smoothing the gain of the other two audio streams, they can be merged into the sample-level mixing node for mixing processing. That is, when mixing the three processed audio streams, the sample-level mixing node is configured to perform the following operations: (1) weighted summation of the three processed audio streams to obtain the initial mixed audio stream; (2) adjusting the amplitude of the initial mixed audio stream to the [-1,1] interval (i.e., limiting the amplitude to the [-1,1] interval) to obtain the mixed audio stream.

[0068] Furthermore, this embodiment replaces the traditional ScriptProcessorNode with AudioWorkletNode to perform high-precision sample-level mixing and silence injection. Specifically: ScriptProcessorNode (an interface in the WebAudio API used for custom audio data processing during audio processing) runs on the main thread and is easily blocked by the UI, causing stuttering and popping sounds, so it has been deprecated; while AudioWorkletNode (an interface in the Web Audio API used to create a node in the audio processing graph, which can connect to other audio nodes and use custom audio processing logic) runs on an independent audio rendering thread, with low latency and stable timing, so it is used as the custom mixing processor, that is, the sample-level mixing node adopts the AudioWorkletNode node; at the same time, for example, mixing is performed in the independent audio rendering thread with a processing granularity of 128 sample frames; and gain switching is smoothed with a short window of 5-20ms to avoid clicking sounds.

[0069] Thus, after obtaining a single mixed audio stream, the MediaStreamAudioDestinationNode (a special node in the Web Audio API whose core function is to convert the audio processed by the Web Audio processing graph into a WebRTC-compatible mediaStream object, which can then be conveniently used for network transmission or local recording) can be used as the sole output endpoint, that is, converted into a single stream (mixedStream) conforming to the WebRTC specification.

[0070] In this embodiment, after obtaining the mixed audio stream, the local playback and echo cancellation unit, for example, is further used to encapsulate the AI ​​audio stream into an RTC-encoded audio frame and inject it into the reference signal channel of the acoustic echo cancellation module within WebRTC for caching, thereby completing the backup of the AI ​​audio stream. In this way, the AI ​​audio stream can be sent directly when the echo cancellation verification fails. Specifically, the RTCRTpScriptTransform interface (a relatively new low-level interface in the Web RTC API that allows you to directly process encoded audio or video data packets in the WebRTC connection by writing JavaScript code) can be used to encapsulate the PCM copy of the AI ​​audio stream into an RTCEncodedAudioFrame (which encapsulates the original PCM audio data (PulseCode Modulation, an uncompressed audio format) into an RTC-compliant encoded audio frame (RTCEencodedAudioFrame) for efficient transmission over the network and correct decoding and playback at the receiving end), and inject it into the reference signal channel of the WebRTC internal AEC module (acoustic echo cancellation module), thereby achieving data caching.

[0071] Based on this, after obtaining the mixed audio stream, caching it, and converting it into a single stream conforming to the WebRTC specification, data can be sent. The process is as follows: The WebRTC communication adapter unit is used to send the mixed audio stream to the remote end. In specific implementation, for example, the WebRTC communication adapter unit registers the single audio track of the mixedStream as a local media source through RTCPeerConnection.addTrack (which is the core method in the WebRTC API used to add local audio and video tracks to the connection and send them to the remote end. It is the "send" entry point for realizing real-time audio and video communication). This ensures that the remote receiving end is unaware of the single-stream format and is compatible with standard SIP / WebRTC gateways and SFU architecture (Selective Forwarding Unit, which is a commonly used architecture pattern in WebRTC audio and video conferencing, also known as star architecture, where audio and video streams are distributed through servers).

[0072] Furthermore, based on the aforementioned transmission scheme, this embodiment can be further optimized by performing echo cancellation verification. The process is as follows: the local playback and echo cancellation unit is also used to perform echo cancellation verification on the mixed audio stream before the WebRTC communication adapter unit sends the mixed audio stream to the remote end, and if the verification fails, read the RTC encoded audio frame from the reference signal channel and transmit it to the WebRTC communication adapter unit so that the WebRTC communication adapter unit can directly send the RTC encoded audio frame to the remote end.

[0073] In this embodiment, a test signal is first acquired, which can be a sinusoidal sweep or pseudo-random sequence in the AI ​​audio track, serving as a preset signal. Then, the test signal and the mixed audio stream are cross-correlation detected, i.e., the cross-correlation coefficient between the two is calculated. Next, a spectrum comparison is performed, i.e., the test signal and the mixed audio stream are respectively subjected to Fast Fourier Transform to obtain the test spectrum and the audio spectrum, so as to calculate the energy ratio of the test spectrum and the audio spectrum at a preset characteristic frequency. Then, the ERLR index of the mixed audio stream is calculated. Finally, the aforementioned cross-correlation coefficient, energy ratio, and ERLR index are weighted and summed to obtain a comprehensive evaluation value. Wherein, when the comprehensive evaluation value is less than a set value, the verification is determined to be passed; otherwise, the verification is determined to be failed. If the verification fails, the RTC encoded audio frame can be directly read from the aforementioned reference signal channel and sent, thereby ensuring that the audio played at the remote end has no echo effect.

[0074] In addition, to address the echo pollution caused by AI voice playback through local speakers, this embodiment also includes a playback path isolation mechanism, namely: The local playback and echo erasure unit is also used to obtain the AI ​​audio stream from the multi-source audio capture and preprocessing unit and play it locally through a playback channel independent of the local microphone acquisition path. In this embodiment, the AI ​​voice (aiStream) is only output through the Web Audio destination node (which is the final output node of the audio processing graph in the Web Audio API, representing the default audio output device of the current browser / operating system, such as a speaker or headphones. All audio signals connected to it will be sent to the physical audio device for playback so that the user can hear it). In this way, the AI ​​audio stream does not pass through any hardware feedback path that may be captured by the microphone. It is only output to the speaker through the standard Web Audio rendering pipeline, which bypasses the "Stereo Mix" or "Loopback" virtual device channel in the operating system audio route that may be captured by the microphone, thereby fundamentally cutting off the acoustic feedback loop.

[0075] Thus, AI echo can be completely eliminated through "adaptive echo filtering + playback isolation + AEC reference injection + send stream purification".

[0076] Finally, monitor the session status. When the user hangs up or the session times out, release MediaStreamTrack (all audio tracks), AudioContext (audio processing graph, i.e., shut down all audio data scheduling and rendering operations), WebSocket (disconnect from the AI ​​server) and other objects in sequence to prevent memory leaks and uncontrolled GPU and CPU resource usage.

[0077] Based on this, the system provided in this embodiment can be applied to an intelligent customer service chat system (user). AI Agent Applications include human agents, real-time AI translation and summary broadcasting in multi-party meetings, virtual anchor live streaming platforms (human voice + background music + AI interactive response), and three-way voice collaboration and assisted diagnosis in online education / telemedicine.

[0078] Thus, through the aforementioned detailed description of the Web-based intelligent voice interaction and mixing control system, this invention pioneers an integrated solution for the Web-based "three-track audio – state machine driven – echo elimination – single-stream output", overcoming the core challenges of multi-source voice fusion and echo suppression in the browser environment, constructing a new generation of intelligent voice interaction front-end audio hub paradigm, and possessing outstanding technical innovation, engineering feasibility and broad industrialization prospects.

[0079] like Figure 2 As shown, the second aspect of this embodiment provides a web-based intelligent voice interaction and mixing control method, wherein the method is executed based on the web-based intelligent voice interaction and mixing control system described in the first aspect of the embodiment, and the operation steps of the method may be, but are not limited to, the steps S1 to S6 below.

[0080] S1. Used to acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server.

[0081] S2. Perform adaptive echo filtering on the initial audio stream to obtain the user audio stream.

[0082] S3. Obtain system timer monitoring events and receive AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling, and system timer monitoring events, determine the current voice interaction state from several preset voice interaction states. Based on the current voice interaction state, determine the gain control parameters corresponding to each of the three audio streams. The gain control parameters are different in different voice interaction states. Each of the three audio streams has a corresponding gain control node.

[0083] S4. Based on the gain control parameters and gain control nodes corresponding to the three audio streams, perform smooth gain processing on the three audio streams to obtain three processed audio streams, so as to perform mixing processing on the three processed audio streams and obtain a mixed audio stream to be transmitted to the WebRTC communication adapter unit.

[0084] S5. Send the mixed audio stream to the remote end. S6. Obtain the AI ​​audio stream from the multi-source audio capture and preprocessing unit, and play it locally through a playback channel independent of the local microphone acquisition path.

[0085] The working process, working details and technical effects of the method provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0086] The third aspect of this embodiment provides a web-based intelligent voice interaction and mixing control device. Taking the device as an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the web-based intelligent voice interaction and mixing control method as described in the second aspect of this embodiment.

[0087] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0088] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0089] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0090] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the Web-based intelligent voice interaction and mixing control method described in the second aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the Web-based intelligent voice interaction and mixing control method as described in the second aspect of the embodiment.

[0091] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0092] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0093] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the Web-based intelligent voice interaction and mixing control method as described in the second aspect of this embodiment. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0094] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / audio> < / audio> < / audio> < / audio> < / audio>

Claims

1. A web-based intelligent voice interaction and mixing control system, characterized in that, include: The multi-source audio capture and preprocessing unit is used to acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server. The local playback and echo removal unit is used to perform adaptive echo filtering on the initial audio stream to obtain the user audio stream; The dialogue state machine controller maintains several voice interaction states, which are used to acquire system timer monitoring events and receive AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling and system timer monitoring events, the current voice interaction state is determined from several voice interaction states. Based on the current voice interaction state, the gain control parameters corresponding to the three audio streams are determined. The gain control parameters are different in different voice interaction states. The three-track dynamic mixer includes gain control nodes corresponding to the user audio stream, background audio stream, and AI audio stream, respectively. It is used to perform smooth gain processing on the three audio streams based on the gain control parameters and gain control nodes corresponding to the three audio streams, so as to obtain three processed audio streams. The three processed audio streams are then mixed to obtain a mixed audio stream that is transmitted to the WebRTC communication adapter unit. WebRTC communication adapter unit, used to send mixed audio streams to a remote location; The local playback and echo erasure unit is also used to acquire the AI ​​audio stream from the multi-source audio capture and preprocessing unit and play it locally through a playback channel independent of the local microphone acquisition path.

2. The intelligent voice interaction and mixing control system based on a Web terminal according to claim 1, characterized in that, When acquiring the AI ​​audio stream pushed by the AI ​​server, the multi-source audio capture and preprocessing unit is configured as follows: The incremental PCM segments pushed by the AI ​​server are received through a streaming interface, wherein the incremental PCM segments represent audio data returned by the AI ​​server. The incremental PCM segment is semantically segmented to obtain several audio cache blocks, wherein each audio cache block is associated with a segmentation sequence number, a start and end timestamp, and the corresponding text; Several audio cache blocks are stored in an indexable buffer according to their segmentation numbers to construct the AI ​​audio stream that supports random positioning.

3. The intelligent voice interaction and mixing control system based on a Web terminal according to claim 1, characterized in that, When adaptive echo filtering is applied to the initial audio stream to obtain the user audio stream, the local playback and echo erasure unit is configured as follows: Based on the AI ​​audio stream, a filtered reference signal is generated; Obtain the filter weight vector at the k-th iteration, where the initial value of k is 1. When k is 1, the filter weight vector at the k-th iteration is the initial vector, and the length of the filter weight vector is the same as the length of the filtered reference signal. Based on the filtered reference signal and the filter weight vector at the kth iteration, echo estimation is performed to obtain the echo signal. Calculate the error signal between the initial audio stream and the echo signal, wherein the error signal is the filtered initial audio stream; Determine if the iteration stopping condition is met; If not, then use the error signal to update the filter weight vector at the kth iteration to obtain the filter weight vector at the (k+1)th iteration. Increment k by 1 and reacquire the filter weight vector at the kth iteration until the iteration stopping condition is met, so that the error signal at the time of the iteration stopping condition is used as the user audio stream.

4. The intelligent voice interaction and mixing control system based on a web terminal according to claim 3, characterized in that, When performing echo estimation based on the filtered reference signal and the filter weight vector at the k-th iteration, the local playback and echo erasure unit is configured as follows: The echo signal is generated according to the following formula; ； In the formula, Indicates an echo signal. This represents the filter weight vector at the k-th iteration. Indicates the filter reference signal. Indicates the transpose operation; Accordingly, when updating the filter weight vector at the k-th iteration using the error signal, the local playback and echo erasure unit is configured as follows: The filter weight vector for the (k+1)th iteration is generated according to the following formula; ； In the formula, This represents the filter weight vector at the (k+1)th iteration. Indicates the update step size. This represents the error signal. For regularization parameters, Represents the magnitude of a vector.

5. The intelligent voice interaction and mixing control system based on a web terminal according to claim 1, characterized in that, Several voice interaction states include: The MIC_ACTIVE state indicates that the user is speaking. In the MIC_ACTIVE state, the gain control parameter of the user audio stream is 1, the gain control parameter of the background audio stream is 0.2 to 0.5, and the gain control parameter of the AI ​​audio stream is 0. The USER_END_SILENCE state represents the silence state after the user finishes speaking. In the USER_END_SILENCE state, the gain control parameters of the three audio streams are all 0, and the duration of the silence state is 50 milliseconds to 200 milliseconds. The AI_THINKING state indicates that the AI ​​server is in the request processing state. In the AI_THINKING state, the gain control parameters of the three audio streams are all 0. The AI_SPEAKING state indicates that the audio is in AI voice playback mode. In the AI_SPEAKING state, the gain control parameter of the user audio stream is 0, the gain control parameter of the background audio stream is 0.05 to 0.3, and the gain control parameter of the AI ​​audio stream is 1. The BBACKGROUND_ONLY state indicates an idle state with no interactive activity. In the BBACKGROUND_ONLY state, the gain control parameter for the background audio stream is 0.1 to 0.3, and the gain control parameter for the user audio stream and AI audio stream is 0.

6. The intelligent voice interaction and mixing control system based on a Web terminal according to claim 5, characterized in that, The AI ​​service signaling includes AI_THINKING signaling and AI_RESPONSE_READY signaling, and the system timer monitoring events include user voice interaction events and AI service signaling interaction events; When the current voice interaction state is determined from several voice interaction states based on events monitored by the user audio stream, AI service signaling, and system timers, the dialogue state machine controller is configured as follows: The user's audio stream is segmented into frames to obtain several audio frames; Calculate the short-time energy of each audio frame and determine the zero-crossing rate of each audio frame; When the short-term energy of several consecutive audio frames is higher than the energy threshold and the zero-crossing rate of several consecutive audio frames is within the preset range, it is determined that there is a voice start event in the user audio stream and the current voice interaction state is determined to be MIC_ACTIVE state; otherwise, the current voice interaction state is determined to be USER_END_SILENCE state. When in the USER_END_SILENCE state, if the AI_THINKING signaling is received, the system switches from the USER_END_SILENCE state to the AI_THINKING state, and uses the AI_THINKING state as the current voice interaction state. When in AI_THINKING state, if AI_RESPONSE_READY signaling is received and the first frame of audio data pushed by the AI ​​server is received, then switch from AI_THINKING state to AI_SPEAKING state, and use AI_SPEAKING state as the current voice interaction state. If no user voice interaction event or AI service signaling interaction event is detected within a preset time period, the system will directly switch to the BACKGROUND_ONLY state, making it the current voice interaction state; and In any state other than MIC_ACTIVE, if a voice start event is detected in the user's audio stream, the system switches from any state to MIC_ACTIVE, and uses MIC_ACTIVE as the current voice interaction state.

7. The intelligent voice interaction and mixing control system based on a web terminal according to claim 1, characterized in that, When determining the gain control parameters corresponding to each of the three audio streams, the smoothing gain time of each of the three audio streams is also determined. The three-track dynamic mixer also includes: sample-level mixing nodes. Specifically, when smoothing the gain of the three audio streams based on their respective gain control parameters and gain control nodes, the three-track dynamic mixer is configured as follows: For the user audio stream, obtain the smoothed gain time corresponding to the user audio stream; Based on the gain control node corresponding to the user audio stream, the amplitude of the user audio stream is adjusted to the target parameter using a Hanning window or linear interpolation within the smooth gain time to obtain the processed user audio stream. The target parameter is the gain control parameter corresponding to the user audio stream. Accordingly, when mixing the three processed audio streams, the sample-level mixing node is configured as follows: The three processed audio streams are weighted and summed to obtain the initial mixed audio stream. The amplitude of the initial mixed audio stream is adjusted to the range of [-1, 1] to obtain the mixed audio stream.

8. The intelligent voice interaction and mixing control system based on a web terminal according to claim 1, characterized in that, The local playback and echo cancellation unit is also used to encapsulate the AI ​​audio stream into RTC-encoded audio frames and inject them into the reference signal channel of the acoustic echo cancellation module in WebRTC for caching, so as to complete the backup of the AI ​​audio stream.

9. A web-based intelligent voice interaction and mixing control system according to claim 8, characterized in that, The local playback and echo removal unit is also used to perform echo cleanup verification on the mixed audio stream before the WebRTC communication adapter unit sends the mixed audio stream to the remote end, and if the verification fails, read the RTC encoded audio frame from the reference signal channel and transmit it to the WebRTC communication adapter unit so that the WebRTC communication adapter unit can directly send the RTC encoded audio frame to the remote end.

10. A web-based intelligent voice interaction and mixing control method, characterized in that, The method is implemented based on the Web-based intelligent voice interaction and mixing control system according to any one of claims 1 to 9, wherein the method includes: Acquire the initial audio stream captured by the user's microphone, the background audio stream from the background audio source, and the AI ​​audio stream pushed by the AI ​​server; Adaptive echo filtering is applied to the initial audio stream to obtain the user audio stream; The system acquires system timer monitoring events and receives AI service signaling sent by the AI ​​server. Based on the user audio stream, AI service signaling, and system timer monitoring events, it determines the current voice interaction state from several preset voice interaction states. Based on the current voice interaction state, it determines the gain control parameters corresponding to each of the three audio streams. The gain control parameters are different in different voice interaction states. Each of the three audio streams has a corresponding gain control node. Based on the gain control parameters and gain control nodes corresponding to the three audio streams, smooth gain processing is performed on the three audio streams to obtain three processed audio streams, which are then mixed to obtain a mixed audio stream that is transmitted to the WebRTC communication adapter unit. Send the mixed audio stream to a remote location; and The AI ​​audio stream is acquired from the multi-source audio capture and preprocessing unit and played locally through a playback channel independent of the local microphone acquisition path.

Images