Full-duplex chinese audio understanding method, device, apparatus and storage medium

Abstract

This application relates to a full-duplex Chinese audio understanding method, apparatus, device, and storage medium. The method includes: acquiring a continuous Chinese audio stream input by a user; inputting the continuous Chinese audio stream into a neural audio encoder, processing it through a hierarchical quantization mechanism, and using the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream; inputting the user audio temporal stream and the system audio temporal stream of the previous moment into a dual-stream temporal backbone network to obtain the system audio temporal stream of the next moment; using a streaming audio decoder to reconstruct the next moment audio temporal stream segment by segment into a continuous speech waveform; and outputting it in real time using a simultaneous generation and playback method. This method achieves simultaneous listening and output, forming a true full-duplex interaction, and can also realize real-time perception, semantic understanding, dialogue reasoning, and streaming speech response generation of Chinese speech without relying on strict round-robin segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio understanding, and relates to a full-duplex Chinese audio understanding method, device, equipment and storage medium. Background Art

[0002] Most current Chinese voice interaction systems follow the traditional cascading route of "ASR - text understanding - TTS". This method can work in short-question and short-answer scenarios, but there are obvious limitations in open-ended, multi-round, and strongly real-time interaction scenarios: (1) High interaction latency, making it difficult to form a natural conversation In the traditional cascading scheme, speech recognition, text understanding, text generation, and speech synthesis belong to different modules, and there is a serial dependency relationship between the modules, resulting in a relatively high end-to-end delay of the overall system. Existing research shows that such voice dialogue systems often have responses in the order of seconds, while the response time of natural human conversations is usually only in the order of hundreds of milliseconds.

[0003] In actual Chinese interaction scenarios, users often hope that the system can "respond while listening" instead of waiting to reply uniformly after a complete sentence ends. Therefore, the traditional architecture is difficult to meet the requirements of high real-time performance.

[0004] (2) Key non-text information in speech is lost Chinese speech understanding does not solely rely on the words themselves. The pauses, stresses, emotions, speech rate changes, filler words, laughter, hesitation sounds, and drawn-out sounds of users will all affect semantic interpretation. For example, the same sentence "Sure" may mean agreement, irony, or hesitation depending on the intonation. If the system only retains the transcribed text while discarding the original acoustic and prosodic information, it is likely to cause understanding deviations. Existing voice dialogue research also points out that when text becomes the only intermediate representation, paralinguistic information and non-speech information will be significantly lost.

[0005] (3) Unable to handle interruptions, overlaps, and echoes in real conversations Real Chinese spoken conversations are not strictly turn-based. Users may interrupt before the system has finished speaking, or may give short feedback such as "um", "yes", "that's not what I mean" while listening. Traditional turn-based voice systems assume that conversations consist of clear single-speaker segments, and this assumption is not suitable for natural conversation phenomena such as overlapping speech, interruptions,抢答, and echoes. Relevant research points out that overlapping speech is not uncommon in real spoken language, and systems based on strict turn boundaries are difficult to effectively model.

[0006] (4) The Chinese speech scenario has stronger uncertainty Compared to phonetic languages ​​like English, Chinese faces the following challenges in voice interaction: there are more homophones, polyphonic characters, and neutral tones; tone changes have a significant effect on semantic differentiation; and omissions, jumps, and interjections frequently occur in spoken language. Summary of the Invention

[0007] To address the problems existing in the above-mentioned traditional methods, the present invention proposes a full-duplex Chinese audio understanding method, apparatus, device, and storage medium.

[0008] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a full-duplex Chinese audio understanding method is provided, including the following steps: Step 1: Obtain the continuous Chinese audio stream input by the user.

[0009] Step 2: Input the continuous Chinese audio stream into the neural audio encoder, process it through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream.

[0010] Step 3: Input the user audio time-series stream and the system audio time-series stream from the previous time step into the dual-stream temporal backbone network to obtain the system audio time-series stream for the next time step. The dual-stream temporal backbone network is used to encode the user audio time-series stream, the system audio time-series stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, and then processed using a gating mechanism and a joint context representation layer to obtain the joint context representation. Semantic planning is performed based on the joint context representation to obtain the semantic planning result. Based on the semantic planning result and the joint context representation, prediction is performed to obtain the low-level acoustic token for the next time step. The dual-stream temporal backbone network introduces a streaming caching mechanism to maintain the historical cache state at each time step.

[0011] Step 4: Use a streaming audio decoder to reconstruct the audio time-series stream of the next moment into a continuous speech waveform segment by segment; and output it in real time by generating and broadcasting it simultaneously.

[0012] On the other hand, a full-duplex Chinese audio understanding device is also provided, including: The continuous Chinese audio stream acquisition module is used to acquire the continuous Chinese audio stream input by the user.

[0013] The discrete encoding module is used to input continuous Chinese audio streams into the neural audio encoder, process them through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream.

[0014] Dual-stream full-duplex joint modeling is used to input the user audio time-series stream and the system audio time-series stream from the previous time step into the dual-stream temporal backbone network to obtain the system audio time-series stream from the next time step. The dual-stream temporal backbone network is used to encode the user audio time-series stream, the system audio time-series stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, and then processed using a gating mechanism and a joint context representation layer to obtain a joint context representation. Semantic planning is performed based on the joint context representation to obtain the semantic planning result. Based on the semantic planning result and the joint context representation, prediction is performed to obtain the low-level acoustic token for the next time step. A streaming caching mechanism is introduced internally into the dual-stream temporal backbone network to maintain the historical cache state at each time step.

[0015] The audio restoration module is used to restore the audio time-series stream of the next moment into a continuous speech waveform segment by segment using a streaming audio decoder; and to output it in real time by generating and broadcasting it simultaneously.

[0016] In another aspect, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-described full-duplex Chinese audio understanding methods.

[0017] Furthermore, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above-described full-duplex Chinese audio understanding methods.

[0018] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned full-duplex Chinese audio understanding method, apparatus, device, and storage medium include the following steps: acquiring a continuous Chinese audio stream input by a user; inputting the continuous Chinese audio stream into a neural audio encoder, processing it through a hierarchical quantization mechanism, and using the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream; inputting the user audio temporal stream and the system audio temporal stream of the previous moment into a dual-stream temporal backbone network to obtain the system audio temporal stream of the next moment; using a streaming audio decoder to reconstruct the next moment audio temporal stream segment by segment into a continuous speech waveform; and outputting it in real time using a simultaneous generation and playback method. This method achieves simultaneous listening and output, forming a true full-duplex interaction, and can also realize real-time perception, semantic understanding, dialogue reasoning, and streaming speech response generation of Chinese speech without relying on strict round-robin segmentation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a traditional voice system and a full-duplex voice dialogue system in one embodiment, wherein... Figure 1 (a) is a schematic diagram of a traditional speech system. Figure 1 (b) is a schematic diagram of a full-duplex voice dialogue system; Figure 2 This is a flowchart illustrating a full-duplex Chinese audio understanding method in one embodiment; Figure 3 This is a schematic diagram of the modeling process for a full-duplex speech understanding model in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0023] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] In one embodiment, such as Figure 1 As shown, a full-duplex Chinese audio understanding method is provided, which may include the following processing steps 1 to 4: Step 1: Obtain the continuous Chinese audio stream input by the user.

[0026] Specifically, this method uses continuous Chinese audio streams as the core input. Instead of separating "recognition, understanding, response, and broadcasting" into independent post-processing modules, it maps the user audio stream, the system's own output stream, and the intermediate semantic representations into the same temporal modeling framework for joint learning.

[0027] Step 2: Input the continuous Chinese audio stream into the neural audio encoder, process it through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream.

[0028] Specifically, the neural audio encoder is mainly responsible for converting continuous Chinese audio streams into discrete high-level semantic tokens and low-level acoustic tokens. The dual-stream temporal backbone network is used to jointly model the external input audio and the system audio temporal stream of the previous time step at the same time, thereby realizing full-duplex speech interaction that can receive, understand, predict, and generate simultaneously.

[0029] The front end of the full-duplex Chinese audio understanding model is a neural audio encoder, which maps the waveform of the input continuous Chinese audio stream into a discrete token sequence suitable for large-scale model processing. This neural audio encoder adopts the classic Wav2vec network structure, consisting of a low-level convolutional feature extraction module, a contextual representation modeling module, and a discrete quantization module. First, the continuous audio waveform undergoes local acoustic feature extraction through a multi-layer one-dimensional convolutional network, compressing the original sampling point sequence into a frame-level latent vector representation. This process preserves low-level acoustic information such as short-term spectral changes, articulation dynamics, pitch variations, and energy distribution in speech. Subsequently, the frame-level latent vector is fed into a Transformer or temporal context network for long-distance dependency modeling, enabling the model to understand the content of the current speech frame by combining the preceding and following audio context, capturing the relationships between continuous syllables, tone changes, pause boundaries, and semantic segments in Chinese speech. Finally, the encoder converts the continuous latent representation into a discrete token sequence through vector quantization or discrete codebook mapping mechanisms, making it a standard input to the dual-stream temporal backbone network.

[0030] In terms of encoding results, the neural audio encoder outputs high-level semantic tokens and low-level acoustic tokens through a hierarchical quantization mechanism. High-level semantic tokens, articulatory units, and sentence-level expressive information help subsequent models determine the user's current intent. Low-level acoustic tokens primarily preserve speech content and semantic trends, mainly retaining prosody, pitch, timbre, speech rate, pauses, emotional tone, and speaker-related acoustic details to support naturalness and coherence in subsequent speech generation. Thus, the original audio is compressed into a unified discrete representation before entering the backbone network, reducing the difficulty of long-term modeling and providing standardized input for subsequent two-stream joint modeling.

[0031] Step 3: Input the user audio time-series stream and the system audio time-series stream from the previous time step into the dual-stream temporal backbone network to obtain the system audio time-series stream for the next time step. The dual-stream temporal backbone network is used to encode the user audio time-series stream, the system audio time-series stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, and then processed using a gating mechanism and a joint context representation layer to obtain the joint context representation. Semantic planning is performed based on the joint context representation to obtain the semantic planning result. Based on the semantic planning result and the joint context representation, prediction is performed to obtain the low-level acoustic token for the next time step. The dual-stream temporal backbone network introduces a streaming caching mechanism to maintain the historical cache state at each time step.

[0032] Specifically, following the neural audio encoder, the full-duplex Chinese audio understanding model employs a dual-stream temporal backbone network. This dual-stream temporal backbone network can adopt a Transformer-based autoregressive structure or a temporal attention structure suitable for streaming processing. The core of this dual-stream temporal backbone network lies in simultaneously receiving two token sequences: the user input stream and the system output stream, and synchronously modeling the information from both streams.

[0033] At time T, the dual-stream temporal backbone network receives the current user audio temporal stream, encoded by a neural audio encoder from the continuous Chinese audio stream input by the user, and the system audio temporal stream generated or being played in the previous time step. Both tokens contain high-level semantic tokens and low-level acoustic tokens. Therefore, the dual-stream temporal backbone network can not only understand what the user is currently saying, but also perceive what the system itself said in the previous time step, whether it is still speaking, and whether it needs to continue outputting speech in the next time step.

[0034] In this dual-stream temporal backbone network, some parameters are used to model the contextual changes of the user's input speech, while others are used to model the dynamic state of the system's output or to-be-output content. A shared representation space is also set up to align and fuse the two temporal information streams. In this way, the model can continuously track what the user is currently speaking, and also perceive what the system has already said and what it is about to say, thus supporting a full-duplex interactive mode that allows for simultaneous listening and speaking.

[0035] The dual-stream parallel full-duplex dialogue architecture supports overlapping speech, interruptions, and echoing. By modeling the user audio stream and the system audio stream separately, it removes the dependence on strict speaking turn boundaries, enabling the model to naturally handle interruptions, interruptions, short echoes, and simultaneous listening and answering in real Chinese dialogues. Compared to traditional single-stream or turn-segmentation schemes, this approach is more suitable for real-world voice interaction environments.

[0036] Furthermore, a streaming caching mechanism is introduced within the dual-stream temporal backbone network. For each input token, the model does not need to wait for the complete speech to finish before inference; instead, it performs incremental computation based on the historical cache state, the current user audio temporal stream, and the current system audio temporal stream. The historical cache stores user-side context information and system-side context information, enabling the model to continuously predict the system audio temporal stream at time T+1 (including the system-generated low-level acoustic token and high-level semantic token at time T+1) with low latency. This design avoids the serial process of "first complete recognition, then complete generation" in traditional speech understanding systems, enabling the model to have real-time auditory perception and real-time speech generation capabilities. The purpose of this mechanism is to reduce redundant computation costs and system response latency, allowing the model to continuously update the context state in the continuous audio stream and predict the system audio temporal stream at time T+1 with low latency. This design avoids the serial process of "first complete recognition, then complete generation" in traditional speech understanding systems, enabling the model to have real-time auditory perception and real-time speech generation capabilities.

[0037] User flow history state is used to record the audio token context that the user has entered up to the current moment; history cache state is used to record the audio token context that the system has played or is about to play; shared semantic state cache is used to save the currently identified user intent, completed response content, incomplete tasks, correction information, and interruption markers; dialogue control state is used to indicate whether the current system is in a control mode of continuing to play, waiting for confirmation, fast replanning, or output pause.

[0038] After constructing the discrete representation of Chinese audio, the system enters the dual-stream full-duplex joint modeling stage. Unlike traditional single-stream speech modeling, this invention does not mix all audio from the entire dialogue into a single sequence for unified processing. Instead, it constructs two synchronously advancing temporal streams: the user audio temporal stream represents the sequence of Chinese speech tokens currently input by the user, and the system audio temporal stream represents the sequence of Chinese speech tokens that the system has already output or plans to output. At each time step, the model simultaneously receives the context states of the two temporal streams and predicts subsequent tokens based on the current dual-stream state. Since the user stream and the system stream exist in parallel within the same temporal framework, the system naturally possesses the ability to listen and speak simultaneously, without needing to wait for the user to finish speaking before starting recognition and response, as is the case with traditional solutions. In other words, the Chinese joint discrete representation generated in the previous step is fed into the user stream and the system stream respectively, enabling the model to synchronously model its own output process while continuously receiving user audio, thereby establishing true full-duplex interactive capability.

[0039] To ensure the continuity of the dual-stream modeling process, the system synchronously maintains four types of state information at each time step: user flow history state, system flow history state, shared semantic state cache, and dialogue control state. The user flow history state records the audio token context input by the user up to the current time; the system flow history state records the audio token context that the system has played or intends to play; the shared semantic state cache stores intermediate semantic results such as currently identified user intent, completed responses, incomplete tasks, correction information, and interruption markers; and the dialogue control state indicates whether the system is currently in a control mode such as continue playing, waiting for confirmation, rapid replanning, or paused output. Therefore, the shared semantic state cache formed in this step is not merely auxiliary information, but rather the direct input foundation for subsequent semantic alignment and implicit planning modules to conduct semantic organization.

[0040] The semantic planning result is the high-level semantic token in the system audio time-series stream at time T+1.

[0041] Step 4: Use a streaming audio decoder to reconstruct the audio time-series stream of the next moment into a continuous speech waveform segment by segment; and output it in real time by generating and broadcasting it simultaneously.

[0042] Specifically, the predicted low-level acoustic tokens and high-level semantic tokens for the next moment are fed into a streaming audio decoder to be reconstructed into a continuous speech waveform segment by segment. In other words, the semantic planning head is responsible for deciding "what to say," while the audio generation head is responsible for deciding "how to say it." The two work together to generate the system's real-time response.

[0043] The aforementioned full-duplex Chinese audio understanding method includes: acquiring a continuous Chinese audio stream input by the user; inputting the continuous Chinese audio stream into a neural audio encoder, processing it through a hierarchical quantization mechanism, and using the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream; inputting the user audio temporal stream and the system audio temporal stream of the previous time step into a dual-stream temporal backbone network to obtain the system audio temporal stream of the next time step; using a streaming audio decoder to reconstruct the audio temporal stream of the next time step into a continuous speech waveform segment by segment; and outputting it in real time using a simultaneous generation and playback method. This method achieves simultaneous listening and output, forming a true full-duplex interaction, and can also realize real-time perception, semantic understanding, dialogue reasoning, and streaming speech response generation of Chinese speech without relying on strict round-robin segmentation.

[0044] This method can be applied to the following types of intelligent audio and video systems: (1) Real-time voice assistant system: such as in-vehicle voice assistant, mobile phone smart assistant, office voice assistant, smart home central control system, etc., which require fast response, interruptibility and continuous dialogue; (2) Chinese voice customer service and companionship system: such as e-commerce customer service, financial consultation, medical guidance, psychological companionship, education Q&A, etc., which require understanding the user's semantic and tone changes during the interaction process; (3) Multi-turn voice agent system: such as a voice intelligent agent with task planning, tool calling and long-term memory, which needs to continuously listen to user commands and support listening and executing at the same time; (4) Chinese voice interaction system in complex environments: such as in a car, conference room, public space, or in noisy or multi-person conversation environments, which requires strong anti-overlapping voice and anti-interference capabilities.

[0045] In one embodiment, step 2 includes: decomposing a continuous Chinese audio stream into interconnected two-layer discrete representations using a multi-level residual quantization structure; wherein the two-layer discrete representations include: a high-level semantic token and a low-level acoustic token; the high-level semantic token is used to represent syllable-level, word-level, and phrase-level semantic content and speaking intention trends; the low-level acoustic token is used to represent timbre, prosody, speech rate, pitch, pauses, and ambient sounds in paralinguistic features; introducing Chinese-specific speech enhancement features during the audio encoding stage; wherein the Chinese-specific speech enhancement features include: fundamental frequency contour and its dynamic change trend, syllable end duration and pause patterns, tone sandhi and neutral tone markings, retroflexion features, and prosodic labels corresponding to interjections and exclamations; forming a user audio temporal stream based on the Chinese-specific speech enhancement features, the high-level semantic token, and the low-level acoustic token.

[0046] Specifically, after obtaining the normalized audio segments, the system inputs the continuous Chinese audio stream into the neural audio encoder to generate a discrete speech token sequence for unified modeling by the subsequent large model. Since the length of the original waveform sequence is too long, directly inputting it into the large model will cause excessive computational costs, overemphasis on temporal dependencies, and insufficient real-time performance. Therefore, the present invention does not directly process the original waveform but uses a hierarchical discretization mechanism to compressively represent Chinese audio.

[0047] The user audio time series stream is a time series discrete token sequence, that is, the joint discrete representation of Chinese audio.

[0048] The neural audio encoder decomposes the input continuous Chinese audio stream into two types of interrelated discrete representations through a multi-level residual quantization structure: one type is high-level semantic tokens, which are used to represent syllable-level, word-level, phrase-level semantic content, as well as the trend of speaking intentions; the other type is low-level acoustic tokens, which are used to represent paralinguistic features such as timbre, rhythm, speech rate, pitch, pause, and ambient sound.

[0049] Through this two-layer discrete representation, in subsequent steps, it is possible to use high-level semantic tokens to judge "what the user said" and low-level acoustic tokens to judge "how the user said it", thereby avoiding the loss of rhythm and emotion information in pure text processing.

[0050] To make the discrete representation more suitable for the Chinese speech scenario, this method further introduces Chinese-specific speech enhancement features in the audio encoding stage. Specifically, it includes: fundamental frequency contour and its dynamic change trend, syllable tail duration and pause pattern, marked tones of liaison and weak stress, erhua features, as well as prosodic labels corresponding to modal particles and interjections. The above enhancement information will form the joint discrete representation of Chinese audio together with high-level semantic tokens and low-level acoustic tokens, that is, the user audio time series stream. In this way, in the subsequent semantic understanding stage, the system can use the joint discrete representation to distinguish Chinese expressions such as "Yes", "Yes?", "Yes, indeed", "Yes, perhaps" that are similar on the surface but have significantly different semantic attitudes. That is to say, the joint discrete representation generated in this step not only exists as the compression result of speech content but will also directly serve as the basic input for the subsequent two-stream full-duplex modeling module.

[0051] In one embodiment, the dual-stream temporal backbone network includes: a user stream coding branch, a system stream coding branch, a cross-stream attention layer, a gating fusion layer, a joint context representation, a semantic planning head, and an audio generation head; the user stream coding branch includes a user stream embedding layer and a user stream causal Transformer layer; the system stream coding branch includes a system stream embedding layer and a system stream causal Transformer layer; step 3 includes: inputting the user audio temporal stream into the user stream coding branch, processing it through the user stream embedding layer to obtain a user stream embedding representation, and encoding the user stream embedding representation and the user-side context information stored in the historical cache using the user stream causal Transformer layer to obtain a higher-level user stream representation; inputting the system audio temporal stream from the previous time step into the system stream coding branch, processing it through the system stream embedding layer to obtain a system stream embedding representation, and encoding the system stream embedding representation and the user-side context information stored in the historical cache using the user stream causal Transformer layer to obtain a higher-level user stream representation; and .... The saved system-side context information is encoded using a system flow causal Transformer layer to obtain a high-level representation of the system flow. The high-level representations of the user flow and the system flow are then input into a cross-flow attention layer to obtain user-side and system-side enhanced representations. These are then input into a gated fusion layer to obtain a fused dual-stream state representation. The fused dual-stream state representation is then input into a joint context representation layer to obtain a joint context representation. This joint context representation includes: the semantic content and acoustic state of the user's current input audio, the semantic progress and acoustic continuity of the system's historical output speech, and the interaction between the user flow and the system flow. The joint context representation is then input into a semantic planning head to obtain the semantic planning result. Finally, the joint context representation and the semantic planning result are input into an audio generation head to obtain the low-level acoustic token of the system at the next time step.

[0052] Specifically, the dual-stream temporal backbone network can be configured with user stream coding branches, system stream coding branches, cross-stream attention layers, gating fusion layers, joint context representation layers, semantic planning heads, and audio generation heads. The overall processing logic can be summarized as follows: first, the temporal states of the user stream and system stream are modeled separately; then, the two streams are aligned through cross-stream interaction; subsequently, key information is selected and fused through a gating mechanism, ultimately forming a joint context representation used to predict the output token for the next time step.

[0053] The gated fusion layer is implemented using MLP, the joint context representation layer is implemented using a normal transformer module, and the semantic planning head and audio generation head can both be implemented using the multi-head attention mechanism in transformer.

[0054] First, the input to the user stream encoding branch is the user audio temporal stream at the current time or within the current time window, i.e., the semantic and acoustic tokens obtained from the external input audio through the neural audio encoder. After the user audio temporal stream enters the user stream embedding layer, it is mapped into a hidden vector of a uniform dimension, and then overlaid with temporal position encoding, token type encoding, and stream type encoding. The purpose of this step is to convert discrete tokens into continuous representations that can be computed by the Transformer network, while distinguishing whether the token belongs to the user stream, a semantic token, or an acoustic token. The output of the user stream embedding layer is the initial hidden state on the user side, which is then input to the user stream causal Transformer layer.

[0055] The user flow causal Transformer layer is primarily used to model the temporal context of user input speech. Its input consists of the hidden state output from the user flow embedding layer and user-side context information stored in the history cache. The output is a high-level representation of the user flow that incorporates historical speech context. Due to the causal attention structure, this layer can only focus on current and historical user input tokens at the current moment, and cannot use future tokens, thus ensuring the model can perform streaming inference. The purpose of this layer is to capture user speech content, pronunciation rhythm, intonation changes, pause boundaries, and speaking state, enabling the model to form preliminary semantic judgments based on existing fragments even before the user has finished speaking a sentence.

[0056] Secondly, the input to the system stream encoding branch is the system audio temporal stream that the system has generated or is currently playing at the previous moment, as well as the system-side context information stored in the historical cache (i.e., the system audio temporal stream generated before the current moment (containing the system-side high-level semantic token and low-level acoustic token) after passing through the system stream embedding layer). This input also contains the system-side high-level semantic token and low-level acoustic token. After the system audio temporal stream of the previous moment enters the system stream embedding layer, it is mapped to a system-side hidden vector of a unified dimension, and time position encoding, token type encoding, and stream type encoding are also superimposed; the output of the system stream embedding layer is the system-side initial hidden state, which is concatenated with the system-side context information stored in the historical cache and then input into the system stream causal Transformer layer.

[0057] The System Flow Causal Transformer layer is primarily used to maintain the system's own output memory and vocal state. Its inputs are the hidden states of the System Flow embedding layer and the system-side context information stored in the history cache; its output is the high-level representation of the System Flow. This layer focuses on what the system expressed in the previous moment, whether the current response is still continuing, whether the semantic output progress is complete, and whether the acoustic performance needs to continue the timbre, rhythm, and pause patterns of the previous moment. Through this branch, the model can avoid problems such as semantic repetition, content jumps, prosodic breaks, or inconsistent vocal states during continuous generation.

[0058] Following the two causal Transformer branches, the model incorporates a cross-stream attention layer. This layer takes as input high-level representations of the user stream and the system stream, and outputs enhanced representations of both the user and system streams after the two-stream interaction. Its purpose is to align the user input state and the system output state within the same temporal space, enabling the model to simultaneously understand "what the user is saying" and "what the system is currently saying." Specifically, the user-side enhanced representation can perform cross-attention calculations using the system stream representation as the key and value, thereby perceiving the system's current vocal content and output progress. Similarly, the system-side enhanced representation can perform cross-attention calculations using the user stream representation as the key and value, thereby perceiving in real-time whether the user continues speaking, interrupts, asks new questions, or breaks the current response.

[0059] Following the cross-stream interaction, the model further dynamically fuses the two streams of information through a gated fusion layer. The input to the gated fusion layer consists of the user-side enhanced representation and the system-side enhanced representation output from the cross-stream attention layer, and its output is the fused dual-stream state representation. The purpose of this layer is to automatically determine whether the model should focus more on user input or system output based on the current dialogue state. For example, when the user is still speaking, the gated fusion layer increases the weight of the user stream information, allowing the model to prioritize listening and understanding; when the user pauses and the system needs to generate a response, the gated fusion layer increases the weight of system stream and semantic planning-related information, putting the model into a response generation state; when the user inserts new speech during system output, the gated fusion layer re-increases the weight of the user stream, thereby supporting interruption recognition, interruption handling, and response adjustment.

[0060] Following the gated fusion layer is a joint context representation layer. This layer takes the gated fusion state of the two streams as input and outputs a unified context vector (i.e., the joint context representation) for the current time step. This joint context representation contains three types of information: first, the semantic content and acoustic state of the user's current audio input; second, the semantic progress and acoustic continuity of the system's historical output speech; and third, the interaction between the user stream and the system stream, such as whether the current state is waiting, responding, interrupted, continuing generation, or pausing speech. This joint context representation is the core input to the subsequent prediction module, determining which token the system should output at time T+1.

[0061] The semantic planning head takes a joint context representation as input and outputs the semantic planning result, i.e., the internal semantic representation of the next time step, including Chinese text tokens, compressed semantic tokens, structured intent labels, or dialogue control tokens. The semantic planning head does not directly output the final speech; instead, it first forms an intermediate semantic trajectory for the current time step based on the joint context representation, used to explicitly describe what the model "understands" and "what it plans to express next." For example, when the user continues speaking, the semantic planning head can output a control signal to wait or continue listening; after the user pauses and expresses a complete intent, the semantic planning head can output the semantic token corresponding to the response content; when a user interruption is detected, the semantic planning head can replan the response direction. Therefore, this structure adds a layer of semantic constraint between acoustic modeling and speech generation, making the subsequent generation process based not only on local acoustic continuity but also on controlled generation based on explicit semantic goals. The semantic planning head is responsible for deciding "what to say," and the audio generation head is responsible for deciding "how to say it." The two are interconnected and work together to complete the real-time response generation of the system. Therefore, this structure is equivalent to adding a layer of semantic constraints between acoustic modeling and speech generation, so that the subsequent generation process of the model is not only based on local acoustic continuity, but also on controlled generation based on explicit semantic goals.

[0062] In the audio generation head, the input is the concatenation of the joint context representation and the semantic planning result output by the semantic planning head, and the output is the low-level acoustic token of the system flow at the next time step.

[0063] After the dual-stream temporal backbone network outputs the high-level semantic token and low-level acoustic token for the next time step, the low-level acoustic token is further input into the streaming audio decoder, where it is reconstructed frame by frame into the output audio waveform, and continues to participate in modeling as part of the system stream input for the next round. Through this closed-loop structure, the model can simultaneously complete listening, understanding, planning, and speaking in a continuous Chinese audio stream, forming a truly full-duplex Chinese audio understanding and generation mechanism oriented towards real-time voice interaction.

[0064] In one embodiment, before step 4, the system further includes: dynamically processing interruptions, interjections, and short feedback in real spoken dialogue phenomena, implementing interruption decisions based on the dialogue control state, and storing the interruption decisions in a shared semantic state cache; specifically, when the system is in the output state and the user starts speaking midway, the system quickly identifies the new input content by combining the energy changes, prosodic changes, and semantic importance in the user's current audio stream; if the system detects that the voice energy and semantic relevance of the user stream increase simultaneously, the system determines that the input has a high interaction priority and dynamically increases the weight of the user stream in subsequent joint modeling; if it is determined to be an echoing short feedback, the system switches to a fast replanning mode and locally adjusts the system output while retaining the original main response target; if it is determined to be background noise, echo, or meaningless short disturbance, the system maintains the original system stream and continues to output without triggering the main response interruption; through the processing mechanism, the system implements interruption decisions based on the dialogue control state maintained in the previous time step and stores the results of the interruption decisions in a shared semantic state cache.

[0065] Specifically, based on the dual-stream state maintenance mechanism, the system further realizes dynamic processing of real spoken dialogue phenomena such as interruptions, interjections, and short feedback. When the system is in the output state, if the user starts speaking midway, the model will not simply interrupt the entire process and restart. Instead, it will first combine the energy changes, prosodic changes, and semantic importance in the user's current audio stream to quickly identify the new input content. If the system detects that the speech energy and semantic relevance of the user stream increase simultaneously, it will determine that the input has a high interaction priority and dynamically increase the weight of the user stream in subsequent joint modeling. If it is judged to be an echoing short feedback such as "um," "yes," "no," or "wait a minute," the system will not completely terminate the current main response but will switch to a fast replanning mode, locally adjusting the system output while retaining the original main response objective. If it is judged to be background noise, echo, or meaningless short disturbances, the system will maintain the original system stream and continue to output without triggering the main response interruption. Through the above mechanism, the system realizes the interruption decision based on the dialogue control state maintained in the previous step, and the result of the interruption decision will be further written back to the shared semantic state cache, providing updated contextual conditions for subsequent semantic planning and speech generation.

[0066] Considering that the system speaker output may re-enter the microphone via spatial reflection in a real-world deployment environment, this invention also incorporates robust designs for overlapping speech and echo interference during dual-stream modeling. Specifically, during the training phase, the system constructs data samples simulating echoes, speech with varying degrees of overlap, background noise of different intensities, and time-shifted perturbations, enabling the model to learn to distinguish between user streams and system streams under complex sound field conditions. After this training, the model can effectively identify which audio segments originate from actual user speech and which are merely environmental refeedback of the system's own speech during online inference, by combining historical cache states with the current user stream input, thereby avoiding self-activation, false interruptions, or misidentification.

[0067] In one embodiment, inputting the joint context representation into the semantic planning head to obtain the semantic planning result includes: inputting the joint context representation into the semantic planning head, processing it using a multi-head attention mechanism, and obtaining the semantic planning result.

[0068] In one embodiment, the method further includes performing Chinese scene enhancement processing before the semantic planning result is input into the audio generation head. The specific process of Chinese scene enhancement processing includes: performing tone and homonym correction using joint information of high-level semantic tokens, low-level prosodic tokens, and contextual semantic states; performing incremental Chinese sentence segmentation and punctuation restoration in real time during streaming processing; using the results of sentence segmentation and punctuation restoration to control the pause length and tone rhythm during final broadcast; and jointly modeling Chinese tone and emotion. Specifically, for input with obvious emotional coloring, not only is its text semantics analyzed, but also the speaker's emotional trend is extracted by combining pitch changes, rhythm patterns, and acoustic intensity, and the emotional trend is written into the internal semantic planning result as a control signal so that the subsequently generated system speech can adaptively adjust to a soothing, confirmatory, explanatory, or prompting style.

[0069] Specifically, after completing semantic alignment and implicit planning, the system will also perform Chinese scenario enhancement processing on the results to further adapt to the real Chinese application environment. First, the system uses the combined information of high-level semantic tokens, low-level prosodic tokens, and context semantic states to perform tone and homophone ambiguity correction. For example, when a syllable may correspond to multiple Chinese characters or multiple semantic interpretations, the system will constrain the candidate interpretations based on the current task context, speaking rhythm, and historical conversation content, so as to correctly merge similar-sounding expressions such as "bank / yinhang" and "fact / current affairs". Secondly, the system performs incremental Chinese sentence segmentation and punctuation restoration in real time during the streaming process. Instead of waiting for the end of the entire sentence, it synchronously identifies the short sentence boundaries, pause points, interrogative mood, and emphasis positions during the continuous generation process. The results of sentence segmentation and punctuation restoration will, on the one hand, improve the structural clarity of the internal semantic planning results, and on the other hand, be directly fed back to the subsequent speech decoding module to control the pause length and tone rhythm during the final broadcast.

[0070] In addition, the system also jointly models Chinese tone and emotion. For inputs with obvious emotional colors such as "a little anxious", "very aggrieved", "not very sure", and "quite happy", the model not only analyzes the text semantics, but also combines pitch changes, rhythm patterns, and acoustic intensity to extract the emotional trend of the speaker, and writes this trend as a control signal into the internal semantic planning results, so that the subsequent generated system voice can be adaptively adjusted to an appeasing, confirmatory, explanatory, or prompt style.

[0071] In one embodiment, step 4 includes: using a streaming audio decoder to gradually restore the low-level acoustic tokens and high-level semantic tokens of the next moment into a continuous speech waveform; the continuous speech waveform continues to participate in the modeling as part of the next round of system stream input; the continuous speech waveform is output in real time in a way of generating and broadcasting simultaneously; specifically including: when the accumulated continuous speech waveform reaches the preset broadcast threshold, the first segment of the broadcast is immediately started to enable the user to obtain interactive feedback, and the system retains the interruptible synthesis cache; if it is detected that the user interrupts during the broadcast, the dialogue control state will trigger cache termination or partial regeneration, enabling the system to quickly switch to a new response path without interrupting the overall interactive coherence; during the streaming output process, a low-latency continuous update mechanism is also executed; at the same time, a risk control output strategy is introduced. When the system determines that the current internal semantic confidence is insufficient or the user's intention is not yet fully clear, it does not directly output conclusive content, but preferentially generates a transitional Chinese response.

[0072] Specifically, the purpose of the audio generation head is to transform the "semantic content to be expressed" into an "acoustic representation that can be reconstructed by a streaming audio decoder." In terms of processing logic, the audio generation head determines the next response content based on the semantic planning results and, combined with the acoustic state in the joint context representation, predicts the corresponding prosody, pitch, speech rate, pauses, and timbre continuity information. In other words, the semantic planning head is responsible for deciding "what to say," while the audio generation head is responsible for deciding "how to say it." The two work together to complete the system's real-time response generation.

[0073] After completing comprehension, planning, and Chinese language enhancement, the system enters the streaming speech decoding and output control stage. At this point, a semantically constrained sequence of system audio tokens has been generated. The streaming audio decoder reconstructs this sequence segment by segment into a continuous speech waveform and outputs it in real time using a simultaneous generation and playback method. To avoid the high latency of traditional solutions that require waiting for the entire sentence to be generated before playback, this invention employs a minimum playable segment triggering mechanism. When the accumulated generated audio tokens reach a preset playable threshold, the system immediately initiates the first segment playback, allowing the user to receive interactive feedback as quickly as possible. Simultaneously, the system retains an interruptible synthesis buffer; that is, subsequent speech segments that have been generated but not yet played are not permanently stored but dynamically managed according to the dual-stream control state. If a user interruption is detected during playback, the aforementioned dialogue control state triggers buffer termination or partial regeneration, allowing the system to quickly transition to a new response path without interrupting the overall interactive continuity.

[0074] During streaming output, the system also executes a low-latency continuous update mechanism. That is, within each short time window, the system re-examines the current user stream state, system stream state, and shared semantic state cache to determine whether fine-tuning of unreleased content is necessary. Thus, voice output is no longer a static linear process, but a dynamic generation process coupled in real-time with the listening process and continuously corrected. Simultaneously, to reduce the risk of erroneous output, this invention also introduces a risk control output strategy. When the system determines that the current internal semantic confidence is insufficient, or the user's intent is not yet fully clear, it does not directly output conclusive content, but instead prioritizes generating transitional Chinese responses such as "Let me confirm," "You mean…?" and "I understand…". This strategy, on the one hand, maintains the fluency of interaction, and on the other hand, provides a time window for further confirmation of the user's true intent.

[0075] In one embodiment, such as Figure 3 As shown, the full-duplex Chinese audio understanding model includes: a neural audio encoder, a dual-stream temporal backbone network, and an audio generation head; enabling the system to simultaneously complete Chinese audio perception, context modeling, semantic understanding, and streaming speech generation within the same model framework.

[0076] In terms of training implementation, this method preferably adopts a phased training strategy to ensure that the model gradually acquires Chinese semantic capabilities, Chinese audio capabilities, and full-duplex interaction capabilities. First, the language model foundation is pre-trained with Chinese text capabilities or integrated with existing large-scale models possessing strong Chinese capabilities, enabling it to master Chinese lexical, syntactic, knowledge, and logical reasoning abilities. Subsequently, large-scale Chinese speech data is used to train the neural audio encoder and audio token modeling capabilities, enabling the model to develop a basic perception ability of Mandarin and diverse spoken Chinese. Building upon this, full-duplex dialogue training data with parallel user and system flows is constructed, and the model undergoes joint training to focus on learning interruption scenarios, overlapping speech, short feedback echoes, intention prediction when the user hasn't finished speaking, and rapid replanning behavior after system correction. Then, Chinese semantic alignment and implicit planning capabilities are trained using audio-text alignment data, enabling the model to learn to form stable local Chinese semantic trajectories before outputting speech. Finally, instruction fine-tuning and scenario reinforcement are performed using data from question-and-answer, customer service, in-vehicle, and office scenarios, allowing the system to develop more appropriate response styles, response timings, and broadcast strategies under different business conditions. Therefore, the various capabilities developed during the training phase will be uniformly invoked during the online inference phase, ensuring a smooth connection between the entire process from input to output.

[0077] This model achieves continuous interaction with low latency through mechanisms such as discrete audio tokenization, hierarchical generation, dual-stream synchronous modeling, and streaming decoding. Simultaneously, relying on semantic planning channels and a language model foundation, the system not only "can hear and speak," but also "can understand, reason, and respond continuously based on context." Therefore, this invention is both a voice interaction system and a foundational Chinese speech model with the potential of an intelligent agent.

[0078] It should be understood that, although the above Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0079] In one embodiment, a full-duplex Chinese audio understanding device is also provided, comprising: The continuous Chinese audio stream acquisition module is used to acquire the continuous Chinese audio stream input by the user.

[0080] The discrete encoding module is used to input continuous Chinese audio streams into the neural audio encoder, process them through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream.

[0081] Dual-stream full-duplex joint modeling is used to input the user audio time-series stream and the system audio time-series stream from the previous time step into the dual-stream temporal backbone network to obtain the system audio time-series stream from the next time step. The dual-stream temporal backbone network is used to encode the user audio time-series stream, the system audio time-series stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, and then processed using a gating mechanism and a joint context representation layer to obtain a joint context representation. Semantic planning is performed based on the joint context representation to obtain the semantic planning result. Based on the semantic planning result and the joint context representation, prediction is performed to obtain the low-level acoustic token for the next time step. A streaming caching mechanism is introduced internally into the dual-stream temporal backbone network to maintain the historical cache state at each time step.

[0082] The audio restoration module is used to restore the audio time-series stream of the next moment into a continuous speech waveform segment by segment using a streaming audio decoder; and to output it in real time by generating and broadcasting it simultaneously.

[0083] In one embodiment, the discrete encoding module is further configured to decompose a continuous Chinese audio stream into interconnected two-layer discrete representations using a multi-level residual quantization structure. The two-layer discrete representations include a high-level semantic token and a low-level acoustic token. The high-level semantic token represents syllable-level, word-level, and phrase-level semantic content, as well as the trend of speaking intent. The low-level acoustic token represents timbre, prosody, speech rate, pitch, pauses, and ambient sounds in the paralinguistic features. Chinese-specific speech enhancement features are introduced during the audio encoding stage. These features include: fundamental frequency contour and its dynamic variation trend, syllable end duration and pause patterns, markers for tone sandhi and neutral tone phenomena, retroflexion features, and prosodic labels corresponding to interjections and exclamations. A user audio temporal stream is formed based on the Chinese-specific speech enhancement features, the high-level semantic token, and the low-level acoustic token.

[0084] In one embodiment, the dual-stream temporal backbone network includes: a user stream coding branch, a system stream coding branch, a cross-stream attention layer, a gated fusion layer, a joint context representation, a semantic planning head, and an audio generation head; the user stream coding branch includes a user stream embedding layer and a user stream causal Transformer layer; the system stream coding branch includes a system stream embedding layer and a system stream causal Transformer layer; dual-stream full-duplex joint modeling is also used to input the user audio temporal stream into the user stream coding branch, process it through the user stream embedding layer to obtain a user stream embedding representation, and encode the user stream embedding representation and the user-side context information stored in the history cache using the user stream causal Transformer layer to obtain a higher-level user stream representation; input the system audio temporal stream from the previous time step into the system stream coding branch, process it through the system stream embedding layer to obtain a system stream embedding representation, and encode the system stream embedding representation and the user-side context information stored in the history cache using the user stream causal Transformer layer to obtain a higher-level user stream representation; and input the system audio temporal stream from the previous time step into the system stream coding branch, process it through the system stream embedding layer to obtain a system stream embedding representation, and encode the system stream embedding representation and ... encode the system stream embedding representation and the history cache using the system stream causal Transformer layer to obtain a higher-level user stream representation. The system-side context information stored in the history cache is encoded using a system flow causal Transformer layer to obtain a high-level representation of the system flow. The high-level representations of the user flow and the system flow are input into a cross-flow attention layer to obtain user-side enhanced representations and system-side enhanced representations. The user-side enhanced representations and system-side enhanced representations are input into a gated fusion layer to obtain a fused dual-stream state representation. The fused dual-stream state representation is input into a joint context representation layer to obtain a joint context representation. The joint context representation includes: the semantic content and acoustic state of the user's current input audio, the semantic progress and acoustic continuity of the system's historical output speech, and the interaction relationship between the user flow and the system flow. The joint context representation is input into a semantic planning head to obtain a semantic planning result. The joint context representation and the semantic planning result are input into an audio generation head to obtain the low-level acoustic token of the system at the next time step.

[0085] In one embodiment, the audio restoration module is preceded by: dynamically processing interruptions, interjections, and short feedback in real spoken dialogue phenomena, implementing interruption decisions based on the dialogue control state, and storing the interruption decisions in a shared semantic state cache; specifically, when the system is in the output state and the user starts speaking midway, the system quickly identifies new input content by combining the energy changes, prosodic changes, and semantic importance in the user's current audio stream; if the system detects that the voice energy and semantic relevance of the user stream increase simultaneously, the system determines that the input has a high interaction priority and dynamically increases the weight of the user stream in subsequent joint modeling; if it is determined to be an echoic short feedback, the system switches to a fast replanning mode, locally adjusting the system output while retaining the original main response target; if it is determined to be background noise, echo, or meaningless short disturbance, the system maintains the original system stream and continues to output without triggering the main response interruption; through the processing mechanism, the system implements interruption decisions based on the dialogue control state maintained in the previous time step, and stores the results of the interruption decisions in a shared semantic state cache.

[0086] In one embodiment, inputting the joint context representation into the semantic planning head to obtain the semantic planning result includes: inputting the joint context representation into the semantic planning head, processing it using a multi-head attention mechanism, and obtaining the semantic planning result.

[0087] In one embodiment, the device further includes performing Chinese scene enhancement processing before the semantic planning result is input into the audio generation head. The specific process of Chinese scene enhancement processing includes: performing tone and homonym correction using joint information of high-level semantic tokens, low-level prosodic tokens, and contextual semantic states; performing incremental Chinese sentence segmentation and punctuation restoration in real time during streaming processing; using the results of sentence segmentation and punctuation restoration to control the pause length and tone rhythm during final broadcast; and jointly modeling Chinese tone and emotion. Specifically, for input with obvious emotional coloring, not only is its text semantics analyzed, but also the speaker's emotional trend is extracted by combining pitch changes, rhythm patterns, and acoustic intensity, and the emotional trend is written into the internal semantic planning result as a control signal so that the subsequently generated system speech can adaptively adjust to a soothing, confirmatory, explanatory, or prompting style.

[0088] In one embodiment, the audio restoration module is further configured to use a streaming audio decoder to restore the low-level acoustic token and high-level semantic token of the next moment into a continuous speech waveform segment by segment; the continuous speech waveform continues to participate in modeling as part of the system stream input in the next round; the continuous speech waveform is output in real time using a method of generating and playing simultaneously; specifically, when the accumulated continuous speech waveform reaches a preset playable threshold, the first segment is immediately started to provide interactive feedback to the user, and the system retains an interruptible synthesis buffer; if a user interruption is detected during the playback process, the dialogue control state will trigger buffer termination or partial regeneration, so that the system can quickly switch to a new response path without interrupting the overall interactive continuity; during the streaming output process, a low-latency continuous update mechanism is also executed; at the same time, a risk control output strategy is introduced, when the system judges that the current internal semantic confidence is insufficient, or the user's intention is not yet fully clear, it does not directly output conclusive content, but prioritizes generating transitional Chinese responses.

[0089] It is understood that for a detailed explanation of the full-duplex Chinese audio understanding device, please refer to the corresponding explanations of the various embodiments of the full-duplex Chinese audio understanding method above, and will not be repeated here. Each module in the above-described full-duplex Chinese audio understanding device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0090] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the method embodiment.

[0091] It is understood that, in addition to the memory and processor mentioned above, the computer equipment described above also includes other hardware and software components not listed in this specification. The specific components can be determined according to the model of the image processing computer in different application scenarios, and will not be listed and described in detail in this specification.

[0092] In one embodiment, a computer-readable storage medium stores a computer program thereon, and when a processor executes the computer program, it can also implement the steps in the above method embodiments.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus DRAM (RDRAM), and interface DRAM (DRDRAM), etc.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A full-duplex Chinese audio understanding method, characterized in that, Including the following steps: Step 1: Obtain the continuous Chinese audio stream input by the user; Step 2: Input the continuous Chinese audio stream into the neural audio encoder, process it through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream; Step 3: Input the user audio timing stream and the system audio timing stream of the previous moment into the dual-stream timing backbone network to obtain the system audio timing stream of the next moment; The dual-stream temporal backbone network is used to encode the user audio temporal stream, the system audio temporal stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, then processed using a gating mechanism and a joint context representation layer to obtain a joint context representation. Semantic planning is performed based on the joint context representation to obtain a semantic planning result. Prediction is then performed based on the semantic planning result and the joint context representation to obtain the low-level acoustic token for the next time step. The dual-stream temporal backbone network internally incorporates a streaming caching mechanism to maintain the historical cache state at each time step. Step 4: Use a streaming audio decoder to reconstruct the audio time-series stream of the next moment into a continuous speech waveform segment by segment; and output it in real time by generating and broadcasting it simultaneously.

2. The full-duplex Chinese audio understanding method of claim 1, wherein, Step 2 includes: A multi-level residual quantization structure is used to decompose a continuous Chinese audio stream into interconnected two-layer discrete representations. The two-layer discrete representations include: a high-level semantic token and a low-level acoustic token. The high-level semantic token is used to represent syllable-level, word-level, and phrase-level semantic content as well as speaking intention trends. The low-level acoustic token is used to represent timbre, prosody, speech rate, pitch, pauses, and ambient sounds in paralinguistic features. Chinese-specific speech enhancement features are introduced in the audio encoding stage; wherein the Chinese-specific speech enhancement features include: fundamental frequency profile and its dynamic change trend, syllable end duration and pause pattern, tone sandhi and neutral tone phenomenon markers, retroflexion features, and prosodic tags corresponding to interjections and exclamations. The user's audio time-series stream is formed based on Chinese-specific speech enhancement features, high-level semantic tokens, and low-level acoustic tokens.

3. The full-duplex Chinese audio understanding method of claim 1, wherein, The dual-stream temporal backbone network includes: a user stream coding branch, a system stream coding branch, a cross-stream attention layer, a gated fusion layer, a joint context representation, a semantic planning head, and an audio generation head; the user stream coding branch includes a user stream embedding layer and a user stream causal Transformer layer; the system stream coding branch includes a system stream embedding layer and a system stream causal Transformer layer; step 3 includes: The user audio time-series stream is input into the user stream coding branch, processed by the user stream embedding layer to obtain the user stream embedding representation. The user stream embedding representation and the user-side context information stored in the history cache are encoded by the user stream causal Transformer layer to obtain the user stream high-level representation. The system audio timing stream from the previous moment is input into the system stream coding branch. After processing by the system stream embedding layer, the system stream embedding representation is obtained. The system stream embedding representation and the system-side context information stored in the history buffer are encoded by the system stream causal Transformer layer to obtain the system stream high-level representation. The user flow high-level representation and the system flow high-level representation are input into the cross-flow attention layer to obtain the user-side enhanced representation and the system-side enhanced representation; The user-side enhanced representation and the system-side enhanced representation are input into the gated fusion layer to obtain the fused dual-stream state representation; The fused dual-stream state representation is input into the joint context representation layer to obtain the joint context representation; the joint context representation includes: the semantic content and acoustic state of the user's current input audio, the semantic progress and acoustic continuity of the system's historical output speech, and the interaction relationship between the user stream and the system stream. The joint context representation is input into the semantic planning header to obtain the semantic planning result; The joint context representation and the semantic planning result are input into the audio generation head to obtain the low-level acoustic token of the system at the next moment.

4. The full-duplex Chinese audio understanding method of claim 1, wherein, Before step 4, the process also includes: dynamically processing interruptions, interjections, and short feedback in real-world spoken dialogues; implementing interruption decisions based on the dialogue control state; and storing the interruption decisions in a shared semantic state cache; specifically including: When the system is in output mode and the user starts speaking midway, the new input content is quickly identified by combining the energy changes, prosodic changes and semantic importance in the user's current audio stream. If the system detects that the speech energy and semantic relevance of the user stream increase simultaneously, it determines that the input has a high interaction priority and dynamically increases the weight of the user stream in subsequent joint modeling. If the feedback is determined to be short and consistent, the system switches to a fast replanning mode, making partial adjustments to the system output while retaining the original main feedback objective. If the disturbance is determined to be background noise, echo, or meaningless short-lived disturbance, the original system flow will continue to be output without triggering the main response interrupt. Through the processing mechanism, the system implements interruption decisions based on the dialogue control state maintained in the previous time step, and stores the result of the interruption decision in the shared semantic state cache.

5. The full-duplex Chinese audio understanding method of claim 1, wherein, The joint context representation is input into the semantic planning header to obtain the semantic planning result, including: The joint context representation is input into the semantic planning head and processed using a multi-head attention mechanism to obtain the semantic planning result.

6. The full-duplex Chinese audio understanding method of claim 1, wherein, The method further includes performing Chinese scene enhancement processing before the semantic planning result is input into the audio generation head. The specific process of Chinese scene enhancement processing includes: Tone and homonym correction are performed using the combined information of high-level semantic tokens, low-level prosodic tokens, and contextual semantic state. Incremental Chinese sentence segmentation and punctuation restoration are performed in real time during streaming processing; the results of sentence segmentation and punctuation restoration are used to control the pause length and tone rhythm during the final broadcast. Joint modeling of Chinese tone and emotion; specifically, for input with obvious emotional coloring, not only is its text semantics analyzed, but also the speaker's emotional trend is extracted by combining pitch changes, rhythm patterns and acoustic intensity, and the emotional trend is written into the internal semantic planning result as a control signal, so that the subsequently generated system speech can adaptively adjust to a soothing, confirming, explanatory or prompting style.

7. The full-duplex Chinese audio understanding method of claim 1, wherein, Step 4 includes: A streaming audio decoder is used to recover the low-level acoustic token and high-level semantic token of the next time step by step into a continuous speech waveform; the continuous speech waveform continues to participate in modeling as part of the system stream input in the next round; The system outputs continuous speech waveforms in real time using a simultaneous generation and playback approach. Specifically, when the accumulated continuous speech waveform reaches a preset playable threshold, the first segment is immediately played to provide interactive feedback to the user, while the system retains an interruptible synthesis buffer. If a user interruption is detected during playback, the dialogue control state triggers buffer termination or partial regeneration, allowing the system to quickly transition to a new response path without interrupting the overall interactive continuity. During streaming output, a low-latency continuous update mechanism is also implemented. Furthermore, a risk control output strategy is introduced: when the system determines that the current internal semantic confidence is insufficient or the user's intent is not yet fully clear, it does not directly output conclusive content but prioritizes generating transitional Chinese responses.

8. A full-duplex Chinese audio understanding device, characterized in that, include: The continuous Chinese audio stream acquisition module is used to acquire the continuous Chinese audio stream input by the user. The discrete encoding module is used to input the continuous Chinese audio stream into the neural audio encoder, process it through a hierarchical quantization mechanism, and use the resulting high-level semantic token sequence and low-level acoustic token sequence as the user audio temporal stream. Dual-stream full-duplex joint modeling is used to input the user audio time-series stream and the system audio time-series stream of the previous moment into the dual-stream time-series backbone network to obtain the system audio time-series stream of the next moment; The dual-stream temporal backbone network is used to encode the user audio temporal stream, the system audio temporal stream, and the historical cache state using a dual-branch coding structure. The encoding results are aligned using a cross-attention mechanism, and then processed using a gating mechanism and a joint context representation layer to obtain a joint context representation. Semantic planning is performed based on the joint context representation to obtain a semantic planning result. Based on the semantic planning result and the joint context representation, prediction is performed to obtain the low-level acoustic token for the next time step. The dual-stream time-series backbone network introduces a streaming caching mechanism to maintain the historical cache state at each time step; The audio restoration module is used to restore the audio time-series stream of the next moment into a continuous speech waveform segment by segment using a streaming audio decoder; and to output it in real time by generating and broadcasting it simultaneously. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the full-duplex Chinese audio understanding method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the full-duplex Chinese audio understanding method according to any one of claims 1 to 7.

Images