End-to-end streaming speech recognition and natural language understanding joint optimization method and system

Abstract

This invention discloses an end-to-end streaming speech recognition and natural language understanding joint optimization method and system. The method includes acquiring an acoustic feature sequence extracted from the original speech signal and inputting the acoustic feature sequence into a pre-trained shared encoder; performing deep encoding on the input acoustic feature sequence through the shared encoder to generate a unified high-dimensional representation that serves both speech recognition and natural language understanding tasks; and performing ASR decoding and NLU decoding in parallel based on the unified high-dimensional representation. The NLU decoding includes: performing intent prediction and slot filling on the currently processed unified high-dimensional representation at a preset fixed time interval or after receiving a predefined speech segment to obtain a preliminary intent probability distribution and identified slot segments; calculating the comprehensive confidence of the preliminary intent probability distribution and identified slot segments; and immediately submitting the intent or slot when the comprehensive confidence reaches a first preset threshold to achieve "understanding while listening".

Description

Technical Field

[0001] This invention relates to the field of speech data processing technology, specifically to a method and system for joint optimization of end-to-end streaming speech recognition and natural language understanding. Background Technology

[0002] With the rapid development of artificial intelligence technology, voice interaction has become an important way of human-computer interaction, and is widely used in fields such as intelligent customer service, smart speakers, in-vehicle systems, and smart wearable devices. In these applications, Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) are the core pillar technologies.

[0003] However, existing voice interaction systems generally adopt a pipelined processing mode that connects ASR (Automatic Speech Recognition) and NLU (Non-Logic Logic Array). In this mode, the raw speech signal is first processed by the ASR module, converting it into a text sequence; then, the text output by the ASR module serves as the input to the NLU module for intent recognition and slot filling. Although this step-by-step, serial pipelined processing mode is technically mature, its inherent defects are becoming increasingly apparent, severely restricting the user experience and system performance of voice interaction, and failing to meet the needs of demanding scenarios.

[0004] For example, ASR (Automatic Speech Recognition) requires complete speech processing before transmitting it to NLU (Natural Language Understanding), resulting in a cumulative delay between the two processing stages. Especially in scenarios with long speech or complex sentences, the user must wait for the entire sentence to finish and for both stages of processing before the system can provide a comprehension result. This causes system lag, preventing true "listening and understanding," severely impacting the smoothness and real-time performance of the interaction. It is particularly problematic for streaming scenarios with high low-latency requirements (such as intent recognition in real-time calls), negatively affecting the user experience. Furthermore, ASR and NLU modules typically have their own independent feature extraction and encoder components, leading to duplication of computational resources and model parameters. This not only increases the system's deployment and maintenance costs but also limits its application in resource-constrained environments (such as edge devices and mobile terminals), resulting in high hardware costs. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a joint optimization method and system for end-to-end streaming speech recognition and natural language understanding to solve the problems in existing technologies.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An end-to-end streaming speech recognition and natural language understanding joint optimization method includes:

[0008] Acquire the acoustic feature sequence extracted from the original speech signal, and input the acoustic feature sequence into a pre-trained shared encoder;

[0009] The shared encoder performs deep encoding on the input acoustic feature sequence to generate a unified high-dimensional representation that serves both speech recognition and natural language understanding tasks.

[0010] Based on the unified high-dimensional representation, ASR decoding and NLU decoding are performed in parallel.

[0011] The NLU decoding includes:

[0012] At a preset fixed time interval or after receiving a predefined voice segment, intent prediction and slot filling are performed on the currently processed unified high-dimensional representation to obtain a preliminary intent probability distribution and identified slot segments.

[0013] Calculate the combined confidence level of the preliminary intent probability distribution and the identified slot segment; wherein the combined confidence level is determined based on ASR acoustic confidence level, semantic consistency and historical context;

[0014] When the overall confidence level reaches the first preset threshold, the intent or slot is submitted immediately;

[0015] When there is a conflict between the current overall confidence level and the cached overall confidence level within a preset range, a backtracking correction is triggered to dynamically correct the submitted intent or slot.

[0016] In one embodiment, prior to the step of inputting the acoustic feature sequence into the pre-trained shared encoder, the method further includes:

[0017] A pre-trained encoder model is obtained by self-supervised learning pre-training based on unlabeled speech data of a predetermined scale;

[0018] The output of the pre-trained encoder is connected to the ASR decoder and the NLU decoder respectively, and the pre-trained encoder, the ASR decoder and the NLU decoder are jointly trained using a multi-task joint loss function to obtain the pre-trained shared encoder.

[0019] In one embodiment, the joint loss function includes:

[0020] ;

[0021] in, For the loss of speech recognition task, To identify the loss of the task, The loss due to slot filling tasks. , and These are the corresponding weighting coefficients.

[0022] In one embodiment, prior to the step of performing ASR decoding and NLU decoding in parallel based on the unified high-dimensional representation, the method further includes: information exchange and enhancement of the intermediate representation of the speech recognition task and the intermediate representation of the natural language understanding task.

[0023] In one embodiment, when there is a conflict between the current overall confidence level and the cached overall confidence level within a preset range, a backtracking correction is triggered, including:

[0024] When subsequent input information causes a decrease in the overall confidence of a submitted intent or slot and / or the emergence of a new intent or slot with high overall confidence;

[0025] Calculate the difference between the new high overall confidence level and the overall confidence level of the submitted intent or slot;

[0026] If the difference is greater than a preset value, the submitted intent or slot will be dynamically corrected.

[0027] In one embodiment, the NLU decoding further includes:

[0028] When multiple consecutive comprehensive confidence levels are lower than the second preset threshold, the intent or slot is recalculated based on the complete output of the shared encoder.

[0029] In one embodiment, it further includes:

[0030] A domain knowledge base is constructed using a hierarchical tree structure or a knowledge graph structure; wherein the domain knowledge base includes a lexical layer, an entity layer, a relation layer, and a semantic layer;

[0031] The domain knowledge base is dynamically loaded and updated with terminology dictionaries, entity lists, and thesauruses for specific business domains.

[0032] During the encoding and decoding process, knowledge injection is performed based on the domain knowledge base through encoder bias, decoder constraints, or vector retrieval enhancement.

[0033] In one embodiment, it further includes:

[0034] Pre-defined unlabeled samples are collected from the production environment, labeled by experts, and the labeled samples are used to perform incremental training or fine-tuning on the shared encoder at a preset period. The pre-defined unlabeled samples include low-confidence samples, erroneous samples that require manual correction, and samples requiring operator intervention.

[0035] In one embodiment, it further includes:

[0036] Construct user profiles and adjust the direction of semantic understanding based on these user profiles;

[0037] Speech emotion features are extracted from the output of the shared encoder, concatenated with the semantic features after interaction, and input into the classification model to output emotion labels. The emotion labels are then input into the NLU decoder. The speech emotion features include tone, speech rate, and pitch frequency variation amplitude, and the emotion labels include positive, negative, neutral, and anxious.

[0038] An end-to-end streaming speech recognition and natural language understanding joint optimization system includes:

[0039] An acoustic feature extraction module is used to acquire an acoustic feature sequence extracted from the original speech signal and input the acoustic feature sequence into a pre-trained shared encoder.

[0040] The shared encoder module, connected to the acoustic feature extraction module, is used to perform deep encoding on the input acoustic feature sequence to generate a unified high-dimensional representation that serves both speech recognition and natural language understanding tasks.

[0041] The ASR decoding head module, connected to the shared encoder module, is used to perform ASR decoding based on the unified high-dimensional representation;

[0042] The NLU decoding head module is connected to the shared encoder module and is used to perform NLU decoding based on the unified high-dimensional representation; wherein, the ASR decoding head module and the NLU decoding head module perform ASR decoding and NLU decoding in parallel;

[0043] The NLU decoding head module includes:

[0044] The segmented intent prediction unit is used to perform intent prediction and slot filling on the currently processed unified high-dimensional representation at a preset fixed time interval or after receiving a predefined voice segment, so as to obtain a preliminary intent probability distribution and identified slot segments.

[0045] A confidence calculation unit is used to calculate the combined confidence of the preliminary intent probability distribution and the identified slot segment; wherein the combined confidence is determined based on ASR acoustic confidence, semantic consistency and historical context;

[0046] The submission unit is used to immediately submit the intent or slot when the overall confidence level reaches a first preset threshold.

[0047] The dynamic correction unit is used to trigger backtracking correction when there is a conflict between the current overall confidence level and the cached overall confidence level within a preset range, so as to dynamically correct the submitted intent or slot.

[0048] Compared with existing technologies, the beneficial effects of this invention are as follows:

[0049] By using a shared encoder (self-supervised pre-trained) and parallel ASR-NLU decoding, the number of model parameters and computation can be effectively reduced. By segmenting and predicting predefined speech segments at preset fixed time intervals and based on comprehensive confidence judgment, the intent or slot can be submitted in real time or dynamically corrected, thereby achieving streaming intent recognition. This ensures that the processing of the current time step input relies only on historical / current information and does not rely on future data, ensuring continuous output results during the user's speech and achieving "understanding while listening". Attached Figure Description

[0050] Figure 1 The flowchart illustrates the process of a joint optimization method for end-to-end streaming speech recognition and natural language understanding provided by this invention.

[0051] Figure 2 The flowchart of NLU decoding provided by this invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] With the rapid development of artificial intelligence technology, voice interaction has become an important way of human-computer interaction, and is widely used in fields such as intelligent customer service, smart speakers, in-vehicle systems, and smart wearable devices. In these applications, Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) are the core technologies.

[0054] However, existing voice interaction systems generally adopt a pipelined processing mode with ASR (Automatic Speech Recognition) and NLU (Non-Logic Logic) connected in series. In this mode, the raw speech signal is first processed by the ASR module, converting it into a text sequence; then, the text output by the ASR module serves as the input to the NLU module for intent recognition and slot filling. Although this step-by-step processing mode is technically mature, its inherent defects are becoming increasingly apparent, severely restricting the user experience and system performance of voice interaction.

[0055] Latency Accumulation and Insufficient Real-Time Performance: The ASR and NLU modules need to complete their respective processing independently, resulting in cumulative latency between the two stages. Especially in scenarios with long speech or complex sentences, the user must wait for the entire sentence to finish and for both stages of processing before the system can provide a comprehension result. This causes system response lag, making it impossible to achieve true "listen-and-understand," severely impacting the smoothness and real-time performance of the interaction, particularly for streaming scenarios with high low-latency requirements (such as intent recognition in real-time calls).

[0056] Error accumulation and propagation: The ASR module inevitably generates errors during speech recognition (such as misidentification of homophones, and recognition biases caused by accents or noise). These errors directly serve as input to the NLU module, causing it to perform semantic analysis on inaccurate text, thus triggering secondary errors in the NLU stage. This "error accumulation and propagation" mechanism significantly reduces the overall accuracy of intent understanding, problems particularly prominent in intelligent outbound calling and customer service scenarios in professional fields such as government and enterprise services, finance, and healthcare. These scenarios not only require high concurrency processing capabilities but also need to handle massive amounts of technical terminology, diverse dialects and accents, and strictly adhere to business compliance requirements. The latency and error accumulation inherent in traditional sequential processing directly impact service quality and user satisfaction, making it difficult to meet the extreme requirements of high accuracy, high real-time performance, and high reliability in these scenarios.

[0057] Information loss and insufficient semantic understanding: In traditional pipelined models, the ASR module typically only outputs the best text transcription result (1-besthypothesis), while a large amount of non-textual information contained in the speech signal (such as intonation, stress, pauses, speaker emotion, acoustic confidence, etc.) and alternative path information (N-besthypotheses or decoder grids) generated during ASR decoding are often truncated or ignored at the ASR-to-NLU interface. As a result, the NLU module cannot utilize this rich low-level acoustic and secondary language information for deeper and more robust semantic understanding, leading to limitations in its capabilities for ambiguity resolution, emotion recognition, or intent fine-tuning.

[0058] Model redundancy and resource consumption: ASR and NLU modules typically have their own independent feature extraction and encoder parts, resulting in duplication of computational resources and model parameters. This not only increases the deployment and maintenance costs of the system, but also limits its application in resource-constrained environments (such as edge devices and mobile terminals).

[0059] Insufficient domain adaptability and dynamism: When faced with specialized terminology and complex business logic in specific vertical domains (such as government, enterprise, finance, and healthcare), existing systems often require time-consuming separate ASR and NLU model training. Furthermore, their terminology database updates often require offline retraining or inflexible hot-loading mechanisms, making it difficult to adapt to rapidly changing business needs and user language.

[0060] In view of the above problems, there is an urgent need for a speech understanding technology that can deeply integrate ASR and NLU, achieve end-to-end joint optimization, and have ultra-low latency and high robustness, so as to overcome the limitations of the traditional pipeline mode and provide a more intelligent and smoother human-computer interaction experience.

[0061] like Figures 1 to 2 As shown, the present invention provides a joint optimization method for end-to-end streaming speech recognition and natural language understanding, comprising:

[0062] S100: Obtain the acoustic feature sequence extracted from the original speech signal, and input the acoustic feature sequence into the pre-trained shared encoder;

[0063] In one embodiment, prior to the step of inputting the acoustic feature sequence into the pre-trained shared encoder, the method further includes:

[0064] A pre-trained encoder model is obtained by self-supervised learning pre-training based on unlabeled speech data of a predetermined scale;

[0065] The output of the pre-trained encoder is connected to the ASR decoder and the NLU decoder respectively, and the pre-trained encoder, the ASR decoder and the NLU decoder are jointly trained using a multi-task joint loss function to obtain a pre-trained shared encoder.

[0066] In this embodiment, deep self-supervised learning pre-training (e.g., based on advanced architectures such as Wav2Vec2.0, HuBERT, Transformer, or Conformer) is performed on massive amounts of unlabeled speech data. This invention preferably uses a Transformer or Conformer architecture, but any functionally equivalent sequence coding architecture based on a self-attention mechanism should be considered within the scope of this invention. This allows the model to learn general and robust speech representations. This pre-training method significantly improves the model's generalization ability and adaptability in low-resource languages, accent variations, and complex noise environments.

[0067] As needed, multimodal fusion can be extended to reserve technical interfaces and architectural flexibility for richer interactive scenarios in the future. Other modal inputs besides voice can also be integrated, such as lip reading, gesture recognition, facial expressions, or auxiliary text commands in videos. Through a unified cross-modal attention mechanism, different modal features can be deeply fused within the model to achieve unified multimodal semantic understanding, thereby extending to more complex interactive scenarios such as video customer service, intelligent meeting assistants, and augmented reality (AR / VR) voice interaction.

[0068] In one embodiment, the joint loss function includes:

[0069] ;

[0070] in, For the loss of speech recognition task, To identify the loss of the task, The loss due to slot filling tasks. , and These are the corresponding weighting coefficients.

[0071] In this embodiment, model training employs a comprehensive joint loss function that combines losses from speech recognition tasks (e.g., Connectionist Temporal Classification, CTC loss, Transducer loss) and natural language understanding tasks (e.g., cross-entropy loss for intent classification, sequence labeling loss for slot filling). This joint optimization forces the model to learn general speech representations that can efficiently support both downstream tasks simultaneously, thereby significantly improving overall performance. In typical application scenarios, intent recognition accuracy can be improved by at least 15% compared to step-by-step processing.

[0072] in, , and As dynamically adjustable weight coefficients, the weights (λ values) of the loss for each task can be adaptively adjusted during training based on the convergence of each task, task difficulty, or business importance, in order to balance the learning progress of different tasks and optimize overall performance.

[0073] S200. The input acoustic feature sequence is deeply encoded by a shared encoder to generate a unified high-dimensional representation that serves both speech recognition and natural language understanding tasks.

[0074] In this embodiment, the original acoustic features (such as Mel filter bank features, spectrograms, etc.) are directly received by a shared encoder and encoded into a series of high-dimensional, semantically rich speech representations. This shared encoder is jointly optimized and trained, and its output serves both speech recognition and semantic understanding tasks. This effectively avoids the redundant computation between the ASR encoder and NLU encoder in traditional architectures, significantly reducing the number of model parameters and inference time.

[0075] S300, based on unified high-dimensional representation, performs ASR decoding and NLU decoding in parallel;

[0076] In this embodiment, the present invention achieves direct mapping from speech signals to semantic intent by constructing a unified end-to-end architecture, significantly reducing the overall latency from user speech to intent being understood; improving the accuracy of intent recognition under complex accents, noise, and colloquial expressions; and effectively reducing model redundancy and computational resource consumption, thereby greatly improving the user experience, intelligence level, and system efficiency of the voice interaction system.

[0077] In one embodiment, the step of performing ASR decoding and NLU decoding in parallel based on a unified high-dimensional representation further includes:

[0078] Based on multi-head attention mechanisms or gating units, information exchange and enhancement are performed between the intermediate representations of speech recognition tasks and the intermediate representations of natural language understanding tasks.

[0079] In this embodiment, a multi-dimensional feature interaction layer is introduced above the shared encoder. Information interaction and enhancement between ASR and NLU tasks can be achieved through multi-head attention mechanisms, gating units, feature concatenation, weighted fusion, etc., and is not limited to the above methods. Any equivalent technical solution that can achieve information interaction and enhancement between ASR and NLU tasks should fall within the protection scope of this invention. Among them, multi-head attention mechanisms or gating units are preferred embodiments. This mechanism enables deep information exchange and mutual enhancement between the intermediate representations of the speech recognition task and the intermediate representations of the natural language understanding task. For example, acoustic information from the ASR branch can guide the NLU branch to identify potential intentions earlier; conversely, semantic context information from the NLU branch can also assist the ASR branch in making more accurate word selection or correcting early recognition errors in ambiguous acoustic signals.

[0080] This mechanism employs a two-way independent enhancement and shared feedback flow approach:

[0081] Positive enhancement: The intermediate representation of the ASR decoder (such as the text fragment "first business subsidy" with high acoustic confidence) is passed to the NLU decoder through a multi-head attention mechanism to provide textual semantic support for intent prediction ("query_subsidy policy") and improve the accuracy of intent recognition.

[0082] Reverse enhancement: The intermediate representation of the NLU decoder (such as the initial identification of the "subsidy" intent) is fed back to the ASR decoder, which constrains the ASR to prioritize words related to the intent (such as excluding homophone errors such as "first business subsidy"), and corrects early identification biases.

[0083] Shared feedback: Enhanced features after interaction (such as the joint representation of "first business subsidy + query intent") are temporarily stored in the "feature interaction module cache" for subsequent segmented prediction calls, avoiding repeated calculations and reducing model redundancy.

[0084] Among them, S320 and NLU decoding include:

[0085] S321. At a preset fixed time interval or after receiving a predefined voice segment, perform intent prediction and slot filling on the currently processed unified high-dimensional representation to obtain a preliminary intent probability distribution and identified slot segments.

[0086] In this embodiment, the NLU part of the present invention employs incremental intent prediction. After the model receives continuous acoustic frames or chunks of data, it outputs the preliminary intent probability distribution and identified slot segments of the currently processed portion in real time at a fixed and configurable frequency (preferably every 50 milliseconds).

[0087] The predefined speech segments employ a multi-condition adaptive triggering method, encompassing three triggering scenarios:

[0088] Time-triggered: By default, segmented analysis is triggered once every 50 milliseconds (basic interval for streaming processing), which is suitable for continuous speech without obvious pauses (such as quickly stating business requirements).

[0089] Pause trigger: The pause trigger is triggered by identifying pauses in the user's speech through Voice Activity Detection (VAD) (e.g., pause duration ≥ 150 milliseconds). This is applicable to natural sentence segmentation (e.g., "I would like to ask how to apply for the first business subsidy").

[0090] Keyword triggering: When ASR / NLU jointly identifies an "intent start word" (such as "subsidy" or "registration" in government scenarios, or "transfer" or "repayment" in financial scenarios), segmented analysis is immediately triggered to prioritize capturing core semantics.

[0091] Among them, the three triggering methods take effect in parallel, with the "earliest condition met" as the criterion, to ensure a balance between the real-time performance and semantic integrity of streaming processing.

[0092] S322. Calculate the overall confidence of the preliminary intent probability distribution and the identified slot segments; wherein, the overall confidence is determined based on ASR acoustic confidence, semantic consistency and historical context;

[0093] In this embodiment, the overall confidence score is calculated using a fusion mechanism of weighted summation and dynamic threshold calibration. The specific calculation formula is as follows:

[0094] Overall confidence = α × Acoustic confidence (mean of frame-level confidence in ASR decoding output) + β × Semantic consistency (matching score with entities / intents in the domain knowledge base, in the range of 0-1) + γ × Historical context consistency (semantic overlap between the current prediction and the intents of the previous 3 segments, in the range of 0-1).

[0095] The weighting coefficients are dynamically adjusted: in normal scenarios, α∈[0.5,0.6], β∈[0.25,0.35], and γ∈[0.1,0.15]; in complex noise / ambiguity scenarios, the β weight is increased to 0.4-0.5, and the α weight is decreased to 0.4-0.45 to ensure semantic consistency is prioritized. When the overall confidence level reaches a preset high threshold (dynamically adjusted from 0.8 to 0.95, such as 0.9 by default in government and enterprise outbound call scenarios), intent / slot submission is triggered.

[0096] S323. When the overall confidence level reaches the first preset threshold, immediately submit the intent or slot.

[0097] In this embodiment, a comprehensive confidence index combining acoustic information, language model constraints, and semantic context is defined. When the comprehensive confidence of a preliminary intent or slot segment reaches a first preset threshold (preferably dynamically adjusted between 0.8 and 0.95), the system immediately submits the intent or slot, thereby achieving extremely low intent recognition latency (TTFI - Time-to-First-Intent, which can be less than 400 milliseconds, preferably less than 300 milliseconds), meeting stringent SLA targets, such as 99.9% TTFI less than 400 milliseconds.

[0098] S324. When there is a conflict between the current overall confidence level and the cached overall confidence level within a preset range, a backtracking correction is triggered to dynamically correct the submitted intent or slot; specifically, this includes:

[0099] S3241. When subsequent input information causes a decrease in the overall confidence of a submitted intent or slot and / or the emergence of a new intent or slot with high overall confidence;

[0100] S3242. Calculate the difference between the new high overall confidence level and the overall confidence level of the submitted intent or slot.

[0101] S3243. If the difference is greater than the preset value, the submitted intent or slot will be dynamically corrected.

[0102] In this embodiment, to address early errors that may be caused by streaming prediction (e.g., a user says "I" before saying "I want to do business"), a robust dynamic error correction mechanism is designed so that after subsequent input information flows in, the model will re-evaluate and correct the previously submitted intent or slot based on the new acoustic and semantic context.

[0103] By caching recent voice segments and their corresponding initial intents / slots, backtracking correction will be triggered immediately when new information causes a significant decrease in the confidence of historical judgments or the emergence of new high-confidence intents. For example, if the new overall confidence of subsequent segment predictions (e.g., 0.92 for "Second Business Incentive Policy") is higher than the overall confidence of the submitted intent (e.g., 0.88 for "First Business Subsidy Policy") and the difference is ≥0.05, or if the overall confidence of the submitted intent drops below the threshold due to new information (e.g., from 0.9 to 0.75), backtracking correction will be triggered immediately, notifying the dialogue management module to update the response logic.

[0104] In one embodiment, NLU decoding further includes:

[0105] S325. When multiple consecutive comprehensive confidence levels are lower than the second preset threshold, recalculate the intent or slot based on the complete output of the shared encoder, wherein the second preset threshold is less than the first preset threshold.

[0106] In this embodiment, the present invention introduces an intelligent backoff and degradation strategy to ensure system stability in environments with extreme noise, complex accents, or high ambiguity. Specifically, when the overall confidence of the convection prediction is consistently lower than the second preset threshold and cannot be effectively corrected, the system can automatically switch to the "final decoding + batch NLU" mode, that is, wait for the user to finish speaking before performing complete speech recognition and NLU processing, in order to ensure the accuracy of recognition and understanding in complex situations, sacrificing some real-time performance to ensure the correctness of the final result, and achieving a success rate of over 98% in switching between online and offline modes.

[0107] The entire end-to-end model is designed as a causal model, meaning that when processing input at any current time step, it relies only on current and previous input information, and never on future information. This ensures that the system can continuously and progressively output preliminary recognition results and intent judgments as the user speaks.

[0108] This invention effectively reduces the number of model parameters and computational load by using a shared encoder (self-supervised pre-training) and parallel ASR-NLU decoding, achieving an overall computational latency reduction of 40% and a reduction in repetitive computations of over 30%. Through segmented prediction of predefined speech segments at preset fixed time intervals, and based on comprehensive confidence judgments, it performs real-time submission or dynamic correction of intent or slots, thereby achieving streaming intent recognition. This ensures that processing the current time step input relies only on historical / current information, not future data, guaranteeing continuous output results during user speech and ensuring a stable TTFI (Time-to-Fit Interaction) below 300ms, meeting the SLA requirement of TTFI < 400ms in 99.9% of scenarios. This completely solves the common technical problems in existing technologies, such as error accumulation, latency superposition, information transmission loss, and the inability to achieve true "listening and understanding" due to independent development and step-by-step serialization of ASR and NLU. Compared with the traditional ASR+NLU serial architecture, this invention can improve the intent recognition accuracy by 15% and the slot filling F1 value by 12% in typical scenarios such as government and enterprise outbound calls, while reducing the overall computation latency by 40%, effectively solving the problems of error accumulation and propagation.

[0109] Furthermore, this invention targets real-time voice interaction scenarios, optimizing the end-to-end model through streaming processing to achieve ultra-low latency "listen-and-understand" capabilities, thereby minimizing the overall latency from the user's voice to the system's understanding of their intent. Through streaming segmented intent analysis (preferably every 50 milliseconds) and confidence-driven instant submission (the first preset threshold is preferably between 0.8 and 0.95), the system's average first intent submission time (TTFI) can be stably kept below 300 milliseconds, meeting the stringent SLA requirements of high real-time interaction scenarios such as government and enterprise outbound calls (e.g., 99.9% TTFI less than 400 milliseconds).

[0110] In one embodiment, it further includes:

[0111] A domain knowledge base is constructed using a hierarchical tree structure or a knowledge graph structure; the domain knowledge base includes a lexical layer, an entity layer, a relation layer, and a semantic layer;

[0112] Dynamically load and update the domain knowledge base with terminology dictionaries, entity lists, and thesaurus for specific business domains;

[0113] During encoding and decoding, knowledge injection is performed based on the domain knowledge base through encoder bias, decoder constraints, or vector retrieval enhancements.

[0114] In this embodiment, terminology dictionaries, entity lists, and thesauruses for specific business domains (such as government and enterprise, finance, healthcare, e-commerce customer service, and in-vehicle interaction) are dynamically loaded, thereby avoiding the need to retrain the model.

[0115] It should be noted that the domain knowledge injection process adopts a layered targeted injection approach, specifically:

[0116] Encoder bias: It applies to both the "shared encoder" and the "NLU decoder head" stages. In the shared encoder stage, a weight bias of 0.2-0.3 is applied to the acoustic feature segments that match domain terms (such as "third business subsidy") during attention calculation. In the NLU decoder head stage, the feature weights are further strengthened for the semantic representations corresponding to domain entities (such as "fourth business registration") to ensure that specialized terms are identified first.

[0117] Decoder constraints: Apply simultaneously to both the ASR decoding head and the NLU decoding head—During ASR decoding, words from the domain terminology dictionary are forced to be selected first (e.g., "fifth business processing" rather than homophones); During NLU decoding, the slot filling result must conform to the domain entity list (e.g., "subsidy type" can only select predefined values ​​such as "first business subsidy" and "second business reward").

[0118] Vector retrieval enhancement: This only applies to the NLU decoding head. It supplements the semantic context and corrects ambiguous predictions (such as distinguishing between "subsidy application" and "subsidy query") by using the domain knowledge base entity vectors (such as the policy terminology vector library) and the current semantic representation through an approximate nearest neighbor search (Top-3 matching).

[0119] Among these features, the dynamic loading mechanism of the domain knowledge base allows government and enterprise clients to update their professional terminology database online within minutes (such as newly released policy or product terms). The model takes effect immediately without retraining, and the recognition rate for new terms can be improved to over 90%. This domain knowledge is deeply injected into the end-to-end model through soft constraints (such as encoder bias and mask enhancement) or hard constraints (such as decoder post-processing and forced decoding paths), significantly improving the accuracy of professional term recognition and semantic understanding. In typical applications, it can improve the domain term recognition rate by more than 20%.

[0120] In addition, by constructing a semantic enhancement library containing information such as common question-answer pairs and business process paths in the domain, richer contextual semantic information can be provided to the model through vector retrieval or knowledge graph embedding, effectively making up for the lack of general models in specific domain knowledge.

[0121] As needed, the system is trained using a large amount of speech data containing different regional dialects, accents, speech rates, and intonations during the training phase. Simultaneously, speaker adaptation and personalized modeling techniques are integrated to quickly learn user speech features in the early stages of a conversation, dynamically adjusting model parameters to further improve robustness in recognizing individual differences.

[0122] As needed, advanced speech enhancement and noise suppression techniques (such as deep learning denoising, blind source separation, and multi-channel signal processing) can be incorporated into the shared encoder during the acoustic feature extraction stage, or noise-insensitive attention mechanisms can be designed within the model to ensure high-accuracy recognition and understanding even in noisy environments (such as telephone channel noise, background voices, and music) and complex acoustic environments (such as multi-channel input and acoustic echo cancellation). Furthermore, speaker diarization techniques can be used to distinguish different speakers and maintain independent acoustic and semantic contexts for each speaker to support multi-party concurrent dialogue scenarios.

[0123] In one embodiment, the method further includes: collecting preset unlabeled samples in the production environment, obtaining labeled samples after expert labeling, and using the labeled samples to perform incremental training or fine-tuning on the shared encoder at a preset period; wherein, the preset unlabeled samples include low-confidence samples, erroneous samples that require manual correction, and customer service intervention samples, and the overall confidence of the low-confidence samples is lower than a first preset threshold.

[0124] In this embodiment, a continuous learning and adaptive evolution mechanism is designed, encompassing two pipelines: offline large-scale training and online incremental fine-tuning. This enables online learning and user profile-driven personalization. By collecting samples with low confidence, high error correction rates, or requiring manual intervention from production environments, and after precise expert annotation, the model is periodically subjected to incremental training or fine-tuning to achieve "continuous evolution." The model also supports version management, facilitating automatic rollback to a previous stable version when issues are discovered. Furthermore, differential privacy algorithms can be employed during this process to protect individual information from reverse engineering when aggregating user data.

[0125] In one embodiment, the method further includes constructing a user profile and adjusting the semantic understanding direction based on the user profile;

[0126] In this embodiment, based on multi-dimensional user profile information such as the user's historical conversations, behavioral preferences, user roles (e.g., customer, salesperson), and user tags in the CRM system, the focus of semantic understanding is dynamically adjusted to achieve user profile-driven personalization. For example, higher priority is given to questions from specific users about specific products, or frequently used expressions by users are identified first, thereby significantly improving the personalized interaction experience and the accuracy of understanding.

[0127] Among these, adjusting the direction of semantic understanding based on user profiles includes:

[0128] Intent priority adjustment: Based on user roles (such as VIP customers, new users) and historical intents, dynamically adjust the order of intent candidates. For example, if a user has consulted "sixth business repayment" 3 times in the past, when mentioning "repayment" again, the intent priority of "sixth business repayment" is higher than that of "seventh business repayment", and the weight is increased by 0.3-0.4.

[0129] Slot sensitivity adjustment: Based on user behavior preferences, high-frequency slots (such as "repayment date" and "account number") are given higher recognition sensitivity. For example, when "the 5th of each month" is recognized, the "repayment date" slot is filled first, and the confidence threshold is reduced by 5%-10% (e.g., from 0.9 to 0.85) to reduce repetitive statements by users. Implementation method: Attention interaction is performed between the "user embedding vector" (a 128-dimensional vector that integrates historical dialogue and CRM tags) and the semantic representation of the NLU decoder to dynamically adjust the weights without modifying the model structure, ensuring lightweight adaptation.

[0130] In one embodiment, the method further includes extracting speech emotion features from the output of the shared encoder, concatenating the speech emotion features with the semantic features after interaction, inputting the concatenation into a classification model, outputting emotion labels, and inputting the emotion labels into the NLU decoder; wherein, the speech emotion features include tone, speech rate and pitch frequency variation amplitude, and the emotion labels include positive, negative, neutral and anxious.

[0131] In this embodiment, by integrating a sentiment analysis unit into the feature interaction module or NLU decoding head, it is possible to classify the user's current tone, speech rate and acoustic features into emotions, and generate emotion tags (such as positive, negative, neutral, etc.) for the dialogue management module to adjust the response style and strategy, so as to achieve a more natural and empatiska interaction.

[0132] Among them, the sentiment features and semantic features adopt attention fusion, dimensional concatenation and two-step fusion mechanism:

[0133] Step 1: The sentiment analysis unit extracts 16-dimensional sentiment features (quantified values ​​of tone, speed, and pitch changes, mapped to probability vectors corresponding to 4 categories of labels: "positive / negative / neutral / anxious"), and interacts with the 256-dimensional semantic features of the NLU decoding head through a multi-head attention mechanism to calculate sentiment-semantic attention weights (e.g., the weight of "anxiety" emotion corresponding to "complaint" intention is increased by 0.25).

[0134] The second step is to concatenate the emotional features (16 dimensions) and semantic features (256 dimensions) after interaction to form a 272-dimensional "emotion-semantic joint feature". This feature is then input into an NLU classifier and outputs intent / slot and emotion label (e.g., "query_subsidy policy + neutral") for the dialogue management module to adjust the response style (e.g., use standard language for neutral emotions and reassuring language for anxious emotions).

[0135] This invention can be efficiently integrated with modules such as front-end audio acquisition, back-end dialogue management, domain knowledge base, and speech synthesis, and has broad future scalability. The end-to-end speech recognition and natural language understanding capabilities provided by this invention can be deeply coordinated with the system's underlying robust acoustic processing and speech activity detection capabilities. The joint optimization results of end-to-end ASR and NLU (i.e., user intent and slots) can be seamlessly and with ultra-low latency fed into the dialogue management module to guide it in generating response logic. Then, a unified text-to-speech (TTS) service generates the response in an integrated manner, which is finally presented to the user, forming an efficient, smooth, and natural "hear-understand-speak-present" closed-loop process.

[0136] Depending on the needs, each core sub-module of this invention (such as the shared encoder, ASR / NLU decoder, streaming module, domain adaptation module, and monitoring and auditing component) can adopt a loosely coupled microservice or plug-in interface design. This modular architecture allows for easy independent upgrades, replacements, or performance optimizations of any module in the future without affecting the stable operation of the entire system. Simultaneously, when a backend service module fails, it can automatically switch to backup service or local degradation mode, ensuring high availability and continuity of the system.

[0137] It should be noted that a unified, versionable interface specification should be designed for different deployment scenarios (e.g., cloud, edge, hybrid deployment) and different user roles (e.g., developers, operations personnel, business personnel). Preferably, standard communication protocols such as RESTful API or gRPC should be used. The input interface should receive real-time audio streams or pre-processed acoustic features, as well as optional contextual information such as session ID, user ID, and domain tags. The output interface should output structured data in real time, including recognized text, intent (including confidence), slots (including confidence), and session state.

[0138] The end-to-end model provided by this invention effectively supports contextual understanding in multi-turn dialogues. By introducing a lightweight context cache or memory-Augmented mechanism within the model, it can perform joint inference by combining the current input with limited historical dialogue context. This enables the system to possess basic cross-turn slot tracking and reference resolution capabilities; for example, when the user omits the subject in subsequent turns, the model can still accurately identify its referent. Furthermore, for different application scenarios (such as long-form dialogue meetings), the size and level of the context window can be dynamically adjusted to balance real-time performance and contextual coherence.

[0139] It is important to understand that, building upon existing technologies such as edge encryption and federated learning, this invention further enhances data security and compliance. The system identifies and employs a data minimization strategy in real time, desensitizing, anonymizing, or directly removing unnecessary sensitive personal information (such as ID card numbers, bank card numbers, and phone numbers) before data is uploaded to the cloud for analysis. All interactive data is tagged and digitally signed to ensure the immutability of audit logs. Furthermore, a dedicated compliance audit module is designed to support the secure sharing of relevant audit data with regulatory agencies, provided it complies with regulations. Audit logs also feature encrypted archiving and multi-level storage strategies to meet long-term compliance requirements and secure backtracking. To further enhance data security, Trusted Execution Environments (TEEs) or homomorphic encryption technologies can be utilized to ensure data security and privacy during model inference in the cloud, strengthening security, compliance, and privacy protection.

[0140] The system was designed with clearly defined Service Level Agreement (SLA) goals from the outset, such as ensuring a 99.9% Time-to-Flight Interaction (TTFI) of less than 400 milliseconds, or an online / offline mode switching success rate of over 98%, to guarantee an exceptional user experience and system reliability. The system includes a built-in canary deployment module, supporting the gradual, small-scale rollout and canary release of new models, algorithms, or features. Operations personnel can compare the performance of different versions in key metrics (such as intent recognition accuracy, TTFI, and resource consumption) in real time, enabling rapid iteration and optimization to ensure a smooth launch of new features. The end-to-end joint optimization method designed in this invention has high versatility. Besides intelligent outbound calling, this method can be seamlessly applied to any voice interaction application that requires understanding the context of multi-turn dialogues, such as intelligent customer service, smart speakers, in-vehicle voice assistants, and meeting minutes systems.

[0141] Furthermore, to address future technological challenges, this invention possesses the following expansion capabilities:

[0142] It supports multilingual switching, automatically recognizes the user's language and loads the corresponding language model or terminology library, enabling global voice interaction capabilities.

[0143] Through techniques such as model pruning and quantization, the model size and computational complexity are deeply optimized, enabling deployment on resource-constrained mobile devices, IoT devices, or edge computing environments, achieving basic interactive capabilities in environments with no or weak network coverage. Simultaneously, the system supports heterogeneous computing acceleration, dynamically adapting and scheduling various hardware such as GPUs, TPUs, FPGAs, and dedicated NPUs for inference tasks. At the edge, dynamic quantization and segmented model loading effectively control memory usage and power consumption, balancing real-time performance and energy efficiency.

[0144] This paper describes a scenario of device automation triggered by voice, supports integration with smart home collaboration protocols, and demonstrates that the method of this patent is not limited to dialogue understanding, but can also drive intelligent control and linkage commands for IoT devices.

[0145] An end-to-end streaming speech recognition and natural language understanding joint optimization system includes:

[0146] The acoustic feature extraction module is used to acquire the acoustic feature sequence extracted from the original speech signal and input the acoustic feature sequence into the pre-trained shared encoder.

[0147] In this embodiment, the acoustic feature extraction module receives the raw speech signal from a microphone array or audio stream, and performs preprocessing and feature transformation on the raw speech signal; wherein,

[0148] Preprocessing includes noise reduction, echo cancellation (such as echo cancellation algorithms based on adaptive filtering or deep learning, which can support echo cancellation under multi-channel input), speech activity detection (VAD), and gain control. Preferably, a deep learning-based VAD algorithm is used, which can accurately identify the start and end of speech segments. In multi-party dialogue scenarios, speaker separation technology can be further integrated to distinguish the speech streams of different speakers.

[0149] Feature transformation involves converting the preprocessed speech signal into time-frequency domain acoustic features. Preferably, an 80-dimensional Mel filter bank feature (MFEC) is used, extracting one frame every 10 milliseconds and performing mean-variance normalization. Alternatively, context-sensitive features extracted from Fbank or self-supervised pre-trained models such as Wav2Vec2.0 can be used.

[0150] The shared encoder module, connected to the acoustic feature extraction module, is used to perform deep encoding on the input acoustic feature sequence to generate a unified high-dimensional representation that serves both speech recognition and natural language understanding tasks.

[0151] In this embodiment, the input acoustic feature sequence is deeply encoded by a shared encoder to generate a high-dimensional, context-aware unified representation that is applicable to both ASR and NLU tasks.

[0152] Preferably, a sequence-to-sequence (Seq2Seq) architecture primarily employing attention mechanisms such as Conformer or Transformer is used. This encoder comprises multiple encoder blocks, each consisting of a self-attention layer, a feedforward network, and residual connections. It effectively captures long-range dependencies in speech signals.

[0153] Input: Receive the acoustic feature sequence output by the acoustic feature extraction module.

[0154] Output: Generates a series of fixed-dimensional context vectors, which serve as shared inputs for the ASR decoder and the NLU decoder.

[0155] Self-supervised pre-training: Self-supervised pre-training is performed on large-scale unlabeled speech datasets (e.g., mask prediction tasks based on Wav2Vec2.0 or HuberT) to enable the encoder to learn rich, general speech representations. This pre-trained model serves as the base model, which is then fine-tuned on downstream task datasets.

[0156] In one embodiment, a feature interaction module is also included, located after the shared encoder and before or between the ASR decoding head and the NLU decoding head, for facilitating information sharing and mutual enhancement between the two tasks.

[0157] Preferably, a multi-head attention mechanism is employed. When generating speech recognition results, the ASR decoder can calculate the attention weight between its current state and the intermediate representation of the NLU decoder, thereby incorporating semantic information. Conversely, when performing intent prediction and slot filling, the NLU decoder can also focus on the acoustic or secondary text representation output by the ASR decoder.

[0158] Gating units (such as GRU or LSTM units) can also be introduced to control the degree of fusion between ASR and NLU information flows, allowing the model to adaptively learn when and how to utilize cross-task information.

[0159] Following the feature interaction module, a sentiment analysis unit can be added. This unit analyzes the emotional cues (such as intonation, speech rate, and pitch variations) in the speech features output by the shared encoder, extracts the emotional features, and fuses them with semantic features to output emotion labels (such as positive, negative, neutral, anxious, etc.). These emotion labels will be passed as additional information to the NLU decoder and dialogue management system to achieve more empathetic and personalized interactions.

[0160] The ASR decoding head module, connected to the shared encoder module, is used to perform ASR decoding based on a unified high-dimensional representation;

[0161] In this embodiment, the ASR decoding head receives the output of the shared encoder and the feature interaction module and is responsible for generating the text transcription result.

[0162] Preferably, a streaming-friendly decoder architecture such as CTC or RNN-Transducer is employed. These architectures allow the model to begin outputting partial text results as soon as it receives partial speech input, achieving incremental recognition.

[0163] During the decoding process, external language models (such as N-gram language models or Transformer-based neural network language models) can be integrated to improve the fluency and accuracy of text transcription.

[0164] The NLU decoding head module, connected to the shared encoder module, is used to perform NLU decoding based on a unified high-dimensional representation; the ASR decoding head module and the NLU decoding head module perform ASR decoding and NLU decoding in parallel.

[0165] In this embodiment, the NLU decoding head receives the output of the shared encoder and the feature interaction module, and is responsible for predicting user intent and filling slots in real time.

[0166] Preferably, a Transformer-based sequence labeling model is used, which can perform intent classification and slot sequence labeling simultaneously.

[0167] The NLU decoding head module includes:

[0168] The segmented intent prediction unit is used to perform intent prediction and slot filling on the currently processed unified high-dimensional representation at a preset fixed time interval or after receiving a predefined speech segment, so as to obtain a preliminary intent probability distribution and identified slot segments.

[0169] In this embodiment, the NLU decoder does not wait for the entire speech input to finish before analysis. Instead, it operates in parallel with the ASR decoder, performing intent and slot prediction on the shared encoder output corresponding to the currently processed speech segment at fixed time intervals or after receiving a predefined speech segment (e.g., preferably every 50 milliseconds or every 200 milliseconds). This "segmented" processing allows the model to continuously output preliminary results of intent analysis as the user speaks. At each segment point, the model outputs the preliminary intent probability distribution and the identified slot segments corresponding to the current speech segment, thereby implementing an incremental prediction effect.

[0170] The confidence calculation unit is used to calculate the overall confidence of the preliminary intent probability distribution and the identified slot segments; wherein, the overall confidence is determined based on ASR acoustic confidence, semantic consistency and historical context;

[0171] In this embodiment, the system calculates a comprehensive confidence index for each initial intent and slot. This index comprehensively considers acoustic confidence (from ASR), semantic consistency (matching degree with the domain knowledge base), and historical context information.

[0172] The submission unit is used to immediately submit the intent or slot when the overall confidence level reaches the first preset threshold.

[0173] In this embodiment, when the overall confidence of a certain initial intent or slot reaches a preset high threshold (preferably 0.85) and has a certain degree of semantic completeness, the system immediately submits the intent or slot to the dialogue management module to achieve ultra-low latency intent recognition and meet strict SLA targets, such as 99.9% TTFI less than 400 milliseconds.

[0174] The dynamic correction unit is used to trigger backtracking correction when there is a conflict between the current overall confidence level and the cached overall confidence level within a preset range, so as to dynamically correct the submitted intent or slot.

[0175] In this embodiment, when the analysis results of subsequent speech segments (especially high-confidence results) conflict with earlier submissions in the cache, the system triggers a dynamic error correction mechanism. For example, if the confidence of an earlier submitted intent A decreases and the confidence of a new intent B increases significantly, the system will correct to intent B and notify the dialogue management module to make corresponding adjustments. This mechanism ensures the accuracy and robustness of streaming prediction.

[0176] Depending on the needs, a domain knowledge management module may also be included, which is used to dynamically load and manage knowledge specific to a particular business domain to enhance the domain adaptability of the model.

[0177] Business users or administrators can dynamically upload and update domain-specific terminology dictionaries, entity lists, and thesauruses via an interface. These domain knowledge bases are stored in a structured format (e.g., JSON, YAML, or a database).

[0178] Preferably, the domain knowledge base adopts a hierarchical tree structure or a knowledge graph structure to ensure the hierarchy and relevance of knowledge. The lexical layer includes professional terms, abbreviations, product names, etc., supporting fuzzy matching and synonym expansion. The entity layer defines various entity types (such as names of people, places, product models, service types) and their corresponding lexical lists, supporting regular expressions or dictionary matching. The relation layer defines semantic relationships between entities (such as "product" belonging to "series", "service" containing "steps"), interacting with NLU models through knowledge graph embedding or graph neural networks. The semantic layer includes common question-answer pairs, business process keywords, intent-triggered phrases, etc., to provide richer contextual semantic cues.

[0179] Information from the domain knowledge base is injected into the model in the following ways:

[0180] Encoder bias: In a shared encoder or NLU decoder, a dynamic bias is applied to the attention computation or feature representation of tokens or fragments that match domain terms, giving them higher attention weight.

[0181] Decoder constraints: During NLU decoding, the identified entities and slots are strictly consistent with the domain knowledge base definition by forcing the decoding path or post-processing rules.

[0182] Vector retrieval enhancement: Entities and concepts from the domain knowledge base are embedded into the vector space, and during NLU prediction, the model is provided with additional semantic context information through approximate nearest neighbor search.

[0183] The application of this invention will be illustrated below using the 'intelligent government customer service' scenario as an example:

[0184] This system can provide technical support for government service hotlines. First, in the domain knowledge management module, the latest policy document terminology (such as 'third business subsidy'), service process keywords (such as 'fourth business registration', 'fifth business processing'), and other domain knowledge base are dynamically loaded.

[0185] When a citizen calls to inquire, "Hello, I would like to ask how to apply for the newly introduced first business subsidy?":

[0186] ① The shared encoder module processes acoustic features in real time and generates a unified high-dimensional representation.

[0187] ②The segmented intent prediction unit in the NLU decoding head module has already calculated a high-confidence preliminary intent (query_subsidy policy) and slot (subsidy type: first business subsidy) through the confidence calculation unit when the user says "first business subsidy".

[0188] ③ Once the confidence level exceeds the first preset threshold (e.g., 0.9), the system immediately submits the intent to the dialogue management module. At this time, the backend system can start preparing relevant policy materials to achieve extremely low response latency without waiting for the user to finish speaking a whole sentence.

[0189] ④ If the user corrects the information in a subsequent conversation, for example, 'Oh no, I meant to ask about the reward policy for the second business,'

[0190] ⑤ At this point, the dynamic correction unit will cancel or correct the previous intent based on new, more comprehensive, and more confident information, and notify the dialogue management module, thereby ensuring the accuracy and smoothness of the entire interaction process.

[0191] It should be noted that the 'domain knowledge' of this invention can also be multimodal information. For example, in scenarios such as in-vehicle navigation or intelligent customer service, the system can integrate geographic location data related to the context of voice commands or image information provided by the user to achieve more accurate semantic understanding.

[0192] In one embodiment, a continuous learning module is also included to handle continuous model optimization and user experience personalization. A comprehensive offline training pipeline is established, including large-scale historical data collection, cleaning, annotation, model training, validation, and performance evaluation. Low-confidence, high-error-correction-rate, or manually corrected speech / text pairs from the production environment are automatically or semi-automatically collected and sent to experts for precise annotation (including ASR transcription, intent, and slots). Using the annotated new data, the end-to-end model is periodically incrementally trained or fine-tuned to update model parameters. The system incorporates model version management and canary release mechanisms, allowing new models to be gradually deployed on a small scale and their key metrics monitored in real time. When a new model's performance is substandard or anomalies occur, automatic rollback to the previous stable version is supported to ensure service continuity. Differential privacy algorithms are used during data collection, aggregation, and model training to add noise when aggregating user data to protect individual information from reverse engineering and comply with data privacy compliance requirements.

[0193] In one embodiment, a user profiling module is also included, comprising:

[0194] The user profile building unit is used to collect and analyze information such as users' historical dialogue records, behavioral preferences, frequently used words, roles (such as VIP customers, new users), and historical interaction emotion tags to build a multi-dimensional user profile.

[0195] The dynamic adjustment unit is used to dynamically adjust the semantic understanding focus in the NLU decoder based on the user profile. For example, it can give higher priority to questions from specific users about specific products, or prioritize the recognition of commonly used expressions by users. This can be achieved by introducing user embedding or personalized attention mechanisms into the NLU decoder.

[0196] The speaker-independent context maintenance unit is used in multi-person interaction scenarios to maintain independent dialogue history and context information for each identified speaker by combining speaker separation technology, thereby achieving accurate personalized understanding and response in multi-party concurrent dialogues.

[0197] In one embodiment, a system integration and interface module further ensures that the system of the present invention can efficiently collaborate with external systems and possesses high reliability. Specifically,

[0198] Each core sub-module of this invention (such as the shared encoder, ASR / NLU decoder, streaming module, domain adaptation module, and monitoring and auditing component) adopts a loosely coupled microservice or plug-in interface design. This modular architecture facilitates independent upgrades, replacements, or performance optimizations of any module in the future without affecting the stable operation of the entire system. The system has an automatic fault detection and failover mechanism. When a backend service module fails, it can automatically switch to backup service or local degradation mode to ensure high availability and continuity of the system.

[0199] Furthermore, a unified, versionable interface specification should be designed for different deployment scenarios (cloud, edge, hybrid deployment) and different user roles (developers, operations personnel, business personnel). Preferably, standard communication protocols such as RESTful API or gRPC should be used.

[0200] The system receives real-time audio streams or pre-processed acoustic features via an input interface, along with optional contextual information such as session ID, user ID, and domain labels.

[0201] The system outputs structured data in real time, including recognized text, intent (including overall confidence score), slots (including overall confidence score), sentiment tags, and conversation state.

[0202] The end-to-end intent recognition results provided by this invention can be used as core input, seamlessly and with ultra-low latency fed into the downstream dialogue management module to guide its response logic generation. The response text from the dialogue management module is then integrated into a unified text-to-speech (TTS) service to generate a voice response, which is then presented to the user through a front-end interface or device, thus forming an efficient, smooth, and natural "listen-understand-speak-present" collaborative closed loop.

[0203] In one embodiment, a security and compliance module is also included to ensure the security of data and models throughout their entire lifecycle.

[0204] The system employs real-time identification and data minimization strategies to desensitize, anonymize, or directly remove unnecessary sensitive personal information (such as ID card numbers, bank card numbers, and phone numbers) before data is uploaded to the cloud for analysis. All interactive data is tagged and digitally signed to ensure the immutability of audit logs. Audit logs also feature encrypted archiving and multi-level storage strategies (such as tiered cold and hot storage) to meet long-term compliance requirements and security backtracking. The system ensures the security of models during cloud or edge inference. This includes, but is not limited to, using trusted execution environments to isolate models and data, or exploring the use of homomorphic encryption and other technologies that allow models to infer from encrypted data without decryption.

[0205] In one embodiment, a hardware adaptation and performance optimization module is also included to ensure efficient deployment and operation of the system on different hardware platforms.

[0206] This system design supports a variety of heterogeneous hardware accelerators, including but not limited to GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), FPGAs (Field-Programmable Gate Arrays), and dedicated NPUs (Neural Processing Units). The system can automatically schedule and allocate inference tasks to the optimal hardware devices based on the deployment environment and workload, achieving efficient utilization of computing resources.

[0207] When deployed at the edge and on mobile devices, this invention significantly controls memory usage and power consumption while ensuring real-time performance through the following means:

[0208] Model pruning involves removing unimportant connections or neurons from the model, significantly reducing the model size without substantially reducing performance.

[0209] Model quantization involves converting model parameters from high-precision floating-point numbers to low-precision integers (such as INT8) to reduce storage and computational overhead.

[0210] Model segmented loading includes loading parts of the model or modules on demand during inference, reducing peak memory requirements.

[0211] The dynamic inference strategy includes dynamically adjusting the inference accuracy and computation graph based on device load and power consumption to achieve a balance between performance and power consumption.

[0212] This invention not only possesses the aforementioned core functions, but also demonstrates significant system advantages and broad forward-looking expansion potential.

[0213] By using a shared encoder and joint optimization, this invention significantly reduces the number of models, memory footprint, and computational load (e.g., the overall system computation can be reduced by at least 30% compared to traditional pipelines). While achieving lower latency, it also improves the overall system throughput and resource utilization, making it more suitable for high-concurrency scenarios. Furthermore, it completely eliminates the problem of ASR errors propagating and accumulating downstream in the traditional ASR-NLU pipeline; the NLU can directly capture semantics from the speech signal, resulting in a lower overall system error rate.

[0214] The end-to-end joint optimization method designed in this invention has high versatility. Besides intelligent outbound calling, this method can be seamlessly applied to any voice interaction application that requires understanding the context of multi-turn dialogues, such as intelligent customer service, smart speakers, in-vehicle voice assistants, and meeting minutes systems. Furthermore, to address future technological challenges, this invention possesses the following expansion capabilities, including:

[0215] It supports multilingual switching, automatically recognizes the user's language and loads the corresponding language model or terminology library, enabling global voice interaction capabilities.

[0216] This paper describes a scenario of device automation triggered by voice, supports integration with smart home collaboration protocols, and demonstrates that the method of this patent is not limited to dialogue understanding, but can also drive intelligent control and linkage commands for IoT devices.

[0217] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is obvious that many changes and variations can be made based on the above teachings. Although embodiments of the invention have been shown and described, these specific embodiments are merely explanations of the invention and are not intended to limit it. The specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. The purpose of selecting and describing exemplary embodiments is to explain the specific principles of the invention and its practical application, so that those skilled in the art, after reading this specification, can make modifications, substitutions, variations, and various choices and changes to the embodiments as needed without departing from the principles and spirit of the invention, provided that such modifications, substitutions, variations, and choices and changes are within the scope of the claims of the invention and are protected by patent law.

Claims

1. An end-to-end streaming speech recognition and natural language understanding joint optimization method, characterized in that including: Obtain an acoustic feature sequence extracted from an original speech signal, and input the acoustic feature sequence into a pre-trained shared encoder; Deeply encode the input acoustic feature sequence through the shared encoder to generate a unified high-dimensional representation for simultaneously serving a speech recognition task and a natural language understanding task; Based on the unified high-dimensional representation, perform ASR decoding and NLU decoding in parallel; wherein, the NLU decoding includes: At a preset fixed time interval or after receiving a predefined speech segment, perform intent prediction and slot filling on the currently processed unified high-dimensional representation to obtain a preliminary intent probability distribution and an identified slot segment; Calculate the comprehensive confidence of the preliminary intent probability distribution and the identified slot segment; wherein, the comprehensive confidence is determined based on ASR acoustic confidence, semantic consistency, and historical context; When the comprehensive confidence reaches a first preset threshold, immediately submit an intent or a slot; When there is a conflict between the current comprehensive confidence and the cached comprehensive confidence within a preset range, trigger backtracking correction to dynamically correct the submitted intent or slot, including: when subsequent input information causes the comprehensive confidence of the submitted intent or slot to decrease and / or a new intent or slot with a high comprehensive confidence appears; calculate the difference between the new high comprehensive confidence and the comprehensive confidence of the submitted intent or slot; if the difference is greater than a preset value, dynamically correct the submitted intent or slot.

2. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 1, wherein Before the step of inputting the acoustic feature sequence into the pre-trained shared encoder, further include: Perform self-supervised learning pre-training on a preset scale of unlabeled speech data to obtain a pre-trained encoder model; Connect the output of the pre-trained encoder to an ASR decoder head and an NLU decoder head respectively, and jointly train the pre-trained encoder, the ASR decoder head, and the NLU decoder head using a multi-task joint loss function to obtain the pre-trained shared encoder.

3. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 2, characterized in that The joint loss function includes: ； Among them, is the loss of the speech recognition task, is the loss of the intent recognition task, is the loss of the slot filling task, , and are the corresponding weight coefficients.

4. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 1, characterized in that Before the step of performing ASR decoding and NLU decoding in parallel based on the unified high-dimensional representation, further include: Perform information interaction and enhancement on the intermediate representation of the speech recognition task and the intermediate representation of the natural language understanding task.

5. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 1, characterized in that The NLU decoding further includes: When the comprehensive confidence is lower than a second preset threshold for a continuous plurality of times, recalculate the intent or slot based on the complete output of the shared encoder.

6. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 1, characterized in that, Further include: Construct a domain knowledge base using a hierarchical tree structure or a knowledge graph structure; wherein, the domain knowledge base includes a vocabulary layer, an entity layer, a relationship layer, and a semantic layer; Dynamically load and update a term dictionary, an entity list, and a synonym list for a specific business domain in the domain knowledge base; In the encoding and decoding processes, perform knowledge injection processing according to the domain knowledge base in a manner of encoder bias, decoder constraint, or vector retrieval enhancement.

7. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 1, characterized in that Further include: Collect preset unlabeled samples in the production environment, obtain labeled samples after expert annotation, and use the labeled samples to perform incremental training or fine-tuning processing on the shared encoder at a preset cycle; wherein, the preset unlabeled samples include low-confidence samples, error samples that need manual correction, and agent intervention samples.

8. An end-to-end streaming speech recognition and natural language understanding joint optimization method according to claim 2, characterized in that It further includes: Construct a user profile and adjust the semantic understanding direction based on the user profile; Extract speech emotion features from the output of the shared encoder, splice the speech emotion features with the semantic features after interaction, input them into a classification model, output emotion labels, and input the emotion labels into the NLU decoder head; wherein, the speech emotion features include intonation, speech rate, and the amplitude of pitch frequency change, and the emotion labels include positive, negative, neutral, and anxious.

9. An end-to-end streaming speech recognition and natural language understanding joint optimization system, characterized in that, It includes: An acoustic feature extraction module, configured to obtain an acoustic feature sequence extracted based on the original speech signal and input the acoustic feature sequence into a pre-trained shared encoder; A shared encoder module, connected to the acoustic feature extraction module, configured to perform deep encoding on the input acoustic feature sequence to generate a unified high-dimensional representation for simultaneously serving speech recognition tasks and natural language understanding tasks; An ASR decoder head module, connected to the shared encoder module, configured to perform ASR decoding based on the unified high-dimensional representation; An NLU decoder head module, connected to the shared encoder module, configured to perform NLU decoding based on the unified high-dimensional representation; wherein, the ASR decoder head module and the NLU decoder head module perform ASR decoding and NLU decoding in parallel; Wherein, the NLU decoder head module includes: A segment intention prediction unit, configured to perform intention prediction and slot filling on the currently processed unified high-dimensional representation at a preset fixed time interval or after receiving a predefined speech segment, to obtain a preliminary intention probability distribution and an identified slot segment; A confidence calculation unit, configured to calculate the comprehensive confidence of the preliminary intention probability distribution and the identified slot segment; wherein, the comprehensive confidence is determined based on the ASR acoustic confidence, semantic consistency, and historical context; A submission unit, configured to immediately submit an intention or a slot when the comprehensive confidence reaches a first preset threshold; A dynamic correction unit, configured to trigger backtracking correction to dynamically correct the submitted intention or slot when there is a conflict between the current comprehensive confidence and the cached comprehensive confidence within a preset range, including: when subsequent input information causes the comprehensive confidence of the submitted intention or slot to decrease and / or a new intention or slot with a high comprehensive confidence appears; calculating the difference between the new high comprehensive confidence and the comprehensive confidence of the submitted intention or slot; if the difference is greater than a preset value, dynamically correct the submitted intention or slot.

Images