Intelligent voice interaction method and system

Abstract

The application discloses an intelligent voice interaction method and system, and relates to the technical field of voice interaction, and comprises the following steps: acquiring user voice, determining emotion labels and pronunciation law features, and outputting language modes; extracting doubt words, filler features and pronouns based on standard word frequency; determining the accurate meanings of each doubt word based on discrimination rules, and determining the accurate meanings of pronouns according to a hierarchical mechanism; generating response voice and operation instructions according to the accurate meanings, the emotion labels and the filler features; the execution of the operation instructions follows a closed-loop process of instruction analysis, safety verification, decision planning, execution monitoring and feedback updating, closely cooperates with an automatic driving core module, and ensures that the instructions are accurately converted into vehicle behaviors; the user voice mode can be accurately judged, and the problem of insufficient understanding ability for non-standard language and spoken language is effectively solved; the accuracy of semantic analysis is improved, and voice interaction is more in line with natural language logic.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to an intelligent voice interaction method and system. Background Technology

[0002] With the development of autonomous driving technology, voice interaction, as a more natural and convenient control method, has gradually gained attention, and vehicle control systems are becoming increasingly complex. Traditional human-machine interaction methods (such as physical buttons and touchscreens) may distract users and affect safety in driving scenarios. Voice interaction provides a more natural and convenient control method. However, most existing in-vehicle voice systems are limited to infotainment, simple navigation, or vehicle control (such as air conditioning and windows), and their integration with the core decision-making and planning modules of autonomous driving is very low.

[0003] In autonomous driving scenarios, drivers need to interact with the vehicle efficiently and accurately via voice to achieve precise control and real-time feedback of the autonomous driving system. A common problem is rigid command understanding, which can only process preset, fixed phrases and cannot understand natural language commands containing complex intentions and multiple constraints. Command understanding is often detached from the vehicle's current state (such as speed, location, and driving mode), traffic environment, and historical dialogue, easily leading to erroneous operations. Furthermore, it is difficult to safely and compliantly translate high-level user intentions (such as "follow the car in front") into finely defined control parameters that the autonomous driving system can execute.

[0004] However, existing voice systems struggle to meet these requirements, and are unable to accurately understand drivers' non-standard accents. They also lack the ability to handle colloquial and non-standard expressions, and perform poorly in processing complex semantics and referential relationships, thus reducing data processing efficiency and recognition accuracy. These problems severely restrict the widespread application of voice interaction in the field of autonomous driving. Summary of the Invention

[0005] This application provides an intelligent voice interaction method and system, which solves the problem of the functional limitations of existing voice systems, which are unable to accurately understand and quickly process complex semantics and referential relationships. It achieves the technical effect of accurately recognizing user voice patterns, generating context-appropriate response voices and operation commands, and improving the accuracy and naturalness of voice interaction.

[0006] This application provides an intelligent voice interaction method, including:

[0007] S1: Acquire user voice, determine emotional tags and pronunciation pattern features, input pronunciation pattern features into language recognition model and output language pattern; if it is a standard language and standard grammar, mark it as standard pattern; if it is a non-standard language, mark it as non-standard pattern; if it is a standard language and non-standard grammar, mark it as spoken language pattern.

[0008] S2: For the standard mode, the speech text is segmented to obtain standard word frequencies, and response speech and operation instructions are generated by combining sentiment tags; for non-standard language mode and spoken language mode, the features of interrogative words, filler words and pronouns are extracted based on standard word frequencies; the precise meaning of each interrogative word is determined based on the discrimination rules, and the precise meaning of the pronouns is determined according to the hierarchical mechanism; based on the precise meaning, response speech and operation instructions are generated by combining sentiment tags and filler word features.

[0009] S3: Real-time calculation of the decay curve of the resolution accuracy, calculation of hierarchical storage efficiency, data processing efficiency and recognition accuracy based on historical data and decay curve, and determination of the optimal recognition rule based on the peak change trend of the recognition curve.

[0010] Furthermore, an automatic speech recognition model is used to convert user speech in real time and output speech text. During the speech recognition stage, the position of filler words is marked, the prosodic features of filler words are extracted, and a personalized filler word pattern library is established for the user. The user's speech text is obtained and compared with the filler word pattern library to identify filler word density, filler word type, and filler word position as filler word features.

[0011] Furthermore, extracting question words is based on calculating the semantic confidence of words in context using a pre-trained language model. This includes: pre-labeling the dataset, where each word is labeled as either a question word or not, as training data; training the model using the training set; validating the parameters using the validation set; optimizing the model parameters; obtaining a fully trained pre-trained language model; outputting a list of semantic confidence scores for each word; identifying the word with the lowest confidence score as a question word; and using an attention weight matrix to label the positions of words with low confidence scores.

[0012] Furthermore, the precise meaning of each ambiguous word is determined based on the discrimination rules, including: taking the ambiguous word as the center, taking several words before and after it to form a context window, and using a sliding window algorithm to extract the window content in real time; using a functional word vector model to obtain the vector representation of each meaning of the ambiguous word, calculating the similarity between each meaning vector representation and the context vector, setting a pre-set association similarity threshold, and removing the corresponding meaning if the similarity is less than the association similarity threshold; marking the remaining meanings as candidate meanings; inputting the candidate meanings and meaning data features into a conditional random model to obtain a probability ranking; selecting the meaning with the highest probability as the precise meaning and outputting the confidence score; the meaning data features include vocabulary, part-of-speech tags, location information, semantic similarity score, and confidence score.

[0013] Furthermore, the precise meaning of the pronoun is determined according to a hierarchical mechanism, which is as follows: find the historical importance score and level of the pronoun; identify the omitted subject based on semantic role labeling; capture the referential relationship across dialogue turns through an attention mechanism to generate several candidate entities; construct an entity-relationship graph, where nodes are entities and edges are relationships; identify indirect referential relationships; determine neighboring nodes; perform multi-turn message passing through GNN; calculate the matching score between the pronoun and the candidate entities respectively; and select the candidate entity with the highest matching score as the precise meaning of the pronoun.

[0014] Furthermore, the language recognition model includes a non-standard language prosody template library and standard grammar indicators. The similarity between the user's pronunciation pattern features and the non-standard language prosody template library is calculated. A pre-set similarity threshold is used; if the similarity exceeds the threshold, the non-standard language type with the highest similarity is selected as the reference non-standard language and marked as a non-standard pattern; if the similarity does not exceed the threshold, it is marked as a standard language. The pronunciation pattern features are then compared with the standard grammar indicators. If the standard grammar indicators are met, it is standard grammar and marked as a standard pattern; otherwise, it is non-standard grammar and marked as a spoken language pattern. The standard grammar indicators include sentence completeness, lexical standardization, and sentence complexity.

[0015] An intelligent voice interaction system, the system comprising: a voice feature extraction module, used to acquire user voice in real time for preprocessing, and determine emotion tags and pronunciation pattern features;

[0016] The language pattern recognition module is used to determine the language pattern of the user's speech based on pronunciation rules and grammatical indicators, and to extract key semantic information.

[0017] The word frequency segmentation module is used to generate standard word frequencies based on different language patterns and their word segmentation standards, and to extract features of question words, filler words, and pronouns;

[0018] The semantic reference resolution module is used to determine the precise meaning of each ambiguous word according to the discrimination rules and to determine the precise meaning of the pronouns according to the hierarchical mechanism.

[0019] The response generation and optimization module is used to generate emotionally matched and context-adapted response voices and operation commands based on semantic representation, emotion tags and vehicle status, and to optimize the interactive experience.

[0020] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0021] By constructing a non-standard language prosody template library and setting standard grammatical indicators, it is possible to accurately determine whether a user's speech is non-standard language, standard language, or colloquial expression, effectively solving the problem of insufficient understanding of non-standard language and colloquial speech in existing speech systems;

[0022] For the processing of interrogative words and pronouns, a combination of pre-trained language models, discrimination rules, and hierarchical mechanisms is adopted to improve the accuracy of semantic parsing. This enables precise determination of the meaning of interrogative words, capture of cross-talk turns of reference, and accurate reconstruction of the entities referred to by pronouns, making voice interaction more in line with natural language logic.

[0023] In terms of response generation, the system combines emotion tags and filler word features to generate emotion-matched and context-adapted response voices and operation commands. At the same time, considering driving status, a safety verification layer is added to ensure that appropriate responses are provided in different road conditions and emergency situations, thus ensuring driving safety. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a smart voice interaction method in an embodiment of the present invention;

[0025] Figure 2 This is a diagram illustrating the architecture of an intelligent voice interaction system according to an embodiment of the present invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, a more complete description of this application will be given below with reference to the accompanying drawings, which illustrate preferred embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to enable a more thorough and complete understanding of the disclosure of the present invention.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention; the term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0028] Example 1: As Figure 1 As shown, an intelligent voice interaction method includes:

[0029] S1: Acquire user voice, determine emotional tags and pronunciation pattern features, input pronunciation pattern features into language recognition model and output language pattern; if it is a standard language and standard grammar, mark it as standard pattern; if it is a non-standard language, mark it as non-standard pattern; if it is a standard language and non-standard grammar, mark it as spoken language pattern.

[0030] The language recognition model includes a non-standard language prosody template library and standard grammar indicators. It calculates the similarity between the user's pronunciation pattern features and the non-standard language prosody template library, pre-setting a similarity threshold. If the similarity exceeds the threshold, the non-standard language type with the highest similarity is selected as the reference non-standard language and marked as a non-standard language pattern; if it does not exceed the threshold, it is marked as standard language. The pronunciation pattern features are then compared with standard grammar indicators. If they meet the standard grammar indicators, it is standard grammar and marked as a standard pattern; otherwise, it is non-standard grammar and marked as a spoken language pattern. The standard grammar indicators include sentence completeness, lexical standardization, and sentence complexity.

[0031] The standard language refers to Mandarin Chinese, while the non-standard languages ​​refer to various local dialects and foreign languages ​​(such as English, German, French, etc., and corresponding language recognition data needs to be pre-set for different languages). The distinction between standard grammar and non-standard languages ​​is based on the premise of the standard language.

[0032] S2: For the standard mode, the speech text is segmented to obtain standard word frequencies, and response speech and operation instructions are generated by combining sentiment tags; for non-standard language mode and spoken language mode, the features of interrogative words, filler words and pronouns are extracted based on standard word frequencies; the precise meaning of each interrogative word is determined based on the discrimination rules, and the precise meaning of the pronouns is determined according to the hierarchical mechanism; based on the precise meaning, response speech and operation instructions are generated by combining sentiment tags and filler word features.

[0033] The real-time acquired user speech needs to be preprocessed. This preprocessing includes initial noise reduction based on a noise reduction algorithm, real-time enhancement of the speech signal, suppression of background noise (such as engine noise and wind noise in a vehicle environment), and improvement of the signal-to-noise ratio (SNR) of the input speech. Preferably, a lightweight convolutional neural network is used for real-time filtering. First, the input speech signal is processed by frame segmentation, with a frame length of 25ms and a frame shift of 10ms. A one-dimensional convolutional layer is used to extract time-frequency features, with a kernel size of 5 and a stride of 2. Temporal dependencies are captured through a gated recurrent unit (GRU) layer to suppress non-stationary noise. An encoder-decoder structure is adopted, where the encoder compresses noise features, and the decoder reconstructs clean speech. An adversarial training strategy is used, where the discriminator distinguishes between clean and noisy speech, and the generator optimizes the noise reduction effect. The final output is preprocessed speech with an improved SNR of more than 15dB, with processing latency controlled within 20ms.

[0034] An automatic speech recognition model is used to convert user speech into interactive text in real time, outputting the speech-text. Absolute accuracy must be ensured during the recognition process; unrecognizable words are marked to guarantee text recognition accuracy. During text recognition, filler words are identified and marked. The positions of filler words are marked during the speech recognition stage, and the prosodic features (duration, pitch, energy) of the filler words are extracted to establish a personalized filler word pattern library for the user. User speech text is acquired and compared with the filler word pattern library to identify filler word density, filler word type, and filler word position as filler word features. The filler word density is the number of filler words per unit time, which can be a fixed time, such as five seconds, or based on the average value of historical statistics on the duration of the user's speech. The filler word type is the distribution of different filler words, such as users frequently adding "ah" at the end of sentences to distinguish different users' speaking habits. The filler word position is the distribution pattern at the beginning, middle, and end of sentences.

[0035] The speech emotion feature vector is obtained based on user speech recognition. Sentence-level and character-level emotion features are obtained from the speech and text respectively. These features are then concatenated and normalized at the text embedding layer to generate a text emotion feature vector. The speech and text emotion feature vectors are input into a multilayer perceptron for fusion and classification, outputting emotion labels (e.g., positive, negative, neutral, angry, etc.). The classifier uses the Softmax function to output labels in probabilistic form, ensuring high confidence. Existing emotion models can be used to generate the emotion feature vectors. For example, the Wav2Vec-2.0 model can be used to extract emotion features from interactive speech to obtain speech emotion feature vectors, and MegatronBert can be used to extract text emotion feature vectors. The specific type of emotion feature extraction model is not limited in this embodiment.

[0036] Based on the user's speech, basic acoustic features and prosodic feature sets are extracted to determine the pronunciation pattern features; the prosodic feature set includes phoneme-level prosodic features, syllable-level prosodic features and sentence-level prosodic features, and the pronunciation pattern features include tone pattern vectors, rhythm pattern vectors and intensity pattern vectors.

[0037] The fundamental acoustic features include fundamental frequency sequence, energy characteristics, spectral characteristics, and duration characteristics. The fundamental frequency sequence refers to the trajectory of the fundamental frequency of vocal cord vibration in a speech signal over time. The RAPT algorithm is used to extract the fundamental frequency value of each frame, calculate the autocorrelation function of the speech signal, and identify periodic peaks. Dynamic programming is used to optimize the continuity of the fundamental frequency trajectory. The Viterbi algorithm is used for path search to ensure the smoothness of the fundamental frequency curve. The dynamic changes in the fundamental frequency profile are emphasized: the first-order difference (instantaneous rate of change) and the second-order difference (acceleration of change) are calculated to identify points of abrupt change. The dynamic changes in fundamental frequency reflect emotional intensity and non-standard language characteristics. For example, the fundamental frequency changes drastically under anger, and specific tone patterns in non-standard languages ​​exhibit regular changes. Identifying points of abrupt change and the changing trends throughout a sentence can serve as unique features for different non-standard languages.

[0038] The energy feature calculates the short-time energy (sum of squares of intra-frame samples) for each frame. The energy distribution pattern reflects changes in speech intensity and is used to detect stress positions and syllable intensity patterns in non-standard language recognition. The spectral feature extracts 12-dimensional MFCC features to characterize timbre characteristics; it also extracts auxiliary features such as spectral centroid and roll-off points. The spectral centroid reflects the brightness of the speech, and the roll-off points indicate the distribution of high-frequency components. The duration feature obtains phoneme-level duration information through phoneme boundary detection. It uses a deep learning-based forced alignment algorithm with a Bi-LSTM model. The input is the MFCC feature, and the output is the phoneme boundary probability. The Viterbi algorithm decodes the most probable phoneme sequence and boundary time points. The end time is subtracted from the start time, and normalization is performed to obtain the phoneme duration.

[0039] The phoneme-level prosodic features include fundamental frequency patterns, energy distribution, and phoneme transition patterns. Specifically, the fundamental frequency pattern involves piecewise linear fitting of the fundamental frequency sequence within each phoneme, extracting slope and curvature parameters to characterize the tone variation patterns of the phoneme. For example, Cantonese has a larger rising tone curvature, while standard languages ​​have a near-zero level tone slope. The energy distribution calculates the mean, variance, and rising / falling slope of the energy within a phoneme, reflecting non-standard language sensitivity features, such as detecting the intensity of plosive plosives through the amplitude and timing of energy surges. The phoneme transition pattern refers to the transition characteristics between adjacent phonemes in non-standard languages, including coarticulation effects. For example, it detects non-standard language sensitivity features such as plosive plosive intensity and fricative duration, calculating the spectral change rate and energy continuity at phoneme boundaries for measurement.

[0040] The syllable-level prosodic features include syllable fundamental frequency profile, syllable energy envelope, and syllable duration pattern. Specifically, the syllable fundamental frequency profile is obtained by extracting the maximum, minimum, range, and shape (e.g., level tone, rising tone, falling tone) of the fundamental frequency within a syllable. This is described by curve fitting coefficients of the fundamental frequency points. A quadratic polynomial is used to fit the fundamental frequency trajectory, and the coefficients are extracted as features to distinguish different tone patterns in tonal languages, such as the four tones in standard languages. The syllable energy envelope calculates the position of the syllable energy peak and the envelope slope (the average slope of the rising and falling energy segments), reflecting the stress pattern. The syllable duration pattern analyzes the ratio of syllable duration to adjacent syllables to capture rhythmic patterns unique to non-standard languages ​​(e.g., the relatively even distribution of syllable duration in Cantonese).

[0041] The sentence-level prosodic features include the fundamental frequency trend of the entire sentence, pause patterns, and intonation profiles. The fundamental frequency trend of the entire sentence is determined by performing a polynomial fit on the fundamental frequency sequence of the entire sentence, extracting the global slope and fluctuation amplitude, fitting the fundamental frequency curve using cubic spline interpolation, and extracting trend parameters, such as distinguishing between declarative and interrogative sentences and determining emotional intensity. The pause patterns detect the duration of pauses between words (the duration of silent segments between words) and the location of pauses (the distribution pattern of pauses in the sentence), analyzing pause patterns unique to non-standard languages ​​(such as the shorter pauses in Sichuan dialect). The intonation profile is calculated by comparing the intonation of the current sentence with a standard language template using the DTW algorithm to calculate the deviation.

[0042] The pronunciation features include tone pattern vectors, rhythm pattern vectors, and intensity pattern vectors. Specifically, the tone pattern vector is based on phoneme-level fundamental frequency features to identify tone types and change patterns, including intensity changes of level, rising, falling, rising-falling, and falling-rising tones. The rhythm pattern vector calculates a rhythm index based on syllable duration features to quantify the rhythmic characteristics of non-standard languages, including syllable duration variance, pause frequency, and speech rate change rate. The intensity pattern vector is based on energy distribution analysis results to identify stress position patterns, including stress position, number of energy peaks, and energy change slope.

[0043] The pronunciation pattern features are used to describe the salient and representative features of the current user's speech. Determining the pronunciation pattern features includes: extracting basic acoustic features and prosodic feature sets from the original speech, which can be the current user's speech or the user's historical speech; performing cross-level association and weighted fusion of multi-granularity features; and extracting pronunciation pattern features through a pattern recognition algorithm. The tone pattern vector is mainly generated based on phoneme-level and syllable-level fundamental frequency features. Phoneme-level fundamental frequency patterns and syllable-level fundamental frequency contours are used as feature inputs. A Gaussian mixture model (GMM) is used to perform cluster analysis on the fundamental frequency features (using the K-Means clustering algorithm, the number of clusters needs to be pre-defined, preferably 5). Each cluster center represents a typical tone pattern (level tone, rising tone, falling tone). The matching degree between the current speech and each tone pattern is calculated through maximum a posteriori probability estimation. The top K tones with the highest matching degree are selected to form a tone pattern vector. Here, K can be the number of clusters. If the number of clusters is not preset, a corresponding threshold needs to be set for selection. The maximum number of clusters cannot exceed five. The 5-dimensional vector represents the intensity of the five basic tone patterns. The value of each dimension is normalized by the softmax function, which represents the probability of belonging to that tone pattern. Example: The fourth tone (falling tone) in standard language will show a high value in the falling tone intensity dimension, while the entering tone in Cantonese shows significant performance in both the level tone intensity and falling tone intensity dimensions.

[0044] The rhythm pattern vector is generated based on duration and pause features. Phoneme duration features, syllable duration patterns, and sentence-level pause patterns are used as feature inputs. The statistical features of each rhythm parameter (such as speech rate change rate and pause patterns) are calculated to obtain the rhythm pattern vector.

[0045] The intensity pattern vector is generated based on energy features and stress patterns. Phoneme-level energy distribution, syllable-level energy envelope, and sentence-level energy trend are used as feature inputs. The dynamic time warping (DTW) algorithm is used to align the energy envelope template, and principal component analysis (PCA) is used to extract the main patterns of energy distribution, identify stress patterns, and generate intensity pattern vectors.

[0046] The three types of vectors are concatenated and fused to obtain pronunciation pattern features. Phoneme-level features provide detailed information, syllable-level features reflect local patterns, and sentence-level features reflect the overall pattern. The contribution of each level of features is dynamically adjusted through attention weights.

[0047] The purpose of further extracting pronunciation pattern features based on the basic acoustic features and prosodic feature set in this embodiment is to improve the accuracy of user speech recognition and the processing efficiency of user speech, and further reduce processing time.

[0048] Speech data from various non-standard languages ​​(such as Cantonese, Wu, and Xiang) are collected and labeled with their respective types. Basic acoustic and prosodic feature sets are extracted for each non-standard language to construct a non-standard language prosodic template library. Similarity calculations are performed between user pronunciation patterns and the pre-built non-standard language prosodic template library, such as calculating the cosine similarity with each non-standard language template. The non-standard language type with the highest similarity is selected as a reference, and a probability vector is generated. A non-standard language probability vector is added to represent the confidence level of belonging to each non-standard language.

[0049] The similarity threshold was determined based on test results from a large-scale non-standard language speech database (covering 10 major non-standard languages, such as Cantonese, Wu, and Xiang). Specifically, a large-scale speech database containing 10 major non-standard language types was constructed, with speech samples collected from no fewer than 200 native speakers for each non-standard language, resulting in a total sample size exceeding 10,000. Each sample was labeled with its true language type (standard language or specific non-standard language) as the baseline ground truth for ROC analysis. Cosine similarity was calculated between the user's pronunciation pattern features (tone pattern vector, rhythm pattern vector, and intensity pattern vector) and each template in the non-standard language prosody template library to obtain a similarity score. Within the range of 0.00 to 1.00, all possible similarity thresholds were iterated with a step size of 0.01. For each threshold, precision and recall were calculated. Through ROC curve analysis, experiments demonstrated that a similarity threshold of 0.75 balanced precision and recall, avoiding false negatives (misclassification of non-standard languages ​​as standard languages) caused by excessively high thresholds or false positives caused by excessively low thresholds. Similarity calculation employs a cosine similarity algorithm, comparing the matching degree between user pronunciation pattern features (tone, rhythm, and intensity vectors) and template features in a non-standard language prosody template library. During calculation, feature parameters are converted into corresponding feature vectors before cosine similarity calculation. The template library is generated through cluster analysis (such as K-means), with each non-standard language category containing at least 100 typical samples. If the similarity between a user's pronunciation pattern features and a certain non-standard language template is greater than 0.75, the non-standard language type with the highest similarity is selected as the reference non-standard language and marked as a non-standard language pattern. If the similarity of all non-standard languages ​​is not greater than 0.75, it is marked as a standard language and proceeds to the grammatical indicator evaluation stage.

[0050] Standard grammatical metrics are used to determine whether standard language conforms to normative expression, including three dimensions: sentence completeness, lexical standardization, and sentence complexity. A sentence is marked as a standard pattern when all of the following thresholds are met: Sentence completeness refers to the completeness of the subject-verb-object structure, calculated through dependency parsing. The completeness score is based on the coverage ratio of core components (such as subject, verb, and object) in the syntax tree. Based on standard Chinese grammar, a complete sentence must contain at least one subject-verb structure; however, voice commands in autonomous driving often omit the subject (e.g., playing a song). Experimental data shows that a score of 0.65 can effectively distinguish between complete sentences and elliptical sentences (e.g., subjectless sentences commonly found in spoken language; "turn on the air conditioner" scores 0.8). Preferably, the driving system pre-stores common standard voice control commands. During recognition, if a preset voice control command is detected, the system can directly respond and operate the command based on the emotion tag, setting the emotional response of the voice, thus reducing processing time.

[0051] For lexical standardization and sentence complexity, corresponding thresholds were set based on experimental data. Lexical standardization refers to the proportion of standard vocabulary (vocabulary included in a dictionary) used in the text, i.e., the ratio of the number of standard words to the total number of words, excluding non-standard words such as slang and internet slang. Research on the purity of the Chinese language indicates that the proportion of standard words in standard text is usually greater than 80%; below this value, it tends to be colloquial. Sentence complexity is measured by the average sentence length (number of words) and the proportion of complex sentences (sentences containing clauses or conjunctions). The average sentence length of standard Chinese text is 5-10 words, and the proportion of complex sentences is 30%-50%; below this value, it is often a simplified expression of colloquial language. Sentences are segmented based on punctuation, and a list of conjunctions (such as "because" and "but") is used to identify complex sentences. The embodiments of this application mainly target the identification and response to complex sentences with non-preset instructions or non-preset vocabulary.

[0052] When the standard language does not meet the standard grammatical indicators, the embodiments of this application determine whether it is a spoken language pattern. To further improve the recognition accuracy, spoken language grammatical indicators are set, including filler word density, fragmentation degree, and word order disorder. The spoken language grammatical indicators can be set by reversing the threshold setting methods of the above standard grammatical indicators; this application will not elaborate further here. All thresholds are embedded in a feedback loop and updated quarterly based on user group data (such as region and age) to avoid rigidity. The similarity threshold prioritizes controlling the false detection rate (ensuring robust standard language recognition), while the grammatical threshold prioritizes controlling the false negative rate (ensuring sensitivity to spoken language patterns).

[0053] In some embodiments, extracting ambiguous words involves calculating the semantic confidence of words in context based on a pre-trained language model. This includes: pre-labeling a dataset, where each word is labeled as either ambiguous or not, as training data; using the training set to train the model and optimize its parameters; using the validation set to validate the parameters; further optimizing the model parameters; finally obtaining a fully trained pre-trained language model; outputting a list of semantic confidence scores for each word; identifying the word with the lowest confidence score as an ambiguous word; using an attention weight matrix to label the positions of low-confidence words; and building an ambiguous word dictionary containing common ambiguous words and their possible meanings.

[0054] The precise meaning of each ambiguous word is determined based on discrimination rules, including: taking a number of words before and after the ambiguous word to form a context window (the window size is adjustable and dynamically set according to the type of ambiguous word: 5-word windows are generally used for polysemous words, and 3-word windows are used for homophones). A sliding window algorithm is used to extract the window content in real time; a functional word vector model is used to obtain the vector representation of each meaning of the ambiguous word, and the similarity between each meaning vector representation and the context vector is calculated. A pre-set association similarity threshold is used; if the similarity is less than the association similarity threshold, the corresponding meaning is eliminated; the remaining meanings are marked as candidate meanings; the candidate meanings and meaning data features are input into a conditional random model to obtain a probability ranking; the meaning with the highest probability is selected as the precise meaning, and the confidence score is output; the meaning data features include vocabulary, part-of-speech tags, location information, semantic similarity score, and confidence score. TF-IDF weights are introduced into the similarity calculation to highlight the influence of keywords within the window; the threshold is dynamically adjusted and optimized based on the validation set F1 score. The Conditional Randomized Model (CRF) outputs the probability distribution of the meaning of each questionable word based on the candidate meanings and meaning data features, combined with contextual information, such as P(meaning 1) = 0.8, P(meaning 2) = 0.2. The L-BFGS algorithm is used to optimize the CRF parameters. The feature template includes unary features (current word) and binary features (relationship between adjacent words), which will not be elaborated on in this application.

[0055] Specifically, BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. It captures contextual information from text through a bidirectional encoder. The input consists of a word-segmented list of words derived from user speech recognition, along with contextual information (such as historical dialogue data, scene labels, and pronunciation patterns) to enhance context awareness. A pre-trained language model is used as the foundation, and a fully connected layer is added after the BERT output. A sigmoid function is used to output a confidence score (0-1) for each word. A lower score indicates a more ambiguous semantic meaning. Multi-head self-attention is employed, with a scaling factor set, and attention weights are normalized using Softmax. When focusing on semantically ambiguous locations, an attention weight threshold (e.g., below 0.5 times the standard deviation of the average weight) is set to mark low-confidence words. A pre-labeled dataset, where each word is marked as either ambiguous or not, was used as training data. 100,000 user speech texts were randomly selected from the in-vehicle voice interaction logs and manually labeled by five linguistics professionals. The labeling standard was: if a word's meaning in the current context is ambiguous, it has a rare usage, or the user's pronunciation is unclear, making its meaning difficult to determine, it was marked as an "ambiguous word"; otherwise, it was marked as a "non-ambiguous word." The labeled data was then divided into training, validation, and test sets in an 8:1:1 ratio. The training set was used to train the model and optimize its parameters, while the validation set was used to validate the parameters and further optimize the model parameters. The model training hyperparameter settings included a batch size of 32 and an initial learning rate of... The AdamW optimizer was used with a weight decay factor of 0.01. Training consisted of 5 epochs, with evaluation using a validation set after each epoch. The model with the highest F1 score on the validation set was selected as the final model. An early stopping mechanism was employed during training; training was halted if the F1 score on the validation set did not improve after two consecutive epochs. The final result was a fully trained pre-trained language model that output a list of semantic confidence scores for each word, with the lowest confidence score identified as a question word. Simultaneously, an attention weight matrix was used to label the positions of low-confidence words. The BERT model can capture long-distance contextual dependencies, and its attention mechanism can automatically focus on semantically ambiguous points (such as the ambiguity of polysemous words in a specific context).

[0056] Filler words refer to words or sound fragments in natural language expression that do not carry core semantic information but have specific pragmatic functions. In intelligent voice interaction, filler word analysis is of great significance for understanding users' expression habits, emotional states, and thought processes. The filler word features include filler word density, filler word type, and filler word position. During the speech recognition stage, filler word positions are marked, prosodic features are extracted, and a personalized filler word pattern library is established for each user. The user's voice text is then acquired and compared with the filler word pattern library to identify filler word features.

[0057] The precise meaning of a pronoun is determined based on a hierarchical mechanism, which involves: finding the historical importance score and level of the pronoun; identifying omitted subjects based on semantic role labeling; capturing referential relationships across dialogue turns through an attention mechanism to generate several candidate entities; constructing an entity-relationship graph, where nodes represent entities and edges represent relationships; identifying indirect referential relationships; determining neighboring nodes; performing multi-turn message passing through a GNN; calculating the matching score between the pronoun and the candidate entities; and selecting the candidate entity with the highest matching score as the precise meaning of the pronoun.

[0058] In some embodiments, the historical importance score and hierarchical level of pronouns are retrieved. The importance of the entity referred to by the pronoun is identified based on the historical importance score, and the hierarchical level is combined to concentrate limited computational resources on key referential relationships, reducing unnecessary data processing time and improving the accuracy of identifying omitted subjects and finding referential relationships. Semantic role labeling is the process of identifying the semantic relationships between predicates and other components in a sentence. In referential resolution, its core role is to identify the absence of subjects due to omission, thereby accurately restoring the referential relationships. A Transformer-based SRL model is used to encode the input sentence, identifying each verb as a predicate and labeling each predicate with its argument roles, including: Agent, Patient, Time, and Location. Based on a Chinese ellipsis grammar rule base, ellipsis patterns are determined. A rule engine is used to match the ellipsis pattern in the current sentence. When ellipsis is detected, the missing position and type are marked. Candidate entities are searched based on a context window, looking for matching positions within the first three sentences of the current sentence and predicting possible positions within the last two sentences. Semantic similarity is used to calculate the matching degree between candidate entities and missing positions, and the candidate entity with the highest similarity is selected as the restored ellipsis subject. Ellipsis subject recognition ensures the integrity of the referential chain in referential resolution, improves resolution accuracy, and supports natural dialogue understanding.

[0059] Referential relations are not about the entities themselves, but rather the reference relationships between entities, including coreference relations: different expressions refer to the same entity (such as "Li Bai" and "he"); part-whole relations: such as "car" and "tire"; attribute relations: such as "apple" and "red"; and temporal relations: such as "eat first" and "rest then". Capturing cross-talk turn referential relationships through an attention mechanism specifically includes: encoding the current referent as a query vector Q, and encoding entities in the historical dialogue as key vectors K and value vectors V, with each entity containing its semantic vector and positional encoding; calculating attention weights, using scaled dot product attention to calculate the association strength between the current referent and historical entities: calculating the dot product of Q and each K to measure semantic similarity, incorporating relative positional encoding to reflect the influence of temporal distance, and applying a softmax function to normalize to attention weights; sorting historical entities according to attention weights, selecting the top K entities with the highest weights as a candidate set (usually K=5), with each candidate entity accompanied by its attention weight as confidence; constructing a cross-talk turn referential relationship graph, using a graph attention network to aggregate multiple neighbor information, and capturing indirect referential relationships (such as referential chains established through intermediate entities).

[0060] The attention mechanism enables long-distance dependency modeling, effectively capturing referential relationships across multiple dialogue turns; dynamic weight allocation adjusts the importance of historical entities based on the current context, and the attention weights intuitively display the basis for referential relationships.

[0061] Construct an entity-relationship graph where nodes represent entities and edges represent relationships. Identify indirect referential relationships and determine neighboring nodes. Specifically, create a unique node for each identified entity, with node attributes including entity name, semantic vector, creation time, and usage frequency. Use a deep learning model to generate a 256-dimensional semantic vector representation. Edge relationships include co-referential edges (connecting different expressions pointing to the same entity), semantic edges (based on semantic relationships between entities), temporal edges (representing the temporal order of entities), and dialogue edges (recording the co-occurrence of entities in the same dialogue). Discover indirect referential relationships through graph path search to improve retrieval efficiency.

[0062] Graph Neural Networks (GNNs) are deep learning models specifically designed for graph-structured data. Their core idea is to update node representations by aggregating information from neighboring nodes. Specifically, node representations are initialized using their semantic vectors, with positional encoding used to distinguish the same entity across different dialogue rounds. An iterative message passing process is then implemented. In the k-th iteration, the message is calculated as follows: for each node, representations of its neighbors are collected; a message function is used to calculate the message from each neighbor; attention weights are used to weight and aggregate neighbor messages; the aggregated message is combined with the current node state; a gating mechanism is used to control the degree of information update; and the updated node representation is output. Multiple iterations can discover two-hop indirect referential relationships based on capturing direct referential relationships, integrating global referential network information. Typically, three iterations achieve optimal results. The updated pronoun node representations and candidate entity node representations are used to calculate the cosine similarity as the matching score.

[0063] A memory importance assessment model is constructed to generate an importance score for each pronoun and classify them into different storage levels. The storage levels include a short-term memory layer, a medium-term memory layer, and a long-term memory layer. The utilization rate of each pronoun in each memory layer is monitored. If the utilization rate is lower than 40%, it is cleaned up. If the utilization rate is higher than 70%, it is automatically migrated to the upper memory layer, such as from the short-term memory layer to the medium-term memory layer. The utilization rate refers to the frequency of user use, which is obtained based on user speech recognition.

[0064] The utilization threshold for tiered storage is determined based on statistical analysis of user voice interaction data. Specifically, by collecting voice interaction logs from 100 users over 30 consecutive days, the frequency distribution of pronouns in each memory tier is statistically analyzed. The percentile method is used to determine the critical points for cleaning and migration: pronouns with a usage frequency below 40% are defined as low-frequency, corresponding to the lower limit of the cleaning threshold; pronouns with a usage frequency above 70% are defined as high-frequency, corresponding to the upper limit of the migration threshold. To adapt to the interaction habits of different users, the system supports a dynamic adjustment mechanism: after every 50 rounds of dialogue, the frequency distribution of pronouns in each memory tier for the current user is recalculated. If the average utilization rate of a memory tier deviates from the initial threshold by more than ±10% in three consecutive statistical analyses, a threshold update is triggered, adjusting the threshold to the 30th and 70th percentiles of the current frequency distribution. The update takes effect immediately.

[0065] In constructing the memory importance assessment model, the formula for calculating the importance score is as follows: ,in, It represents the i-th pronoun. It is an importance score. Refers to the chain integrity score. It is a frequency score. It is a context relevance score. It is a personalized weighted score. It is the score for temporal continuity; These are the weight coefficients for the corresponding scores, initialized to... The sum of the weight coefficients is 1, which emphasizes the contribution of the chain integrity score and the context relevance score. In actual use, it needs to be optimized according to the actual feedback results. This application does not impose specific restrictions here.

[0066] The formula for calculating the chain integrity score is as follows:

[0067]

[0068] in, It refers to the reference chain length, that is, the number of connections of an entity in the reference chain. For example, if "it" appears three times consecutively in the context, then the reference chain length is 3. This refers to the number of chain breaks; the higher the value, the lower the score. This is the length weighting coefficient, initially set to 0.5; This is the breakage penalty coefficient, initially set to 0.3. When presetting the weighting coefficients, the optimal weighting coefficient value is selected as the initial coefficient for this voice interaction based on historical experimental statistics, i.e., the initial setting value.

[0069] The formula for calculating the frequency score is as follows:

[0070]

[0071] in, It is the number of times the i-th pronoun is accessed within the time window t, and the time window is the same as the context window mentioned above. It is the maximum number of visits for the i-th pronoun, and it is much greater than 1. The maximum number of visits is dynamically set according to the statistical distribution of historical actual dialogue data. For example, the 95th percentile of the historical dialogue visits for different pronouns is taken as the maximum number of visits, and it is dynamically updated to adapt to the interaction habits of different users. This is the time decay constant, set according to the time interval between multiple voice interactions by the user, initially set to 200 seconds; It is the difference between the current time and the last access time.

[0072] The formula for calculating the context relevance score is as follows:

[0073]

[0074] in, This refers to the number of keywords in the current conversation. The keywords are standard words with clear meanings in the user's speech, which can provide accurate meanings to help understand pronouns. Cosine similarity is used to measure the directional similarity between two non-zero vectors in a multidimensional space. The value range is [-1, 1], where 1 indicates that they are in the same direction, -1 indicates that they are in opposite directions, and 0 indicates that they are orthogonal and have no correlation. Only the semantic vectors that have correlation and the keywords are extracted to calculate the correlation score. It refers to the semantic vector of pronouns; It is the vector of the j-th keyword in the current dialogue context; It is the keyword weight, calculated based on TF-IDF.

[0075] The formula for calculating the personalized weight score is as follows:

[0076]

[0077] in, This refers to the degree of personalization of pronouns for users. It is set by normalization based on the total frequency of occurrence of pronouns and the frequency of occurrence in a single sentence, with a value range of [0, 1]. The personalization threshold, based on statistical analysis of historical experimental data, is set to 0.6. Specifically, during the model training phase, a voice interaction dataset containing 500 users is used to label the degree of personalization of each pronoun to the user (judged manually as "high" or "low"). The threshold parameter is optimized through grid search so that the threshold that achieves the highest F1 score on the validation set is 0.6. The personalization threshold supports dynamic adjustment. After processing every 1000 user voice messages, the personalization weight model is retrained using the most recently accumulated user data, and the values ​​are updated using the same grid search method to ensure that the personalization weight score remains synchronized with the user's actual usage habits. is the steepness coefficient of the Sigmoid function, set as a baseline value of 5. Based on the mathematical properties of the Sigmoid function and the optimal parameter selection of the experimental data, using the aforementioned speech interaction dataset, a grid search method is employed to traverse the steepness coefficient range [1, 10] with a step size of 0.5. The optimization objective is the discriminative power of the personalized weight scores on the dataset (i.e., the ratio of high-personalization pronoun scores to low-personalization pronoun scores). Experimental results show that when the baseline value is 5, the discriminative power reaches its maximum value, and the transition smoothness of scores near the threshold is moderate, avoiding score binarization caused by excessive steepness and insufficient improvement in high-personalization pronoun scores due to excessive smoothness.

[0078] The formula for calculating the temporal continuity score is as follows:

[0079]

[0080] in, It is the current timestamp; It is the timestamp of the last occurrence of the pronoun; This is the time scale parameter, set to 100 seconds.

[0081] Based on precise meaning, response voices and operation commands are generated by combining emotional tags and filler word features. The emotional aspect of the response voice needs to be set according to the emotional tags to generate response voices that match the user's emotions. For example, a soothing voice is output for negative emotions, and an affirmative voice is output for positive emotions. Specifically, emotional tags need to be transformed from discrete tags into quantifiable prosodic control parameters: emotional intensity mapping: mapping emotional categories (such as positive and negative) to intensity values ​​(0-1 scale), for example: neutral is 0.3, mild positive is 0.6, and strong positive is 0.9; in addition to basic emotions, driving-related dimensions are added: based on voice rate and energy changes, urgency, anxiety (assessed through filler word frequency and pitch fluctuation), and trust (dynamically adjusted based on historical interaction success rates) are set within the range of [0, 1].

[0082] Based on real-time vehicle data, environmental data, and user commands, operational instructions are executed according to priority. The execution of these instructions follows a closed-loop process: instruction parsing, safety verification, decision planning, execution monitoring, and feedback updates. This process works closely with core autonomous driving modules (such as perception, decision-making, and control) to ensure accurate translation of instructions into vehicle behavior. The collaboration between operational instructions and the core autonomous driving modules is achieved through a unified interface layer, employing a service-based communication architecture (ROS2) as the underlying communication protocol. Instructions are synchronously invoked via ROS2's Service / Client mechanism, ensuring verifiable execution results. The instruction data format uses JSON structured encoding, with each instruction containing the following fields: command_id (unique identifier), command_type (e.g., "ACCELERATE"), parameters (parameter dictionary, such as acceleration value, lane number), priority (value range 0-100, higher values ​​indicate higher priority), timestamp (generation timestamp), and user_id (user identifier). The core autonomous driving modules provide standard service interfaces, which the voice system calls through a service client. When a user command conflicts with a vehicle safety system (such as automatic emergency braking or collision avoidance warning), a safety-priority adjudication mechanism is employed. The specific process is as follows: After a command is generated, it first enters the safety verification layer, which simultaneously obtains the current status of the vehicle safety system (e.g., whether AEB is activated or ESC is engaged). If the safety system status is "activated," adjudication is made according to the conflict level: Level 1 conflict (e.g., the user requests acceleration but AEB is braking) directly rejects the command and generates a voice prompt "The current safety system is engaged, and this operation cannot be performed"; Level 2 conflict (e.g., the user requests a lane change but blind spot monitoring alarm) downgrades the command to a suggestion mode, only issuing a voice prompt "Vehicle in blind spot, lane change is risky," without actual execution; no conflict or Level 3 conflict (e.g., the user requests adjusting the air conditioning temperature, which is unrelated to the safety system) is executed normally. All conflict adjudication results are recorded in the system log for subsequent optimization of the command generation strategy. Safety verification rules are set to perform multi-layer checks on the parsed operation commands to ensure that the commands comply with safety regulations, including: static verification, checking whether the command complies with traffic rules and vehicle capabilities (e.g., speed limits, lane legality). Dynamic risk verification involves real-time analysis of the vehicle environment (obstacle and road condition data acquired through the perception module) and limiting the scope of commands based on user identity (e.g., driver or passenger) (e.g., only the driver can perform emergency operations). For navigation commands (e.g., route change), the optimal path is recalculated, taking into account real-time traffic and energy consumption. For control commands (e.g., acceleration), vehicle behavior strategies (e.g., following distance, acceleration curve) are adjusted to ensure smooth execution. For example, acceleration commands require optimizing the acceleration curve based on the current traffic flow speed.If a user issues multiple commands consecutively (such as avoiding obstacles and then accelerating), a priority scheduling algorithm (such as weighted round-robin based on urgency) is used to ensure that the commands are executed in an orderly manner.

[0083] Filler words are not only indicators of language habits, but also important clues to emotional states. Filler words should be classified and processed accordingly, including: hesitant filler words ("uh", "ah"), which require more explicit guidance; cohesive filler words ("that", "just"), which maintain the coherence of the conversation; and emotional filler words ("oh my god", "wow"), which enhance emotional resonance. We should establish user filler word usage profiles, learn personalized filler word patterns, calculate the filler word density index (number of occurrences per unit time), and analyze filler word position patterns (beginning of sentence, middle of sentence, end of sentence).

[0084] A correlation model between sentiment tags and filler word features is established to generate joint sentiment-filler word features. This includes generating a sentiment-filler word association matrix and setting corresponding response strategies, such as: high anxiety sentiment + high filler word density: requiring a reassuring response; positive sentiment + low filler word density: a concise and affirmative response is appropriate; neutral sentiment + filler words in specific positions: maintaining an informative tone. The feature fusion algorithm uses an attention mechanism to weightedly fuse multiple features, with sentiment features having a weight of 0.6 and filler word features having a weight of 0.4; the weights are dynamically adjusted based on the dialogue context.

[0085] A safety verification layer is added before the response is generated to monitor the driving status and obtain vehicle data in real time: vehicle speed, road conditions, obstacle distance, calculate driving emergency index (0-1 scale), classify response priorities, reduce response delay, automatically smooth tone when the emotional intensity is too high, simplify language structure in complex road conditions, and prioritize the clarity of instructions in emergency situations.

[0086] The final response is generated based on fusion features, with prosodic parameters controlled as follows: Fundamental frequency: For every 0.1 increase in emotional intensity, the fundamental frequency range expands by 5%; Speech rate: When the filler word density is >0.3 words / second, the speech rate decreases by 20%; Pauses: When anxiety levels are high, comforting pauses (0.3-0.5 seconds) are added. Personalized speech synthesis: Non-standard language mode: Adapts to regional intonation variations; Age mode: Younger users speak faster, while older users speak with a more stable intonation; Habit mode: Preserves the user's unique language rhythm.

[0087] The method further includes: S3: Real-time calculation of the decay curve of the pronoun resolution accuracy; calculation of hierarchical storage efficiency, data processing efficiency, and recognition accuracy based on historical data and the decay curve; determination of the optimal recognition rule based on the peak change trend of the recognition curve; improvement of pronoun recognition accuracy; and reduction of data processing time. The recognition rule includes the storage and utilization efficiency of pronouns at each memory level, as well as the historical dialogue usage range. This embodiment aims to dynamically determine the optimal recognition rule by comprehensively analyzing the trade-off between hierarchical storage efficiency, data processing speed, and recognition accuracy through real-time calculation of the pronoun resolution accuracy decay curve. The core of the solution lies in establishing a closed-loop optimization system: real-time monitoring of pronoun resolution performance decay, triggering rule adjustments based on decay characteristics, and continuous optimization through a feedback mechanism, ultimately improving pronoun recognition accuracy and processing time. The solution combines statistical modeling, real-time analysis, and adaptive learning to ensure the system maintains high performance in long-term dialogues.

[0088] The decay curve describes the decreasing accuracy of referential resolution as the number of dialogue rounds increases. Accuracy decay is mainly caused by factors such as increased memory burden, contextual complexity, and broken referential chains. Real-time calculation of the decay curve is fundamental to optimization. Specifically, after each dialogue round, the referential resolution result (correct or incorrect), the dialogue round timestamp, resolution time, and memory layer information used are recorded. A sliding window method (window size set to the most recent 10 dialogue rounds) is used to calculate instantaneous accuracy. The formula is: An exponential decay model was used to fit the accuracy curve, and online least squares method was used to update the parameters in real time. The model form is as follows:

[0089]

[0090] Where t is the number of dialogue rounds, A(t) is the reference resolution accuracy after round t, A0 is the initial accuracy, and λ is the decay rate. For steady-state accuracy, the model parameters are updated online using least squares. After each dialogue round, the data points (ti, Ai) of the current sliding window (the last 10 dialogue rounds) are used as input, and the parameters are updated in real time by minimizing the squared error. and The update formula employs the Recursive Least Squares (RLS) algorithm, enabling the model to adapt to recent dialogue trends. The optimal recognition rule is determined by selecting the rule combination that maximizes the overall performance index within the peak trend of the decay curve. The peak trend is determined by detecting the point where the second derivative of the decay curve is zero; this point corresponds to the inflection point where accuracy transitions from rapid to slow decline, indicating that the current memory layer configuration has reached its optimal balance. Rule optimization is triggered when the system detects that the current recognition rule (i.e., the memory layer partitioning strategy, cleanup, and migration thresholds) causes the overall performance index to fall below 90% of the historical best. The optimization process uses a grid search approach, exploring new rule combinations within a preset parameter space: the short-term memory retention time is adjusted between 10-30 rounds, the cleanup threshold between 30%-50%, and the migration threshold between 60%-80%. Using the most recent 500 dialogue rounds as the evaluation set, the rule combination that maximizes the overall performance index is selected to replace the current rule; the replacement takes effect at the start of the next dialogue round. After each dialogue round, the curve is refitted based on the window data.

[0091] The hierarchical storage efficiency, data processing efficiency, and recognition accuracy are calculated. The peak value trend of the recognition curve is analyzed to determine extreme points and select the optimal time node. For example, the optimal historical data node for pronoun A is historical dialogue data within the last 12 hours, which greatly reduces data processing time, improves data processing efficiency, and quickly identifies the meaning of user speech, providing a response. Hierarchical storage efficiency is calculated by: calculating the utilization rate of each memory layer (the ratio of used memory to total memory) and the storage efficiency score (the percentage of fixed storage in each memory layer; ideal storage efficiency is 60%, and lower or higher scores indicate uneven storage allocation). Data processing efficiency is calculated by: calculating the average processing time (the average time from query to completion of pronoun resolution) and query efficiency (the ratio of successful queries to total queries, measuring the effectiveness of the memory-level query strategy). The pronoun resolution described in this application refers to the process of pronouns going from unclear meaning to accurate recognition. Recognition accuracy analysis: Short-term accuracy: based on the resolution results of the most recent 3 rounds of dialogue. Long-term accuracy: based on the resolution results of the entire dialogue history. Error types (such as false positives and false negatives) are analyzed using a confusion matrix to identify weaknesses in the rules. A comprehensive analysis is performed every 5 rounds of dialogue: the objective function value is calculated, and storage efficiency scores and decay rates are checked. A diagnostic report is generated to guide rule adjustments. Based on the decay curve and the comprehensive analysis results, the recognition rules are dynamically adjusted to optimize the objective function and determine the optimal recognition rules.

[0092] In this embodiment, an automatic speech recognition model is used to convert speech into text, and the positions of filler words are marked and prosodic features are extracted to establish a personalized filler word pattern library. A pre-trained language model is used to calculate the semantic confidence of words to extract question words, and discrimination rules and conditional randomization models are applied to determine the precise meaning of these question words. A hierarchical mechanism is adopted, combining semantic role labeling, attention mechanisms, entity relationship graphs, and graph neural networks to determine the precise meaning of pronouns. A memory importance assessment model is constructed to hierarchically store and manage pronouns. Finally, response voices and operation commands are generated based on the precise meaning, sentiment tags, and vehicle status, and the decay curve of pronoun resolution accuracy is calculated in real time to optimize the recognition rules.

[0093] It solves the limitations of existing in-vehicle voice interaction functions and the problem of low integration with the core decision-making and planning modules of autonomous driving; it overcomes the difficulties of inaccurate understanding of non-standard language and colloquial expressions, and inaccurate parsing of doubt words and pronouns; it improves the situation where the response voice and operation instructions do not match the user's emotions and context; it enhances the performance and stability of the system in long-term use, reduces data processing time, and improves the accuracy of pronoun recognition.

[0094] It achieves a high degree of integration between the intelligent voice interaction system and the autonomous driving system; it can accurately identify user voice patterns, accurately parse semantics, and generate emotionally matched and context-appropriate response voices and operation commands; it effectively improves the accuracy, naturalness, and security of voice interaction; and by dynamically optimizing recognition rules, it enhances the overall performance and stability of the system, providing users with a higher quality and more efficient intelligent voice interaction experience.

[0095] Example 2: Figure 2 As shown, this embodiment also provides an intelligent voice interaction system, the system comprising:

[0096] The speech feature extraction module is used to acquire user speech in real time for preprocessing, determine emotional tags and pronunciation pattern features; it takes the original speech signal as input and outputs the preprocessed speech, basic acoustic features (fundamental frequency, energy, spectrum, duration), prosodic feature set (phoneme level, syllable level, sentence level features) and pronunciation pattern features (tone, rhythm, intensity vector) to provide high-quality speech signals and basic feature data for subsequent processing.

[0097] The language pattern recognition module is used to determine the language pattern (standard pattern, non-standard language pattern, spoken language pattern) of the user's speech based on pronunciation pattern features and grammatical indicators, and to extract key semantic information; it takes pronunciation pattern features and speech text as input, and outputs language pattern labels (standard / non-standard language / spoken language), standard word frequency, list of interrogative words, filler word features (density, type, position) and list of pronouns.

[0098] The word frequency segmentation module is used to generate standard word frequencies based on different language patterns and their word segmentation standards, and to extract features of question words, filler words, and pronouns; it directly segments words and counts word frequencies for standard patterns; and for non-standard languages ​​and spoken language patterns, it extracts question words, filler words (marks positions and extracts prosodic features), and pronouns (preliminary identification based on context window and semantic role annotation).

[0099] The semantic reference resolution module is used to determine the precise meaning of each ambiguous word according to the discrimination rules and to determine the precise meaning of the pronouns according to the hierarchical mechanism. It takes as input a list of pronouns, context text, a dictionary of ambiguous words and filler word features, and outputs the precise meaning of the pronouns, the parsing results of the ambiguous words and the fused semantic representation.

[0100] The response generation and optimization module generates emotionally matched and context-appropriate response voices and operation commands based on semantic representation, emotion tags, and vehicle status, and optimizes the interactive experience. It takes semantic representation, emotion tags, filler word features, and vehicle data (vehicle speed, road conditions) as input and outputs response voices (prosody parameters, personalized intonation) and operation commands. Emotion tags (positive / negative / neutral / angry) are mapped to prosody control parameters (such as fundamental frequency range, speech rate, and pauses) to generate response voices that match the user's emotions (e.g., outputting soothing voices when experiencing negative emotions). Response strategies are adjusted based on filler word type (hesitant, connecting, emotional) and position (sentence beginning / middle / end) (e.g., reducing speech rate and adding comforting pauses when filler word density is high). Vehicle status is monitored in real time (e.g., simplifying language structure and prioritizing command clarity in emergency situations), and response latency is reduced through a safety verification layer to ensure interactive safety and efficiency. Personalized voices are generated based on the user's non-standard language habits (e.g., Cantonese intonation), age characteristics (younger users speak faster), and historical interaction data to enhance the user experience.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An intelligent voice interaction method, characterized in that, include: S1: Acquire user speech, determine emotional tags and pronunciation pattern features, input pronunciation pattern features into language recognition model and output language pattern; If it is a standard language and standard grammar, it is marked as standard mode; if it is a non-standard language, it is marked as non-standard mode; if it is a standard language and non-standard grammar, it is marked as spoken mode. S2: For standard mode, the speech text is segmented to obtain standard word frequencies, and combined with emotion tags to generate response speech and operation instructions; for non-standard language mode and spoken language mode, the features of interrogative words, filler words and pronouns are extracted based on standard word frequencies. The precise meaning of each question word is determined based on the discrimination rules, and the precise meaning of the pronouns is determined based on the hierarchical mechanism. Based on the precise meaning, the response voice and operation instructions are generated by combining the emotion tags and filler word features. The operation instructions are executed with priority based on real-time vehicle data, environmental data and user instructions. S3: Real-time calculation of the decay curve of the resolution accuracy, calculation of hierarchical storage efficiency, data processing efficiency and recognition accuracy based on historical data and decay curve, and determination of the optimal recognition rule based on the peak change trend of the recognition curve.

2. The method as described in claim 1, characterized in that, The system uses an automatic speech recognition model to convert user speech into text in real time, marks the position of filler words during the speech recognition stage, extracts the prosodic features of filler words, and establishes a personalized filler word pattern library for users. The system acquires user voice text and compares it with a filler word pattern library to identify filler word density, filler word type, and filler word position as filler word features.

3. The method as described in claim 1, characterized in that, Extracting questionable words is based on calculating the semantic confidence of words in context using a pre-trained language model. This process includes: pre-labeling a dataset, where each word is labeled as either a questionable word or not, as training data; training the model using the training set; validating the parameters using the validation set; optimizing the model parameters; obtaining a fully trained pre-trained language model; outputting a list of semantic confidence scores for each word; identifying the word with the lowest confidence score as a questionable word; and using an attention weight matrix to label the positions of words with low confidence scores.

4. The method as described in claim 1, characterized in that, The precise meaning of each ambiguous word is determined based on the discrimination rules, including: taking the ambiguous word as the center, taking several words before and after it to form a context window, and using a sliding window algorithm to extract the window content in real time; using a functional word vector model to obtain the vector representation of each meaning of the ambiguous word, calculating the similarity between each meaning vector representation and the context vector, setting a pre-set association similarity threshold, and removing the corresponding meaning if the similarity is less than the association similarity threshold; marking the remaining meanings as candidate meanings; inputting the candidate meanings and meaning data features into a conditional random model to obtain a probability ranking; selecting the meaning with the highest probability as the precise meaning and outputting the confidence score; the meaning data features include vocabulary, part-of-speech tags, location information, semantic similarity score, and confidence score.

5. The method as described in claim 1, characterized in that, The precise meaning of pronouns is determined according to a hierarchical mechanism, which is as follows: the historical importance score and level of the pronoun are found; the omitted subject is identified based on semantic role labeling; the referential relationship across dialogue rounds is captured through an attention mechanism to generate several candidate entities; an entity-relationship graph is constructed, where nodes are entities and edges are relationships; indirect referential relationships are identified; neighboring nodes are determined; and multi-round message passing is performed through a GNN to calculate the matching score between the pronoun and the candidate entities. The candidate entity with the highest matching score is selected as the precise meaning of the pronoun.

6. The method as described in claim 1, characterized in that, A memory importance assessment model is constructed to generate an importance score for each pronoun and classify them into different storage levels. These storage levels include short-term memory, medium-term memory, and long-term memory. The utilization rate of each pronoun in each memory level is monitored; if the utilization rate is below 40%, it is cleaned up; if the utilization rate is above 70%, it is automatically migrated to a higher memory level. The formula for calculating the importance score is as follows: ,in, It represents the i-th pronoun. It is an importance score. Refers to the chain integrity score. It uses frequency scoring. It is a context relevance score. It is a personalized weighted score. It is the score for temporal continuity; These are the weight coefficients for the corresponding scores, initialized to... And the sum of the weighting coefficients is 1.

7. The method as described in claim 1, characterized in that, The language recognition model includes a non-standard language prosody template library and standard grammar indicators. It calculates the similarity between the user's pronunciation pattern features and the non-standard language prosody template library, pre-setting a similarity threshold. If the similarity exceeds the threshold, the non-standard language type with the highest similarity is selected as the reference non-standard language and marked as a non-standard pattern; if the similarity does not exceed the threshold, it is marked as a standard language. The pronunciation pattern features are then compared with the standard grammar indicators. If the standard grammar indicators are met, it is considered standard grammar and marked as a standard pattern. Conversely, if the grammar is not standard, it is marked as spoken language; the standard grammar indicators include sentence completeness, lexical standardization, and sentence complexity.

8. The method as described in claim 1, characterized in that, Based on the user's speech, basic acoustic features and prosodic feature sets are extracted to determine the pronunciation pattern features. The prosodic feature set includes phoneme-level prosodic features, syllable-level prosodic features, and sentence-level prosodic features. The pronunciation pattern features include tone pattern vectors, rhythm pattern vectors, and intensity pattern vectors. The basic acoustic features include fundamental frequency sequence, energy features, spectral features, and duration features.

9. The method as described in claim 8, characterized in that, The tone pattern vector is generated based on phoneme-level and syllable-level fundamental frequency features. The phoneme-level fundamental frequency pattern and syllable-level fundamental frequency profile are used as feature inputs. A Gaussian mixture model is used to perform cluster analysis on the fundamental frequency features. Each cluster center represents a tone pattern. The matching degree between the current speech and each tone pattern is calculated through maximum a posteriori probability estimation. The top K tones with the highest matching degree are selected to form the tone pattern vector. The rhythm pattern vector is generated based on duration features and pause features. Phoneme duration features, syllable duration patterns and sentence-level pause patterns are used as feature inputs. The statistical features of each rhythm parameter are calculated to obtain the rhythm pattern vector. The intensity pattern vector is generated based on energy features and accent patterns, with phoneme-level energy distribution, syllable-level energy envelope, and sentence-level energy trend as feature inputs. The energy envelope template is aligned using a dynamic time warping algorithm, and principal component analysis is used to extract the main patterns of energy distribution, identify accent patterns, and generate intensity pattern vectors.

10. An intelligent voice interaction system, applied to the method as described in any one of claims 1 to 9, characterized in that, The system includes: a speech feature extraction module, used to acquire user speech in real time for preprocessing, and determine emotion tags and pronunciation pattern features; The language pattern recognition module is used to determine the language pattern of the user's speech based on pronunciation rules and grammatical indicators, and to extract key semantic information. The word frequency segmentation module is used to generate standard word frequencies based on different language patterns and their word segmentation standards, and to extract features of question words, filler words, and pronouns; The semantic reference resolution module is used to determine the precise meaning of each ambiguous word according to the discrimination rules and to determine the precise meaning of the pronouns according to the hierarchical mechanism. The response generation and optimization module is used to generate emotionally matched and context-adapted response voices and operation commands based on semantic representation, emotion tags and vehicle status, and to optimize the interactive experience.

Images