Vehicle-mounted streaming language dialogue interaction system and method based on large language model

Abstract

The invention discloses a vehicle-mounted streaming language dialogue interaction system and method based on a large language model, and the system comprises the large language model disposed at a vehicle-mounted end, and the large language model is used for carrying out the local processing and cloud processing of information inputted by a user, so as to obtain a processing result. Wherein the large language model is connected with the edge calculation and privacy protection module, the edge calculation and privacy protection module is used for carrying out edge calculation on a processing task in local equipment and distributing a data processing reasoning process to local and cloud service for processing, and the proportion of a local processing part is larger than that of the cloud service. The advantages of the LLMs are fully utilized, the special requirements of the vehicle-mounted environment can be met, and the method has important significance for improving the intelligent level of vehicle-mounted man-machine interaction and the user experience.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation systems and vehicle-mounted human-computer interaction technology, and in particular to a vehicle-mounted streaming language dialogue interaction system based on a large language model and a method thereof. Background Art

[0002] With the rapid development of automobile intelligence and networking technology, in-vehicle human-computer interaction systems have become increasingly important in improving driving safety and user experience. Existing in-vehicle voice interaction systems are usually based on fixed instruction sets or traditional speech recognition technology, lacking the ability to deeply understand natural language and semantic parsing. This limitation makes the system unable to effectively handle complex user intentions, the interaction method is rigid, and users must remember specific instruction formats, resulting in a poor user experience.

[0003] In addition, traditional systems have obvious deficiencies in real-time performance and response accuracy. Due to the limitations of algorithms and computing resources, these systems perform poorly in processing multi-round conversations, context associations, and fuzzy semantics, and cannot meet the user's needs for fast and accurate information acquisition during driving. Especially in the driving environment, users need to focus more on road conditions, and complex interaction processes may distract drivers and bring safety risks.

[0004] In recent years, large-scale pre-trained language models (LLMs) have made breakthrough progress in the field of natural language processing, demonstrating powerful language understanding and generation capabilities. However, directly applying LLMs to in-vehicle environments faces many challenges, such as limited computing resources, large model size, and high real-time requirements. This makes how to efficiently deploy and run large language models in in-vehicle environments an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an in-vehicle streaming language dialogue interaction system based on a large language model, which can fully utilize the advantages of LLMs and meet the special needs of the in-vehicle environment. It is of great significance to improve the intelligence level of in-vehicle human-computer interaction and user experience.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: an in-vehicle streaming language dialogue interaction system based on a large language model, the system includes a large language model deployed on the vehicle side, the large language model is used to perform local processing and cloud processing on the information input by the user to obtain the processing result, wherein the large language model is connected to the edge computing and privacy protection module, the edge computing and privacy protection module is used to perform edge computing on the processing task in the local device, and distribute the data processing reasoning process to local and cloud services for processing, wherein the local processing part accounts for a larger proportion than the cloud service proportion.

[0007] The vehicle also has a speech recognition module, a large language model-driven dialogue management module, a language generation module, and a speech synthesis module.

[0008] The speech recognition module is used to convert the user's speech into text and input it into the dialogue management module driven by the large language model; the dialogue management module is connected to the large language model, and forwards the mechanical context text to the large language model for semantic analysis to obtain the intention recognition result; the language generation module generates the language text corresponding to the reply user input based on the acquired intention recognition result using the generation capability of the large language model; the speech synthesis module is used to convert the language text into speech output.

[0009] The system also includes a multimodal interaction module, the output end of which is connected to the dialogue management module, and the multimodal interaction module is used to recognize user input information other than voice and convert it into text to send to the dialogue management module.

[0010] The edge computing and privacy protection module includes a local processing unit, a data encryption unit and a privacy protection unit; wherein: the local processing unit is the locally deployed part of the large language model, and the large language model is called in the local processing unit for language recognition processing. When the local processing unit is unable to perform calculation processing, part or all of the processing tasks are uploaded to the cloud-deployed part of the large language model for calculation processing through the data encryption unit and the privacy protection unit.

[0011] An interaction method for an in-vehicle streaming language dialogue interaction system based on a large language model includes an initialization stage. In the initialization stage, speech recognition, large language model reasoning, language generation and speech synthesis modules are installed and configured on the in-vehicle device, and the large language model is loaded according to the hardware configuration of the in-vehicle device. The language recognition processing tasks are allocated to the local and cloud through edge computing technology, wherein the proportion of processing tasks allocated to the local is greater than that of the cloud; the installed system is tested to ensure that speech recognition, semantic analysis, dialogue management and generated responses are accurately and smoothly linked to enter the user interaction stage.

[0012] The user interaction phase includes:

[0013] Voice input and recognition: Users send commands to the system through voice. The voice recognition module converts the voice input into text in real time and passes it to the dialogue management module for processing;

[0014] Intent recognition and strategy generation: The dialogue management module uses a large language model to perform semantic analysis on the text, identify the user's intent, and generate appropriate dialogue strategies based on the context;

[0015] Language generation and speech synthesis: Based on the generated strategy, the language generation module generates a natural language response that fits the context, and the speech synthesis module converts the response into speech output.

[0016] The user interaction use stage also includes: the user interacts through gestures or text input and is recognized by the multimodal interaction module, the multimodal interaction recognition module is used to process and convert the user's gestures or text input, and then converted into text data and sent to the dialogue management module, and then the large language model is used to perform text and speech analysis processing.

[0017] During the user interaction phase, the system response speed and performance in the vehicle environment are monitored, and the large language model is updated or optimized based on the system response.

[0018] The user evaluation feedback information after each voice feedback during the mobile phone user interaction process is used to update and optimize the large language model to match user needs based on the user's evaluation of each interaction.

[0019] The advantages of the present invention are: 1. Streaming dialogue processing architecture

[0020] The system uses streaming dialogue processing technology, which can perform real-time analysis and response while voice input is in progress, reducing latency and improving user experience. Compared with traditional voice systems, users can get feedback without waiting for voice input to complete, which enhances the interactivity and response speed of the system.

[0021] 2. Optimized deployment of large language models

[0022] By optimizing the deployment and reasoning process of large language models for the vehicle environment, the real-time response problem of large models under limited computing resources is solved. Through the application of model compression, quantization and edge computing technology, efficient reasoning performance is achieved to meet the requirements of vehicle systems for low latency and high accuracy.

[0023] 3. Multimodal Fusion and Dynamic Context Management

[0024] The system supports multi-modal input such as voice, text, gesture, etc., and realizes complex multi-round dialogue through dynamic context management function. This function can track user intentions in real time, perform semantic association and reasoning, ensure accurate capture of user needs in different dialogue rounds, and provide a more natural and intelligent interactive experience.

[0025] 4. Tuning the language model for vehicle scenarios

[0026] The large language model is customized and tuned based on the in-vehicle scenario. The model can make personalized responses based on the driver's voice characteristics and usage scenarios (navigation, music, driving assistance, etc.), improving the system's specificity and adaptability.

[0027] 5. Multi-language support and natural language generation capabilities

[0028] The system has built-in support for multi-language processing capabilities. Combined with the powerful natural language generation function of the large language model, it can achieve natural conversation and information provision in multiple languages ​​and is suitable for global in-vehicle application environments.

[0029] 6. Security and privacy protection mechanism

[0030] The system adopts a strict privacy protection mechanism in the vehicle environment, using edge computing and localized reasoning technology to ensure that the user's voice data and interaction information are not transmitted to external servers, reducing the risk of privacy leakage and improving the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0032] Figure 1 This is a block diagram of the vehicle-mounted streaming voice dialogue system of the present invention;

[0033] Figure 2 This is a flow chart of the vehicle-mounted streaming voice dialogue of the present invention;

[0034] Figure 3 For performance monitoring and automatic updating of swim lane diagrams of the present invention;

[0035] Figure 4 This is a flow chart of privacy protection and data encryption of the present invention;

[0036] Figure 5 Incremental update flow chart for model update;

[0037] Figure 6 A schematic diagram of a feedback loop for updating and optimizing the system of the present invention;

[0038] Figure 7 This is a flow chart of the collaboration between the vehicle-mounted system and the cloud of the present invention;

[0039] Figure 8 Update feedback flow chart for the model of the present invention. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0041] The in-vehicle streaming language dialogue interaction system based on a large language model provided in this embodiment is applied in a vehicle to provide language interaction between a user in the vehicle and the vehicle, and realize human-computer voice interaction. The system includes a large language model deployed on the vehicle side, and the large language model is used to process the information input by the user locally and in the cloud to obtain a processing result, wherein the large language model is connected to an edge computing and privacy protection module, and the edge computing and privacy protection module is used to perform edge computing on the processing task in the local device, and distribute the data processing reasoning process to local and cloud services for processing, wherein the local processing part accounts for a larger proportion than the cloud service proportion, and by processing most of the user's voice interaction demand tasks locally, the cloud task volume can be reduced, reducing costs while protecting privacy.

[0042] The vehicle also has a speech recognition module, a large language model-driven dialogue management module, a language generation module, and a speech synthesis module.

[0043] The speech recognition module is used to convert the user's speech into text and input it into the dialogue management module driven by the large language model; the dialogue management module is connected to the large language model, and forwards the mechanical context text to the large language model for semantic analysis to obtain the intention recognition result; the language generation module generates the language text corresponding to the reply user input based on the acquired intention recognition result using the generation capability of the large language model; the speech synthesis module is used to convert the language text into speech output.

[0044] Furthermore, the system also includes a multimodal interaction module, the output end of the multimodal interaction module is connected to the dialogue management module, and the multimodal interaction module is used to recognize user input information other than voice and convert it into text and send it to the dialogue management module.

[0045] The edge computing and privacy protection module includes a local processing unit, a data encryption unit and a privacy protection unit; wherein: the local processing unit is the locally deployed part of the large language model, and the large language model is called in the local processing unit for language recognition processing. When the local processing unit is unable to perform calculation processing, part or all of the processing tasks are uploaded to the cloud-deployed part of the large language model for calculation processing through the data encryption unit and the privacy protection unit.

[0046] An interaction method for an in-vehicle streaming language dialogue interaction system based on a large language model includes an initialization stage. In the initialization stage, speech recognition, large language model reasoning, language generation and speech synthesis modules are installed and configured on the in-vehicle device, and the large language model is loaded according to the hardware configuration of the in-vehicle device. The language recognition processing tasks are allocated to the local and cloud through edge computing technology, wherein the proportion of processing tasks allocated to the local is greater than that of the cloud; the installed system is tested to ensure that speech recognition, semantic analysis, dialogue management and generated responses are accurately and smoothly linked to enter the user interaction stage.

[0047] The user interaction stage includes: voice input and recognition: the user issues instructions to the system through voice, the voice recognition module converts the voice input into text in real time, and passes it to the dialogue management module for processing; intention recognition and strategy generation: the dialogue management module performs semantic analysis on the text through a large language model, identifies the user's intention, and generates appropriate dialogue strategies based on the context; language generation and speech synthesis: based on the generated strategy, the language generation module generates a natural language response that conforms to the context, and the speech synthesis module converts the response into speech output.

[0048] The user's interaction through gestures or text input is recognized by the multimodal interaction module, and the multimodal interaction recognition module is used to process and convert the user's gestures or text input, and then convert it into text data and send it to the dialogue management module, and then the large language model performs text and speech analysis.

[0049] The present invention proposes an innovative solution to the deficiencies of existing in-vehicle voice interaction systems in terms of complex intent understanding, real-time performance, response accuracy, and multimodal support. Traditional systems usually rely on fixed instruction sets, lack a deep understanding of natural language, cannot handle complex multi-round conversations, and are prone to response errors or delays when faced with ambiguous or non-standardized user expressions, which seriously affects the user experience. In addition, these systems are relatively single in terms of multimodal input support and are difficult to adapt to the diverse interaction needs during driving. By introducing a large-scale pre-trained language model, the present invention significantly improves the system's ability to understand and generate user natural language, and supports continuous conversations and intent recognition in complex scenarios. At the same time, the system optimizes model deployment and realizes low-latency streaming processing to ensure that users can obtain real-time responses in the in-vehicle environment. By supporting multimodal input such as voice, text, and gestures, the present invention enhances the flexibility of the system and meets the needs of different driving scenarios. In addition, the system also uses edge computing technology to reduce the need for data upload, protect user privacy and data security, and expand the possibility of global applications through multi-language support.

[0050] System structure:

[0051] 1. Speech recognition module: responsible for converting user speech into text input to ensure the accuracy of speech input in the vehicle environment.

[0052] 2. Large language model-driven dialogue management module: This module is the core of the system and uses large-scale pre-trained language models (LLMs) for natural language processing. Large language models not only have powerful language understanding capabilities and can handle complex natural language inputs, but also can achieve multi-round dialogue interactions through context tracking. Compared with traditional rule-driven systems, the innovation of this invention is to utilize the semantic parsing capabilities of large language models, so that the system can dynamically adapt to user needs and flexibly generate suitable dialogue strategies.

[0053] 3. Language generation module: Relying on the generation capability of the large language model, it automatically generates responses that conform to natural language habits based on user input. The training of the large language model enables it to have powerful language generation capabilities, accurately convey complex concepts, and improve the user conversation experience.

[0054] 4. Speech synthesis module: Converts the generated text responses into natural speech output, completing the closed loop of streaming conversation.

[0055] 5. Multimodal interaction module: supports multimodal input, such as voice, text, gesture, etc., so that the system can adapt to diverse in-vehicle interaction needs.

[0056] 6. Edge computing and privacy protection module: By performing edge computing on local devices for part of the processing tasks of the large language model, the amount of data uploaded can be reduced, user privacy can be protected, and the operating efficiency of the large model can be optimized. Edge computing: By embedding high-performance computing units in vehicle-mounted devices (such as vehicle-mounted host and processing unit), some data can be processed locally. The goal of edge computing is to reduce dependence on network connections and improve real-time performance and response speed.

[0057] Privacy protection: Ensure that user privacy is not leaked through data classification protection mechanism, localized sensitive data processing, and encrypted transmission. Use technologies such as Federated Learning or Differential Privacy to avoid direct exposure of personal data when training models in the cloud.

[0058] The data of voice interaction is distributed and processed according to the task allocation principle. The task allocation principle is based on the real-time performance, bandwidth conditions and data sensitivity of the vehicle system. The task allocation rules are as follows:

[0059] (1) Task types that are processed locally first

[0060] High real-time requirements: For example, voice command processing for driver assistance (such as navigation route update and air conditioning adjustment).

[0061] Strong privacy: such as the owner’s biometric information (fingerprints, facial features) or driving habits analysis.

[0062] Low computational complexity: tasks that can be completed by the local device within a bounded latency (such as simple speech recognition or ambient noise filtering).

[0063] (2) Types of tasks processed in the cloud

[0064] High computational complexity: such as multi-language translation, large-scale semantic understanding, or complex gesture recognition.

[0065] · Need for global data integration: For example, map updates, real-time traffic condition predictions, and other functions that require large-scale cloud data support.

[0066] Non-real-time requirements: such as system model updates, performance optimization, or batch data analysis.

[0067] Dynamic adjustment mechanism of task allocation: Dynamic evaluation model: Through the task allocation model embedded in the system, the task allocation strategy is evaluated in real time according to the characteristics of the task (real-time, privacy, computational complexity) and system status (network bandwidth, device load).

[0068] Local / Cloud Load Balancing: Dynamically monitor the local and cloud loads through the task queue to prevent resource overload on one side. For example, when the cloud is highly loaded or the network is unstable, more tasks will be executed locally; conversely, complex tasks will be submitted to the cloud first when the network is in good condition.

[0069] Collaborative computing: Some tasks can be divided into multiple subtasks, where the parts that require immediate feedback (such as keyword recognition) are completed locally, and subsequent complex analysis (such as semantic understanding) is submitted to the cloud and the result feedback is used to enhance the user experience.

[0070] Technical implementation details

[0071] (1) Edge computing module

[0072] Embed hardware accelerators (such as GPU and NPU) in local devices to support real-time computing.

[0073] Optimized model pruning technology: Prune the pre-trained large language model and retain the most relevant weights to adapt to the performance of the vehicle equipment.

[0074] (2) Privacy Protection Module

[0075] Encrypted communication: All data transmission between local and cloud is end-to-end encrypted (such as TLS protocol).

[0076] Sensitive data isolation: Store and process private data such as biometric information through a local security chip (TPM or similar module).

[0077] Federated learning: User data is used locally for model training, and the cloud only receives updated model weights instead of raw data.

[0078] Differential privacy: Introducing tiny noise during data transmission or sharing to ensure that the data cannot be reversed and reconstructed.

[0079] Typical application scenario analysis

[0080] Scenario 1: Voice navigation instructions

[0081] Local processing: keyword extraction of voice commands (such as "navigate to XX").

[0082] Cloud processing: calculate the optimal path and feedback navigation results.

[0083] Scenario 2: Gesture Recognition

[0084] Local processing: preliminary recognition and classification of gestures (such as detecting a waving motion).

[0085] Cloud processing: Analyze the specific intent based on the environmental context of the user's gesture (such as time and location).

[0086] Scenario 3: User driving habits analysis

[0087] Local processing: Record and analyze daily driving behavior.

[0088] Cloud processing: Integrate data into a large sample database to optimize fleet operation plans.

[0089] The technical advantages of the above solution are:

[0090] Real-time: Edge computing reduces network latency and improves interactive experience.

[0091] Privacy protection: Localized processing can avoid data leakage to the greatest extent possible.

[0092] Cost optimization: Reduce cloud computing resource usage and network bandwidth consumption through dynamic allocation.

[0093] The technical solution of the present invention significantly enhances the natural language processing and generation capabilities of the in-vehicle streaming language dialogue system by introducing a large language model. The system first obtains user input through the speech recognition module and converts the speech into text. Then, the dialogue management module driven by the large language model performs deep semantic analysis on the text, combines contextual information and multi-round dialogue history to accurately identify user intentions. Unlike traditional in-vehicle systems, the large language model can not only process complex natural language input, but also has contextual memory and reasoning capabilities, supporting continuous dialogue and complex commands.

[0094] Based on the generative capabilities of the large language model, the system is able to generate natural and coherent responses. The language generation module makes full use of the pre-trained knowledge base of the large language model, making the system highly intelligent and adaptable when facing different scenarios and different language requirements. The speech synthesis module converts the generated text into speech to achieve smooth real-time conversation.

[0095] The innovation of this technical solution lies in the integration of large language models and their optimized application. Through model pruning, quantization and other technologies, large models can run efficiently in resource-constrained vehicle-mounted devices. At the same time, combined with the streaming processing architecture, it ensures that the system can respond efficiently with low latency in the vehicle environment. Through the powerful semantic parsing and generation capabilities of the large language model, the present invention solves many deficiencies of existing in-vehicle voice interaction systems in complex dialogue processing, response accuracy and user experience.

[0096] Implementation method:

[0097] 1. Voice Recognition

[0098] Input processing: Receive user voice input through the vehicle microphone, and use advanced noise suppression and voice enhancement technology to ensure accurate voice signal acquisition in a noisy vehicle environment. This step needs to consider the diversity of voice input, such as different accents, speaking speeds, etc.

[0099] Speech-to-text: Use a deep learning-based end-to-end speech recognition model (Transformer-based ASR) to convert the collected speech signals into text in real time. This model should have the ability to adapt to the vehicle environment and be able to handle background noise and interference from multiple sound sources.

[0100] Text output: The recognized text data is passed to the dialogue management module to provide a basis for the next step of semantic analysis.

[0101] 2. Dialogue Management

[0102] Context maintenance: The system automatically maintains the context information of the conversation, including the user's historical input and the system-generated responses. Based on the memory and reasoning capabilities of the large language model, the system is able to track the history of multiple rounds of conversations and ensure the coherence of the conversation.

[0103] Intent recognition: The intent of user input is recognized through the large language model (Llama). The model learns the complex structure of natural language from pre-training data and has the ability to understand multiple semantic expressions and fuzzy intents, ensuring accurate recognition of user needs.

[0104] Dialogue strategy generation: Dynamically generate appropriate dialogue strategies based on intent recognition results and dialogue context. This process relies on the reasoning ability of a large language model to ensure that the system can flexibly handle dialogues in various scenarios and provide personalized responses.

[0105] 3. Language Generation

[0106] Text generation: Use a large language model to generate text responses that conform to natural language habits. Based on contextual information and intent recognition results, the system generates semantically coherent and easy-to-understand conversation text.

[0107] Output optimization: Perform secondary optimization on the generated text to ensure that the sentences are natural, accurate, and meet the user's expectations. The optimization process may include grammar checking, context adjustment, etc.

[0108] 4. Speech Synthesis

[0109] Text-to-speech: The generated text response is passed to the speech synthesis module. Using text-to-speech (TTS) technology, the text is converted into natural and clear speech output. The TTS system should have high-quality speech synthesis capabilities to provide a smooth listening experience.

[0110] Voice playback: The synthesized voice response is played through the vehicle audio system, ensuring the sound is clear and the volume is moderate to meet the needs of the vehicle environment.

[0111] 5. Multimodal Interaction

[0112] Input fusion: The system supports multiple input methods (voice, text, gesture, etc.), and integrates different input forms through the multimodal interaction module. Users can interact with the system by selecting the most suitable input method.

[0113] Processing coordination: Coordinate the processing of data from different input sources to ensure that the system can correctly identify and respond to various input types.

[0114] 6. Edge computing and privacy protection

[0115] Local processing: Part of the computing tasks of the large language model are performed locally on the vehicle-mounted device, and model optimization technology (pruning, quantization) is used to reduce the computing burden and improve processing efficiency.

[0116] Data encryption: Encrypt user data to ensure that data is not accessed without authorization during transmission and storage. Implement data encryption and decryption mechanisms to protect user privacy.

[0117] Privacy protection: Reduce the need to upload data to the cloud, and use edge computing technology to handle most computing tasks to reduce the risk of privacy leakage. Set up permission management and data access control mechanisms to ensure the security of user data. In the intelligent vehicle system, edge computing and privacy protection modules work together to improve response speed, reduce bandwidth usage, and protect user privacy. The following is the working principle of this module and how to allocate computing tasks, dynamically adjust local and cloud computing, and implement privacy protection and data encryption. Edge computing refers to moving data processing tasks from the cloud to the proximal device where the data is generated (i.e., the vehicle-mounted computing device) to achieve real-time data analysis and lower latency. Vehicle-mounted devices (such as vehicle-mounted computing units, sensors, cameras, microphones, etc.) can respond more quickly through edge computing. Basic principles of task allocation: The edge computing module needs to dynamically decide which tasks can be processed locally and which tasks need to be uploaded to the cloud for processing. It is mainly based on the following factors:

[0118] 1. Real-time requirements:

[0119] Highly real-time tasks (such as speech recognition, motion tracking, and feedback from driver assistance systems) are processed locally first, thus avoiding response delays caused by network latency.

[0120] Some non-real-time tasks (such as long-term analysis of user behavior data or background updates of vehicle systems) can be uploaded to the cloud for processing.

[0121] 2. Computational complexity:

[0122] Local processing is suitable for tasks with low computational complexity, such as speech recognition, image preprocessing, vehicle speed perception, navigation instruction parsing, etc. Based on existing in-vehicle hardware (such as GPU, NPU, etc.), these operations can be completed with low latency.

[0123] Cloud processing is suitable for large-scale data analysis that requires higher computing resources, such as the training of deep learning models and the optimization of the entire vehicle network.

[0124] 3. Bandwidth and network conditions:

[0125] When network bandwidth is sufficient, computing tasks can be assigned to the cloud (such as natural language processing and multimodal understanding of speech), especially for tasks that require a lot of computing resources.

[0126] If the network bandwidth is poor or there is network delay, more tasks will be processed locally first to ensure the stability and timely response of the in-vehicle system.

[0127] Detailed mechanism for local and cloud task allocation:

[0128] Low-latency tasks, such as keyword extraction in speech recognition, action commands (such as waving to control the air conditioner), and environmental noise filtering, must be processed locally. In-vehicle devices will use efficient inference engines (such as TensorRT, NVIDIA Jetson, etc.) to quickly respond to user input.

[0129] Complex computing tasks: such as semantic understanding based on deep learning, real-time traffic prediction, map updates, vehicle behavior analysis and other tasks, when the amount of computing is large, will be uploaded to the cloud for processing. At this time, the on-board system will first upload the data from the local device to the cloud through compression and encryption, and the cloud will analyze and feedback the results based on the latest data.

[0130] Dynamic load adjustment: If the load on the local device is too high (for example, multiple complex tasks are being performed simultaneously), the system will automatically adjust the distribution of tasks, and some tasks will be pushed to the cloud for processing. Conversely, when the computing resources of the vehicle-mounted device are sufficient, the system will give priority to keeping more tasks for local execution. To ensure that most tasks are processed locally, in order to maximize the advantages of edge computing and minimize data upload to the cloud, thereby reducing the pressure on network bandwidth and the risk of privacy leakage, we adopt the following technical strategies:

[0131] 1. Task pruning: The tasks executed locally are optimized and streamlined, and only the most core processing steps are retained. For example, large-scale neural networks do not directly run the entire model locally, but reduce the amount of calculation through model pruning, quantization and other technologies.

[0132] 2. Optimize the algorithm: put some computationally intensive operations (such as feature extraction in image processing) on ​​local hardware (such as NPU,

[0133] GPU) to accelerate execution and break down tasks into smaller tasks that can be executed efficiently.

[0134] 3. Separation of inference and training: The onboard equipment only runs optimized inference models (Inference), which are migrated to the local after training, avoiding the complex calculation of repeated training. Only when the model is updated will it be retrained through the cloud and the model update package will be downloaded to the local device.

[0135] 4. Separation of model and data preprocessing: Perform data preprocessing (such as denoising, dimensionality reduction, feature extraction, etc.) in the local device and transmit the simplified data to the cloud, reducing the redundant part of the uploaded data.

[0136] The implementation of privacy protection and data encryption includes:

[0137] Privacy protection mechanism: Data localization processing: Sensitive data (such as driver authentication and biometric information) will be encrypted and processed locally to avoid transmitting these sensitive data to the cloud. Use hardware security modules (HSM) or security chips (such as TPM) to encrypt and store data, and verify identity through biometric recognition, vehicle fingerprints, etc. Federated learning: The federated learning mechanism is adopted to enable model training to be completed locally without uploading personal data to the cloud. Post-training updates (such as model parameters) will be uploaded to the cloud, and the cloud will merge local learning results based on these updates.

[0138] Differential privacy: In order to protect user privacy, the system perturbs local data through differential privacy technology to ensure the privacy of data when uploading it to the cloud. Even if the data in the cloud is leaked, personal information cannot be reversed.

[0139] End-to-end encryption: When data is transmitted from local devices to the cloud, all transmitted data is protected by end-to-end encryption (such as TLS, SSL) to prevent the data from being intercepted or tampered with during transmission.

[0140] Data encryption and startup mechanism:

[0141] Real-time encryption: All sensitive data (such as driving instructions, voice commands, vehicle location, etc.) is encrypted immediately when collected. Symmetric encryption (such as AES) or asymmetric encryption (such as RSA) is usually used to protect data. Data is always encrypted during transmission and storage to avoid any potential risk of leakage.

[0142] Encryption trigger mechanism: Privacy protection and data encryption will be activated at the first moment of data collection. For example, when the in-vehicle microphone recognizes the driver's voice command, the system first determines whether the command contains sensitive information. If so, the encryption program is immediately activated to ensure that the command data is protected before uploading to the cloud.

[0143] Real-time decryption and encryption feedback: Some local task processing may require temporary decryption of sensitive data (such as vehicle owner voice verification). After the verification is completed, the results are immediately encrypted and uploaded to the cloud to ensure data security.

[0144] Specific implementation steps

[0145] 1. Initialization phase:

[0146] Module deployment: Install and configure speech recognition, large language model inference, language generation, and speech synthesis modules on the vehicle device. Ensure that each module can run in a resource-constrained environment and achieve their interoperability.

[0147] Model loading and optimization: Load the optimized large language model (LLM with model compression and quantization) according to the hardware configuration of the vehicle-mounted equipment. Most of the reasoning process is distributed locally through edge computing technology to reduce dependence on cloud services.

[0148] System integration testing: Integration testing is performed in a real vehicle environment to ensure that speech recognition, semantic parsing, dialogue management, and generated responses can be linked accurately and smoothly.

[0149] 2. User interaction stage:

[0150] Voice input and recognition: Users give commands to the system through voice. The voice recognition module converts the voice input into text in real time and passes it to the dialogue management module for processing.

[0151] Intent recognition and strategy generation: The dialogue management module uses a large language model to perform semantic analysis on the text, identify the user's intent, and generate appropriate dialogue strategies based on the context.

[0152] Language generation and speech synthesis: Based on the generated strategy, the language generation module generates a natural language response that fits the context, and the speech synthesis module converts the response into speech output.

[0153] Multimodal interaction support: If the user interacts through gestures or text input, the system can recognize multimodal signals, comprehensively process different input methods, and realize dynamic input fusion and response generation.

[0154] 3. Optimization and maintenance phase:

[0155] Performance monitoring and model update: Regularly monitor the system's response speed and performance in the vehicle environment, update and optimize the large language model in a timely manner, and improve reasoning capabilities and response efficiency. Use model pruning and quantization technology to ensure the efficient operation of the large model. Monitor the system response speed and performance in the vehicle environment during the user interaction phase, and update or optimize the large language model based on the system response. Automatically update the model based on the monitored response speed and performance, including monitoring the following data:

[0156] Response speed monitoring: Real-time tracking of the system’s response time from user input (such as voice, gesture, text) to output (such as voice feedback, control instructions), with a particular focus on delays in task completion.

[0157] Computing resource usage monitoring: Monitor the hardware resource usage of vehicle-mounted devices, including CPU, GPU, memory, network bandwidth and other indicators. When the computing resource requirements for model inference are too high, the overall response speed may be affected.

[0158] User experience feedback: Analyze the accuracy and quality of system responses through user interaction history and feedback. When users use the system in different environments (such as cities, highways, parking lots, etc.), they may perceive different response speeds and accuracy.

[0159] These monitoring data are collected in real time through the intelligent monitoring module and stored in the local storage of the vehicle system or in a secure database in the cloud.

[0160] Analysis and evaluation of performance monitoring data

[0161] Performance monitoring data is regularly transmitted to the system's central processing module, which is responsible for real-time analysis of the collected data.

[0162] The specific analysis process is as follows:

[0163] Real-time evaluation: By analyzing response speed, resource usage, and user experience, the system can evaluate the current operating efficiency of the model. For example, if the response speed exceeds a predetermined threshold (such as 500ms), or the system resource utilization is too high (such as more than 80% CPU or memory usage), it indicates that the system has a performance bottleneck.

[0164] Performance heatmap generation: Generate performance heatmaps in the vehicle environment, recording the performance of different tasks (such as speech recognition, intent parsing, text generation) under different conditions (such as low bandwidth, high noise environment, different vehicle speeds, etc.). Heatmaps can help developers identify the weak links of the model in specific situations.

[0165] Automatic model update mechanism

[0166] Based on the performance evaluation results, the system will decide whether to trigger a model update based on the following situations:

[0167] Performance failure triggers updates: When the system response speed drops significantly, or when too many computing resources are consumed during model reasoning, the system will automatically determine whether the model needs to be optimized or updated. For example:

[0168] o If the response time of real-time feedback exceeds the set threshold (such as 2 seconds), or the accuracy of user feedback is lower than the preset standard (such as 80%), the system automatically updates the model.

[0169] oBased on the performance monitoring data, the system can automatically enable "performance optimization strategies", such as model pruning, quantization technology, or adjusting some parameters to improve the model's reasoning efficiency.

[0170] Adaptive optimization: In some cases, the system needs to optimize certain parts of the model instead of just updating the entire model. For example, the speech recognition part may need to be more adaptable to dialects or noise, while the dialogue management module needs to be more capable of understanding complex user needs. In this case, adaptive optimization will only update the relevant parts of the model locally, avoiding the burden of a full update.

[0171] Model updating methods and strategies

[0172] Model pruning and quantization:

[0173] Model pruning: Reducing the size of the model while maintaining or improving performance by removing some redundant neurons or weights.

[0174] In a vehicle environment, due to limited computing resources, model pruning can effectively reduce the model's demand for resources.

[0175] Quantization technology: By converting high-precision floating-point weights into low-precision integers, the model's memory and computing resources are reduced while ensuring the speed and accuracy of model reasoning. The quantized model can significantly speed up the reasoning process, especially when the resources of the vehicle equipment are limited.

[0176] Incremental updates: To avoid the inconvenience caused by frequent updates, the system uses incremental updates to only update those parts of the model that do not meet the performance requirements. Incremental updates can gradually optimize the model without affecting other functions of the vehicle system.

[0177] Multi-version coexistence and A / B testing: The system supports storing multiple versions of models locally and automatically selects the best model version through the A / B testing mechanism. For example, some users can continue to use the old version before the new model is released to ensure that there will be no compatibility or stability issues. At the same time, the system can analyze the performance of different versions in different scenarios based on real-time monitoring data and ultimately select the best version.

[0178] Automatic update trigger conditions and execution process

[0179] Periodic checks: The system automatically checks the model performance at fixed intervals (e.g. every 24 hours). If the threshold is exceeded, the system will initiate an automatic update process.

[0180] Real-time triggering: When certain high-priority tasks (such as emergency voice command processing) fail to respond in time, the system can trigger updates in real time to ensure efficient operation of the system at critical moments. The specific steps are:

[0181] 1. Data collection and analysis: Collect the latest performance monitoring data from on-board equipment and evaluate whether it meets the update standards.

[0182] 2. Model evaluation and selection: Based on the collected data, select the model version or update strategy (such as pruning, quantization, etc.) that best suits the vehicle environment.

[0183] 3. Model update and replacement: Load the optimized model to the local device, replace the original model, and perform a complete verification process to ensure that the new model performs better than the old model in all in-vehicle tasks.

[0184] 4. Performance monitoring recovery: The updated model will re-enter the performance monitoring process to ensure that the updated model can continue to optimize and maintain efficiency in the vehicle environment.

[0185] Privacy protection and data security during the update process

[0186] Encrypted update process: The model update package is transmitted using encryption to ensure no data leakage during the model update process.

[0187] Privacy protection: All user interaction data, including voice data, location information, etc., are only processed locally and will not be directly uploaded to the cloud for model training or updating. During the model update process, the training data used strictly follows privacy protection technologies such as differential privacy and federated learning to ensure that user data will not be abused.

[0188] Security and privacy review: Regularly review the privacy protection mechanism of the system to ensure that the user's interactive data is encrypted, transmitted and stored to prevent data leakage. At the same time, check the correctness of edge computing tasks to ensure their processing efficiency and security.

[0189] User feedback and system improvement: Collect user feedback during the interaction process, identify potential problems and improvement points of the system, and optimize and update the system based on feedback to further improve the user experience. The user evaluation feedback information after each voice feedback during the mobile phone user interaction process is used to update and optimize the large language model to match user needs based on the user's evaluation of each interaction. Updating the model based on user feedback information includes:

[0190] 1. Collect user feedback information:

[0191] After each voice interaction, users can evaluate the effect of this interaction through a rating mechanism (e.g., 5-star rating). The system also records the user's interaction log, including input content, generated responses, interaction context, etc.

[0192] 2. Identify areas for improvement:

[0193] Using user rating data and interaction logs, the system analyzes potential problems with the model using the following methods:

[0194] Analysis of reasons for low scores: For low-scoring interactions, extract the deviation between the responses generated by the system and the responses expected by users.

[0195] Common problem classification: Similar problems are grouped into the same category, such as "semantic understanding deviation", "context tracking failure", etc.

[0196] 3. Update optimization strategy:

[0197] Adjust model parameters: Optimize the weights or specific parameters of an existing model based on feedback results.

[0198] Data incremental training: Collect annotated interaction data to fine-tune the model and enhance its adaptability to user needs.

[0199] Rule or template optimization: For fixed-pattern problems, use rule correction to generate results.

[0200] 4. Iterative verification: The updated model needs to verify its improvement effect in a test environment, and A / B testing should be used to evaluate whether the optimization meets the expected goals.

[0201] Update method description:

[0202] 1. Model parameter optimization:

[0203] Online learning: Use user feedback data to directly adjust the weights of specific layers of the model (such as the classification weights of the output layer).

[0204] Dynamic parameter update: Dynamically adjust the confidence threshold of model generation based on user ratings, for example, lowering the risk of generated content in low-rated scenarios.

[0205] 2. Data enhancement and retraining:

[0206] Annotate feedback data: Add annotations to users’ low-rated interactions (e.g., “question type,” “expected output”).

[0207] Data enhancement: Expand more similar data through semantic similarity (such as synonym replacement, sentence transformation).

[0208] Fine-tune the model: Use labeled data for small-scale incremental training to optimize performance for specific scenarios.

[0209] 3. Rule correction and template optimization:

[0210] Define rules: For clear low-scoring issues (such as certain keywords being misunderstood), use rules to optimize the output.

[0211] Template matching: Optimize response templates for common problems and improve adaptability to fixed scenarios.

[0212] Rating weights and priorities: User ratings can be assigned weights, for example, high-frequency users have a higher rating weight than low-frequency users. User feedback with multiple low-rated interactions is treated as high-priority data.

[0213] Privacy protection: Before the feedback data is uploaded to the cloud, differential privacy technology is used to protect user data to avoid leakage of personal information.

[0214] Real-time and batch updates: Real-time updates: For significant errors (such as multiple low scores from users), quick corrections are triggered and take effect in real time. Batch updates: After regularly collecting enough feedback data, the full model is retrained and a new version is released.

[0215] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. The vehicle-mounted streaming language dialogue interaction system based on a large language model is characterized by: The system includes a large language model deployed on the vehicle side, which is used to perform local and cloud processing on information input by the user to obtain processing results, wherein the large language model is connected to an edge computing and privacy protection module, and the edge computing and privacy protection module is used to perform edge computing on processing tasks in a local device, and distribute the data processing reasoning process to local and cloud services for processing, wherein the proportion of local processing is greater than that of cloud services.

2. The vehicle-mounted streaming language dialogue interaction system based on a large language model as claimed in claim 1, characterized in that: The vehicle also has a speech recognition module, a large language model-driven dialogue management module, a language generation module, and a speech synthesis module. The speech recognition module is used to convert the user's speech into text and input it into the dialogue management module driven by the large language model; the dialogue management module is connected to the large language model, and forwards the mechanical context text to the large language model for semantic analysis to obtain the intention recognition result; the language generation module generates the language text corresponding to the reply user input based on the acquired intention recognition result using the generation capability of the large language model; the speech synthesis module is used to convert the language text into speech output.

3. The vehicle-mounted streaming language dialogue interaction system based on a large language model as claimed in claim 2, characterized in that: The system also includes a multimodal interaction module, the output end of which is connected to the dialogue management module, and the multimodal interaction module is used to recognize user input information other than voice and convert it into text to send to the dialogue management module.

4. The vehicle-mounted streaming language dialogue interaction system based on a large language model as claimed in claim 2, characterized in that: The edge computing and privacy protection module includes a local processing unit, a data encryption unit and a privacy protection unit; wherein: the local processing unit is the locally deployed part of the large language model, and the large language model is called in the local processing unit for language recognition processing. When the local processing unit is unable to perform calculation processing, part or all of the processing tasks are uploaded to the cloud-deployed part of the large language model for calculation processing through the data encryption unit and the privacy protection unit.

5. The method for party interaction in a vehicle-mounted streaming language dialogue interaction system based on a large language model according to any one of claims 1 to 4, characterized in that: It includes an initialization phase. During the initialization phase, speech recognition, large language model inference, language generation and speech synthesis modules are installed and configured on the vehicle-mounted equipment. According to the hardware configuration of the vehicle-mounted equipment, the large language model is loaded, and the language recognition processing tasks are allocated to the local and cloud through edge computing technology, among which the proportion of processing tasks allocated to the local is greater than that of the cloud. The installed system is tested to ensure that speech recognition, semantic analysis, dialogue management and generated responses are accurately and smoothly linked to enter the user interaction phase.

6. The method for party interaction in a vehicle-mounted streaming language dialogue interaction system based on a large language model as claimed in claim 5, characterized in that: The user interaction phase includes: Voice input and recognition: Users send commands to the system through voice. The voice recognition module converts the voice input into text in real time and passes it to the dialogue management module for processing; Intent recognition and strategy generation: The dialogue management module uses a large language model to perform semantic analysis on the text, identify the user's intent, and generate appropriate dialogue strategies based on the context; Language generation and speech synthesis: Based on the generated strategy, the language generation module generates a natural language response that fits the context, and the speech synthesis module converts the response into speech output.

7. The method for party interaction in a vehicle-mounted streaming language dialogue interaction system based on a large language model as claimed in claim 6, characterized in that: The user interaction use stage also includes: the user interacts through gestures or text input and is recognized by the multimodal interaction module, the multimodal interaction recognition module is used to process and convert the user's gestures or text input, and then converted into text data and sent to the dialogue management module, and then the large language model is used to perform text and speech analysis processing.

8. The method for party interaction in a vehicle-mounted streaming language dialogue interaction system based on a large language model according to any one of claims 5 to 7, characterized in that: During the user interaction phase, the system response speed and performance in the vehicle environment are monitored, and the large language model is updated or optimized based on the system response.

9. The method for party interaction in a vehicle-mounted streaming language dialogue interaction system based on a large language model according to any one of claims 5 to 7, characterized in that: The user evaluation feedback information after each voice feedback during the mobile phone user interaction process is used to update and optimize the large language model to match user needs based on the user's evaluation of each interaction.

Images