Vehicle-mounted intelligent OBU voice interaction device and method integrated with large language model

Abstract

This application discloses an in-vehicle intelligent OBU voice interaction device and method integrating a large language model, belonging to the fields of intelligent transportation systems and in-vehicle human-machine interaction technology. The device includes a main control unit, an ETC transaction processing unit, a voice acquisition / synthesis module, a local edge computing unit, a multi-source external data fusion module, a context awareness module, and an intelligent fault diagnosis module. The intelligent fault diagnosis module monitors the ETC transaction status in real time, and when an anomaly occurs, it collects multi-dimensional fault parameters and inputs them into a large model to generate a personalized guidance solution. The context awareness module inputs vehicle status, environmental information, and historical interactions as context into the large model to achieve context-aware responses. The multi-source external data fusion module obtains real-time traffic conditions, weather, and other information. This application upgrades the OBU from a single payment terminal to an intelligent travel assistant, possessing functions such as proactive fault diagnosis, personalized guidance, and context-aware interaction, significantly improving user experience and driving safety.

Description

TECHNICAL FIELD

[0001] The application relates to an OBU voice interaction device and method integrated with a large language model, and belongs to the technical field of intelligent transportation systems and vehicle-mounted human-computer interaction. BACKGROUND

[0002] Current OBU (On Board Unit, vehicle-mounted unit) devices have relatively single functions, mainly focusing on non-stop charging scenarios in expressways and parking lots. The interaction mode with drivers is extremely limited, usually only relying on a buzzer prompt or a simple LED indicator light, lacking effective information transmission capability. Although some OBU have integrated Bluetooth modules, their uses are mostly limited to parameter configuration, and cannot support real-time information interaction during driving.

[0003] Although some OBU on the market have voice broadcast functions, they are generally based on a preset instruction set or a keyword matching mechanism, and can only realize one-way broadcast, without true natural language understanding and dialogue capability. Especially when ETC transactions fail or devices are abnormal, existing devices can only play fixed and general prompt sounds (such as "tag invalid" and "card reading error"), cannot identify specific fault reasons, and cannot provide targeted and guided solutions, resulting in user confusion and difficulty in timely solving problems.

[0004] In the implementation of the present application, the inventors found that the prior art at least has the following problems: OBU can only trigger fixed voice broadcast based on preset states, cannot receive user voice input, does not have voice recognition (ASR), natural language understanding (NLU) and dialogue management (DM) capabilities, and cannot realize the closed-loop interaction of "user asking-system understanding-dynamic response"; OBU functions are limited to ETC transactions themselves, and cannot provide real-time and dynamic travel-related information for users during driving; when ETC transactions are abnormal, it cannot intelligently analyze specific error codes, communication logs, card states or environmental context, and cannot provide step-by-step and operable solution guidance to users through natural language. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide an OBU voice interaction device and method integrated with a large language model to solve the technical problems of weak interaction capability of existing OBU, inability to intelligently diagnose faults, and inability to provide personalized services.

[0006] The technical solution adopted by the present application to solve its technical problems is: On one hand, an in-vehicle intelligent OBU voice interaction device integrating a large language model is provided, including: a main control unit, an ETC transaction processing unit, a voice acquisition module, a voice synthesis module, a local edge computing unit, a multi-source external data fusion module, a context awareness module, and an intelligent fault diagnosis module. The ETC transaction processing unit communicates with the roadside unit to complete transactions and reports the transaction status to the main control unit in real time; the voice acquisition module acquires user voice commands; the voice synthesis module converts system-generated text into voice output; the local edge computing unit is equipped with a lightweight voice recognition engine, a natural language understanding model, and dialogue management logic; the multi-source external data fusion module connects to external data sources via API interfaces to obtain real-time traffic and weather information; the main control unit internally deploys a large language model inference engine to perform natural language understanding of user voice commands and generate responses or control commands; the context awareness module acquires real-time... The system collects vehicle status information, driving environment information, and user historical interaction records, and inputs the acquired multimodal information into a large language model inference engine as contextual prompts. The intelligent fault diagnosis module is connected to the ETC transaction processing unit and configured to monitor the ETC transaction status in real time. When an ETC transaction anomaly is detected, the system automatically starts and collects fault parameters, including error type codes, device operating status parameters, signal strength values, the time of the most recent successful transaction, and user historical operation records. After deduplication and structuring, the collected data is input into the large language model inference engine and, combined with predefined fault diagnosis prompt templates, generates a personalized step-by-step voice guidance solution.

[0007] On the other hand, a vehicle-mounted intelligent OBU voice interaction method based on the above-mentioned device is provided, including the following steps: Step S1, the system is in a low-power monitoring state, and the voice acquisition module collects the in-vehicle voice signal in real time. When a valid voice start point is detected, the voice acquisition process is started; Step S2, the collected voice data is converted into text commands, and the context awareness module uses the current vehicle status, environmental information, and historical dialogue records as supplementary context input; Step S3, according to the intent category output by natural language understanding, a hybrid processing decision is executed: if the intent is a locally processable type, the local processing steps are executed, and the main control unit directly generates the response text; if the intent is a cloud-based large model processing type, then the cloud-based large model processing steps are executed. The cloud processing steps are as follows: The user text and context are encapsulated by the communication module and sent to the cloud-based large model interface; the response text generated by the cloud is received. If the intent is a fault diagnosis trigger, the intelligent fault diagnosis process is triggered. Step S4: For fault diagnosis triggers, the intelligent fault diagnosis process is executed: multi-dimensional fault parameters related to ETC transaction anomalies are collected, deduplicated, and structured before being input into the large language model inference engine to generate a personalized step-by-step voice guidance solution as the response text. Step S5: The generated response text is converted into a voice broadcast output through the speech synthesis module. Step S6: The interaction information is recorded and the local log is updated. The log records use a cyclic overwrite mechanism to retain the most recent N records.

[0008] Preferably, the intelligent fault diagnosis module further includes a multi-round interactive supplementation unit. When the large language model inference engine finds insufficient data during the diagnosis process, it actively initiates a query through the speech synthesis module to supplement user information, and then merges the supplemented information and re-inputs it into the large language model inference engine.

[0009] Preferably, the context-aware module is also used to dynamically adjust the broadcast strategy according to the vehicle speed: if the vehicle speed is greater than the first threshold, the generated voice guidance scheme is broken down into short sentences for broadcast, or a message indicating that details have been sent to the user terminal is displayed; if the vehicle speed is lower than the second threshold, all steps are broadcast in their entirety.

[0010] This application constructs an intelligent interaction hub for edge-cloud collaboration by deeply embedding a large language model into a resource-constrained on-board unit (OBU). This allows traditional ETC devices to transcend the one-way prompting function and evolve into intelligent travel terminals with natural language understanding and generation capabilities. Utilizing the reasoning power of the large model, the single error message when an ETC transaction fails is transformed into a multi-dimensional personalized solution integrating device status, user history, and driving context, achieving a qualitative leap in fault diagnosis from "passive alarm" to "active guidance." Simultaneously, through context-aware multi-turn dialogue management, real-time fusion of vehicle speed, location, time, and historical dialogue information maintains the dialogue state, supports coherent and natural multi-turn interactions, and integrates multi-source external data to provide intelligent query services. In low-power standby mode, combined with voice activity detection and directional command recognition, natural interaction without wake-words is achieved, comprehensively improving the user experience while ensuring driving safety.

[0011] One of the above technical solutions has the following advantages or beneficial effects: (1) This application collects multi-dimensional fault parameters (including error codes, device status, and user historical operation records) through the intelligent fault diagnosis module, and generates personalized step-by-step guidance schemes in combination with the big language model, which significantly improves the efficiency of resolving ETC transaction anomalies and reduces user confusion and manual customer service consultations; (2) The context perception module uses vehicle status, environmental information, and historical interaction records as context inputs of the big model, realizing personalized interaction that is highly matched with the driving situation, such as automatically adjusting the broadcast mode according to vehicle speed to ensure driving safety; (3) The multi-source external data fusion module enables the OBU to provide value-added services such as real-time road conditions and weather, upgrading the OBU from a single payment terminal to a smart travel assistant; (4) The hybrid processing mechanism of end-to-cloud collaboration flexibly schedules local data according to intent category. With cloud resources, it not only ensures real-time performance but also takes into account the ability to solve complex tasks. At the same time, the local log loop coverage and periodic upload mechanism supports continuous model optimization. (5) This application organically integrates multiple technical means (intelligent fault diagnosis, context awareness, multi-source data fusion, and wake-free interaction) into the OBU, which produces a synergistic effect: fault diagnosis relies on the real-time vehicle status and historical records provided by context awareness, multi-source data fusion enriches the response content, and wake-free interaction improves usability. The modules cooperate with each other to achieve technical effects far exceeding the simple superposition of individual modules. For example, when driving at high speed, the system can automatically simplify the fault guidance broadcast according to the vehicle speed and send detailed steps to the mobile phone, which not only ensures safety but also provides complete information. This is something that no single module can achieve. In addition, the combination of fault diagnosis and context awareness further improves driving safety: when the system detects an abnormal ETC transaction while driving at high speed, it will automatically compress the guidance information according to the vehicle speed to avoid the driver being distracted for a long time. At the same time, it will push detailed solutions to the mobile phone for viewing after parking. This improvement in safety is the result of the synergistic effect of multiple technical features. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating the structure of an in-vehicle intelligent OBU voice interaction device that integrates a large language model, according to an exemplary embodiment. Figure 2 This is a flowchart illustrating an in-vehicle intelligent OBU voice interaction method integrating a large language model, according to an exemplary embodiment. Figure 3 This is a system architecture diagram of an in-vehicle intelligent OBU voice interaction device according to an exemplary embodiment; Figure 4 This is an ETC transaction flowchart illustrated according to an exemplary embodiment. Detailed Implementation

[0013] To more clearly illustrate the technical features of this application, the following detailed description is provided through specific embodiments and in conjunction with the accompanying drawings.

[0014] like Figure 1 As shown in the figure, an embodiment of this application provides an in-vehicle intelligent OBU voice interaction device integrating a large language model, including a main control unit, an ETC transaction processing unit, a voice acquisition module, a voice synthesis module, a local edge computing unit, a multi-source external data fusion module, a context awareness module, and an intelligent fault diagnosis module; The ETC transaction processing unit is used to communicate with the roadside unit to complete the transaction and report the transaction status to the main control unit in real time. The voice acquisition module is used to acquire user voice commands; The speech synthesis module is used to convert the text generated by the system into speech output. The local edge computing unit is equipped with a lightweight speech recognition engine, a natural language understanding model, and dialogue management logic; The multi-source external data fusion module is used to connect to external data sources through API interfaces to obtain real-time traffic and weather information; The main control unit is equipped with a large language model inference engine, which is used to perform natural language understanding of user voice commands and generate responses or control commands. The context awareness module is used to acquire vehicle status information, driving environment information, and user historical interaction records in real time, and input the acquired multimodal information into the large language model inference engine as contextual prompts; The intelligent fault diagnosis module is connected to the ETC transaction processing unit and configured to monitor the ETC transaction status in real time. When an ETC transaction anomaly is detected, it automatically starts and collects fault parameters, including error type code, device operating status parameters, signal strength value, the time of the most recent successful transaction, and user historical operation records. After the collected data is deduplicated and structured, it is input into the large language model inference engine and combined with predefined fault diagnosis prompt templates to generate a personalized step-by-step voice guidance solution.

[0015] The local edge computing unit is equipped with a low-power AI chip, running a lightweight speech recognition engine, a natural language understanding model, and dialogue management logic; the large language model inference engine uses a quantized Transformer model deployed on the main control unit. Specifically, GPTQ or AWQ algorithms can be used to quantize the model in 4-bit or 8-bit form, or knowledge distillation can be used to transfer the capabilities of the large model to lightweight models such as TinyBERT, thereby reducing the consumption of computing resources while ensuring inference performance.

[0016] The context awareness module is also used to dynamically adjust the broadcast strategy according to the vehicle speed: if the vehicle speed is greater than the first threshold, the generated voice guidance scheme is broken down into short sentences for broadcast, or a message indicating that details have been sent to the user terminal is displayed; if the vehicle speed is lower than the second threshold, all steps are broadcast in full.

[0017] The intelligent fault diagnosis module also includes a multi-round interactive supplementation unit. When the large language model inference engine finds insufficient data during the diagnosis process, it actively initiates a query through the speech synthesis module to supplement the user information, and then merges the supplemented information and re-inputs it into the large language model inference engine.

[0018] The user's historical interaction records are stored in the local storage module in a cyclic overwrite manner, including the text, intent type, processing result and success mark of the last N interactions, and can be periodically uploaded to the cloud for model optimization.

[0019] The main control unit continuously monitors in-vehicle voice through a low-power voice activity detection circuit. When a clearly directional instruction is recognized, it enters the voice processing flow to achieve wake-word-free interaction. The clearly directional instruction refers to a voice segment that contains a preset wake-up keyword or conforms to a specific voice pattern and is determined by a lightweight neural network model to have an interactive intent.

[0020] The information acquired by the multi-source external data fusion module includes at least one of the following: real-time traffic conditions, weather warnings, service area facility information, and ETC billing information.

[0021] The device also includes a communication module that supports DSRC or 5G-V2X communication for interaction with roadside units, and supports Bluetooth connection with vehicle audio systems and user mobile phones.

[0022] like Figure 2 As shown in the figure, an embodiment of this application provides a vehicle-mounted intelligent OBU voice interaction method based on the above-described device, comprising the following steps: Step S1: The system is in a low-power monitoring state and collects in-vehicle voice signals in real time through the voice acquisition module. When a valid voice start point is detected, the voice acquisition process is started.

[0023] Specifically, in step S1, a low-power voice activity detection circuit continuously monitors the system. When a clearly directional instruction is detected, the voice acquisition process is initiated directly without a wake-up word. The clearly directional instruction refers to a valid instruction determined by a two-level wake-up mechanism: the first level is energy detection, and the second level is instruction intent recognition based on a lightweight neural network. When a voice segment is determined to have a clear interactive intent after passing both levels of detection, it is considered a valid voice start point. The low-power monitoring state is achieved through a low-power voice activity detection circuit within the main control unit. This circuit continuously analyzes the audio signal input from the microphone. When the voice energy exceeds a preset threshold and matches the voice characteristics, it is determined to be a valid voice start point. The detection of a valid voice start point includes: initiating acquisition after detecting a preset wake-up word, or initiating acquisition after recognizing a directional instruction using wake-up-free technology. The wake-up-free technology analyzes the voice stream in real time based on a deep neural network model to identify instructions containing specific intents. After initiating the voice acquisition process in step S1, a silence timeout T is set. When a continuous silence duration exceeds T, acquisition automatically ends, and the acquired voice data is sent to the next processing step.

[0024] In step S2, the collected voice data is converted into text commands, and the context awareness module uses the current vehicle status, environmental information, and historical dialogue records as supplementary context inputs.

[0025] Specifically, in step S2, the historical dialogue records are read from the local log, including the most recent 5-10 interaction records, timestamps, user question text, intent type, processing result, and success indicator. The collected voice data is converted into text commands using an end-to-end speech recognition model based on Transformer, deployed on a local edge computing unit, supporting noise suppression and multi-accent recognition. The vehicle status information acquired by the context-aware module includes vehicle speed, geographical location, and time; driving environment information includes at least one of weather conditions, road type, and traffic congestion level. The historical dialogue records are read from the local log and sorted by timestamp. The most recent N records are concatenated into a natural language format contextual prompt, which is input into the large language model inference engine along with the current text command. The contextual supplementary input also includes real-time traffic and weather information obtained from the multi-source external data fusion module, which is formatted as a text description and appended before or after the user command.

[0026] Step S3: Based on the intent category output by natural language understanding, perform a hybrid processing decision: if the intent is locally processable, execute the local processing steps and the main control unit directly generates the response text; if the intent is cloud-based large model processing, execute the cloud processing steps, encapsulate the user text and context through the communication module and send it to the cloud-based large model interface, and receive the response text generated by the cloud; if the intent is fault diagnosis triggering, trigger the intelligent fault diagnosis process.

[0027] Specifically, in step S3, the locally processable categories include device status queries, basic settings, and simple fault diagnosis, which are directly executed by the main control unit calling the corresponding hardware interfaces to generate concise response text. The cloud-based large-scale model processing categories include complex queries, fault assistance requests, and requests requiring external data fusion. These are sent to the cloud-based large-scale model interface after encapsulating the user text and context through the communication module, and the response text generated by the cloud is received. The intent category output by the natural language understanding is obtained through a natural language understanding model deployed on the local edge computing unit. This model is based on a lightweight BERT model for intent classification and entity extraction. The hybrid processing decision is based on a preset rule engine: intents for querying device status, setting parameters, and simple faults are classified as locally processable categories; intents for querying external information, complex fault assistance requests, and requests requiring real-time data are classified as cloud-based large-scale model processing categories; and intents containing fault keywords or system-detected ETC transaction failures are classified as fault diagnosis trigger categories. In step S3, when the intent category is locally processable, the system directly calls the corresponding hardware interface or local function according to the intent mapping table to generate response text without going through large model inference, so as to ensure real-time performance. When the intent category is cloud-based large model processing, the system encapsulates the user's text command, the context provided by the context awareness module, and the recent history into JSON format, sends it to the cloud-based large model API through the communication module, and sets a timeout. If the timeout occurs, it will be downgraded to local processing or return a preset prompt.

[0028] Step S4: For fault diagnosis triggers, execute the intelligent fault diagnosis process: collect multi-dimensional fault parameters related to ETC transaction anomalies, and after deduplication and structuring, input them into the large language model inference engine to generate a personalized step-by-step voice guidance solution as the response text.

[0029] Specifically, the intelligent fault diagnosis process described in step S4 includes: Step S41: After the fault is triggered, collect device status parameters, card information, OBU information, user operation history and context information. The device status parameters include the battery level of the ETC transaction processing unit, firmware version, anti-tamper status, and signal strength of the communication module. The card information includes the card balance, card validity period, and card type. The OBU information includes the OBU validity period and device serial number. Step S42: Preprocess the collected raw data and encapsulate it into JSON format. The preprocessing of the collected raw data includes: removing duplicate data, filtering outliers, filling missing values, and encapsulating it into JSON format according to a predefined data structure, wherein the field names adopt standardized naming. Step S43: Input the encapsulated data into the lightweight fault pre-diagnosis model of the local edge computing unit for preliminary judgment. The lightweight fault pre-diagnosis model is a hybrid model based on rule engine and decision tree. The rule engine directly matches fixed words according to error code, and the decision tree is used to handle fault scenarios with multiple conditions. Step S44: If the local pre-diagnosis is successful, generate the response text directly; otherwise, send the encapsulated data to the cloud-based large model for diagnosis via the communication module. Step S45: If the cloud call times out or fails, use the local pre-diagnosis result or the preset fallback statement; Step S46: If insufficient data is found during the diagnosis process, actively initiate multiple rounds of interactive supplementation of information. The active initiation of multiple rounds of interactive supplementation of information includes: generating natural language query statements according to the diagnostic needs, broadcasting them through the speech synthesis module, waiting for the user's voice reply, and merging the reply content with the original data after speech recognition. Step S47: parse the results returned by the large model, generate response text, and broadcast it. Step S48: After the broadcast is completed, collect user feedback, record the processing results and update the log. The method for collecting user feedback is: after the broadcast is completed, ask "Is the problem solved?" in voice, and recognize the user's answer "yes / no" or similar semantics, while recording the user's subsequent operations.

[0030] Step S5: The generated response text is converted into a speech broadcast output through the speech synthesis module.

[0031] Specifically, in step S5, the broadcasting strategy is dynamically adjusted based on vehicle speed: if the vehicle speed exceeds the first threshold, the long text is split into short sentences for broadcast or a message indicating that details have been sent is displayed; if the vehicle speed is below the second threshold, all steps are broadcast in their entirety, and detailed steps are simultaneously sent to the user's mobile phone. The first threshold is 80 km / h, and the second threshold is 60 km / h. When the vehicle speed is between these two, a medium broadcasting strategy is adopted, that is, key information is retained but secondary information is simplified. Before the voice broadcast output in step S5, the broadcasting volume and speaking speed are automatically adjusted based on the environmental noise level provided by the context awareness module: the volume is increased when the noise is high, and the volume is decreased when the noise is low, and the speaking speed is appropriately slowed down; if a user interruption command is received during the broadcasting process, the system immediately stops the current broadcasting and enters the corresponding processing: if it is "repeat", the broadcasting is restarted; if it is "stop", the broadcasting ends.

[0032] Step S6: Record the interaction information and update the local log. The log records use a circular overwrite mechanism to retain the most recent N records, where N is an integer between 100 and 1000.

[0033] Specifically, in step S6, the log records include user question text, intent type, response text, processing success identifier, and subsequent user feedback, and are periodically uploaded to the cloud for model fine-tuning and rule optimization.

[0034] Specifically, in step S6, the periodic upload to the cloud refers to: at preset intervals, compressing and encrypting the local logs before uploading them to the cloud server. The log records are stored in the local storage module in a circular queue with a queue length of N, where N is an integer between 100 and 1000. When a new record is written, the oldest record is automatically overwritten.

[0035] Step S6 also includes: automatically filtering out failed cases or low-scoring cases based on user feedback and successful processing indicators recorded in the logs, and attaching labels when uploading them for targeted fine-tuning of the cloud model.

[0036] Preferably, the in-vehicle intelligent OBU voice interaction method integrating a large language model further includes step S7 after step S6: the cloud server receives the uploaded logs, updates the global model based on differential privacy or federated learning technology, and sends the updated model parameters to the OBU to achieve continuous model optimization.

[0037] like Figure 3 As shown, this embodiment provides an in-vehicle intelligent OBU voice interaction system integrating a large language model. At the hardware level, this system integrates a voice acquisition module, a voice synthesis module, a local edge computing unit, a multi-source external data fusion module, a main control unit, a communication module, a storage module, a context-aware module, an intelligent fault diagnosis module, and an ETC transaction processing unit.

[0038] The voice acquisition module uses a multi-microphone array to acquire user voice commands in real time, and processes them with noise reduction and echo cancellation; it can maintain low-power monitoring in sleep mode. The speech synthesis module converts system-generated text into natural speech output, which is then directly output through the built-in speaker. The local edge computing unit is equipped with a low-power AI chip, running a lightweight speech recognition (ASR) engine, a natural language understanding (NLU) model, and dialogue management (DM) logic. The multi-source external data fusion module connects to external data sources through API interfaces to obtain real-time traffic conditions, weather, vehicle status, and other information. The main control unit, as the control core of the system, uses an automotive-grade multi-core processor to coordinate the collaborative work of various modules. The communication module supports DSRC / 5G-V2X communication to interact with roadside units, and also supports Bluetooth connection to the vehicle audio system and user mobile phones. The storage module stores lightweight local model parameters, user historical data, device status logs, local knowledge base, etc. The main control unit internally deploys a large language model inference engine, which adopts a lightweight deployment architecture, including a quantized Transformer model or a small-parameter language model based on knowledge distillation. This engine is used to perform natural language understanding of user voice commands, identify user intent, and generate corresponding responses or control commands.

[0039] Furthermore, the system also includes a context-aware module, used to acquire vehicle status information, driving environment information (such as vehicle speed, geographical location, and time), and user historical interaction records in real time. The context-aware module inputs the acquired multimodal information into the large language model inference engine as contextual prompts to generate context-relevant personalized responses.

[0040] The intelligent fault diagnosis module is connected to the ETC transaction processing unit and configured to monitor the ETC transaction status in real time. When an ETC transaction anomaly is detected, the intelligent fault diagnosis module automatically starts, collects fault parameters, including error type codes, device operating status parameters, signal strength values, the time of the most recent successful transaction, and the user's historical operation records. After deduplication and structuring, the collected data is input into the large language model inference engine, combined with predefined fault diagnosis prompt templates, to generate a personalized step-by-step voice guidance solution.

[0041] The ETC transaction processing unit is the core radio frequency component of a traditional OBU, responsible for completing toll collection via 5.8GHz microwave communication with the RSU. In this system, this unit reports the transaction status to the main control unit in real time, providing a data source for intelligent fault diagnosis.

[0042] Furthermore, the system supports a wake-word-free interaction mode. The main control unit continuously monitors in-vehicle voice through a low-power voice activity detection circuit. When a clearly directional command is recognized, it is determined to be a valid interaction command and enters the voice processing flow.

[0043] based on Figure 3 The system shown in this embodiment provides a vehicle-mounted intelligent OBU voice interaction method, including the following steps: I. Voice wake-up and data collection: The system operates in a low-power monitoring state. Upon triggering with a preset wake-up word or identifying a valid voice start point using wake-up-free technology, the voice acquisition process begins, and a voice silence timeout (N seconds) is set. The system then acquires in-vehicle voice signals in real time via the voice acquisition module, performs front-end signal processing, and obtains the voice data to be recognized.

[0044] II. Speech Recognition: Voice data is converted into text commands and input into the large language model inference engine. Simultaneously, the context-aware module supplements the input with current vehicle status, environmental information, and historical dialogue records. The system reads the last five interaction records from a local log file. The logs are stored using a circular overwrite method, and each record includes: timestamp, user question text, intent type, processing result, and success indicator.

[0045] III. Hybrid Processing Decisions: The system executes hybrid processing decision logic based on the intent category output by NLU: locally processable categories, cloud-based large model processing categories, and fault diagnosis trigger categories.

[0046] Locally processed requests include device status queries ("card expiration", "check balance"), basic settings ("increase volume"), and simple fault diagnosis (fixed prompts for error codes). These requests are executed directly by the main control unit calling the corresponding hardware interface to generate a concise response text. For example: (1) Check balance: Read the balance information in the ETC card and generate "Your ETC card balance is 328 yuan"; (2) If the device does not raise the barrier and the error code is E007 (insufficient balance): the message "Your ETC card balance is insufficient, this deduction has failed, please recharge in time" will be generated.

[0047] Large-scale cloud-based model processing includes complex queries ("road conditions ahead", "charging stations near the airport"), troubleshooting requests ("why didn't the barrier gate open?"), and requests requiring external data fusion. These requests enter the cloud processing flow. (1) The data is encapsulated in JSON format, including user text, recent interaction records, etc.; (2) Network request: Initiate an HTTPS request to the cloud large model interface, set a timeout to avoid long waiting times; (3) The cloud-based big data model is combined with the knowledge base to generate targeted response text; (4) Verify the validity of the response text and generate colloquial text; (5) Set timeout reminder.

[0048] Fault diagnosis trigger type: If the user's problem includes a fault-related description (such as "the barrier did not lift" or "the deduction failed"), or if the context awareness module detects an ETC transaction failure event, the intelligent fault diagnosis process will be triggered.

[0049] IV. Log Updates and Feedback: Regardless of whether the processing is done locally or in the cloud, the system will update the local log with the following information: user question text, intent type, response text, processing success indicator, and subsequent user feedback.

[0050] The logs use a circular overwrite mechanism, retaining the most recent N records, and also support periodically uploading logs to the cloud backend for model fine-tuning and rule optimization.

[0051] V. Speech Synthesis and Broadcasting: The generated response text is fed into the speech synthesis module, converted into natural speech, and played through the in-vehicle audio system or the OBU's built-in speaker. It also supports Bluetooth connection to the in-vehicle audio system for playback.

[0052] During the broadcast, the context awareness module continuously monitors the vehicle speed: if the vehicle speed is >80km / h, the system automatically breaks the long text into short sentences for broadcast, or prompts "Details have been sent to your mobile phone, please check it" to avoid distracting the driver with long-term voice messages.

[0053] The following is combined with, for example Figure 4 The ETC transaction process shown describes in detail the intelligent fault diagnosis process: "The intelligent fault diagnosis module collects equipment status parameters and error types, combines them with user operation history, and generates personalized step-by-step guidance information through a large language model."

[0054] 1. Fault Triggering and Data Collection: The fault diagnosis process can be triggered in two ways: Active trigger: The user speaks a description of the fault (such as "Why didn't the lever lift just now?"), which is recognized by the NLU as an intent to "help with the fault". Passive triggering: When the context-aware module detects an ETC transaction failure event (the ETC transaction processing unit reports an error code), it automatically starts the diagnostic process.

[0055] Upon triggering, the intelligent fault diagnosis module immediately collects the following multi-dimensional data: (1) Device status parameters: Read from the ETC transaction processing unit and storage module: Error type code: such as E001 (card reading failure), E003 (transaction failure), E007 (insufficient balance); Device working status: battery level; Last successful transaction time: used to determine whether the fault is new or historical.

[0056] (2) Card information and OBU information: Card balance, card validity period, card type (prepaid card / account card); OBU validity period, tamper status (normal / removed), firmware version.

[0057] (3) User operation history: Extract the 10 most recent user operation records from the local log, focusing on: whether the card has been inserted or removed recently; whether the OBU has been reset or upgraded recently.

[0058] (4) Contextual information: Real-time data provided by the contextual perception module: vehicle speed (to determine whether the vehicle is passing through the ETC lane at high speed); geographical location (to determine whether the vehicle is in a specific fault-prone section); time (to determine whether the vehicle is in a low-temperature period at night).

[0059] 2. Data encapsulation and preprocessing: The collected raw data is preprocessed according to the defined data format and encapsulated in JSON format.

[0060] 3. Local lightweight model pre-diagnosis: The encapsulated JSON data is first input into the lightweight fault pre-diagnosis model of the local edge computing unit. This model, based on a rule-based + machine learning hybrid architecture, can quickly identify common faults. Directly map the error code to a fixed message (e.g., E007 → "Insufficient balance, please recharge"). For combined conditions (such as "weak signal / low battery + card insertion / removal"), the model outputs a pre-diagnostic conclusion.

[0061] If the local pre-diagnosis is successful, a response text will be generated directly, and the process will proceed to step 7 to generate and play the response text. Otherwise, the process will proceed to the cloud-based large-scale model diagnostic workflow.

[0062] 4. Cloud-based large-scale model diagnostics: For complex faults that cannot be determined locally, the system sends a packaged JSON message to the cloud-based big data model API via the communication module, with a timeout of 60 seconds. The cloud-based big data model then performs inference using its built-in ETC fault knowledge base, generating response data in JSON format.

[0063] 5. Timeout / Failure Handling: If the cloud call times out (>60 seconds) or returns an error, the system will use a local pre-diagnostic result or a pre-set fallback message: "The current network is poor. You can call the ETC customer service hotline ***** for consultation, or view the diagnostic report later through the mobile app." 6. Multiple rounds of interactive information supplementation: If insufficient data is detected during the large model diagnostic process, the system will proactively initiate multiple rounds of interaction: System: "Have you recently had your windshield tinted?" User: "I've already put it on, just last week." The system will merge the supplementary information into the original JSON and then re-call the large model in the cloud.

[0064] 7. Generate response text and broadcast: The system parses the JSON returned by the large model, extracts the response data, and converts it into natural speech for broadcast through the speech synthesis module.

[0065] Broadcasting strategy: If the vehicle speed is >80km / h, only step 1 (emergency operation) will be announced, and the remaining steps will be simplified to "detailed steps have been sent to the mobile app"; If the vehicle speed is ≤60km / h, the entire process will be broadcast aloud, with a pause of N seconds between each step. Detailed steps will also be sent to the mobile app.

[0066] 8. User feedback: After the broadcast is complete, the system will ask in voice, "Has the problem been resolved?" Collect user feedback: When users say "solved", record successful cases for model fine-tuning. If a user says "unresolved" or "the steps are too complicated," prompt them to ask if they want to be transferred to a human customer service representative, or provide navigation to the nearest ETC service center.

[0067] Feedback results are written to local logs and uploaded to the cloud periodically, forming a closed loop of continuous optimization.

[0068] This application deeply embeds a large language model into the resource-constrained on-board unit (OBU) to construct an intelligent interaction hub for edge-cloud collaboration. This enables traditional ETC devices to transcend the one-way prompting function and evolve into intelligent travel terminals with natural language understanding and generation capabilities. Utilizing the reasoning capabilities of the large model, the single error message when an ETC transaction fails is transformed into a multi-dimensional personalized solution that integrates device status, user history, and driving context, achieving a qualitative leap in fault diagnosis from "passive alarm" to "active guidance." Simultaneously, through context-aware multi-turn dialogue management, real-time fusion of vehicle speed, location, time, and historical dialogue information maintains the dialogue state, supports coherent and natural multi-turn interactions, and integrates multi-source external data (road conditions, weather, charging piles, bills, etc.) to provide intelligent query services. In low-power standby mode, combined with voice activity detection and directional command recognition, natural interaction without wake-words is achieved, comprehensively improving the user experience while ensuring driving safety.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation methods of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A vehicle-mounted intelligent OBU voice interaction device integrating a large language model, characterized in that, include: The ETC transaction processing unit is used to communicate with the roadside unit to complete the transaction and report the transaction status to the main control unit in real time. The voice acquisition module is used to collect user voice commands; The speech synthesis module is used to convert the text generated by the system into speech output; The local edge computing unit is equipped with a lightweight speech recognition engine, a natural language understanding model, and dialogue management logic; The multi-source external data fusion module is used to connect to external data sources through API interfaces to obtain real-time traffic and weather information; The main control unit is equipped with a large language model inference engine, which is used to perform natural language understanding of user voice commands and generate responses or control commands. The context awareness module is used to acquire vehicle status information, driving environment information, and user history interaction records in real time, and input the acquired multimodal information into the large language model inference engine as contextual prompts; The intelligent fault diagnosis module is connected to the ETC transaction processing unit and configured to monitor the ETC transaction status in real time. When an ETC transaction anomaly is detected, the system automatically starts and collects fault parameters, including error type code, device operating status parameters, signal strength value, the time of the most recent successful transaction, and user historical operation records. After the collected data is deduplicated and structured, it is input into the large language model inference engine. Combined with predefined fault diagnosis prompt templates, a personalized step-by-step voice guidance solution is generated.

2. The in-vehicle intelligent OBU voice interaction device integrating a large language model according to claim 1, characterized in that, The intelligent fault diagnosis module also includes a multi-round interactive supplementation unit. When the large language model inference engine finds insufficient data during the diagnosis process, it actively initiates a query through the speech synthesis module to supplement the user information, and then merges the supplemented information and re-inputs it into the large language model inference engine.

3. The in-vehicle intelligent OBU voice interaction device integrating a large language model according to claim 1, characterized in that, The main control unit continuously monitors in-vehicle voice through a low-power voice activity detection circuit. When a clearly directional instruction is recognized, it enters the voice processing flow to achieve wake-word-free interaction. The clearly directional instruction refers to a voice segment that contains a preset wake-up keyword or conforms to a specific voice pattern and is determined by a lightweight neural network model to have an interactive intent.

4. The in-vehicle intelligent OBU voice interaction device integrating a large language model according to any one of claims 1-3, characterized in that, The information acquired by the multi-source external data fusion module includes at least one of the following: real-time traffic conditions, weather warnings, service area facility information, and ETC billing information.

5. A vehicle-mounted intelligent OBU voice interaction method based on the device according to any one of claims 1-4, characterized in that, Includes the following steps: Step S1: The system is in a low-power monitoring state and collects in-vehicle voice signals in real time through the voice acquisition module. When a valid voice start point is detected, the voice acquisition process is started. Step S2: The collected voice data is converted into text commands, and the context awareness module uses the current vehicle status, environmental information and historical dialogue records as supplementary context inputs. Step S3: Based on the intent category output by natural language understanding, perform a hybrid processing decision: if the intent is locally processable, execute the local processing steps and the main control unit directly generates the response text; if the intent is cloud-based large model processing, execute the cloud processing steps, encapsulate the user text and context through the communication module and send it to the cloud-based large model interface, and receive the response text generated by the cloud; if the intent is fault diagnosis triggering, trigger the intelligent fault diagnosis process. Step S4: For fault diagnosis triggers, execute the intelligent fault diagnosis process: collect multi-dimensional fault parameters related to ETC transaction anomalies, and after deduplication and structuring, input them into the large language model inference engine to generate a personalized step-by-step voice guidance solution as the response text. Step S5: The generated response text is converted into speech and output through the speech synthesis module; Step S6: Record the interaction information and update the local log. The log records use a circular overwrite mechanism to retain the most recent N records.

6. The in-vehicle intelligent OBU voice interaction method according to claim 5, characterized in that, In step S3, the locally processable classes include device status query, basic settings, and simple fault diagnosis, which are directly executed by the main control unit by calling the corresponding hardware interface to generate concise response text; the cloud-based large model processing classes include complex queries, fault assistance, and requests requiring external data fusion, which encapsulate the user text and context through the communication module and send it to the cloud-based large model interface to receive the response text generated by the cloud.

7. The in-vehicle intelligent OBU voice interaction method according to claim 5, characterized in that, In step S4, the intelligent fault diagnosis process specifically includes: Step S41: After the fault is triggered, collect device status parameters, card information, OBU information, user operation history and context information; Step S42: Preprocess the collected raw data and encapsulate it into JSON format; Step S43: Input the encapsulated data into the lightweight fault pre-diagnosis model of the local edge computing unit for preliminary judgment; Step S44: If the local pre-diagnosis is successful, generate the response text directly; otherwise, send the encapsulated data to the cloud-based large model for diagnosis via the communication module. Step S45: If the cloud call times out or fails, use the local pre-diagnosis result or the preset fallback statement; Step S46: If insufficient data is found during the diagnosis process, actively initiate multiple rounds of interaction to supplement information; Step S47: parse the results returned by the large model, generate response text, and broadcast it. Step S48: After the broadcast is completed, collect user feedback, record the processing results, and update the log.

8. The in-vehicle intelligent OBU voice interaction method according to claim 5, characterized in that, In step S5, the broadcasting strategy is dynamically adjusted according to the vehicle speed during the broadcast: if the vehicle speed exceeds the first threshold, the long text is split into short sentences for broadcast or a message indicating that details have been sent is displayed; if the vehicle speed is below the second threshold, all steps are broadcast in full, and detailed steps are sent to the user's mobile phone at the same time.

9. The in-vehicle intelligent OBU voice interaction method according to claim 5, characterized in that, In step S6, the log records include user question text, intent type, response text, processing success identifier, and subsequent user feedback, and are periodically uploaded to the cloud for model fine-tuning and rule optimization.

10. The in-vehicle intelligent OBU voice interaction method according to any one of claims 5-9, characterized in that, It also includes step S7: The cloud server receives the uploaded logs, updates the global model based on differential privacy or federated learning technology, and sends the updated model parameters to the OBU to achieve continuous model optimization.

Images