Vehicle interaction method, device and system fusing vehicle-mounted real-time state and electronic equipment

Abstract

The invention provides a vehicle interaction method, device and system fusing a vehicle-mounted real-time state and electronic equipment. The method comprises the following steps: receiving a voice instruction of a user; calling vehicle state information related to the voice instruction; the vehicle state information comprises vehicle current state information and long-term state information; the voice instruction and the vehicle state information are uploaded to a cloud server, so that the cloud server conducts reasoning on the voice instruction and the vehicle state information through an LLM model, and a vehicle control instruction is generated; and the vehicle receives and executes the vehicle control instruction. According to the method, the vehicle state information related to the voice instruction is called, and the LLM model is used for speculating the voice instruction and the vehicle state information to generate the vehicle control instruction, so that the core reasoning capability of the LLM model is deeply applied to a vehicle-mounted core interaction task instead of a search engine which only can check data; the value of the intelligent brain is thoroughly activated, and the defect of LLM capability waste in an existing scheme is overcome.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit and in-vehicle human-machine interaction technology, specifically to a vehicle interaction method, device, system and electronic device that integrates real-time in-vehicle status. Background Technology

[0002] In the current field of intelligent cockpit and in-vehicle human-machine interaction technology, in-vehicle voice interaction systems have become a key component for improving driving safety and passenger convenience. With continuous technological advancements, users' expectations for in-vehicle voice systems have expanded beyond basic command execution; they now expect a more natural, intelligent, and context-aware interactive experience. Currently, to balance response speed, network dependence, and the depth of natural language understanding, in-vehicle voice interaction systems generally adopt a hybrid edge-cloud integrated speech recognition and understanding solution. Its specific architecture and working principle are as follows:

[0003] The system deploys lightweight ASR (Action Speech Recognition) and NLU (Natural Language Understanding) models locally on the vehicle's infotainment system. This is used to handle basic vehicle control commands that require high response speeds and do not require internet connectivity, such as "open the window" or "turn up the volume." When the local NLU cannot understand the command, or when the command requires online information support (e.g., "navigate to Tiananmen Square"), the system sends the speech or recognized text to a cloud server. The cloud-based proprietary NLU model, which offers more powerful and broader coverage, performs deep analysis and returns the results to the vehicle's infotainment system for execution. To enhance the system's knowledge-based question-answering capabilities, the existing technology also integrates a relatively small and fast cloud-based LLM (Large Language Model). The system first makes a preliminary judgment on the user's intent. If the intent pertains to a routine task within the vehicle domain, it is handled by the hybrid proprietary NLU system. If it is an open-domain encyclopedic knowledge question-answering task, it is forwarded to the LLM module for processing. The LLM then returns the query results to the vehicle's infotainment system in text form for broadcast or display.

[0004] Although the hybrid solution integrating LLM represents the current mainstream level of in-vehicle voice technology, its design still has the following profound inherent defects, resulting in the level of intelligence and user experience falling far short of the ideal state. Summary of the Invention

[0005] This invention provides a vehicle interaction method, device, system, and electronic device that integrates real-time vehicle status, in order to address the shortcomings of existing technologies in terms of intelligence and user experience.

[0006] This invention provides a vehicle interaction method that integrates real-time vehicle status, comprising:

[0007] Receive user voice commands;

[0008] Retrieve vehicle status information related to the voice command; the vehicle status information includes the vehicle's current status information and long-term status information.

[0009] The voice commands and vehicle status information are uploaded to the cloud server so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands.

[0010] The vehicle receives and executes the vehicle control commands.

[0011] According to the vehicle interaction method for integrating real-time vehicle status provided by the present invention, retrieving vehicle status information related to the voice command includes:

[0012] Convert the voice command into ASR text data;

[0013] Based on the ASR text, select vehicle current status information related to the ASR text data from the entire set of vehicle current operating statuses, and select long-term status information related to the ASR text data from the entire set of vehicle operating statuses.

[0014] The vehicle's current status information includes: standardized vehicle attributes, media playback status, door and window status, air conditioning status, foreground and background application status, and audio focus status.

[0015] According to the vehicle interaction method for integrating real-time vehicle status provided by the present invention, the step of using an LLM model to infer the voice commands and vehicle status information to generate vehicle control commands includes:

[0016] Based on the ASR text and the vehicle status information, an enhanced prompt word is constructed;

[0017] The LLM model performs inference on the enhanced prompt words and obtains the inference results stored in JSON format;

[0018] The reasoning result is parsed, and the vehicle control command is generated based on the reasoning result.

[0019] According to the vehicle interaction method for fusing real-time vehicle status provided by the present invention, the collection of long-term status information includes:

[0020] Monitor key vehicle events; these key events include: navigation events, status threshold events, time / date events, driving mode change events, and user habit events.

[0021] Collect contextual data related to the key events;

[0022] The context data is uploaded to the cloud server as the long-term state information.

[0023] According to the vehicle interaction method for integrating real-time vehicle status provided by the present invention, the navigation event includes: navigation start point, end point, waypoints, and remaining mileage;

[0024] The state threshold events include: the vehicle entering a low battery or low fuel state.

[0025] The time / date events mentioned include: entering the morning rush hour on weekdays, or long-distance travel time during holidays;

[0026] The driving mode change event includes: switching to Sport mode or Off-road mode;

[0027] The user habit events include events corresponding to a user habitually performing a certain operation at a specific time / place.

[0028] According to the vehicle interaction method for integrating real-time vehicle status provided by the present invention, the vehicle receiving and executing the vehicle control command includes:

[0029] Based on the vehicle control command, the vehicle's local API is invoked;

[0030] Execute the API and return the result to the user.

[0031] The present invention also provides a vehicle interaction device that integrates real-time vehicle status, comprising:

[0032] The receiving unit is used to receive the user's voice commands;

[0033] The retrieval unit is used to retrieve vehicle status information related to the voice command; the vehicle status information includes the current status information and long-term status information of the vehicle.

[0034] The uploading unit is used to upload the voice commands and vehicle status information to the cloud server, so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands.

[0035] An execution unit is used to receive and execute the vehicle control commands.

[0036] The present invention also provides a vehicle interaction system that integrates real-time vehicle status, comprising:

[0037] The receiving module is used to receive the user's voice commands;

[0038] A local vehicle status collector is used to collect vehicle status information related to the voice command; the vehicle status information includes the vehicle's current status information and long-term status information.

[0039] A dynamic prompt word builder is used to construct enhanced prompt words based on the voice commands and vehicle status information;

[0040] The vehicle status cloud synchronization bus is used to upload the constructed enhanced prompt words to the cloud server;

[0041] A cloud server is used to infer the enhanced prompts using an LLM model and generate vehicle control commands.

[0042] An execution module is used for the vehicle to receive and execute the vehicle control commands.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle interaction method for integrating real-time vehicle status as described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle interaction method for integrating real-time vehicle status as described above.

[0045] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a vehicle interaction method for integrating real-time vehicle status as described above.

[0046] The vehicle interaction method, device, system, and electronic device provided in this application integrate real-time vehicle status. By retrieving vehicle status information related to voice commands and using an LLM model to infer voice commands and vehicle status information, vehicle control commands are generated. This allows the core reasoning capability of the LLM model to be deeply applied to core vehicle interaction tasks, rather than being just a search engine for information. This fully activates the value of its intelligent brain and solves the defect of wasted LLM capabilities in existing solutions. Attached Figure Description

[0047] Figure 1 This is one of the schematic diagrams of the vehicle interaction method that integrates real-time vehicle status provided by the present invention;

[0048] Figure 2 This is a schematic diagram of the long-term status information collection process for this application;

[0049] Figure 3 The second schematic diagram of the vehicle interaction method that integrates real-time vehicle status provided by the present invention;

[0050] Figure 4 This is a structural schematic diagram of the vehicle interaction device that integrates real-time vehicle status provided by the present invention.

[0051] Figure 5 This is a schematic diagram of the vehicle interaction system that integrates real-time vehicle status provided by the present invention.

[0052] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] In the current field of in-vehicle voice interaction technology, the core demand is to balance response speed (ensuring rapid command execution), network dependence (reducing the need for continuous network connectivity), and natural language understanding depth (accurately interpreting user intent). To meet this demand, the industry has generally phased out single solutions that are purely local (limited functionality, insufficient understanding depth) or purely cloud-based (reliant on network, high response latency), and has formed and adopted a hybrid "edge-cloud integrated" voice recognition and understanding solution as the mainstream technical path.

[0055] The existing solutions are as follows:

[0056] A lightweight ASR (Audio Recognition) and NLU (Natural Language Understanding) model is deployed locally on the vehicle's infotainment system. It prioritizes two types of commands: basic vehicle control commands that do not require internet access (such as "open the window" or "turn up the volume"), and real-time operation commands requiring high response speed. These latter commands can be recognized and executed locally, avoiding network latency. In two special cases, the system triggers cloud access: complex commands that the local NLU model cannot understand, and commands requiring online information support (such as "navigate to Tiananmen Square," which requires real-time map data). In these cases, the system sends the user's voice or recognized text to the cloud server. The proprietary NLU model deployed in the cloud (with stronger computing power and broader coverage) performs deep analysis of the command and returns the results to the vehicle's infotainment system for execution. This hybrid proprietary NLU system has been specifically optimized for in-vehicle scenarios, focusing on handling explicit commands in the in-vehicle domain. It can accurately and efficiently handle most interactive commands in core in-vehicle scenarios such as navigation, media, communication, and vehicle control, making it a core module ensuring the stable operation of basic in-vehicle voice functions. Based on this hybrid proprietary NLU system, a lightweight LLM (Large Language Model) is used as an encyclopedic knowledge add-on module to fill the gap in the infrastructure's capabilities for "general knowledge question answering." It does not participate in core in-vehicle command processing. The specific logic and role are as follows: On the basis of the "local + cloud proprietary NLU" infrastructure, a lightweight cloud LLM is additionally integrated. The system uses an "Intent Routing" mechanism to allocate tasks: First, the user's intent is initially judged. If it is determined to be a routine task in the in-vehicle domain (such as "play the next song" or "turn on the air conditioner"), it is directly handed over to the hybrid proprietary NLU system for processing. Only when the intent is clearly identified as "open domain encyclopedic knowledge question answering" (i.e., general knowledge queries unrelated to in-vehicle functions, such as "introduce Zhang San's albums" or "explain photosynthesis"), is the question forwarded to the add-on LLM module. In this solution, LLM is not a core decision-making or understanding module, but plays only a "parallelized, single-function" auxiliary role, similar to a search engine or encyclopedia Q&A robot. It is only responsible for handling general knowledge query needs that cannot be covered by hybrid proprietary NLU, and finally sends the query results to the vehicle terminal in text form, and feeds them back to the user through broadcast or screen display.

[0057] While the aforementioned hybrid solutions integrating LLM represent the current mainstream level of in-vehicle voice technology, they still suffer from profound inherent design flaws, resulting in a level of intelligence and user experience that falls far short of expectations. Specifically, firstly, the core capabilities of LLM are wasted, and its application scenarios are extremely narrow. Existing solutions fundamentally misposition LLM, placing it merely as a supplementary element at the end of the interaction process rather than a core processing module. Functionally, LLM is only responsible for handling non-in-vehicle general knowledge questions (such as encyclopedic Q&A) that hybrid proprietary NLU systems cannot cover. This is equivalent to downgrading the intelligent brain, which possesses core capabilities such as contextual understanding, fuzzy reasoning, and situational awareness, to a search engine that can only retrieve information. This design completely severs the connection between LLM and core in-vehicle interaction tasks: for example, user commands related to the vehicle (such as "It's a bit stuffy in the car" or "Is the road ahead congested?") cannot be handled by LLM's reasoning ability. Its most valuable advantages—understanding complex scenarios and deriving implicit needs—are completely idle, leading to a serious waste of technical resources. Secondly, proprietary NLU models cannot cope with complex contextual scenarios. The core advantage of proprietary NLU models is their ability to handle explicit in-vehicle commands (such as "open the window" or "navigate to the company"). However, due to limitations in their model architecture, they lack the deep reasoning capabilities of LLM models. In complex scenarios that require combining real-time vehicle status and potential user needs, they are prone to misunderstandings or failures.

[0058] Typical Scenario 1 (Contextual Association Failure): When the vehicle is in a state of "high-speed driving + windows fully open", the user says "the wind noise is too loud". The proprietary NLU cannot associate the real-time context such as "vehicle speed, window status" and can only match preset instructions from the keyword "noise". It may incorrectly recommend "turn down the volume" instead of accurately executing "close the windows".

[0059] Typical Scenario 2 (Failure of Potential Needs Derivation): When a user says "I'm a little tired," the proprietary NLU cannot combine data such as "driving time (e.g., driving continuously for 2 hours) and in-car air quality (e.g., high CO2 concentration)" to deduce the user's potential need to "refresh or improve the in-car environment." It can only fail to respond due to "vague instructions" or only provide irrelevant basic function options.

[0060] Secondly, the dual-core split leads to a fragmented interactive experience. Existing solutions are not a unified intelligent system, but rather a simple patchwork of two independent modules: a proprietary NLU and an LLM. An unbridgeable information gap exists between them: the proprietary NLU is responsible for performing in-vehicle operations (actions), while the LLM is responsible for answering general questions (chatting). The former cannot access the latter's reasoning capabilities, and the latter cannot obtain the former's vehicle status data (such as vehicle speed and air conditioning settings), resulting in functional fragmentation. Furthermore, users must actively adapt to the system logic during interaction. For example, if they want to know how to adjust the air conditioning when fatigued, they must first determine if it's an operational question and use instructions understandable to the proprietary NLU (such as turning on the external air circulation), rather than directly using natural language (such as "I'm tired, can you adjust the air conditioning to be more comfortable?"). If an operational request is mistakenly sent to the LLM, only a text response is received, and no actual operation can be performed. This user-request-system adaptation model, rather than the system actively adapting to the user, leads to a fragmented and disjointed interactive experience. To address these issues, this application provides a vehicle interaction method that integrates real-time in-vehicle status, as follows: Figure 1 As shown, it includes the following steps:

[0061] Step 101: Receive the user's voice commands.

[0062] Specifically, the system captures the user's voice signal through the voice acquisition hardware (such as a microphone array) on the vehicle's infotainment system, and then the built-in ASR (speech recognition) engine converts the voice signal into a machine-readable text string, such as converting the user's "The wind noise is too loud" into the corresponding text.

[0063] Step 102: Retrieve vehicle status information related to the voice command; the vehicle status information includes the current status information and long-term status information of the vehicle.

[0064] Specifically, the system retrieves two core data categories: current vehicle status information and long-term status information. Current status information is dynamic data that changes in real time (such as vehicle speed, window status, air conditioning temperature, remaining navigation range, and media playback progress). Long-term status information is static data that remains constant, such as vehicle model, seat adjustment limits, maximum range, and supported vehicle control functions—basic attributes preset at the factory. The system does not indiscriminately retrieve all status information; instead, it filters relevant data based on the voice command in step 101. For example, if the user's voice command is "the wind noise is too loud," it prioritizes retrieving current status information directly related to wind noise, such as vehicle speed, window status, and current driving mode. If the user's command is "is the range enough to reach the destination?", it simultaneously retrieves the current remaining battery / fuel level (current status) and the vehicle's energy consumption per 100 kilometers (inherent information).

[0065] Step 103: Upload the voice commands and vehicle status information to the cloud server so that the cloud server can use the LLM model to infer the voice commands and vehicle status information and generate vehicle control commands.

[0066] Specifically, the cloud-based LLM model in this application is no longer just an add-on for processing encyclopedic knowledge, but rather serves as the core decision engine, combining uploaded vehicle status information to perform contextualized reasoning on voice commands. For example, upon receiving the voice command "I'm a little tired" along with the status data of driving continuously for 2 hours (current state) and supporting seat massage (inherent information), the LLM will infer the user's potential need to relieve fatigue, and then generate a combined control command to activate seat massage and play refreshing music. If it receives the command "The wind noise is too loud" along with the data of a vehicle speed of 80 km / h and the left front window open (current state), it will directly infer and generate a precise control command to close the left front window. The LLM ultimately generates executable vehicle control commands (such as "close the left front window" or "raise the air conditioning temperature by 2 degrees"), thus achieving a functional leap from knowledge-based question answering to decision execution.

[0067] Step 104: The vehicle receives and executes the vehicle control command.

[0068] Specifically, after receiving vehicle control commands from the cloud via the network, the in-vehicle infotainment system uses an intent-to-API mapping engine to parse the commands into locally callable API commands (e.g., parsing "close the left front window" into the WindowControl(leftFront,close) command that calls the vehicle manufacturer's middleware API). The parsed API commands directly drive the vehicle's hardware or software to perform operations (e.g., controlling the window motor to close the window or adjusting the air conditioning compressor power). At the same time, a TTS engine can generate voice feedback (e.g., "The left front window has been closed for you"), allowing the user to perceive the operation result and forming a complete interactive loop.

[0069] In existing solutions, proprietary NLUs, lacking reasoning capabilities, are prone to failure in scenarios involving "fuzzy instructions + contextual association" (e.g., misinterpreting "too much wind noise" as "turn down the volume"). This application, however, provides data support for reasoning by retrieving the current state and inherent information, avoiding the drawbacks of blind judgment without context. Furthermore, the LLM handles the reasoning, its powerful contextual understanding capabilities accurately interpreting fuzzy and implicit instructions (e.g., "I'm a little tired" corresponds to "turn on the seat massage"), without relying on the keyword matching logic of proprietary NLUs. This makes the system more capable in complex contextual scenarios, completely solving the shortcomings of proprietary NLUs. Additionally, in existing technologies, because proprietary NLUs (for actions) and LLMs (for chat) are independent modules, an information gap exists, requiring users to actively adapt to the system. In contrast, this application eliminates the need for users to determine which module to assign instructions to; users simply express their needs naturally (e.g., "I'm tired," "too much wind noise"), and the system automatically completes the entire process of "receiving - retrieving state - reasoning - executing." This achieves natural user interaction and completely improves the experience gap problem of existing solutions.

[0070] This application generates vehicle control commands by retrieving vehicle status information related to voice commands and using an LLM model to infer the voice commands and vehicle status information. This allows the core reasoning capabilities of the LLM model to be deeply applied to core in-vehicle interaction tasks, rather than being just a search engine for information. This fully activates the value of its intelligent brain and solves the problem of wasted LLM capabilities in existing solutions.

[0071] Preferably, the method of retrieving vehicle status information related to voice commands in this application includes: converting the voice commands into ASR text data; filtering out current vehicle status information related to the ASR text data from the entire set of current vehicle operating states, and selecting long-term status information related to the ASR text data from the entire set of current vehicle operating states; the current vehicle status information includes: standardized vehicle attributes, media playback status, door and window status, air conditioning status, front and back-end application status, and audio focus status.

[0072] Specifically, the complete set of vehicle current status data includes all dynamic real-time data that the vehicle's infotainment system can collect (such as vehicle speed, range, location, and hardware status). The system will filter out directly related dynamic data based on keywords and semantic logic in the ASR text. The filtering scope focuses on standardized / high-frequency data in the following dimensions:

[0073] Standardized vehicle attributes: such as vehicle speed, fuel / battery charge, gear;

[0074] Media playback status: such as the currently playing application, track name, and play / pause status;

[0075] Vehicle door and window status: such as the open / closed status of the left front / right front / rear windows, and the sunroof status;

[0076] Air conditioning status: such as current temperature, fan speed, mode (internal / external circulation), seat heating / ventilation status;

[0077] Foreground and background application status: such as whether a navigation app or a music app is currently running in the foreground, and whether any communication software is running in the background;

[0078] Audio focus status: such as the current audio output device (speaker / headphones), volume level, and whether a call is occupying the audio channel.

[0079] For example, if the ASR text is "I can't hear the passenger," the system will filter relevant data from the current state set, such as the left front window status (whether it's open), vehicle speed (whether it's traveling at high speed), and audio focus status (whether there's high-volume media playback). The long-term state information set consists of fixed attribute data preset at the vehicle's factory (such as vehicle model, seat adjustment limits, supported vehicle control functions, maximum range, etc.), and the system also filters related data based on the ASR text. For example, if the ASR text is "Is the range enough to reach the next service area?", the system will filter the vehicle's energy consumption per 100 kilometers (inherent information) + current remaining battery power (current state information), providing complete data support for LLM's inference of whether the range is sufficient. This application provides LLM with contextualized data strongly correlated with user needs by filtering relevant state information, rather than undifferentiated full data, thus enabling rapid inference of precise control commands instead of simply generating text responses. This allows LLM's contextual understanding and fuzzy reasoning capabilities to be truly applied to core in-vehicle tasks, thereby completely solving the defect of wasted LLM capabilities in existing solutions.

[0080] Furthermore, this application, on the one hand, directly provides a basis for intent understanding by filtering out relevant state information; on the other hand, it avoids interference from the full state data on reasoning, allowing LLM to focus on the core scenario. This greatly improves the system's understanding accuracy when dealing with ambiguous and implicit instructions, solving the problem of proprietary NLU failing in complex scenarios.

[0081] Preferably, this application utilizes an LLM model to infer from voice commands and vehicle status information to generate vehicle control commands, including: constructing enhanced prompt words based on the ASR text and the vehicle status information; inferring from the enhanced prompt words using the LLM model to obtain inference results stored in JSON format; parsing the inference results and generating vehicle control commands based on the inference results.

[0082] Specifically, after receiving enhanced prompts, the cloud-based LLM model no longer simply outputs text responses. Instead, it performs contextualized reasoning by combining the vehicle status information within the prompts. First, it identifies the user's core needs; then, it determines feasible actions based on long-term status information; and finally, it outputs the structured reasoning results in JSON format. This structured output ensures the standardization of the reasoning results (avoiding ambiguity) and provides a unified data interface for subsequent parsing and execution, achieving seamless integration of reasoning and execution. The system does not simply concatenate ASR text with vehicle status information; instead, it integrates them according to a standardized format (JSON) that LLM can efficiently understand, forming enhanced prompts that include user intent and real-time context. This structured format allows LLM to clearly identify what the user is saying, the vehicle's current state, and what the vehicle can do, avoiding reasoning bias caused by cluttered information and providing a complete contextual profile for subsequent accurate decision-making.

[0083] This application provides LLM with vehicle contextual data through enhanced prompt words, enabling it to perform specific reasoning for in-vehicle scenarios (rather than general knowledge answers). The structured reasoning output requires LLM to generate actionable operational decisions, rather than textual information, thus upgrading LLM from an information query tool to an in-vehicle intelligent decision-making hub. Moreover, the enhanced prompt words provide a complete context of ASR text plus vehicle status, allowing LLM to reason based on real-time data, avoiding blind judgment without context. The structured reasoning results clearly define operational details, eliminating the need to rely on proprietary NLU "keyword matching logic." Even when faced with ambiguous commands (such as "it's a bit stuffy inside the car"), LLM can combine current and long-term vehicle status information to infer precise vehicle control commands.

[0084] Preferably, Figure 2 This is a schematic diagram of the long-term status information collection process for this application, such as... Figure 2 As shown, the collection of long-term status information includes: monitoring key vehicle events; the key events include: navigation events, status threshold events, time / date events, driving mode change events, and user habit events; collecting context data related to the key events; and uploading the context data as the long-term status information to the cloud server.

[0085] Specifically, the system continuously monitors a set of predefined key events in the background through a local vehicle state collector, which reflect the long-term / stable scenario characteristics of the vehicle, rather than indiscriminately capturing all state changes. These key events focus on the following core scenario dimensions:

[0086] Navigation events: When a user sets or starts a new navigation in a navigation application (such as "set destination to XX shopping mall" or "start navigation to the company"), the route, destination and other information associated with these events remain stable throughout the navigation process.

[0087] State threshold events: When a vehicle enters a specific critical state and this state lasts for a long time (such as a range of less than 50 kilometers or a fuel level of less than 1 / 4), these are basic attribute data that require long-term monitoring.

[0088] Time / Date Events: Entering a time period with fixed scenario attributes (such as weekday morning rush hour 7:00-9:00, long-distance travel during holidays), the scenario requirements corresponding to these events (such as "avoiding traffic jams during morning rush hour" and "entertainment during holidays") are stable.

[0089] Driving mode change event: When a user switches to a fixed driving mode of the vehicle (such as switching from "Comfort mode" to "Sport mode" or "Off-road mode"), the driving mode will remain unchanged for a relatively long period of time, which is a stable attribute of vehicle operation.

[0090] User habit events: Actions that users repeatedly perform at fixed times / locations (such as playing favorite morning music at 8:00 every day, or opening car windows for ventilation when passing through XX intersection). This type of data, which reflects user behavior preferences, has long-term stability.

[0091] The method provided in this application packages the collected context data, actively uploads it to a cloud server, and associates and caches it with the current user session or user profile, thereby ensuring that the inherent information stored in the cloud is always synchronized with the user's latest long-term scenario needs.

[0092] Preferably, the navigation events include: the starting point, ending point, waypoints, and remaining mileage of the navigation; the state threshold events include: the vehicle entering a low battery or low fuel state; the time / date events include: entering the morning rush hour on a weekday, or the long-distance travel time during holidays; the driving mode change events include: switching to sport mode or off-road mode; and the user habit events include: events corresponding to the user habitually performing a certain operation at a specific time / location.

[0093] Preferably, the process of receiving and executing vehicle control commands in this application includes: calling a local vehicle API according to the vehicle control commands; executing the API; and feeding back the results of the operation to the user.

[0094] Specifically, after receiving vehicle control commands (usually structured data in JSON format) from the cloud, the vehicle first extracts key information from the commands through a parsing module. This information includes the operation domain (e.g., window control, air conditioning adjustment), specific actions (e.g., closing, raising), and operation targets (e.g., left front window, temperature parameters). This information is then mapped to locally callable API interfaces. Local API calls cover three core types of interfaces, ensuring that vehicle control needs at different levels can be met.

[0095] Standardized system APIs, such as AndroidCarAPI and AutomotiveAPI, are used to call control functions of standardized vehicle attributes (such as adjusting vehicle speed and changing gears).

[0096] Car manufacturer middleware API: Used to call the car manufacturer's proprietary hardware control functions (such as seat massage mode for specific car models, custom air conditioning scenarios);

[0097] System-level service APIs, such as AudioManager (audio control) and MediaSessionManager (media playback), are used to call operations at the vehicle infotainment software level (such as switching music tracks and adjusting volume).

[0098] The invoked local APIs communicate directly with the vehicle's hardware controller or software services to drive actual operations. For example, after calling the window control API, the hardware controller sends an electrical signal to control the motor of the left front window until the window is completely closed; after calling the air conditioning adjustment API, the air conditioning system adjusts the compressor power or the damper angle to change the temperature / fan speed. After the operation is completed, the system will provide feedback to the user in two ways to ensure the integrity of the interactive experience:

[0099] Voice feedback: Based on LLM inference results, voice broadcasts are generated through a TTS (text-to-speech) engine, allowing users to quickly obtain operation results through hearing. Visual feedback: Operation results (such as window closing status icons and current air conditioning temperature) are displayed on the vehicle's infotainment screen, forming a dual confirmation of voice and vision, avoiding confusion for users who cannot hear voice feedback due to ambient noise.

[0100] Because existing solutions only output encyclopedic text answers and cannot participate in vehicle control, the core capabilities of LLM are severely wasted. This application addresses this by using API calls to directly translate control commands generated by LLM into vehicle operations. Furthermore, existing proprietary NLU solutions not only have shortcomings in understanding fuzzy commands but may also fail to execute commands due to API mismatches (e.g., recognizing a "relieve fatigue" need but being unable to call the seat massage API). This application addresses this by, firstly, incorporating long-term state information into the LLM's command generation to ensure executability and avoid generating unfeasible intentions; secondly, the intent-to-API mapping engine covers both standardized and vehicle-customized APIs, matching various vehicle control needs. Even in complex scenarios (e.g., "activate the air conditioning dust prevention mode in off-road mode"), precise execution can be achieved by calling the vehicle-customized middleware API. This allows the system to not only understand but also execute commands in complex scenarios, completely resolving the limitations of proprietary NLU. After operation, the system automatically provides voice / visual feedback, eliminating the need for users to actively check the status and effectively improving the user's in-vehicle interaction experience.

[0101] The solution provided by this invention will be described in its entirety below. Figure 3 This is the second schematic diagram of the vehicle interaction method that integrates real-time vehicle status provided by the present invention, as shown below. Figure 3 As shown, the solution includes the following steps:

[0102] Step 1: The vehicle system continuously monitors various key events, including navigation events (such as the start and end points of navigation), status threshold events (such as low battery / low fuel), time / date events (such as weekday morning rush hour), driving mode change events (such as switching to sport mode), and user habit events (such as habitual operations at specific times / locations).

[0103] Step 2: After a critical event is detected, collect contextual data related to the critical event. This data includes current vehicle status information (such as standardized vehicle attributes, media playback status, etc.) and long-term status information (such as vehicle model, supported vehicle control functions, etc.).

[0104] Step 3: Upload the collected vehicle data to the LLM (Large Language Model) server in the cloud for use in LLM inference during the subsequent main voice interaction process.

[0105] Step 4: The user issues a voice command to the vehicle to start the voice interaction process.

[0106] Step 5: The vehicle uses Automatic Speech Recognition (ASR) technology to convert the user's voice commands into ASR text data.

[0107] Step 6: Retrieve the vehicle's current status information related to the voice command, including associating the ASR text data with relevant vehicle current status information (such as media playback status, door and window status, etc.) and long-term status information (such as inherent vehicle attributes) selected from the complete set of vehicle current operating status.

[0108] Step 7: Based on the ASR text and the obtained vehicle status information (including instantaneous vehicle information and vehicle data uploaded by the proactive context synchronization subprocess), construct enhanced cue words to provide richer information for LLM inference.

[0109] Step 8: The cloud-based LLM model infers the enhanced prompts, integrates the vehicle's current state and existing vehicle data, and obtains the inference results stored in JSON format. These results clearly define the vehicle control intent, such as the operating domain, specific actions, and objectives.

[0110] Step 9: Send the vehicle control intent obtained from LLM inference to the vehicle.

[0111] Step 10: The vehicle calls the local API of the vehicle according to the received vehicle control command and performs the corresponding operation, such as controlling the doors and windows, air conditioning, etc.

[0112] Step 11: After the operation is completed, the result will be fed back to the user via voice.

[0113] Step 12: Complete the voice interaction process and end this interaction.

[0114] The vehicle interaction device integrating real-time vehicle status provided by the present invention will be described below. The vehicle interaction device integrating real-time vehicle status described below can be referred to in correspondence with the vehicle interaction method integrating real-time vehicle status described above. Figure 4 As shown, this application provides a vehicle interaction device that integrates real-time vehicle status, including:

[0115] The receiving unit 401 is used to receive the user's voice commands;

[0116] The retrieval unit 402 is used to retrieve vehicle status information related to the voice command; the vehicle status information includes the current status information and long-term status information of the vehicle.

[0117] Upload unit 403 is used to upload the voice command and vehicle status information to the cloud server, so that the cloud server can use the LLM model to reason about the voice command and vehicle status information and generate vehicle control commands.

[0118] The execution unit 404 is used to receive and execute the vehicle control command.

[0119] Figure 5 This is a schematic diagram of the vehicle interaction system structure that integrates real-time vehicle status provided by the present invention, as shown below. Figure 5 As shown, the system includes:

[0120] Receiver module 501 is used to receive user voice commands;

[0121] A vehicle status local collector 502 is used to collect vehicle status information related to the voice command; the vehicle status information includes the current status information and long-term status information of the vehicle.

[0122] A dynamic prompt word builder 503 is used to construct enhanced prompt words based on the voice command and vehicle status information;

[0123] The vehicle status cloud synchronization bus 504 is used to upload the constructed enhanced prompt words to the cloud server;

[0124] The cloud server 505 is used to infer the enhanced prompt words using an LLM model and generate vehicle control commands.

[0125] The execution module 506 is used for the vehicle to receive and execute the vehicle control commands.

[0126] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a vehicle interaction method that integrates real-time vehicle status, the method including:

[0127] Receive user voice commands;

[0128] Retrieve vehicle status information related to the voice command; the vehicle status information includes the vehicle's current status information and long-term status information.

[0129] The voice commands and vehicle status information are uploaded to the cloud server so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands.

[0130] The vehicle receives and executes the vehicle control commands.

[0131] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the vehicle interaction method integrating real-time vehicle status provided by the above methods, the method comprising:

[0133] Receive user voice commands;

[0134] Retrieve vehicle status information related to the voice command; the vehicle status information includes the vehicle's current status information and long-term status information.

[0135] The voice commands and vehicle status information are uploaded to the cloud server so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands.

[0136] The vehicle receives and executes the vehicle control commands.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a vehicle interaction method for integrating real-time vehicle status provided by the methods described above, the method comprising:

[0138] Receive user voice commands;

[0139] Retrieve vehicle status information related to the voice command; the vehicle status information includes the vehicle's current status information and long-term status information.

[0140] The voice commands and vehicle status information are uploaded to the cloud server so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands.

[0141] The vehicle receives and executes the vehicle control commands.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle interaction method integrating real-time in-vehicle status, characterized in that, include: Receive user voice commands; Retrieve vehicle status information related to the voice command; The vehicle status information includes the vehicle's current status information and long-term status information; The voice commands and vehicle status information are uploaded to the cloud server so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands. The vehicle receives and executes the vehicle control commands.

2. The vehicle interaction method integrating real-time vehicle status according to claim 1, characterized in that, The retrieval of vehicle status information related to the voice command includes: Convert the voice command into ASR text data; Based on the ASR text, select vehicle current status information related to the ASR text data from the entire set of vehicle current operating statuses, and select long-term status information related to the ASR text data from the entire set of vehicle operating statuses. The vehicle's current status information includes: standardized vehicle attributes, media playback status, door and window status, air conditioning status, foreground and background application status, and audio focus status.

3. The vehicle interaction method integrating real-time vehicle status according to claim 2, characterized in that, The step of using an LLM model to reason about the voice commands and vehicle status information to generate vehicle control commands includes: Based on the ASR text and the vehicle status information, an enhanced prompt word is constructed; The LLM model performs inference on the enhanced prompt words and obtains the inference results stored in JSON format; The reasoning result is parsed, and the vehicle control command is generated based on the reasoning result.

4. The vehicle interaction method integrating real-time vehicle status according to claim 1, characterized in that, The collection of the long-term status information includes: Monitor key vehicle events; these key events include: navigation events, status threshold events, time / date events, driving mode change events, and user habit events. Collect contextual data related to the key events; The context data is uploaded to the cloud server as the long-term state information.

5. The vehicle interaction method integrating real-time vehicle status according to claim 4, characterized in that, The navigation events include: the navigation start point, the destination, the waypoints, and the remaining mileage; The state threshold events include: the vehicle entering a low battery or low fuel state. The time / date events mentioned include: entering the morning rush hour on weekdays, or long-distance travel time during holidays; The driving mode change event includes: switching to Sport mode or Off-road mode; The user habit events include events corresponding to a user habitually performing a certain operation at a specific time / place.

6. The vehicle interaction method for integrating real-time vehicle status according to any one of claims 1-5, characterized in that, The vehicle receiving and executing the vehicle control commands includes: Based on the vehicle control command, the vehicle's local API is invoked; Execute the API and return the result to the user.

7. A vehicle interaction device integrating real-time in-vehicle status, characterized in that, include: The receiving unit is used to receive the user's voice commands; The retrieval unit is used to retrieve vehicle status information related to the voice command; The vehicle status information includes the vehicle's current status information and long-term status information; The uploading unit is used to upload the voice commands and vehicle status information to the cloud server, so that the cloud server can use the LLM model to reason about the voice commands and vehicle status information and generate vehicle control commands. An execution unit is used to receive and execute the vehicle control commands.

8. A vehicle interaction system integrating real-time in-vehicle status, characterized in that, include: The receiving module is used to receive the user's voice commands; A local vehicle status collector is used to collect vehicle status information related to the voice command; The vehicle status information includes the vehicle's current status information and long-term status information; A dynamic prompt word builder is used to construct enhanced prompt words based on the voice commands and vehicle status information; The vehicle status cloud synchronization bus is used to upload the constructed enhanced prompt words to the cloud server; A cloud server is used to infer the enhanced prompts using an LLM model and generate vehicle control commands. An execution module is used for the vehicle to receive and execute the vehicle control commands.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the vehicle interaction method that integrates real-time vehicle status as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the vehicle interaction method that integrates real-time vehicle status as described in any one of claims 1 to 6.

Images