A method, system and device for recognizing potential intent in an automotive cabin

Abstract

This invention discloses a method for identifying latent intentions in a car cockpit. S1. Constructing a car cockpit intention knowledge graph and configuring a dedicated set of scene features for each type of intention: Sorting out user interaction intentions across all scenarios in the intelligent cockpit, including function control, fault alarm, and travel service intentions; defining the relationships between intentions in the knowledge graph, categorized as positive or negative correlations; establishing a scene feature set for each intention, with feature dimensions including vehicle driving characteristics, vehicle body state characteristics, driver and passenger physiological characteristics, and environmental characteristics. This method overcomes the limitations of existing technologies that can only respond to explicit user commands. It mines latent needs not actively expressed by users through knowledge graph association reasoning, providing scenario-based pre-emptive intelligent services, significantly improving the driving experience. By combining multi-dimensional perception data of vehicle operation, drivers and passengers, and the environment with the intention feature set for quantitative matching, it avoids misjudgments caused by relying solely on semantic reasoning and accurately distinguishes true latent intentions in different scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit human-computer interaction technology, specifically to a method, system, and device for identifying potential intentions in an automotive cockpit. Background Technology

[0002] Intelligent vehicles have become a key direction for promoting industrial upgrading and enhancing product competitiveness. From the perspective of application scenarios, intelligent vehicles mainly cover two major areas: intelligent driving and intelligent cockpit. Intelligent driving focuses on the construction of advanced driver assistance and autonomous driving capabilities, aiming to improve vehicle driving safety, reduce driving burden, and optimize travel efficiency. Intelligent cockpit, on the other hand, focuses on core aspects such as cockpit control, human-machine interaction, and passenger experience, improving the comfort, convenience, and overall experience of vehicle use through intelligent functions and scenario-based services.

[0003] Current cockpit intent recognition technologies are mainly divided into two categories: Based on a fixed instruction dictionary + traditional NLP recognition scheme: It can only match fixed speech to execute corresponding functions. It has high recognition accuracy and fast processing speed, but low recall rate and frequent false triggers. It only passively responds to instructions and cannot uncover hidden needs, resulting in low intelligence. The semantic recognition scheme based on fine-tuning large models has better semantic recall and recognition accuracy than traditional NLP schemes, but it is still limited to parsing explicit user commands. It does not combine the real-time operating status of the vehicle, the physiological state of the driver and passengers, and environmental information to make association inferences. It can only passively execute explicit user commands and cannot actively explore potential contextual intentions, resulting in obvious shortcomings in the interactive experience.

[0004] In summary, existing technologies can only analyze users' explicit intention graphs and lack a complete technical solution that combines multi-dimensional data on vehicles, people, and the environment to infer potential intentions, thus failing to achieve proactive intelligent services in the cockpit. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and apparatus for identifying potential intentions in a car cabin, in order to solve the problems mentioned in the background art.

[0006] This invention provides the following technical solution: a method for identifying potential intentions in a car cabin, characterized by comprising the following steps: S1. Construct a knowledge graph of car cockpit intents, define the positive and negative correlations between various user intents, and construct a corresponding set of scene features for each type of intent; S2. Train a concept graph recognition model based on cockpit interaction samples; S3. Multi-dimensional perception data is collected synchronously through a hardware terminal connected to the vehicle's infotainment system. The multi-dimensional perception data includes vehicle status data, driving dynamic data, driver and passenger status data, and external environment data. S4. Receive user input instructions, identify the user's idea image through the idea image recognition model, and control the vehicle system to execute the cockpit function corresponding to the idea image; S5. Using the idea graph as the retrieval condition, query the intent knowledge graph to extract all candidate potential intents that are positively related to the idea graph; S6. Match the multi-dimensional perception data collected in step S3 with the scene feature set of each candidate potential intent, calculate the feature matching degree, and filter the candidate potential intents with matching degree exceeding the preset threshold as valid potential intents. S7. Perform differentiated processing on the effective potential intentions according to the preset classification strategy: comfort-related potential intentions initiate a confirmation interaction with the user and are executed after receiving the confirmation instruction; safety monitoring-related potential intentions are automatically detected in the background, and an alarm is pushed to the user when an abnormality is detected. S8. Collect user feedback information after the execution of the idea graph and effective potential intents, and evaluate whether to iteratively search for deeper-level related potential intents based on the feedback information to form a closed-loop identification system.

[0007] A system for recognizing latent intentions in a car cabin, for performing the recognition method of claim 1, comprising: a. The knowledge graph and feature library construction module is used to construct a knowledge graph of car cockpit intents and store the scene feature set corresponding to each intent; among them, it is used to construct the cockpit intent knowledge graph, define the positive and negative correlation relationships of intents, construct a scene feature set for each type of intent and store it in the database.

[0008] b. Multimodal perception and acquisition module, used to collect multi-dimensional raw perception data of vehicle, driving, passengers, and environment; including vehicle-machine data interaction unit, positioning and attitude sensing unit, occupant perception unit, and environment sensing unit, responsible for synchronously collecting multi-dimensional raw data of vehicle, driving, passengers, and environment.

[0009] c. Intent Recognition and Reasoning Module: It has a built-in pre-trained main intent recognition model and a knowledge graph reasoning engine, which are used to parse the user's intent graph and match and filter effective potential intents. The pre-trained main intent recognition model is used to parse the user's explicit instructions. The built-in knowledge graph reasoning engine retrieves candidate potential intents based on the intent graph and filters effective potential intents by combining the matching degree of perceptual data features.

[0010] d. Intelligent execution and feedback closed-loop module: Stores hierarchical execution strategies for intents, distinguishes between two processing modes: inquiry confirmation and automatic background execution, and collects user feedback which is fed back to the intent recognition and reasoning module for iterative reasoning logic; Specifically, it configures a hierarchical execution strategy library for intents, distinguishes between inquiry confirmation and automatic background execution; collects user interaction feedback, and feeds the feedback data back to the intent recognition and reasoning module to achieve iterative mining of potential intents.

[0011] A device for recognizing potential intentions in a car cabin includes a domain controller body, a multi-source sensing hardware assembly, a human-machine interaction output unit, and a storage unit. The domain controller itself is equipped with the aforementioned identification system; The multi-source sensing hardware group includes a satellite positioning module, an attitude sensor, an on-board microphone, a millimeter-wave radar, an in-vehicle camera, a CAN bus interaction interface, and an ambient temperature and light sensor. The human-machine interaction output unit includes a central control display screen, a voice broadcast speaker, and instrument alarm indicator lights; The storage unit is used to store the intent knowledge graph, scene feature set, intent graph recognition model weights, and user interaction feedback data.

[0012] As an optimization, the scene feature set includes vehicle driving characteristics, vehicle body fault characteristics, driver and passenger physiological characteristics, and temperature and light environment characteristics.

[0013] Compared with the prior art, the beneficial effects of the present invention are: The method, system, and device for identifying potential intentions in the car cockpit upgrades intelligent services in the car cockpit from passively executing commands to proactively uncovering potential important needs, providing car users with more professional and thoughtful services. For example, when a user requests to "lower the air conditioning temperature," the system can identify a potential need to "close the sunshade" based on the intent indication graph, thus improving the user's driving experience. By matching information collected by the hardware terminal with the intent feature set and calculating the matching degree, the accuracy of potential intent recognition is improved. This is because potential intents cannot be directly derived from user commands but depend on the current driving conditions. For instance, when a user requests to "lower the air conditioning temperature," possible potential needs include playing upbeat music or checking tire pressure. Based on the information collected by the hardware terminal, the user is not currently fatigued, but the vehicle is experiencing significant bumps, and the outside temperature is high. Based on the intent feature database, the system can accurately identify the need for a "tire pressure check." With the help of the intelligent analysis and execution module for "potential needs," after identifying a potential need, the system can automatically determine whether to execute it directly or ask the user for confirmation, greatly simplifying the user's interaction and demonstrating the value of intelligent proactive service. For example, after identifying the potential need to "close the sunshade," the system will proactively confirm with the user before proceeding; after identifying the potential need to "check tire pressure," the system will automatically execute it in the background and remind the user if there are any abnormalities in the feedback information. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the overall business process of the method for identifying potential intentions in a car cabin according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 The present invention provides a technical solution: a method for identifying potential intentions in a car cabin. S1. Construct a knowledge graph of automotive cockpit intents and configure a unique set of scene features for each type of intent: The system analyzes user interaction intents across all scenarios in the intelligent cockpit, including intents related to function control, fault alarms, and travel services. It defines the relationships between these intents in a knowledge graph, categorizing them as positive or negative correlations. For each intent, a set of scenario features is established, with feature dimensions including vehicle driving characteristics, vehicle body status characteristics, physiological characteristics of drivers and passengers, and environmental characteristics. Example: The feature set for the intent "Tire Pressure Check" includes features such as severe vehicle vibration, vehicle veering, sluggish steering, increased braking distance, and high ambient temperature.

[0017] S2. Training the idea graph recognition model: We collected a large number of user voice and touch interaction samples from the cockpit, labeled the explicit idea graphs of the samples, and used natural language processing / fine-tuning of a large model to train an idea graph recognition model, which is used to parse the explicit needs corresponding to the user's direct input commands.

[0018] S3. Collect multi-dimensional sensing data: Multi-dimensional data is collected synchronously through hardware terminals connected to the vehicle's infotainment system: Vehicle-level data: Vehicle status data such as vehicle malfunctions, air conditioning, braking, and steering output by the infotainment system; Driving dynamic data: Driving data such as vehicle speed, road bumps, driving trajectory, and tilt angle output by satellite positioning module and attitude sensor; Passenger status data: User voice, fatigue status, body movements, and facial expressions collected by microphone, millimeter-wave radar, and vehicle camera; External environment data: Perception data of the external environment such as temperature, light intensity, and rainfall.

[0019] S4. Identify and execute the user idea graph: The user inputs commands into the idea graph recognition model, which outputs the idea graph corresponding to the current interaction. The vehicle system then directly executes the cockpit function corresponding to that idea graph.

[0020] S5. Matching candidate latent intents based on intent knowledge graph: Using the identified idea graph as the search keyword, query the intent knowledge graph and extract the set of all candidate potential intents that are positively correlated with the idea graph.

[0021] S6. Feature matching calculates the matching degree to determine the valid latent intent: The multi-dimensional perception data collected in step S3 is matched with the scene feature set corresponding to each candidate potential intent, and the feature matching degree is quantitatively calculated. Candidate intents with matching degrees higher than a preset threshold are selected as the real and effective potential intents in the current scene.

[0022] S7. Perform differentiated processing logic on valid potential intentions: The system has a built-in intent-based hierarchical strategy that distinguishes between two processing modes: Confirmation-based interaction: For potential intents related to comfort adjustments (such as closing the roof sunshade or adjusting the passenger-side air conditioning), a pop-up window / voice prompt is made to the user, and the instruction is executed after confirmation; Background automatic execution: For potential intents related to safety monitoring (such as tire pressure monitoring or brake status self-check), the detection is performed silently in the background, and an alarm is pushed to the user if an abnormality is detected.

[0023] S8. Collect interactive feedback and iteratively mine potential intentions: Record user feedback (confirmation, rejection, no feedback, abnormal alarm) after the execution of the main graph and potential intent. Send the feedback data back to the knowledge graph reasoning module to determine whether it is necessary to continue iterating to retrieve deeper-level related potential intents and complete the closed-loop iteration.

[0024] Typical Implementation Case: The user issues the command "lower the air conditioning temperature"; the idea graph recognition model identifies the idea graph as air conditioning cooling and executes it; the knowledge graph is searched to obtain candidate potential intents: close the sunshade, check tire pressure, play soothing music; hardware perception data shows: high outdoor temperature, continuous bumpy road surface, and no user fatigue characteristics; the three potential intent feature sets are matched respectively, with tire pressure check having the highest matching degree, followed by closing the sunshade; the background automatically performs a tire pressure self-check and simultaneously asks the user via voice whether to close the roof sunshade; feedback is recorded based on the user's selection result to determine whether to continue to explore other related needs.

[0025] A system for recognizing latent intentions in a car cabin, used to implement the aforementioned recognition method, comprises four core modules: a. Atlas and Feature Library Construction Module: This is used to build a cockpit intent knowledge graph, define positive and negative correlations of intents, and build a set of scene features for each type of intent and store them in the database. b. Multimodal sensing and acquisition module: It includes a vehicle-to-everything (V2X) data interaction unit, a positioning and attitude sensing unit, an occupant perception unit, and an environmental sensing unit, and is responsible for synchronously collecting multi-dimensional raw data on the vehicle, driving, passengers, and environment. c. Intent Recognition and Reasoning Module: It has a built-in pre-trained main intent recognition model for parsing explicit user commands; and a built-in knowledge graph reasoning engine for retrieving candidate potential intents based on the main intent graph and filtering effective potential intents by combining the matching degree of perceptual data features. d. Intelligent execution and feedback closed-loop module: Configure an intent-based hierarchical execution strategy library to distinguish between two processing logics: inquiry confirmation and automatic background execution; collect user interaction feedback and feed the feedback data back to the intent recognition and reasoning module to achieve iterative mining of potential intents.

[0026] A device for recognizing potential intentions in a car cockpit, integrated into an in-vehicle intelligent cockpit domain controller, comprises the following hardware components: The domain controller itself contains the four software modules of the aforementioned identification system, providing computing power support; Multi-source sensing hardware group: satellite positioning module, six-axis attitude sensor, vehicle microphone, millimeter-wave human body radar, in-vehicle camera, vehicle CAN bus interaction interface, temperature, light and rain environmental sensor; Human-machine interaction output unit: vehicle central control display screen, cabin voice broadcast speaker, instrument warning indicator; Storage unit: Used to store intent knowledge graph, feature sets of each intent scenario, weights of intent graph recognition model, and historical data of user interaction feedback.

[0027] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying latent intentions in a car cabin, characterized in that, Includes the following steps: S1. Construct a knowledge graph of car cockpit intents, define the positive and negative correlations between various user intents, and construct a corresponding set of scene features for each type of intent; S2. Train a concept graph recognition model based on cockpit interaction samples; S3. Multi-dimensional perception data is collected synchronously through a hardware terminal connected to the vehicle's infotainment system. The multi-dimensional perception data includes vehicle status data, driving dynamic data, driver and passenger status data, and external environment data. S4. Receive user input instructions, identify the user's idea image through the idea image recognition model, and control the vehicle system to execute the cockpit function corresponding to the idea image; S5. Using the idea graph as the retrieval condition, query the intent knowledge graph to extract all candidate potential intents that are positively related to the idea graph; S6. Match the multi-dimensional perception data collected in step S3 with the scene feature set of each candidate potential intent, calculate the feature matching degree, and filter the candidate potential intents with matching degree exceeding the preset threshold as valid potential intents. S7. Perform differentiated processing on the effective potential intentions according to the preset classification strategy: comfort-related potential intentions initiate a confirmation interaction with the user and are executed after receiving the confirmation instruction; safety monitoring-related potential intentions are automatically detected in the background, and an alarm is pushed to the user when an abnormality is detected. S8. Collect user feedback information after the execution of the idea graph and effective potential intents, and evaluate whether to iteratively search for deeper-level related potential intents based on the feedback information to form a closed-loop identification system.

2. A system for recognizing latent intentions in a car cabin, characterized in that, For performing the identification method of claim 1, comprising: a. Knowledge graph and feature library construction module, used to build a knowledge graph of automotive cockpit intents and store the scene feature set corresponding to each intent; b. Multimodal perception and acquisition module, used to collect raw perception data of multiple dimensions of vehicle, driving, passengers and environment; c. Intent recognition and reasoning module, with a built-in trained main intent recognition model and knowledge graph reasoning engine, used to parse user intent graphs and match and filter effective potential intents; d. The intelligent execution and feedback closed-loop module stores the hierarchical execution strategy of intent, distinguishes between two processing modes: inquiry confirmation and automatic background execution, and collects user feedback and feeds it back to the intent recognition and reasoning module to iterate the reasoning logic.

3. A device for identifying latent intentions in a car cabin, characterized in that, Includes the domain controller itself, multi-source sensing hardware group, human-machine interaction output unit, and storage unit: The domain controller body is equipped with the identification system described in claim 2; The multi-source sensing hardware group includes a satellite positioning module, an attitude sensor, an on-board microphone, a millimeter-wave radar, an in-vehicle camera, a CAN bus interaction interface, and an ambient temperature and light sensor. The human-machine interaction output unit includes a central control display screen, a voice broadcast speaker, and instrument alarm indicator lights; The storage unit is used to store the intent knowledge graph, scene feature set, intent graph recognition model weights, and user interaction feedback data.

4. The method, system, and apparatus for identifying latent intentions in a car cabin according to claim 1, characterized in that: The scene feature set includes vehicle driving characteristics, vehicle body fault characteristics, physiological characteristics of drivers and passengers, and temperature and light environment characteristics.

Images