A Visual Semantic Understanding-Based AI-Powered Smart Cockpit Voice-Gesture Multimodal Collaborative Interaction Method

By synchronously collecting and analyzing voice, gesture, and environmental data in the smart cockpit, and using a causal inference engine to construct a multimodal causal graph, the problem of command conflict in multimodal interaction is solved, achieving high-accuracy command processing and improved security.

CN123114408APending Publication Date: 2026-11-17BEIJING E CREDENCE INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611259376.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing multimodal interaction solutions lack a deep understanding of the inherent logical relationship between voice and gesture commands, making it difficult for the system to effectively handle potential conflicts, affecting the reliability of interaction and user experience, and posing safety risks, especially in driving scenarios.

Method used

By synchronously collecting voice, gesture, and environmental data through the in-vehicle sensing system, constructing a multimodal causal graph using visual semantic understanding and a causal reasoning engine, quantifying and evaluating the causal strength between commands, dynamically fusing commands and providing feedback, the system has continuous learning capabilities.

Benefits of technology

It significantly improves the accuracy of command conflict recognition to over 94%, reduces the error rate, enhances user experience and driving safety, and the system can adapt to the interaction habits of different users and continuously optimize.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114408A_ABST
    Figure CN123114408A_ABST
Patent Text Reader

Abstract

This invention relates to the field of automotive human-machine interaction technology and discloses an AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding. The method includes the following steps: synchronously collecting user voice, gesture, and vehicle environment data through in-vehicle sensors; performing semantic parsing to extract structured instructions; inputting the parsing results into a pre-trained causal inference engine, which, based on a multimodal causal graph constructed using a causal discovery algorithm, calculates and evaluates the causal correlation strength between voice and gesture instructions to scientifically distinguish between true causal correlations and spurious correlations; dynamically fusing multimodal instructions based on the causal correlation strength to generate final control instructions; and executing corresponding cockpit device control operations according to the final control instructions, providing multimodal interactive feedback. This invention effectively solves the conflict problem between multimodal instructions by introducing a causal inference mechanism, significantly improving the accuracy and fluency of interaction, and enhancing user experience and driving safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicles and human-computer interaction technology, specifically to an AI-based intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding. Background Technology

[0002] With the rapid development of artificial intelligence technology, smart cockpits are gradually becoming the "second living space" of automobiles. To improve the naturalness and efficiency of interaction, collaborative interaction technologies that integrate multiple modalities such as voice, gestures, and vision have become a research hotspot. Users expect to interact with vehicles using a combination of language and body language, just like conversing with a person.

[0003] However, existing multimodal interaction solutions have significant drawbacks. Most solutions use simple temporal matching or preset weights for command fusion, lacking a deep understanding of the inherent logical relationships between commands. This makes it difficult for the system to effectively handle potential conflicts between voice and gesture commands. For example, when a user says "open the window" but makes a gesture similar to "close" while wiping the glass, existing systems may incorrectly associate the two commands and execute an incorrect operation, or require the user to repeat the command due to inability to judge, thus disrupting the smoothness of the interaction and even introducing safety risks in driving scenarios. Therefore, this paper proposes an AI intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding to solve the above-mentioned problems. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding. This method has the advantages of intelligently understanding the user's true intentions and accurately determining the relationships between multimodal commands, thereby improving interaction reliability and user experience. It solves the significant defects of existing multimodal interaction solutions, most of which use simple temporal matching or preset weights for command fusion, lacking a deep understanding of the inherent logical relationships between commands. This makes it difficult for the system to effectively handle potential conflicts between voice and gesture commands.

[0005] (II) Technical Solution To achieve the aforementioned goal of intelligently understanding the user's true intentions and accurately determining the relationships between multimodal commands, this invention provides the following technical solution: an AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding, characterized by the following steps: Step 1: Synchronously collect user voice data, gesture image data, and vehicle environmental status data through the in-vehicle sensing system; Step 2: Perform semantic parsing on the voice data and gesture image data respectively to extract voice command triples and gesture spatial semantic information; Step 3: Input the parsed voice commands, gesture commands, and environmental state data into the pre-trained causal inference engine. The causal inference engine is based on a multimodal causal graph constructed by the causal discovery algorithm to calculate and evaluate the causal correlation strength between voice commands and gesture commands in order to distinguish between real causal correlations and pseudo-correlation. Step 4: Based on the strength of the causal relationship, dynamically fuse voice commands and gesture commands to generate the final control command; Step 5: Execute the corresponding cockpit equipment control operations according to the final control command, and provide multimodal interactive feedback.

[0006] Preferably, the vehicle environmental status data in step S1 includes: (1) Voice Data: Data acquisition equipment: In-vehicle microphone array (multi-microphone design) Data format: 16kHz sampling rate, 16-bit depth, mono audio stream Data collection area: Covering the area where the driver and front passenger are conversing. (2) Gesture Image Data Data acquisition equipment: In-vehicle cameras, including front-facing or rear-facing cameras. Data format: 1080p resolution, 30 frames per second, RGB format Data collection area: Driver's face and hands (3) Vehicle environmental state data Sensor types: Speed ​​sensor: real-time vehicle speed (0-250km / h), Weather sensor: rain and snow detection (rainfall, snowfall), GPS module: geographic location (latitude, longitude, altitude), Door sensor: door opening and closing status, Passenger sensor: detects the number of occupants in the vehicle via infrared or pressure sensors.

[0007] Preferably, the extraction of gesture spatial semantic information in step S2 specifically includes: Step 21: Apply the improved YOLOv7 model to detect 21 key gesture nodes in real time and extract the three-dimensional spatial coordinates of multiple joints of the hand in real time; Step 22: Based on the CLIP model (multimodal pre-training), the key point vectors are mapped to gesture semantic labels, and the gesture semantic understanding module is constructed to classify the spatial vector sequence of key points into predefined gesture intention types; Step 23: Analyze the temporal changes of hand key points in consecutive frames using an attention LSTM network to identify dynamic gesture trajectories and their parameters.

[0008] Preferably, the extraction of the voice command triplet in step S2 specifically includes: A robust speech recognition model was used to perform speech-to-text conversion in a noisy vehicle environment. Using an intent parsing model based on the Transformer architecture, structured triples containing the object of the operation, the action type, and the parameter value are extracted from the text.

[0009] Preferably, the construction of the causal reasoning engine in step S3 specifically includes the following steps: Step 31: Collect historical multimodal interaction data to form a training dataset; Step 32: Apply the PC algorithm to perform conditional independence tests on the training dataset and construct a partially directed acyclic graph describing the causal relationships between multimodal instructions; Step 33: Calculate the causal strength weights of each edge in the graph based on the Fisher-Z test to form a multimodal causal graph that can be quantified and queried.

[0010] Preferably, the step S3 of distinguishing between true causal relationships and spurious correlations specifically includes: For real-time input voice commands and gesture commands, query the causal path and strength between their corresponding nodes in the multimodal causal graph; If the causal strength exceeds the first preset threshold, it is determined to be a true causal relationship, and the instructions should be executed accordingly. If the causal strength is lower than the second preset threshold and there is an indirect path via environmental state variables, it is determined to be a spurious correlation, and one of the instructions is ignored or reconfirmed.

[0011] Preferably, the dynamic fusion in step S4 specifically adopts the following decision formula: Final instruction = arg max [α·P (语音|因果图) +β·P (手势|因果图) +γ·P (环境|因果图) ] Among them, α, β, and γ are dynamic weighting coefficients, whose values ​​are adjusted in real time by the causal correlation strength calculated in step S3.

[0012] Preferably, the multimodal interaction feedback in step S5 includes: Visual feedback: The device object to be controlled is highlighted on the cockpit center console screen, and the control effect is previewed; Voice feedback: The system reads a summary of the operation to be performed via a TTS engine and provides clarification questions when command conflicts are detected.

[0013] Preferably, the system also includes learning optimization, specifically comprising the following steps: Record the user's implicit feedback behavior regarding the execution results; Based on the implicit feedback behavior, an incremental causal discovery algorithm is used to update the structure and parameters of the multimodal causal graph online.

[0014] (III) Beneficial Effects Compared with existing technologies, this invention provides an AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding, which has the following beneficial effects: 1. This AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding collects and analyzes multimodal data, simultaneously collecting and analyzing voice, gesture, and environmental data, transforming them into structured semantic information; it utilizes pre-trained multimodal causal graphs to quantitatively evaluate the causal strength between commands, scientifically distinguishing between genuine intent collaboration and accidental coincidence; it dynamically adjusts the fusion strategy based on causal strength, outputs and executes the final command, and provides clear interactive feedback; furthermore, the system can update the causal model online based on the user's subsequent behavior, achieving personalized evolution.

[0015] 2. This AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding fundamentally solves the problem of pseudo-correlation between commands through causal reasoning, improving the recognition accuracy of conflicting commands to over 94%; it significantly reduces the rate of misoperation and interaction interruption, eliminating the need for repeated confirmations and providing a smoother user experience; it can reliably understand user commands in complex driving environments, reducing distraction caused by interaction errors and improving driving safety; and the system has continuous learning capabilities, adapting to the interaction habits of different users and becoming increasingly intelligent with use. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the AI ​​intelligent cockpit voice-gesture multimodal collaborative interaction method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 A multimodal collaborative interaction method for voice and gesture in an AI-powered smart cockpit based on visual semantic understanding includes the following steps: Step 1: Synchronously collect user voice data, gesture image data, and vehicle environmental status data through the in-vehicle sensing system; Step 2: Perform semantic parsing on the speech data and gesture image data respectively, and extract speech command triples and gesture spatial semantic information; Step 3: Input the parsed voice commands, gesture commands, and environmental state data into the pre-trained causal inference engine. The causal inference engine, based on the multimodal causal graph constructed by the causal discovery algorithm, calculates and evaluates the causal correlation strength between voice commands and gesture commands to distinguish between true causal correlations and false correlations. Step 4: Based on the strength of causal correlation, dynamically fuse voice commands and gesture commands to generate the final control command; Step 5: Execute the corresponding cockpit equipment control operations according to the final control command and provide multimodal interactive feedback.

[0019] The vehicle environmental status data in step S1 includes: (1) Voice Data: Data acquisition equipment: In-vehicle microphone array (multi-microphone design) Data format: 16kHz sampling rate, 16-bit depth, mono audio stream Data collection area: Covering the area where the driver and front passenger are conversing. (2) Gesture Image Data Data acquisition equipment: In-vehicle cameras, including front-facing or rear-facing cameras. Data format: 1080p resolution, 30 frames per second, RGB format Data collection area: Driver's face and hands (3) Vehicle environmental state data Sensor types: Speed ​​sensor: real-time vehicle speed (0-250km / h), Weather sensor: rain and snow detection (rainfall, snowfall), GPS module: geographic location (latitude, longitude, altitude), Door sensor: door opening and closing status, Passenger sensor: detects the number of occupants in the vehicle via infrared or pressure sensors.

[0020] Specifically: Data synchronization mechanism Timestamp alignment: All sensor data is synchronized via a hardware clock to ensure timestamp accuracy ≤1ms. Data buffer: A circular buffer (FIFO queue) is used to store the data from the most recent second. Data fusion: Real-time acquisition and merging of data streams through a multi-threaded mechanism to ensure the temporal consistency of multimodal data. Data transmission protocol Data format: Binary serialization (such as Protocol Buffers) Transmission method: via vehicle bus (such as CAN Bus) Data verification: CRC check ensures data integrity. Data storage: Temporary storage: Data from the last 10 seconds is cached in memory. Persistent storage: In case of abnormal situations, data is written to a local log file (encrypted storage). Data processing flow Data acquisition: Parallel acquisition by multiple sensors Data Preprocessing: Real-time Filtering and Enhancement Data Synchronization: Timestamp Alignment and Buffer Management Data transmission: Multiplexed transmission protocol Data parsing: Input to the multimodal parsing engine Step S2, which extracts gesture spatial semantic information, specifically includes: Step 21: Apply the improved YOLOv7 model to detect 21 key gesture nodes in real time and extract the three-dimensional spatial coordinates of multiple joints of the hand in real time; Step 22: Based on the CLIP model (multimodal pre-training), the key point vectors are mapped to gesture semantic labels, and the gesture semantic understanding module is constructed to classify the spatial vector sequence of key points into predefined gesture intention types; Step 23: Analyze the temporal changes of hand key points in consecutive frames using an attention LSTM network to identify dynamic gesture trajectories and their parameters.

[0021] Step S2, which extracts the voice command triplet, specifically includes: A robust speech recognition model was used to perform speech-to-text conversion in a noisy vehicle environment. Using an intent parsing model based on the Transformer architecture, structured triples containing the operation object, action type, and parameter value are extracted from the text. The speech triples are then fused with gesture semantic labels to generate a preliminary instruction structure such as "increase volume" + "rotate clockwise".

[0022] The specific steps are as follows: Gesture key point detection module (YOLOv7 improvement) 21-point hand keypoint detection head: Adds a keypoint regression branch to the original YOLOv7 detection head; 3D coordinate reconstruction: Combines binocular vision or depth camera information to map 2D keypoints to 3D space; Real-time optimization: Uses TensorRT acceleration to ensure detection speed >30fps; 3D coordinate transformation: P 3d =[X,Y,Z]=K inv ×[u,v,1] ᵀ ×d Where K inv Let d be the inverse matrix of the camera intrinsic parameters, and d be the depth value. Gesture semantic understanding module (CLIP-based) Joint vectorization: flattening 21 3D coordinates into a 63-dimensional vector; Temporal encoding: adding position encoding to capture temporal relationships; Multimodal alignment: using CLIP pre-trained weights for cross-modal mapping; Trajectory parameterization: Bézier curve fitting: extracting control points of the trajectory; motion features: parameters such as velocity, acceleration, and curvature; Voice command triple extraction Triplet extraction process Input text preprocessing: Tokenization: Use a BERT tokenizer (such as bert-base-chinese) to segment the input text into sub-word units (such as "open car window" → "open", "car", "window").

[0023] Special markers: Add [CLS] and [SEP] markers to both ends of the text for model context recognition.

[0024] BERT Encoding: Vectorization: The word segmentation results are mapped to the input IDs (integer sequences) of the BERT model, and corresponding attention masks are generated (ignoring padding characters).

[0025] Embedding layer: The embedding layer of the BERT model converts the input ID into word vectors of fixed dimensions (such as 768 dimensions).

[0026] Sequence labeling (BIEO): Labeling rules: Each sub-word unit is labeled using the BIEO labeling system (Begin-Inside-End-Other) (e.g., "open" → B-verb, "car" → I-noun).

[0027] Model output: The BERT model outputs the label probability distribution for each word unit (e.g., [0.1, 0.8, 0.1] represents the probability of a B-verb).

[0028] Entity recognition Threshold filtering: The label probability of each sub-word unit is filtered by a threshold (e.g., 0.5) to retain the entity boundaries with high confidence (e.g., "open" → B-verb).

[0029] Entity merging: Merge consecutive entity units using BIEO rules (e.g., "Open the window" → "Open the window").

[0030] Entity classification: The identified entities are divided into predefined categories (such as actions, objects, parameters).

[0031] Relationship Extraction Dependency parsing: Apply BERT-CRF models (such as BertCRF in the transformers library) to parse the dependency relationships between entities (such as the subject-verb relationship between "open" and "car window").

[0032] Relational template matching: Extract structured relations (e.g., "open" → "car window") based on predefined relational templates (e.g., "verb + noun").

[0033] Structured triple generation Triple format: Combine the identified entities and relations into triples (e.g., {"action":"open","object":"window","value":null}).

[0034] Parameter value parsing: Extract parameter values ​​from the text using regular expressions or a rule engine (such as the re library) (e.g., "open car window 30" → {"value":30}).

[0035] Semantic verification: Verify the semantic consistency of triples in conjunction with context (such as environmental state) (e.g., "open the car window" is invalid on a rainy day).

[0036] Fusion strategy: Feature-level fusion: Alignment of gesture and speech features in the embedding space Decision-level fusion: Processing data separately before decision fusion. Attention weights: dynamically adjust the contribution of each modality Error handling mechanism: Confidence assessment: Each module outputs a confidence score. Conflict Resolution: Decision Logic in the Context of Multimodal Information Conflict Fault tolerance mechanism: Degradation handling in case of single-mode failure The construction of the causal reasoning engine in step S3 specifically includes the following steps: Step 31: Collect historical multimodal interaction data to form a training dataset; Step 32: Apply the PC algorithm to perform conditional independence tests on the training dataset and construct a partially directed acyclic graph describing the causal relationships between multimodal instructions; Step 33: Calculate the causal strength weights of each edge in the graph based on the Fisher-Z test to form a quantifiable and queryable multimodal causal graph; For real-time input voice commands and gesture commands, query the causal paths and strengths between their corresponding nodes in the multimodal causal graph; If the causal strength exceeds the first preset threshold, it is determined to be a true causal relationship, and the instructions should be executed accordingly. If the causal strength is lower than the second preset threshold and there is an indirect path via environmental state variables, it is determined to be a spurious correlation, and one of the instructions is ignored or reconfirmed.

[0037] The dynamic fusion in step S4 specifically adopts the following decision formula: Final instruction = arg max [α·P (语音|因果图) +β·P (手势|因果图) +γ·P (环境|因果图) ] Among them, α, β, and γ are dynamic weighting coefficients, whose values ​​are adjusted in real time by the causal correlation strength calculated in step S3.

[0038] The multimodal interaction feedback in step S5 includes: Visual feedback: The device object to be controlled is highlighted on the cockpit center console screen, and the control effect is previewed; Voice feedback: The system reads a summary of the operation to be performed via a TTS engine and provides clarification questions when command conflicts are detected.

[0039] It also includes system learning optimization, specifically including the following steps: Record the user's implicit feedback behavior regarding the execution results; Based on implicit feedback behavior, an incremental causal discovery algorithm is used to update the structure and parameters of the multimodal causal graph online.

[0040] Example Imagine a driving scenario in the rain. The driver says, "It's a bit stuffy," while making a rotating gesture with his hand.

[0041] Data acquisition: The system simultaneously acquires the voice signal, hand image, and environmental data (e.g., the rain sensor detects moderate rain and the vehicle speed is 60km / h).

[0042] Semantic parsing: The speech parsing module outputs a triple: {"State description": "Hot", "Object": "None", "Action": "None"}. This is a state description, not a direct operation command.

[0043] The gesture analysis module recognized the gesture as an intention to "rotate and adjust" and detected that it was pointing at the virtual air conditioning knob on the center console.

[0044] Causal Reasoning: The causal reasoning engine queries multimodal causal graphs. It found a strong causal path between the description of "sultry" state and the gesture of "adjusting the air conditioning" (for example, users were satisfied after this combination appeared multiple times in historical data), with a causal strength of 0.9 (exceeding the first threshold of 0.8). At the same time, the engine found a strong conflict between the "rainy" environment and the "open the window" command. Therefore, the system determined that the voice and gesture were a real causal relationship, both pointing to the "adjust the air conditioning" operation. Dynamic fusion: According to the decision formula, since voice commands are state descriptions, their direct operation probability P(voice|causal graph) is low, while the operation probability of gestures P(gesture|causal graph) is high. Environmental data also supports this operation. Therefore, the final command is determined to be "lower the air conditioning temperature" or "turn on the air conditioning external circulation". Execution and Feedback: The system executes the air conditioning adjustment operation. The air conditioning icon is highlighted on the central control screen, and an animation of temperature drop appears. The TTS voice broadcast says: "The air conditioning temperature has been lowered for you." Learning and optimization: If the driver does not subsequently cancel the operation or expresses satisfaction, the system will fine-tune the causal strength between "hot and stuffy" and "adjust the air conditioning" to make it more accurate in the future.

[0045] In this embodiment, it can be seen that the present invention can understand indirect speech expressions and accurately infer user intentions by combining context and gestures, avoiding errors that may be caused by simple keyword matching (such as opening a car window in the rain).

[0046] In summary, this AI-powered intelligent cockpit voice-gesture multimodal collaborative interaction method based on visual semantic understanding fundamentally solves the problem of spurious correlation between commands through causal reasoning, improving the recognition accuracy of conflicting commands to over 94%. It significantly reduces the rate of misoperation and interaction interruption, eliminating the need for repeated confirmations and providing a smoother user experience. Furthermore, it reliably understands user commands in complex driving environments, reducing distraction caused by interaction errors and improving driving safety. The system also possesses continuous learning capabilities, adapting to the interaction habits of different users and becoming increasingly intelligent with use.

[0047] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for AI intelligent cockpit voice-gesture multi-modal collaborative interaction based on visual semantic understanding, characterized in that: Includes the following steps: Step 1: Synchronously collect user voice data, gesture image data, and vehicle environmental status data through the in-vehicle sensing system; Step 2: Perform semantic parsing on the voice data and gesture image data respectively to extract voice command triples and gesture spatial semantic information; Step 3: Input the parsed voice commands, gesture commands, and environmental state data into the pre-trained causal inference engine. The causal inference engine is based on a multimodal causal graph constructed by the causal discovery algorithm to calculate and evaluate the causal correlation strength between voice commands and gesture commands in order to distinguish between real causal correlations and pseudo-correlation. Step 4: Based on the strength of the causal relationship, dynamically fuse voice commands and gesture commands to generate the final control command; Step 5: Execute the corresponding cockpit equipment control operations according to the final control command, and provide multimodal interactive feedback.

2. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The vehicle environmental status data in step S1 includes: (1) Voice Data: Data acquisition equipment: In-vehicle microphone array (multi-microphone design) Data format: 16kHz sampling rate, 16-bit depth, mono audio stream Data collection area: Covering the area where the driver and front passenger are conversing. (2) Gesture Image Data Data acquisition equipment: In-vehicle cameras, including front-facing or rear-facing cameras. Data format: 1080p resolution, 30 frames per second, RGB format Data collection area: Driver's face and hands (3) Vehicle environmental state data Sensor types: Speed ​​sensor: real-time vehicle speed (0-250km / h), Weather sensor: rain and snow detection (rainfall, snowfall), GPS module: geographic location (latitude, longitude, altitude), Door sensor: door opening and closing status, Passenger sensor: detects the number of occupants in the vehicle via infrared or pressure sensors.

3. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The extraction of gesture spatial semantic information in step S2 specifically includes: Step 21: Apply the improved YOLOv7 model to detect 21 key gesture nodes in real time and extract the three-dimensional spatial coordinates of multiple joints of the hand in real time; Step 22: Based on the CLIP model (multimodal pre-training), the key point vectors are mapped to gesture semantic labels, and the gesture semantic understanding module is constructed to classify the spatial vector sequence of key points into predefined gesture intention types; Step 23: Analyze the temporal changes of hand key points in consecutive frames using an attention LSTM network to identify dynamic gesture trajectories and their parameters.

4. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The extraction of voice command triples in step S2 specifically includes: A robust speech recognition model was used to perform speech-to-text conversion in a noisy vehicle environment. Using an intent parsing model based on the Transformer architecture, structured triples containing the object of the operation, the action type, and the parameter value are extracted from the text.

5. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The construction of the causal reasoning engine in step S3 specifically includes the following steps: Step 31: Collect historical multimodal interaction data to form a training dataset; Step 32: Apply the PC algorithm to perform conditional independence tests on the training dataset and construct a partially directed acyclic graph describing the causal relationships between multimodal instructions; Step 33: Calculate the causal strength weights of each edge in the graph based on the Fisher-Z test to form a multimodal causal graph that can be quantified and queried.

6. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The specific steps in step S3 to distinguish between genuine causal relationships and spurious correlations include: For real-time input voice commands and gesture commands, query the causal path and strength between their corresponding nodes in the multimodal causal graph; If the causal strength exceeds the first preset threshold, it is determined to be a true causal relationship, and the instructions should be executed accordingly. If the causal strength is lower than the second preset threshold and there is an indirect path via environmental state variables, it is determined to be a spurious correlation, and one of the instructions is ignored or reconfirmed.

7. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The dynamic fusion in step S4 specifically adopts the following decision formula: Final instruction = arg max [α·P (语音|因果图) +β·P (手势|因果图) +γ·P (环境|因果图) ] Among them, α, β, and γ are dynamic weighting coefficients, whose values ​​are adjusted in real time by the causal correlation strength calculated in step S3.

8. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: The multimodal interaction feedback in step S5 includes: Visual feedback: The device object to be controlled is highlighted on the cockpit center console screen, and the control effect is previewed; Voice feedback: The system reads a summary of the operation to be performed via a TTS engine and provides clarification questions when command conflicts are detected.

9. The AI intelligent cockpit voice-gesture multi-modal collaborative interaction method based on visual semantic understanding according to claim 1, characterized in that: It also includes system learning optimization, specifically including the following steps: Record the user's implicit feedback behavior regarding the execution results; Based on the implicit feedback behavior, an incremental causal discovery algorithm is used to update the structure and parameters of the multimodal causal graph online.