Intelligent cabin adaptive interaction method and system

Abstract

The invention discloses an intelligent cabin self-adaptive interaction method and system, relates to the technical field of self-adaptive interaction, and aims to monitor passenger, cabin environment and vehicle information in real time, carry out space-time alignment on the information, obtain a panoramic state snapshot, and utilize an end-side AI lightweight model and a scene context manager to carry out scene self-adaptive interaction. Analyzing the panoramic state snapshot to obtain a scene label containing confidence, judging whether a scene in the scene label is an effective scene or not, generating a high-level service intention instruction, meanwhile, obtaining an atomic service calling chain based on dynamic atomic service arrangement of SOA, realizing safety domain cross-domain interaction through Hypervisor and a vehicle-mounted Ethernet, and obtaining a high-level service intention instruction. The AI lightweight model and the service strategy at the end side are optimized, so that a reasonable atomic service call chain can be output in time, the problems of data isomerism and time sequence dislocation are solved, software and hardware decoupling and infinite multiplexing of service capability are realized, the development, operation and maintenance cost is reduced, and the rationality of the atomic service call chain and the riding comfort of passengers are guaranteed.

Description

Technical Field

[0001] This invention relates to the field of adaptive interaction technology, and specifically to an adaptive interaction method and system for an intelligent cockpit. Background Technology

[0002] With the development of the "new four modernizations" of automobiles, intelligent cockpits are evolving from traditional electronic cockpits to "software-defined cockpits". Traditional cockpits consist of distributed electronic control units and physical switches, while modern intelligent cockpits rely on high-performance computing platforms and virtualization technology to realize the multi-screen driving function of instrument panel, central control and HUD. Currently, passengers' needs for cockpits have shifted from a single entertainment function to a personalized, immersive "third living space" with proactive care capabilities.

[0003] Existing technologies, such as the one disclosed in patent application No. 202511346678.X, include a vehicle alert method, device, vehicle, and storage medium. By obtaining the status of whether the vehicle is occupied, the system switches between "intermittent monitoring" and "real-time monitoring" modes to monitor the quality of the in-vehicle environment and to trigger an alarm or activate purification when an abnormality occurs.

[0004] Existing technology, such as patent application number 202511434536.9, discloses a display method, apparatus, vehicle, storage medium, and program product. The method displays a driving mode component on a first display screen of a vehicle, which is used to display the vehicle's driving mode. A vehicle body animation is displayed on a second display screen of the vehicle, which is used to display a vehicle body model. In response to a switch in the driving mode, the display of the driving mode component and the vehicle body animation is updated synchronously.

[0005] Regarding the above solutions, the applicant of this invention has found that the above technology has at least the following technical problems: 1. The existing technology focuses on a single environmental health dimension, and its response mechanism is based on rule triggering with a fixed threshold. It lacks comprehensive consideration of the passenger's current behavior, emotions and vehicle driving scenario, and therefore cannot guarantee the rationality of the atomic service call chain.

[0006] 2. Existing technologies mainly address the display issue, providing passive visual feedback and lacking the ability to understand the user's deeper intentions. For example, while the system can display off-road mode animations, it cannot automatically adjust the combination of driving mode parameters, air conditioning vents, and ambient lighting effects based on the driver's fatigue level or the passenger's motion sickness. Furthermore, existing technologies often employ tightly coupled application development models, making cross-domain functional collaboration difficult and hindering flexible scenario definition. Consequently, they cannot promptly establish reasonable atomic service call chains for the scenario and cannot guarantee passenger comfort. Summary of the Invention

[0007] To address the aforementioned technical shortcomings, the present invention aims to provide an intelligent cockpit adaptive interaction method and system.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In the first aspect, the present invention provides an intelligent cockpit adaptive interaction method, including: S1, multimodal data acquisition and spatiotemporal alignment: information is collected from passengers, cockpit environment and vehicle, and the collected passenger information, cockpit environment information and vehicle information are spatiotemporally aligned to obtain a panoramic state snapshot.

[0009] S2. Edge AI Reasoning and Scene Recognition: Acquire panoramic state snapshots and perform deep semantic reasoning through a lightweight edge AI model to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels.

[0010] S3. Scenario Determination and Service Strategy Generation: Obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time.

[0011] S4, SOA Atomic Service Dynamic Orchestration: Obtain high-level service intent instructions, query the availability status of each atomic service in the current vehicle, analyze the high-level service intent instructions, and orchestrate the call sequence of each atomic service.

[0012] S5, Cross-Domain Collaborative Execution and Interaction: Obtain the call sequence of each atomic service and drive physical devices and digital interfaces to achieve comprehensive collaborative control of vision, hearing, touch and environment.

[0013] S6. Feedback Monitoring and Closed-Loop Optimization: Real-time monitoring of passenger status and behavior, and optimization of edge AI lightweight model and service strategy.

[0014] Secondly, the present invention provides an intelligent cockpit adaptive interaction system, comprising: a multimodal data acquisition and spatiotemporal alignment module for acquiring information on passengers, cockpit environment and vehicle, and spatiotemporally aligning the acquired passenger information, cockpit environment information and vehicle information to obtain a panoramic state snapshot.

[0015] The edge AI reasoning and scene cognition module is used to acquire panoramic state snapshots and perform deep semantic reasoning through the edge AI lightweight model to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels.

[0016] The scenario determination and service strategy generation module is used to obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time.

[0017] The SOA atomic service dynamic orchestration module is used to obtain high-level service intent instructions, query the availability status of each atomic service of the current vehicle, analyze the high-level service intent instructions, and orchestrate the call sequence of each atomic service.

[0018] The cross-domain collaborative execution and interaction module is used to obtain the call sequence of each atomic service and drive physical devices and digital interfaces to perform comprehensive collaborative control of vision, hearing, touch and environment.

[0019] The feedback monitoring and closed-loop optimization module is used to monitor passengers' status and behavior in real time and optimize the edge AI lightweight model and service strategy.

[0020] The database is used to store the pre-defined strategy library and the atomic service library.

[0021] The beneficial effects of this invention are as follows: 1. This invention provides an intelligent cockpit adaptive interaction method and system that monitors passenger, cockpit environment and vehicle information in real time, aligns them spatiotemporally, obtains a panoramic state snapshot, analyzes the panoramic state snapshot using a lightweight AI model and scene context manager to obtain scene labels containing confidence levels, determines whether the scene in the scene label is a valid scene, generates high-level service intent instructions, and obtains atomic service call chains based on SOA dynamic atomic service orchestration. Cross-domain interaction is achieved through Hypervisor and in-vehicle Ethernet, and the lightweight AI model and service strategy are optimized to output reasonable atomic service call chains in a timely manner, solving the problems of data heterogeneity and temporal misalignment, realizing software and hardware decoupling and unlimited reuse of service capabilities, reducing development and maintenance costs, and ensuring the rationality of atomic service call chains and passenger comfort.

[0022] 2. This invention monitors passengers' facial expressions and body language in real time using DMS and OMS cameras, and generates a time-stamped visual feature stream. At the same time, the underlying driver reads the dynamic parameters of the vehicle chassis and the cabin environment parameters in real time through CAN FD. Based on a unified system clock, the visual feature stream, vehicle chassis dynamic parameters and cabin environment parameters at the same moment are timestamped to construct a panoramic state snapshot containing passenger, vehicle chassis and cabin environment information, thus solving the problems of data heterogeneity and timing misalignment.

[0023] 3. This invention deploys a lightweight AI model on the cockpit domain controller NPU and acquires a panoramic state snapshot. The panoramic state snapshot is converted into a high-dimensional feature vector, and the obtained high-dimensional feature vector is input into the lightweight AI model to acquire passenger and vehicle states. The vehicle state and passenger state at each historical monitoring time are obtained from the scene context manager and combined with the lightweight AI model to acquire the confidence of each scene label. The confidence of each scene label is generated and output, ensuring the rationality of the atomic service call chain.

[0024] 4. This invention obtains high-level service intent instructions and queries the availability status of each atomic service in the current vehicle through the vehicle service bus. Each available atomic service is called a marked atomic service. The high-level service intent instructions are split into service instructions through a dynamic service orchestrator, and the execution order of each service instruction is obtained. The atomic service library is retrieved from the database, and each marked atomic service is compared with the atomic service library to obtain the marked service corresponding to each service instruction, which is called a secondary marked atomic service. At the same time, the call sequence of each secondary marked atomic service is arranged according to the execution order of each service instruction to generate an atomic service call chain, thereby realizing software and hardware decoupling and unlimited reuse of service capabilities, and reducing development and maintenance costs. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the implementation steps of the method of the present invention.

[0027] Figure 2 This is a flowchart of an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the system structure connection of the present invention.

[0029] Figure 4 This is a system architecture diagram of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1: Please refer to Figure 1 and Figure 2 As shown, the present invention provides an intelligent cockpit adaptive interaction method, including: S1, multimodal data acquisition and spatiotemporal alignment: information is collected from passengers, cockpit environment and vehicle, and the collected passenger information, cockpit environment information and vehicle information are spatiotemporally aligned to obtain a panoramic state snapshot.

[0032] It should be noted that passenger information includes facial expressions and body language, cabin environment information includes cabin environmental parameters such as carbon dioxide concentration and humidity, and vehicle information includes acceleration and suspension travel.

[0033] In one specific embodiment, the multimodal data acquisition is spatiotemporally aligned, and the specific process is as follows: the facial expressions and body language of passengers are collected in real time through DMS and OMS cameras and generated into a visual feature stream with timestamps. At the same time, the underlying driver reads the dynamic parameters of the vehicle chassis and the cabin environment parameters in real time through CAN FD.

[0034] It should be noted that the passenger's facial expressions and body language are processed by the ISP to generate a time-stamped visual feature stream. The ISP is an image signal processor.

[0035] It should also be noted that the dynamic parameters of the vehicle chassis include acceleration and suspension travel, while the cabin environment parameters include carbon dioxide concentration and humidity.

[0036] Based on a unified system clock, the visual feature streams, vehicle chassis dynamic parameters, and cabin environment parameters at the same moment are timestamped to construct a panoramic state snapshot containing passenger, vehicle chassis, and cabin environment information.

[0037] It should be noted that the specific process of aligning the visual feature stream, vehicle chassis dynamic parameters, and cabin environment parameters at the same moment with timestamps is as follows: The current time of the unified system clock is read in real time, and a timestamp is added to each frame of the visual feature stream. When the underlying driver reads the dynamic parameters of the vehicle chassis and the cabin environment parameters in real time through CAN FD, the timestamp of the unified system clock is recorded synchronously. The visual feature stream, vehicle chassis dynamic parameters, and cabin environment parameters are temporarily stored in the order of timestamps, and time window matching and missing data difference filling are performed. A Cartesian coordinate system is established with the vehicle's center of mass as the origin, and the visual feature stream, vehicle chassis dynamic parameters, and cabin environment parameters are spatially aligned.

[0038] S2. Edge AI Reasoning and Scene Recognition: Acquire panoramic state snapshots and perform deep semantic reasoning through a lightweight edge AI model to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels.

[0039] In a specific embodiment, the edge AI inference and scene cognition process is as follows: a lightweight AI model is deployed on the cockpit domain controller NPU, and a panoramic state snapshot is obtained, which is then converted into a high-dimensional feature vector.

[0040] It should be noted that a training dataset is constructed that includes historical multimodal feature data (such as visual features and vehicle chassis parameters) and corresponding real-world scene labels (such as motion sickness, fatigue, and off-road). This training dataset is used to perform supervised fine-tuning (SFT) on the pre-trained large language model or multimodal model, iteratively updating the model parameters through backpropagation until the model's inference accuracy on the validation set reaches a preset threshold or the loss function converges. Subsequently, to adapt to the computing power and storage resources of the cockpit domain controller NPU, the trained model undergoes quantization and compression processing (such as INT8 or INT4 quantization), finally deploying the aforementioned lightweight edge AI model.

[0041] The high-dimensional feature vector is input into the AI ​​lightweight model, which then outputs scene labels containing confidence scores based on the input high-dimensional feature vector.

[0042] It should be noted that the scene tags include motion sickness, off-road driving, and fatigue.

[0043] It should also be noted that, for example, if the scene label is motion sickness and the confidence level is 0.92, then the scene label including the confidence level would be (motion sickness, confidence level 0.92). This example is for illustrative purposes only and is not the only valid one.

[0044] S3. Scenario Determination and Service Strategy Generation: Obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time.

[0045] In a specific embodiment, the scenario determination and service policy generation process is as follows: obtain scenario tags containing confidence levels and call them confidence scenario tags; match the confidence scenario tags with the scenario context manager and determine whether the match is successful.

[0046] When a match is successful, the scenario confidence score and scenario are extracted from the confidence scenario label. It is then determined whether the scenario in the confidence scenario label is in a service cooldown period. Simultaneously, the scenario confidence score in the confidence scenario label is compared with a preset confidence score threshold. If the scenario confidence score in the confidence scenario label is greater than the preset confidence score threshold and the scenario is not in a service cooldown period, the confidence scenario label is considered a valid scenario label. At this point, a preset policy library is retrieved from the database, and the valid scenario label is matched with the preset policy library to obtain a high-level service intent instruction. Alternatively, the valid scenario label can be input into the edge-side generative AI model, and the high-level service intent instruction can be generated in real time through generative inference. If the scenario confidence score in the confidence scenario label is greater than the preset confidence score threshold and the scenario is in a service cooldown period, the service policy for the valid scenario label is being executed. If the scenario confidence score in the confidence scenario label is not greater than the preset confidence score threshold, the service process is terminated.

[0047] It should be noted that the confidence level in the confidence scene label is used as the scene confidence level.

[0048] It should also be noted that the preset confidence threshold is a critical value used to determine whether the confidence level of the scene is high, and it is set by the staff.

[0049] It should be explained that the pre-built strategy library is used to store high-level service intent instructions corresponding to different scenario tags, and is set by staff.

[0050] Specifically, the effective scenario tags are compared with the scenario tags in the pre-set policy library. If the effective scenario tag is the same as a scenario tag in the pre-set policy library, the high-level service intent instruction corresponding to that scenario tag is obtained from the pre-set policy library.

[0051] It should also be noted that the present invention obtains high-level service intent instructions through any of the following methods: Method 1: Obtain a pre-set strategy library from the database and match the effective scenario tags with the pre-set strategy library to find the corresponding instructions. For example, if the tag is 'motion sickness', the preset instructions 'turn on ventilation and adjust the seat' will be matched. Method 2: Based on generative AI technology (such as large language models), the effective scenario tags and the current vehicle environment data are used as prompts and input into the model. The model infers in real time and generates high-level service intent instructions that conform to the current context. The instructions only describe the service goal (such as 'relieving passenger motion sickness') and do not contain specific underlying hardware control code, thereby achieving decoupling between the decision-making layer and the execution layer. This embodiment adopts Method 1.

[0052] If a match fails, the service process is terminated.

[0053] The specific process for determining whether a match is successful, as described above, is as follows: The confidence scene label is compared with each confidence scene label stored in the scene context manager. If the confidence scene label is different from all the confidence scene labels stored in the scene context manager, the match fails. If the confidence scene label is the same as any of the confidence scene labels stored in the scene context manager, the match is successful. The specific process for determining whether a scenario in the confidence scenario label is in a service cooldown period is as follows: obtain the service policy corresponding to the scenario in the scenario confidence label, and obtain the most recent start time of the cooldown period of the service policy, which is called the marked cooldown start time. At the same time, obtain the current time, and calculate the interval duration based on the current time and the marked cooldown start time.

[0054] The interval is compared with the preset cooling time. If the interval is less than the preset cooling time, it means that the scene in the confidence scene label is in the service cooling period. Otherwise, it means that the scene in the confidence scene label is not in the service scene cooling period.

[0055] It should be noted that the preset cooling-off period is a critical value used to determine whether the service strategy is in a cooling-off period, and it is set by the staff.

[0056] S4, SOA Atomic Service Dynamic Orchestration: Obtain high-level service intent instructions, query the availability status of each atomic service in the current vehicle, analyze the high-level service intent instructions, and orchestrate the call sequence of each atomic service.

[0057] In a specific embodiment, the SOA atomic service dynamic orchestration process is as follows: obtain the high-level service intent instruction, and query the availability status of each atomic service in the current vehicle through the vehicle service bus, and refer to each available atomic service as a tagged atomic service.

[0058] It should be noted that, in the SOA architecture, an atomic service refers to the smallest independently callable software service unit that encapsulates the hardware functions of a vehicle. These hardware functions include functions such as window operation, air conditioning adjustment, fragrance release, seat heating, and ambient lighting color.

[0059] It should also be noted that the availability status of the atomic service refers to whether the car window is malfunctioning and whether the fragrance is fully replenished. A malfunctioning car window means the car window is unusable, a normal car window means the car window is usable, sufficient fragrance is usable, and insufficient fragrance is unusable.

[0060] The high-level service intent instructions are broken down into service instructions by a dynamic service orchestrator, and the execution order of each service instruction is obtained. The atomic service library is retrieved from the database, and each marked atomic service is compared with the atomic service library to obtain the marked service corresponding to each service instruction. These are called secondary marked atomic services. At the same time, the call sequence of each secondary marked atomic service is arranged according to the execution order of each service instruction to generate an atomic service call chain.

[0061] It should be noted that, for example, a command to alleviate motion sickness generates a series of ordered atomic service call requests, including turning on the air conditioning's external air circulation, slightly opening the windows by 5%, activating the seat's soothing massage function, and switching to a stable UI theme. This example is for illustrative purposes only and is not the only valid one.

[0062] It should also be noted that the atomic service library is used to store atomic services corresponding to different service commands.

[0063] Obtain the current vehicle's driving specifications, determine whether the atomic service call chain conforms to the current vehicle's driving specifications, if it does, drive the hardware to perform physical actions according to the atomic service call chain, if it does not, remove the secondary-marked atomic services that do not conform to the current vehicle's driving specifications, form a new atomic service call chain, and drive the hardware to perform physical actions according to the atomic service call chain.

[0064] It should be noted that this method and system are integrated with a traffic management system to obtain the current vehicle's driving rules from the traffic management system.

[0065] It should also be noted that each secondary-marked atomic service in the atomic service call chain is obtained, and it is determined whether each secondary-marked atomic service violates the current vehicle's driving rules. If each secondary-marked atomic service does not violate the current vehicle's driving rules, it means that the atomic service call chain conforms to the current vehicle's driving rules; otherwise, it means that the atomic service call chain does not conform to the current vehicle's driving rules. The secondary-marked atomic services that violate the current vehicle's driving rules are referred to as secondary-marked atomic services that do not conform to the current vehicle's driving rules.

[0066] S5, Cross-Domain Collaborative Execution and Interaction: Obtain the call sequence of each atomic service and drive physical devices and digital interfaces to achieve comprehensive collaborative control of vision, hearing, touch and environment.

[0067] It should also be noted that by obtaining the call sequence of each atomic service and driving the physical devices and digital interface in parallel, on the one hand, each atomic service interface converts software instructions into specific CAN / LIN signals through the gateway, driving hardware such as air conditioning, seats, and windows to perform physical actions and adjust the cabin environment; on the other hand, the 3D UI rendering engine receives display instructions and renders digital twin animations synchronized with the physical actions in real time on the central control screen, such as displaying the airflow effect of 3D window opening, and broadcasting prompts in conjunction with voice services. During this process, the multi-screen linkage manager ensures that the information on the instrument panel and HUD is switched synchronously with the central control screen, providing users with immersive feedback in all aspects of sight, sound, and touch.

[0068] S6. Feedback Monitoring and Closed-Loop Optimization: Real-time monitoring of passenger status and behavior, and optimization of edge AI lightweight model and service strategy.

[0069] In a specific embodiment, the feedback monitoring and closed-loop optimization process is as follows: When executing the service strategy, the facial expressions and body language of passengers are monitored in real time through DMS / OMS, and negative samples and positive samples are obtained based on the facial expressions and body language of passengers at each time, so as to optimize the edge AI lightweight model and service strategy.

[0070] Obtain each negative sample and fine-tune the edge AI lightweight model online according to the RLHF mechanism.

[0071] Obtain each positive sample and strengthen the weight of the service strategy based on each positive sample.

[0072] It should be noted that the RLHF mechanism is used to optimize the lightweight AI model and service strategy on the edge, where RLHF stands for reinforcement learning based on human feedback.

[0073] The specific process for obtaining each negative sample and each positive sample is as follows: At a certain moment, the facial expressions and body language of the passengers at that moment are obtained, and the passenger status is obtained. The passenger status and body language are compared with the passenger status and atomic service call chain before the service strategy is executed. If the passenger status at that moment is better than the passenger status before the service strategy is executed, and the body language conforms to the atomic service call chain, then the passenger information, cabin environment information, vehicle information, and atomic service call chain at that moment are positive samples. Conversely, if they are not, then the passenger information, cabin environment information, vehicle information, and atomic service call chain at that moment are negative samples. Each negative sample and each positive sample is obtained in this way.

[0074] It should be noted that passenger mood is determined based on facial expressions. When a passenger's mood at a given moment is better than it was before the service strategy was implemented, it indicates that the passenger's overall state is better than it was before the strategy was implemented. For example, if a passenger is happy at this moment, but was calm before the strategy was implemented, then the passenger's mood at this moment is better than it was before the strategy was implemented. This example is for illustrative purposes only and is not the only valid approach.

[0075] Please see Figure 3 As shown, the present invention provides an intelligent cockpit adaptive interaction system, including: a multimodal data acquisition and spatiotemporal alignment module for collecting information on passengers, cockpit environment and vehicle, and spatiotemporally aligning the collected passenger information, cockpit environment information and vehicle information to obtain a panoramic state snapshot.

[0076] The edge AI reasoning and scene cognition module is used to acquire panoramic state snapshots and perform deep semantic reasoning through the edge AI lightweight model to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels.

[0077] The scenario determination and service strategy generation module is used to obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time.

[0078] The SOA atomic service dynamic orchestration module is used to obtain high-level service intent instructions, query the availability status of each atomic service of the current vehicle, analyze the high-level service intent instructions, and orchestrate the call sequence of each atomic service.

[0079] The cross-domain collaborative execution and interaction module is used to obtain the call sequence of each atomic service and drive physical devices and digital interfaces to perform comprehensive collaborative control of vision, hearing, touch and environment.

[0080] The feedback monitoring and closed-loop optimization module is used to monitor passengers' status and behavior in real time and optimize the edge AI lightweight model and service strategy.

[0081] The database is used to store the pre-defined strategy library and the atomic service library.

[0082] Example 2: Please refer to Figure 4 As shown, the present invention includes a hardware infrastructure layer, a system software layer, an AI perception and cognition layer, an SOA service architecture layer, and an interaction and execution layer.

[0083] The hardware infrastructure layer includes input, computing, and output.

[0084] The system software layer is based on Hypervisor virtualization technology, which enables the security domain OS and entertainment domain OS to run in isolation on the same chip, ensuring the system's high security and rich ecosystem.

[0085] The AI ​​perception and cognition layer includes multimodal data fusion (synchronizing and spatially aligning heterogeneous sensor data with timestamps) and an edge AI large model engine (this is the "brain" of the system; it does not simply process commands but understands "scenes." For example, it can combine "user frowning" + "vehicle bumps" + "rapid acceleration" to infer the high-level semantic meaning of "user motion sickness"). The SOA service architecture layer includes an atomic service library (which encapsulates hardware capabilities such as "adjusting the air conditioning temperature" and "turning on the seat massage" as independent software service interfaces) and a dynamic service orchestrator (which dynamically calls and combines atomic services such as "turning on external circulation" + "lowering the seat" + "playing soothing music" to generate a service chain based on the inference results of the AI ​​layer, such as "relieving motion sickness").

[0086] The interaction and execution layer includes a 3D UI rendering engine (which receives instructions from the service layer and renders digital twin images through the Unity / Unreal engine, such as a transparent chassis during off-road driving) and multi-screen linkage (which ensures that information from the instrument panel, central control screen and HUD is displayed synchronously, providing an immersive experience).

[0087] This invention provides real-time monitoring of passenger, cabin environment, and vehicle information, spatiotemporally aligns them, and acquires a panoramic state snapshot. Utilizing a lightweight edge AI model and scene context manager, the panoramic state snapshot is analyzed to obtain scene labels containing confidence levels. The system then determines whether the scene in the label is a valid scene, generates high-level service intent instructions, and obtains atomic service call chains based on SOA-based dynamic atomic service orchestration. Cross-domain interaction within the security domain is achieved through the Hypervisor and in-vehicle Ethernet. Furthermore, the lightweight edge AI model and service strategies are optimized to promptly output reasonable atomic service call chains, resolving data heterogeneity and temporal misalignment issues. This achieves hardware-software decoupling and unlimited reuse of service capabilities, reduces development and maintenance costs, and ensures the rationality of atomic service call chains and passenger comfort.

[0088] The examples described in this invention are not limited to the specific embodiments listed above. The examples are merely illustrative to facilitate understanding of the invention and do not constitute a limitation on the scope of protection of this invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of this invention should be included within the scope of protection.

[0089] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in this specification, they should all fall within the protection scope of the present invention.

Claims

1. An adaptive interaction method for an intelligent cockpit, characterized in that, Includes the following steps: S1. Multimodal data acquisition and spatiotemporal alignment: Information on passengers, cabin environment and vehicles is collected, and the collected passenger information, cabin environment information and vehicle information are spatiotemporally aligned to obtain a panoramic state snapshot; S2, Edge AI Reasoning and Scene Recognition: Acquire panoramic state snapshots and perform deep semantic reasoning through edge AI lightweight models to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels; S3, Scenario Judgment and Service Strategy Generation: Obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time. S4, SOA Atomic Service Dynamic Orchestration: Obtain high-level service intent instructions, query the availability status of each atomic service in the current vehicle, and analyze the high-level service intent instructions to orchestrate the call sequence of each atomic service; S5, Cross-domain Collaborative Execution and Interaction: Obtain the call sequence of each atomic service and drive physical devices and digital interfaces to achieve comprehensive collaborative control of vision, hearing, touch and environment; S6. Feedback Monitoring and Closed-Loop Optimization: Real-time monitoring of passenger status and behavior, and optimization of edge AI lightweight model and service strategy.

2. The intelligent cockpit adaptive interaction method according to claim 1, characterized in that, The multimodal data acquisition and spatiotemporal alignment process is as follows: The DMS and OMS cameras collect passengers' facial expressions and body language in real time and generate a time-stamped visual feature stream. At the same time, the underlying driver reads the dynamic parameters of the vehicle chassis and cabin environment parameters in real time through CAN FD. Based on a unified system clock, the visual feature streams, vehicle chassis dynamic parameters, and cabin environment parameters at the same moment are timestamped to construct a panoramic state snapshot containing passenger, vehicle chassis, and cabin environment information.

3. The intelligent cockpit adaptive interaction method according to claim 1, characterized in that, The specific process of edge AI reasoning and scene recognition is as follows: Deploy a lightweight AI model on the cockpit domain controller NPU and acquire a panoramic state snapshot, then convert the panoramic state snapshot into a high-dimensional feature vector. The high-dimensional feature vector is input into the AI ​​lightweight model, which then outputs scene labels containing confidence scores based on the input high-dimensional feature vector.

4. The intelligent cockpit adaptive interaction method according to claim 1, characterized in that, The specific process for scenario determination and service strategy generation is as follows: Obtain scene labels containing confidence levels and call them confidence scene labels. Match the confidence scene labels with the scene context manager and determine whether the match is successful. When a match is successful, the scenario confidence score and scenario are extracted from the confidence scenario label, and it is determined whether the scenario in the confidence scenario label is in a service cooldown period. At the same time, the scenario confidence score in the confidence scenario label is compared with a preset confidence score threshold. When the scenario confidence score in the confidence scenario label is greater than the preset confidence score threshold and the scenario is not in a service cooldown period, it means that the confidence scenario label is a valid scenario label. At this time, a preset policy library is retrieved from the database, and the valid scenario label is matched with the preset policy library to obtain a high-level service intent instruction. Alternatively, the valid scenario label is input into the edge generative AI model, and the high-level service intent instruction is generated in real time through generative inference. When the scenario confidence score in the confidence scenario label is greater than the preset confidence score threshold and the scenario is in a service cooldown period, it means that the service policy of the valid scenario label is being executed. When the scenario confidence score in the confidence scenario label is not greater than the preset confidence score threshold, the service process is terminated. If a match fails, the service process is terminated.

5. The intelligent cockpit adaptive interaction method according to claim 4, characterized in that, The specific process for determining whether a match is successful is as follows: The confidence scene label is compared with each confidence scene label stored in the scene context manager. If the confidence scene label is different from each of the confidence scene labels stored in the scene context manager, the match fails. If the confidence scene label is the same as a certain confidence scene label stored in the scene context manager, the match succeeds.

6. The intelligent cockpit adaptive interaction method according to claim 4, characterized in that, The specific process for determining whether a scenario in the confidence scenario label is in a service cooldown period is as follows: Obtain the service policy corresponding to the scenario in the scenario confidence label, and obtain the most recent cooldown period start time of the service policy, which is called the marked cooldown start time. At the same time, obtain the current time, and calculate the interval duration based on the current time and the marked cooldown period start time. The interval is compared with the preset cooling time. If the interval is less than the preset cooling time, it means that the scene in the confidence scene label is in the service cooling period. Otherwise, it means that the scene in the confidence scene label is not in the service scene cooling period.

7. The intelligent cockpit adaptive interaction method according to claim 1, characterized in that, The dynamic orchestration of SOA atomic services follows the following process: Obtain high-level service intent instructions and query the availability status of each atomic service in the current vehicle through the vehicle service bus. Each available atomic service is referred to as a tagged atomic service. The high-level service intent instructions are broken down into service instructions by a dynamic service orchestrator, and the execution order of each service instruction is obtained. The atomic service library is retrieved from the database, and each marked atomic service is compared with the atomic service library to obtain the marked services corresponding to each service instruction, which are called secondary marked atomic services. At the same time, the call sequence of each secondary marked atomic service is arranged according to the execution order of each service instruction to generate an atomic service call chain. Obtain the current vehicle's driving specifications, determine whether the atomic service call chain conforms to the current vehicle's driving specifications, if it does, drive the hardware to perform physical actions according to the atomic service call chain, if it does not, remove the secondary-marked atomic services that do not conform to the current vehicle's driving specifications, form a new atomic service call chain, and drive the hardware to perform physical actions according to the atomic service call chain.

8. The intelligent cockpit adaptive interaction method according to claim 1, characterized in that, The feedback monitoring and closed-loop optimization process is as follows: When implementing service strategies, DMS / OMS monitors passengers' facial expressions and body language in real time, and obtains negative and positive samples based on passengers' facial expressions and body language at different times to optimize the edge AI lightweight model and service strategies. Obtain each negative sample and fine-tune the lightweight AI model on the edge according to the RLHF mechanism online; Obtain each positive sample and strengthen the weight of the service strategy based on each positive sample.

9. The intelligent cockpit adaptive interaction method according to claim 8, characterized in that, The specific process for obtaining each negative sample and each positive sample is as follows: At a certain moment, the facial expressions and body language of passengers at that moment are acquired, and the passenger status is obtained. The passenger status and body language are compared with the passenger status and atomic service call chain before the service policy is executed. If the passenger status at that moment is better than the passenger status before the service policy is executed, and the body language conforms to the atomic service call chain, then the passenger information, cabin environment information, vehicle information, and atomic service call chain at that moment are positive samples. Conversely, if they are worse, then the passenger information, cabin environment information, vehicle information, and atomic service call chain at that moment are negative samples. Each negative sample and each positive sample is obtained in this way.

10. An adaptive interaction system utilizing the intelligent cockpit adaptive interaction method according to any one of claims 1-9, characterized in that, include: The multimodal data acquisition and spatiotemporal alignment module is used to collect information on passengers, cabin environment and vehicles, and to spatiotemporally align the collected passenger information, cabin environment information and vehicle information to obtain a panoramic state snapshot; The edge AI reasoning and scene cognition module is used to acquire panoramic state snapshots and perform deep semantic reasoning through the edge AI lightweight model to identify driving scenarios and potential passenger intentions, while outputting scene labels containing confidence levels. The scenario determination and service strategy generation module is used to obtain scenario tags containing confidence levels, filter the scenario confidence levels of scenario tags containing confidence levels, determine whether they are valid scenarios, and generate high-level service intent instructions in real time. The SOA atomic service dynamic orchestration module is used to obtain high-level service intent instructions, query the availability status of each atomic service of the current vehicle, analyze the high-level service intent instructions, and orchestrate the call sequence of each atomic service. The cross-domain collaborative execution and interaction module is used to obtain the call sequence of each atomic service and drive physical devices and digital interfaces to perform comprehensive collaborative control of vision, hearing, touch and environment. The feedback monitoring and closed-loop optimization module is used to monitor passengers' status and behavior in real time and optimize the edge AI lightweight model and service strategy. The database is used to store the pre-defined strategy library and the atomic service library.

Images