A cabin-driving integrated intelligent driving system based on a VLA model

Abstract

The application relates to the technical field of intelligent driving and discloses a cabin-driving integrated intelligent driving system based on a VLA model. The system comprises a cabin-driving integrated domain controller, an extracabin sensor group, an intracabin sensor group and a vehicle bus network connected with the cabin-driving integrated domain controller; the cabin-driving integrated domain controller is configured to execute: a multi-modal coding module for performing time synchronization and multi-modal coding on multi-source data from the extracabin sensor group, the intracabin sensor group and the vehicle bus network to obtain unified feature representation; a VLA cognitive center for receiving the unified feature representation and performing multi-modal joint reasoning to generate driving side upper layer decision instructions and cabin side service control instructions; and a functional safety module for performing safety verification on the driving side upper layer decision instructions and the cabin side service control instructions. The application realizes joint arrangement of driving strategy enhancement and cabin service, improves experience consistency, hardware utilization rate and safety redundancy capability.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a cockpit-driver integrated intelligent driving system based on the VLA model. Background Technology

[0002] Most mainstream intelligent vehicles currently adopt a domain control architecture that separates the driver and cabin: the driver domain (ADAS / AD) and the cabin domain deploy independent computing chips, sensors, and software stacks respectively. The driver domain is equipped with external sensors such as forward / surround view cameras, millimeter-wave radar, and ultrasonic radar, responsible for environmental perception and vehicle control; the cabin domain is equipped with in-cabin sensors such as DMS (Driver Monitoring System), OMS (Occupant Monitoring System), and microphone arrays, responsible for human-machine interaction, entertainment, and comfort adjustments. The two domains exchange limited data via CAN / Ethernet, each performing its own function.

[0003] Existing cabin-rider separation architectures and single-domain VLA solutions have the following typical shortcomings: • Perception fragmentation: In traditional architectures, external camera data is "locked" into the intelligent driving domain and used only for perception and obstacle avoidance; internal DMS / OMS data only serves cockpit functions. The lack of unified semantic alignment between the two leads to the linkage between "driver fatigue" and "intelligent driving strategy adjustment" relying on predefined rules and high-latency cross-domain communication, resulting in sluggish response (typical latency of over 500ms).

[0004] • Decision-making conflict: The intelligent driving domain prioritizes traffic efficiency (such as frequently suggesting lane changes), while the cockpit domain prioritizes comfort and safety alerts. Independent decision-making between the two may lead to behavioral conflicts. For example, when the intelligent driving system suggests a lane change based on road conditions, the cockpit may simultaneously broadcast a fatigue warning, resulting in information overload and negatively impacting the user experience.

[0005] • High hardware redundancy and cost: The separation of cockpit and driver requires a dual-chip solution such as Orin (intelligent driving) + Qualcomm 8295 (cockpit). Sensor data cannot be shared, memory and computing power are redundantly configured, and hardware costs remain high, which is about 40% higher than the cost of a single-chip integrated cockpit and driver solution.

[0006] • Insufficient bidirectional perception between the cabin and the outside: The external camera can recognize semantic information such as “stunning sunset”, “construction cones”, and “ground lock status”, but the traditional architecture cannot transform this information into experiences such as AR-HUD enhancement, landscape capture, and parking prompts on the cockpit side; The internal DMS / OMS can recognize states such as “children crying”, “driver talking intensely”, and “suspected fainting”, but the intelligent driving domain is unable to dynamically adjust the driving style (such as smooth mode, defensive mode, MRC emergency takeover) based on this, lacking “empathy ability”.

[0007] • Low data efficiency: The driving domain and cockpit domain are collected, labeled and trained separately. It is difficult to form a unified VLA training sample from multimodal data of the same scene. The data utilization efficiency is low and the model iteration cycle is long. Summary of the Invention

[0008] To address the aforementioned shortcomings in existing technologies, this invention provides a cockpit-driver integrated intelligent driving system based on a VLA model. While retaining real-time AD as the control foundation, it introduces a VLA cognitive hub and a dual-branch decoder to achieve joint orchestration of driving strategy enhancement and cockpit services, breaking down the barriers between the cockpit and driver domains and improving experience consistency, hardware utilization, and safety redundancy.

[0009] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A cockpit-driver integrated intelligent driving system based on a VLA model includes: The cockpit-driver integrated domain controller, the external sensor group connected to the cockpit-driver integrated domain controller, the internal sensor group, and the vehicle bus network; The cockpit-rider integrated domain controller is configured to execute: The multimodal coding module is used to perform time synchronization and multimodal coding on multi-source data from the external sensor group, the internal sensor group and the vehicle bus network to obtain a unified feature representation; The VLA cognitive center is used to receive the unified feature representation and perform multimodal joint reasoning to generate upper-level decision-making instructions on the driver's side and service control instructions on the cockpit side. The functional safety module is used to perform safety verification on the driver-side upper-level decision-making commands and the cockpit-side service control commands; The driver-side upper-level decision-making commands are verified and then sent to the real-time automated driving system to modulate its control strategy; the cockpit-side service control commands are verified and then sent to the cockpit actuators.

[0010] In one possible implementation, the multimodal coding module includes: A unified visual coding unit is used to encode the road scene video stream captured by the external camera and the driver monitoring video stream captured by the internal camera to generate a visual token sequence; The language encoding unit is used to encode language data in text form and generate a sequence of language tokens; The audio encoding unit is used to encode the audio data collected by the in-cabin microphone array and generate audio feature tokens.

[0011] In one possible implementation, the unified visual coding unit is further configured as follows: Based on the driver monitoring video stream, estimate the driver's gaze direction vector; Using the gaze direction vector as the query vector and the road target features obtained by encoding the road scene video stream as the key and value vectors, cross-view attention calculation is performed to obtain the driver's attention confidence for a specific road target.

[0012] In one possible implementation, the driver-side upper-level decision-making instructions include at least one of the following: Driving mode commands, following distance parameters, speed limit parameters, and strategies for navigating complex scenarios.

[0013] In one possible implementation, the functional safety module is configured to perform a three-level safety fence, including a Constitutional AI constraint verification at the model layer, a responsibility-sensitive safety RSS verification at the rule layer, and an independent monitoring controller verification at the hardware layer.

[0014] In one possible implementation, the external sensor group and the internal sensor group are connected to the integrated cockpit domain controller via an Ethernet interface; The vehicle bus network is a CAN bus, used to transmit the vehicle status data; The integrated cockpit-pilot domain controller includes a time synchronization unit, which uses PTP or GPS absolute time as the system time reference to stamp data accessed through different interfaces with a unified timestamp.

[0015] In one possible implementation, the VLA cognitive center is configured as follows: The generated upper-level decision-making instructions on the driver's side are sent to the planning and control module of the real-time autonomous driving system in the form of strategy parameters, which are used to modulate the underlying vehicle control instructions generated by the module. The generated cockpit-side service control commands are sent directly to the corresponding cockpit actuators in a format that includes natural language text and structured data.

[0016] In one possible implementation, the VLA cognitive center is further configured as follows: Based on the encoding results of the external visual data by the multimodal coding module, semantic information of the external environment is identified, and corresponding cockpit-side service control commands are generated based on the semantic information; and, Based on the encoding results of the in-cabin visual data by the multimodal coding module, the status information of the in-cabin personnel is identified, and the corresponding upper-level decision-making instructions on the driver's side are generated based on the status information.

[0017] In one possible implementation, an end-to-end data closed-loop module is also included, which is used to iterate the VLA cognitive center and / or the real-time autonomous driving system by collecting multimodal driving data and generating training data. The multimodal driving data includes at least: external video data, internal video data, voice data, and control sequence data containing vehicle control actions and cockpit control actions; The end-to-end data closed-loop module is configured to perform at least one of the following operations: Shadow mode data acquisition: After detecting that the driver has taken over the vehicle, the system automatically records multimodal driving data before and after the takeover, as well as the driver's feedback information, as data for supervision and fine-tuning. Simulation refeedback: Reconstruct driving scenarios based on collected multimodal driving data and generate training data for rare scenarios.

[0018] In one possible implementation, a privacy compliance processing module is also included, configured as follows: Before the multimodal coding module encodes the in-cabin visual data, the data undergoes real-time desensitization processing to remove or blur biometric information that can identify an individual, and only the desensitized non-sensitive feature data is used for subsequent processing or uploading; and / or, The model is updated in the cloud using a federated learning approach, based on the model update parameters obtained after processing external visual data on the vehicle side. The model of the VLA cognitive center and / or the real-time autonomous driving system is then aggregated and updated in the cloud without uploading the original video data.

[0019] The present invention has the following beneficial effects: (1) Significantly improved experience consistency: Users do not need to distinguish between "intelligent driving function" and "cabin function". They can obtain the joint response of driving and cabin by describing the destination and status in natural language. For example, "There is a traffic jam ahead, and I am a little carsick" can trigger the vehicle speed to decrease, the following distance to increase, the seat ventilation and soothing music at the same time, forming an integrated comfort experience; eliminating the information conflict and behavior inconsistency caused by the independent decision-making of intelligent driving and cabin under the traditional architecture.

[0020] (2) Reduced hardware costs: Compared with the separate cabin and driver solution (such as Orin + 8295 dual chip), the hardware cost can be reduced by about 40% by using a single chip cabin and driver integrated solution such as Thor and Snapdragon Ride Flex; unified memory and zero-copy architecture reduce data transfer and redundant calculations, and improve computing power utilization.

[0021] (3) Safety redundancy upgrade: In-cabin status (such as driver fatigue, suspected fainting, children crying) can directly participate in the adjustment of driving strategy, and the response delay is reduced from the 500ms level of traditional cross-domain solutions to the 50ms level; the active safety of line of sight guidance can dynamically adjust the warning intensity according to whether the driver perceives danger, avoiding excessive intervention or insufficient intervention; in health emergency scenarios, VLA can link intelligent driving MRC and cabin 120 call to form a complete emergency closed loop.

[0022] (4) Two-way empowerment of cabin and interior perception: external perception empowers the cabin: semantic information such as construction area, scenery, and ground lock status can be transformed into AR-HUD prompts, intelligent capture, parking guidance and other experiences; internal perception empowers driving: fatigue, emotions, occupant activities, etc. can drive adaptive driving style (conservative / stable / defensive mode) and comfort prediction control; realizing "external sensors become the eyes of the cabin, and internal sensors become the sixth sense of intelligent driving".

[0023] (5) Improved data efficiency: The same VLA model learns driving and cockpit data at the same time, avoiding repeated collection and multiple annotation processes; the four-dimensional spatiotemporal data flywheel (external + internal + language + action) supports joint training, accelerating model iteration and scene coverage.

[0024] (6) VLA enhances rather than replaces, taking into account real-time performance and intelligence: Real-time AD (BEV + planning and control) independently undertakes the 50Hz level underlying control, ensuring safety redundancy and millisecond-level response; VLA, as the upper cognitive enhancement layer, provides solutions for complex scenarios and intelligent planning decisions at a lower frequency (1-5Hz), making up for the shortcomings of traditional AD in long-tail scenarios and natural language understanding; The two work together: The existing vehicle intelligent driving capabilities are enhanced rather than replaced, retaining the reliability of mature solutions while gaining the generalization and interpretability brought by large models.

[0025] (7) Users describe their parking preferences in natural language (such as "spacious, close to the elevator"), and the system understands and executes them; Emotional human-machine co-driving: fatigue monitoring is linked to driving mode and cabin service; Social interactive driving: pedestrian intentions are identified and the system conveys the signal "I have seen you" to pedestrians in conjunction with HUD virtual zebra crossings, reducing road rage and misjudgment. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the hardware system of the present invention; Figure 2 This is a schematic diagram of the software system of the present invention. Detailed Implementation

[0027] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0028] like Figure 1 and Figure 2 As shown in the figure, an embodiment of the present invention provides a cockpit-driver integrated intelligent driving system based on a VLA model, comprising: The cockpit-driver integrated domain controller, the external sensor group connected to the cockpit-driver integrated domain controller, the internal sensor group, and the vehicle bus network; The cockpit-rider integrated domain controller is configured to execute: The multimodal coding module is used to perform time synchronization and multimodal coding on multi-source data from the external sensor group, the internal sensor group and the vehicle bus network to obtain a unified feature representation; The VLA cognitive center is used to receive the unified feature representation and perform multimodal joint reasoning to generate upper-level decision-making instructions on the driver's side and service control instructions on the cockpit side. The functional safety module is used to perform safety verification on the driver-side upper-level decision-making commands and the cockpit-side service control commands; The driver-side upper-level decision-making commands are verified and then sent to the real-time automated driving system to modulate its control strategy; the cockpit-side service control commands are verified and then sent to the cockpit actuators.

[0029] This embodiment provides a cockpit-driver integrated intelligent driving system based on a VLA model, which is composed of a hardware platform and a software system. The system adopts a layered architecture: the bottom layer is a real-time autonomous driving system (such as BEV / OCC perception + control), responsible for high-frequency vehicle control; the upper layer is a VLA cognitive enhancement layer, responsible for complex scenario solutions, intelligent upper-level planning and decision-making, and joint orchestration of the driver and cockpit. VLA does not replace real-time AD, but enhances the capabilities of existing vehicle autonomous driving systems by providing semantic understanding, multi-step reasoning, and cockpit-inside-outside fusion capabilities. The key innovations of this embodiment include: ① a hardware and software integrated cockpit-driver fusion architecture; ② collaboration between multimodal coding (visual / language / audio) and the VLA cognitive center; ③ a functional safety module as a unified verification checkpoint before execution; ④ bidirectional empowerment of cockpit-inside-outside perception and unified scenario orchestration. The system as a whole consists of two main parts: a hardware system and a software system.

[0030] like Figure 1As shown, the hardware system adopts an architecture centered on an integrated cockpit-driver domain controller, with sensors grouped according to communication methods. The integrated cockpit-driver domain controller subsystem serves as the computing and decision-making center, connecting to the navigation and positioning module, chassis sensors, and environmental sensors via a CAN bus, and connecting to the in-cabin and external sensor subsystems via Ethernet. Each subsystem forms a data path under a unified time reference, supporting real-time synchronization and zero-copy sharing of multimodal data. The human-machine interface input subsystem and in-vehicle communication and time synchronization mechanisms are integrated throughout. The overall topology reflects the hierarchical access principle of "CAN carrying vehicle status and positioning, and Ethernet carrying high-bandwidth sensing data."

[0031] In this embodiment, the external sensor subsystem is responsible for collecting multimodal data of the vehicle's external environment, providing a visual and ranging foundation for driving perception and cockpit enhancement. It is interconnected with the cockpit-driver integrated domain controller via Ethernet to meet the requirements for high-bandwidth video and point cloud data transmission.

[0032] The panoramic camera array, mounted on the windshield and around the vehicle body, is typically configured as 11V (11-channel vision), including two 8MP front-view cameras, four 3MP side-view cameras, and one 2MP rear-view camera, used to generate BEV (bird's-eye view) and surround-view images. It provides a continuous video stream of the external driving scene to the real-time AD system (BEV perception, etc.), supporting obstacle detection, lane line recognition, and drivable area segmentation; simultaneously, it provides scene semantic understanding input (such as construction areas, scenery, and parking lock status) to the VLA, driving the cockpit-side AR-HUD, intelligent capture, parking prompts, and upper-level planning decisions of the VLA. It connects to the cockpit-driver integrated domain controller via Ethernet (e.g., after GMSL / FPD-Link conversion) or a dedicated high-speed video interface, where preprocessing and feature extraction are performed by a unified ISP and GPU / NPU.

[0033] The millimeter-wave radar, employing the 77GHz / 79GHz frequency band, provides target speed, distance, and angle information. It compensates for the limitations of vision in adverse weather, backlighting, and nighttime conditions, providing redundant perception for driving control. The output target list is accessed to the domain controller via Ethernet and fused with visual data under a unified timestamp.

[0034] The ultrasonic radar is deployed as an ultrasonic sensor array in the front and rear bumpers of the vehicle, typically configured as a 12U array. It is used for short-range obstacle detection, primarily serving parking scenarios and low-speed driving, providing centimeter-level ranging information. It connects to the domain controller via Ethernet or CAN to fuse with other external sensing data.

[0035] The optional lidar sensor outputs 3D point cloud data. It provides high-precision 3D environment modeling in L3 and above autonomous driving scenarios, enhancing perception reliability in adverse weather or backlight conditions; the point cloud data can provide richer scene geometry and semantic information for VLA. Access to the domain controller is via Ethernet (e.g., 100 / 1000BASE-T1), aligning the point cloud data and video stream on a unified time base.

[0036] The in-cabin sensor subsystem is responsible for collecting occupant status and interaction information, providing input for adaptive driving strategies and cabin services. It interconnects with the integrated cockpit-driver domain controller via Ethernet, supporting low-latency transmission of high-frame-rate video streams. The Driver Monitoring System (DMS) uses infrared or visible light cameras facing the driver to monitor facial expressions, gaze direction, eyelid status, head posture, etc. It provides semantic information such as driver fatigue, attention distribution, and emotional state to the Vehicle Automation System (VLA), supporting "eye-guided active safety" (e.g., determining warning intensity based on whether the driver notices a dangerous target), "physiological state adaptive driving" (switching to conservative mode when fatigued), and "health emergency takeover" (triggering MRC in case of suspected fainting). It connects to the domain controller via GMSL / FPD-Link or MIPI through Ethernet conversion, aligning with the external video stream on a unified time reference. The Occupant Monitoring System (OMS) uses cameras facing the occupants, typically configured with two 2-megapixel cameras, covering the driver, front passenger, and rear passenger areas. It recognizes occupant postures (such as children crying, using laptops, drinking hot water, etc.) to provide input for "predictive passenger comfort control"; in health emergency scenarios, it combines with the DMS to determine the driver's incapacity status. It is connected to the domain controller along with the DMS, participating in unified multimodal coding. A microphone array, typically consisting of six microphones, is located in the cabin, supporting sound source localization and speech enhancement. It collects voice commands and natural dialogues from the driver and occupants, providing a source for the VLA's language modal input; it also assists in recognizing audio semantics such as children crying and heated conversations. It connects to the domain controller via a dedicated audio interface, aligning with the video stream in time.

[0037] The cockpit-driver integrated domain controller subsystem serves as the system's computing and decision-making center. It employs an integrated solution of One Box / One Chip or One Box / Two Boards to achieve unified scheduling of cockpit-driver computing power. Its hardware architecture, from top to bottom, includes: a gateway and time synchronization layer, a main control chip layer, and a memory and storage layer. The gateway (time synchronization) comprises a communication gateway and time synchronization unit located at the top level of the domain controller. It aggregates multiple data streams from the CAN bus and Ethernet, handling cross-domain data routing. It establishes a unified time reference using mechanisms such as PTP / GPS second pulse, aligning data from external cameras, internal DMS / OMS, chassis bus, and navigation host on a unified timeline. This ensures millisecond-level synchronization between external video (e.g., 30fps) and internal DMS (e.g., 60fps), guaranteeing accurate cross-view attention calculation from "driver's line of sight to road target." One side connects to external buses such as CAN and Ethernet, while the other side sends data to the main control chip via an internal bus. The main control chip is a high-performance automotive SoC, with options including NVIDIA Thor (approximately 2000 TOPS), Qualcomm Snapdragon Ride Flex, Black Sesame Wudang C1200 series, and Huawei Ascend 610, integrating multiple computing and acceleration units such as CPU, GPU / NPU, and ISP. The CPU handles general control and task scheduling; the GPU / NPU performs AI calculations such as VLA model inference, multimodal encoding, and BEV perception; and the ISP performs image signal preprocessing. These three components work together to achieve joint inference for driving control and cockpit services. An internal high-speed bus connects memory, non-volatile storage, and various sensor interfaces, supporting zero-copy data sharing. Memory uses high-speed memory such as LPDDR5X, with a typical capacity of 16-32GB (expandable to higher specifications for higher-end configurations). It meets the real-time processing requirements of VLA model parameter loading, visual feature caching, and multi-channel video streams, providing a unified data read / write space for the CPU / GPU / NPU, supporting 10-second-level cache reuse of BEV feature tokens to avoid redundant encoding. Non-volatile storage exists locally in the form of SSD / eMMC / UFS. Stores operating system, model weights, configuration parameters, and event-level multimodal data (for shadow mode acquisition and simulation backfeedback).

[0038] The navigation and positioning module provides vehicle position, attitude, and route planning information. It interconnects with the cockpit-driver integrated domain controller via a CAN bus, providing positioning and map input for intelligent driving control and VLA scene understanding. The GPS / GNSS receiver uses a high-precision GNSS receiver supporting multiple systems and frequencies, including BDS, GPS, GLONASS, and Galileo. It provides absolute position and time references, offering positioning input for the control module and map matching; centimeter-level positioning accuracy is achieved when RTK mode is supported. An optional high-precision integrated navigation module can be integrated, typically combining GNSS and IMU, meeting ASIL B functional safety levels. In scenarios with weak or transient GNSS signal loss, positioning continuity is maintained through dead reckoning; outputting 100-500Hz high-frequency positioning and attitude data. The inertial navigation system (INS) is an inertial measurement unit with built-in gyroscopes and accelerometers, which can be tightly or loosely coupled with GNSS. It provides state variables such as vehicle attitude, angular velocity, and acceleration, supporting motion compensation for BEV perception and state estimation for planning and control algorithms; it maintains short-term positioning accuracy in GNSS-constrained scenarios such as tunnels and under viaducts. The onboard navigation unit is a navigation ECU that carries map data and path planning. It provides map matching results, path planning, road semantics (speed limits, lane types, etc.), and point of interest information, providing navigation-level input to the VLA and planning and control modules. It communicates with the domain controller via CAN or Ethernet, and the positioning and navigation data are fused with the perception data under a unified timestamp.

[0039] Chassis sensors collect dynamic information about the chassis and vehicle body, which is then connected to the integrated cockpit-driver domain controller via the CAN bus, providing real-time vehicle status input for control algorithms and safety verification. Wheel speed sensors provide four-wheel speeds for vehicle speed estimation and slip ratio calculation. ESP status signals provide the electronic stability system's operating status for safety decisions and braking coordination. Pedal sensors collect accelerator and brake pedal openings for driver intent recognition and intervention decisions. Steering angle sensors collect steering wheel angles for trajectory tracking and driver behavior analysis. Accelerometers provide longitudinal / lateral acceleration for vehicle dynamics estimation and comfort control. Vehicle speed sensors provide longitudinal vehicle speed, which is fused with wheel speeds and GNSS data to form a reliable vehicle speed estimate. All these signals are converged to the domain controller gateway via the CAN / CAN FD bus, aligning with navigation and perception data on a unified time base.

[0040] Environmental sensors collect information about the external environment and connect to the domain controller via the CAN bus, providing auxiliary input for VLA scene understanding and cabin comfort control. Rain sensors detect rainfall intensity, triggering wiper logic and providing "severe weather" scene labels for VLA. Light sensors detect ambient light intensity for automatic headlights, HUD brightness adjustment, and day / night scene recognition. Temperature sensors collect outside temperature for air conditioning control and thermal management strategies. Humidity sensors collect outside humidity, supporting defogging and predictive comfort control. These data, connected to the domain controller via CAN, along with internal and external sensing data, constitute scene labels, supporting VLA cross-domain strategy orchestration.

[0041] The human-machine interface (HMI) input subsystem includes touchscreen controls and physical buttons such as the central control screen and steering wheel buttons, providing interactive input in addition to voice input. It connects to the domain controller via CAN or a dedicated interface, serving as the language / command modality source for VLA along with voice input.

[0042] The CAN / CAN FD bus of the in-vehicle communication and time synchronization subsystem carries chassis signals (wheel speed, steering angle, pedals, ESP status, etc.), body control signals, navigation and positioning data, and environmental sensor information, connecting the navigation and positioning module, chassis sensors, environmental sensors, and domain controller. Ethernet carries external camera video streams, in-vehicle DMS / OMS video streams, millimeter-wave / ultrasonic / LiDAR data, and high-bandwidth map data, connecting the external sensor subsystem, in-vehicle sensor subsystem, and the integrated cockpit-driver domain controller. The gateway and time synchronization utilize a gateway module within the domain controller to aggregate CAN and Ethernet data, using PTP / GPS second pulses to establish a unified time reference, ensuring millisecond-level alignment of multi-source data. A zero-copy architecture is optional; camera RAW data directly enters unified video memory via a PCIe switch. The VLA model simultaneously processes features required by the intelligent driving branch and the cockpit branch, sharing an intermediate representation to avoid redundant ISP processing and cross-domain copying, reducing latency and bandwidth consumption.

[0043] The hardware system is divided into two data paths based on communication topology: Ethernet side – external sensors (surround view camera, millimeter-wave radar, ultrasonic radar, lidar) and internal sensors (DMS, OMS, microphone array) send high-bandwidth video and point cloud data to the domain controller via Ethernet; CAN side – navigation and positioning modules, chassis sensors, and environmental sensors send vehicle status and positioning data to the domain controller via CAN bus. The gateway and time synchronization module within the domain controller establish a unified time base, completing multi-source data fusion under a unified timestamp. The real-time AD system (BEV / OCC + regulatory control) is responsible for 50Hz-level driving control output, which is sent to the chassis via CAN; the VLA cognitive center is responsible for upper-level planning decisions (input to regulatory control) and cockpit service commands (driving HMI, air conditioning, seats, etc. via internal bus). Through a One Box / One Chip or One Box / Two Boards integrated cockpit-driver solution, hardware costs are reduced, data paths are simplified, and real-time performance is improved.

[0044] The aforementioned hardware system provides a unified data path and computing platform for the software system. Based on this, the software system performs multi-source data acquisition, multimodal coding, VLA cognitive reasoning, and dual-domain application output, ultimately driving the vehicle to execute commands after functional safety verification.

[0045] like Figure 2 As shown, the software system runs on the cockpit-driver integrated domain controller, adopting a pipeline architecture of data input → multi-source synchronization → multimodal coding → VLA cognitive center → dual-domain application → functional safety → vehicle execution. It is divided into five layers: data input layer, data preprocessing and synchronization layer, perception and cognition layer, application service layer, and safety and execution layer. The core innovation of this invention lies in achieving cockpit-driver integrated decision-making through multimodal coding and a unified cognitive center, under the division of labor of the "real-time AD foundation layer (BEV / OCC and other end-to-end systems) + VLA cognitive enhancement layer". The real-time autonomous driving foundation layer consists of end-to-end or quasi-end-to-end systems such as BEV (bird's-eye view) perception and OCC (occupancy network) and control algorithms, outputting vehicle control commands at a high refresh rate (e.g., 50Hz) to meet safety and real-time requirements; this layer is a mature solution, and this invention does not replace its function. The VLA cognitive enhancement layer runs on top of the real-time AD, performing multi-step reasoning and semantic understanding at a lower frequency (e.g., 1-5Hz) to output upper-level planning decisions and cockpit service commands. The VLA output serves as the policy input or parameter modulation for the real-time AD, enhancing rather than replacing its capabilities. The real-time AD ensures the reliability and timeliness of the underlying control; VLA provides intelligent enhancements in complex scenarios, natural language interaction, and cockpit-driver linkage.

[0046] In some embodiments of the present invention, the external sensor group and the internal sensor group are connected to the integrated cockpit-rider domain controller via an Ethernet interface; The vehicle bus network is a CAN bus, used to transmit the vehicle status data; The integrated cockpit-pilot domain controller includes a time synchronization unit, which uses PTP or GPS absolute time as the system time reference to stamp data accessed through different interfaces with a unified timestamp.

[0047] This embodiment employs a multi-source data acquisition and time synchronization module to aggregate data from various sensors in the hardware system, corresponding to the data input layer in the software framework diagram. The acquisition scope covers the external sensor subsystem, the internal sensor subsystem, the navigation and positioning module, chassis sensors, and environmental sensors, providing multi-source input under a unified time reference for subsequent multimodal coding and VLA cognition. In the data acquisition function, external video acquisition obtains continuous video streams from forward, side, and rear-view cameras, supporting formats such as RAW, YUV, and RGB. After preprocessing by the ISP, the streams are sent to the multimodal encoder. Internal video acquisition obtains driver and passenger video streams from the DMS and OMS, aligning them with the external video under a unified time reference. Voice acquisition obtains voice streams from the microphone array, converting them to text through speech recognition or directly extracting voice features for injection into VLA. Chassis and body signal acquisition uses CAN to obtain wheel speed, vehicle speed, acceleration, steering wheel angle, pedal opening, ESP status, etc., from chassis sensors for motion decoding and safety verification. Navigation and positioning data acquisition obtains GNSS / INS fusion positioning, map matching results, path planning, and road semantics (speed limits, lane types, etc.) from the navigation and positioning module. Environmental signal acquisition uses CAN to acquire rainfall, light intensity, temperature, humidity, etc., from environmental sensors for VLA scene label generation and cockpit comfort control. The time synchronization mechanism uses PTP or GPS absolute time as the system time reference, and each data source is stamped with a unified timestamp upon generation. Considering sensor acquisition and transmission delays, time offset compensation is performed, and a cross-modal time index is established. A multimodal circular buffer is maintained, and data is sent to subsequent modules after being aligned by timestamp, supporting event-driven backtracking recording.

[0048] In some embodiments of the present invention, the multimodal coding module includes: A unified visual coding unit is used to encode the road scene video stream captured by the external camera and the driver monitoring video stream captured by the internal camera to generate a visual token sequence; The language encoding unit is used to encode language data in text form and generate a sequence of language tokens; The audio encoding unit is used to encode the audio data collected by the in-cabin microphone array and generate audio feature tokens.

[0049] The unified visual coding unit is also configured as follows: Based on the driver monitoring video stream, estimate the driver's gaze direction vector; Using the gaze direction vector as the query vector and the road target features obtained by encoding the road scene video stream as the key and value vectors, cross-view attention calculation is performed to obtain the driver's attention confidence for a specific road target.

[0050] In this embodiment, after multi-source data is synchronized in time, the modal coding layer is processed by the visual coding module, language coding module, and audio coding module respectively, and outputs a unified feature representation, which is then injected into the VLA cognitive center. This layer realizes parallel encoding and feature alignment of multi-modal data, providing input for subsequent joint inference. The visual coding module converts the external and internal video streams into visual token sequences, breaking the traditional separation of "forward-facing cameras only look at the road and DMS only looks at people". The unified ViT encoder uses shared patch embedding to process the forward-facing driving video stream (which can be converted to BEV perspective or retain perspective view) and the internal personnel video stream (3D facial + posture features) simultaneously, outputting a fused token sequence, such as [CLS_road][ROI_driver][Token_t1][Token_t2]..., forming a unified token space. The cross-view attention mechanism uses the driver's gaze vector as the query and the road target embedding as the key / value, calculates cross-attention, and outputs the confidence score of "whether the driver has noticed a dangerous target", supporting active safety guided by gaze. For example, when a "ghost peek" (a sudden, unexpected movement of pedestrians) is detected, a slight warning is issued if the driver's gaze is already in that direction; if the gaze is on a mobile phone or to the side, a strong intervention is initiated, including emergency braking, a red flashing HUD, and seat vibration. The final output is a structured environmental perception result (targets, lane lines, drivable areas, etc.) and cabin status (fatigue level, gaze level, occupant activity, etc.), for use by the VLA cognitive center and the real-time AD system. The language encoding module converts driver voice commands, touch inputs, and system announcements into language tokens, providing the VLA with semantic representations of intent and commands. It employs a large language model tokenizer or a dedicated language encoder to support semantic parsing and embedding of natural language commands (such as "There's a traffic jam ahead, I'm a little carsick"). The final output is a sequence of language tokens, which, along with visual and audio tokens, undergo joint attention calculation within the VLA. The audio encoding module processes the speech stream collected by the microphone array and environmental audio, extracting speech features or acoustic event features (such as a child crying or a heated conversation), providing the VLA with audio modal input. Text can be extracted using speech recognition models such as Whisper, or audio embeddings can be directly extracted using acoustic encoders such as HuberT; the timestamps are aligned with the vision and language modules. The final output is an audio feature token or speech-to-text, which participates in VLA multimodal inference. This embodiment persistently stores multimodal data and system outputs, supporting shadow mode acquisition, simulation backfeeding, and model iteration. The stored content includes a four-dimensional spatiotemporal data structure with dimensions such as exterior_video, interior_video, language, and action; event-level corner case data; and SFT samples such as user takeover and language feedback.The data storage module is located near the coding layer and cockpit services, providing local caching and upload interfaces for end-to-end data closure.

[0051] The real-time autonomous driving system (ARAS) serves as the real-time execution layer for vehicle control, employing end-to-end or quasi-end-to-end perception and control architectures such as BEV (Bird-of-Eye View) and OCC (Occupancy Network), independent of VLA's inference speed. BEV perception converts multi-camera inputs into a bird-of-eye view representation, outputting object detection, lane lines, and drivable areas; optional fusion with LiDAR point clouds enhances 3D perception. The OCC predicts a 3D spatial occupancy grid, identifying unknown obstacles and temporary construction areas, compensating for the limitations of traditional object detection in recognizing objects outside the training set. Based on BEV / OCC perception results and planning targets, the control module outputs low-level control commands such as steering wheel angle and throttle / brake inputs at frequencies of 50Hz and above. Emergency safety responses (such as AEB) are implemented using rules or lightweight networks, with a trigger latency of less than 50ms, bypassing VLA to ensure safety redundancy. Upper-level decisions from VLA (such as driving mode, strategy parameters, and planning preferences) are input to the control module, which converts them into executable trajectories and control commands. VLA enhances the decision-making capabilities of real-time AD but does not replace its control execution.

[0052] In some embodiments of the present invention, the VLA cognitive center is configured as follows: The generated upper-level decision-making instructions on the driver's side are sent to the planning and control module of the real-time autonomous driving system in the form of strategy parameters, which are used to modulate the underlying vehicle control instructions generated by the module. The generated cockpit-side service control commands are sent directly to the corresponding cockpit actuators in a format that includes natural language text and structured data.

[0053] The VLA cognitive center is further configured as follows: Based on the encoding results of the external visual data by the multimodal coding module, semantic information of the external environment is identified, and corresponding cockpit-side service control commands are generated based on the semantic information; and, Based on the encoding results of the in-cabin visual data by the multimodal coding module, the status information of the in-cabin personnel is identified, and the corresponding upper-level decision-making instructions on the driver's side are generated based on the status information.

[0054] The driver-side upper-level decision-making instructions include at least one of the following: Driving mode commands, following distance parameters, speed limit parameters, and strategies for navigating complex scenarios.

[0055] In this embodiment, the VLA cognitive center is based on a lightweight VLA model of MoE (Hybrid Expert), with approximately 7B-13B parameters, adapted to in-vehicle computing power. Due to the limited inference speed of VLA, it does not directly output 50Hz-level driving control, but instead outputs upper-level planning decisions and cockpit service commands. Vision-language alignment adopts architectures such as Q-Former, compressing visual features into a fixed number of Query Tokens (e.g., 32) and injecting them into a large language model to achieve a unified representation of vision and language. The pre-training data for embedding driving and cockpit knowledge includes driving scene video-subtitle pairs, cockpit interaction logs, vehicle control signal-language description pairs, and joint cockpit / interior scene descriptions (e.g., "rainy night + child falling asleep + driver nervousness"). This embodiment coordinates VLA output with real-time AD. On the driver's side, VLA outputs upper-level decisions, such as driving mode (conservative / stable / defensive), strategy parameters (speed limit, following distance, lane change permission), and complex scenario solutions (such as narrow road passing strategies and semantic-level parking preferences). The above outputs are input to the real-time AD's control module, where they are converted into 50Hz lower-level control. On the cockpit side, VLA directly outputs 5Hz-level cockpit services (HUD content, air conditioning, seats, music, etc.), which can be handled by VLA due to its lower refresh rate requirements. This embodiment uses cross-domain scenario understanding to uniformly encode "external + internal + language" into scenario labels, such as [severe weather], [night], [child passenger], [driver nervousness]. Based on these, a joint strategy between the driving and cockpit domains is generated to ensure behavioral consistency.

[0056] This embodiment employs dual-domain action decoding and execution. The VLA outputs upper-level driving decisions and cockpit control, while the lower-level driving control is executed by real-time AD, achieving integrated cockpit-driver collaboration. The dual-domain actions are shown in Table 1.

[0057] Table 1

[0058] Taking the driver-cabin linkage as an example, the input is that the user says, "There's a traffic jam ahead, and I'm feeling a bit carsick." The VLA output (driver's side) makes a higher-level decision—limiting the speed to 20km / h, increasing the following distance, and using a smooth braking curve; this decision input is controlled by real-time AD, generating a 50Hz lower-level control and executing it. The VLA output (cabin side) turns on the seat ventilation (level 3), plays soothing music, and the HUD displays "Anti-motion sickness mode switched."

[0059] A typical example of how external perception empowers the cockpit is a scenario-based AR-HUD. When it recognizes "construction ahead + irregular cones," the HUD generates a natural language prompt: "Road construction 150 meters ahead, we suggest changing lanes to the left. A safe distance has been provided for you." When the intelligent driving recorder recognizes "stunning sunset / rainbow" in a congested area, the central control panel pops up: "Beautiful scenery detected, want to take a picture and share?" When the parking environment visualization recognizes "parking lock in the raised state," it announces in voice: "Parking space parking lock not lowered, we have found a nearby empty space for you." The sentry mode has been upgraded to trigger ambient light flashing + horn warning when it recognizes "person approaching and trying to open the car door," and pushes an alarm video with voice explanation to the owner's APP.

[0060] A typical output of internal perception-enabled driving is physiological state adaptation. When the DMS detects a fatigue level >0.7, it switches to conservative mode (reduced speed limit, increased following distance, and prohibition of lane changes), and the cabin activates alertness mode. Passenger comfort prediction execution, based on OMS identification of passengers drinking hot water / using a laptop, employs a gentle deceleration strategy to avoid sudden braking. Emergency health takeover execution, in conjunction with DMS and OMS, activates MRC (deceleration, pull over, hazard lights) when the driver is deemed incapacitated, and the cabin automatically dials 120 (emergency services) and announces the driver's location.

[0061] In some embodiments of the present invention, the functional safety module is configured to perform a three-level safety fence, including a Constitutional AI constraint verification at the model layer, a responsibility-sensitive safety RSS verification at the rule layer, and an independent monitoring controller verification at the hardware layer.

[0062] The functional safety module is located between the real-time autonomous driving system and the cockpit services and vehicle execution, serving as a unified verification checkpoint before execution to ensure that driving and cockpit commands are issued under safety constraints. All control outputs must undergo functional safety verification before driving the vehicle. It employs a three-level safety fence, including: 1. Model layer: Constitutional AI constrains VLA outputs, prohibiting the generation of dangerous upper-level decisions (such as illegal lane change suggestions). 2. Rule layer: The traditional RSS (Responsibility Sensitive Safety) model serves as a fallback; the final control output of the real-time AD system must pass RSS verification before being sent to the chassis; the decision parameters input from VLA to the rule control must also meet safety constraints. 3. Hardware layer: Independent MCU monitoring; when the difference between the real-time AD output and chassis feedback exceeds a threshold, MRC (Minimum Risk State) is triggered.

[0063] The outputs of real-time AD and cabin services converge to the functional safety module. After passing the three-level fence verification, they are distributed to chassis execution (steering, braking, throttle, etc.) and cabin execution (HMI, air conditioning, seats, etc.) respectively, forming a closed loop of application services → functional safety → vehicle execution.

[0064] In some embodiments of the present invention, the system of the present invention further includes an end-to-end data closed-loop module, used to iterate the VLA cognitive center and / or the real-time autonomous driving system by collecting multimodal driving data and generating training data; The multimodal driving data includes at least: external video data, internal video data, voice data, and control sequence data containing vehicle control actions and cockpit control actions; The end-to-end data closed-loop module is configured to perform at least one of the following operations: Shadow mode data acquisition: After detecting that the driver has taken over the vehicle, the system automatically records multimodal driving data before and after the takeover, as well as the driver's feedback information, as data for supervision and fine-tuning. Simulation refeedback: Reconstruct driving scenarios based on collected multimodal driving data and generate training data for rare scenarios.

[0065] The end-to-end data closed-loop module employs a four-dimensional spatiotemporal data flywheel. The data structure includes exterior_video (forward continuous video), interior_video (cabin occupant status), language (driver's voice / system announcements), and action (including vehicle and cockpit control sequences), which are persistently stored by the data storage module. It can perform shadow mode acquisition: after each user takeover, it automatically records "voice feedback + takeover action" as SFT data and writes it to the data storage. Simulation backfeedback: using NeRF and other methods to reconstruct the cabin and interior / exterior scenes, it generates corner case training data in batches to support continuous iteration of VLA and BEV / OCC models.

[0066] In some embodiments of the present invention, the system further includes a real-time optimization module. During model distillation, the VLA teacher model (13B) is distilled into a student model (approximately 1.8B) to adapt for local operation on the cockpit-driver integrated chip, meeting the inference latency requirements of upper-level planning decisions and cockpit services. During speculative decoding: the VLA's cockpit UI generation uses a draft model, the main model is used for inference in complex scenarios, and parallel inference is used to improve VLA throughput; the 50Hz control at the driving level is independently handled by the real-time AD, without relying on the VLA inference speed. During token caching, the BEV feature tokens of the driving scenario are reused within 10 seconds to avoid redundant encoding and reduce the latency of shared features between the VLA and the real-time AD.

[0067] In one possible implementation, the system of the present invention further includes a privacy compliance processing module, configured to: Before the multimodal coding module encodes the in-cabin visual data, the data undergoes real-time desensitization processing to remove or blur biometric information that can identify an individual, and only the desensitized non-sensitive feature data is used for subsequent processing or uploading; and / or, The model is updated in the cloud using a federated learning approach, based on the model update parameters obtained after processing external visual data on the vehicle side. The model of the VLA cognitive center and / or the real-time autonomous driving system is then aggregated and updated in the cloud without uploading the original video data.

[0068] The privacy compliance processing module is used for real-time desensitization (facial blurring) of in-cabin videos at the edge, extracting only pose tokens for cloud training. External camera data updates the model through federated learning; the original video does not leave the vehicle. It supports data collection and storage strategies that comply with GDPR, personal information protection laws, and other requirements.

[0069] The typical workflow of the system during vehicle operation follows the pipeline structure of the software framework diagram, forming a complete closed loop from data input to vehicle execution: Step S1: Multi-source data acquisition and synchronization (data input layer → preprocessing and synchronization layer) - Continuously acquire data from external cameras, millimeter-wave / ultrasound / LiDAR, internal DMS / OMS, microphone array, chassis sensors, navigation and positioning modules, environmental sensors, etc.; - The multi-source data acquisition and time synchronization module assigns a unified timestamp to each data source, writes it to a circular cache, and establishes a cross-modal time index.

[0070] Step S2: Multimodal Coding (Perception and Cognition Layer Front End) - The visual coding module processes external and internal video, outputs a fusion token, and calculates the correlation between the driver's line of sight and road targets across perspectives; - The language coding module and audio coding module process voice / touch commands and environmental audio respectively, outputting language and audio tokens; - The coding results are integrated with the data storage module, supporting shadow mode and event-level playback.

[0071] Step S3: VLA Cognitive Reasoning (Core of Perception and Cognition Layer) - Inject visual, linguistic, and audio tokens into the VLA cognitive center; - Perform multimodal joint reasoning based on scene tags (weather, occupant status, driver status, etc.) to generate a joint strategy for driving and the cockpit.

[0072] Step S4: Dual-Domain Application Output (Application Service Layer) - The real-time AD system (BEV / OCC + regulatory control) outputs 50Hz low-level driving control (steering, throttle, braking, etc.); - The VLA cognitive center outputs upper-level planning and decision-making (1-5Hz) to the regulatory control module, enhancing its decision-making capabilities; it also outputs 5Hz cockpit services (HUD, air conditioning, seats, music, etc.); - The driver and cockpit coordinate and execute under a unified scenario understanding to avoid conflicts.

[0073] Step S5: Functional Safety Verification and Vehicle Execution (Safety and Execution Layer) - Driver and cockpit outputs converge to the functional safety module and are verified through a three-level safety fence (model layer, rule layer, hardware layer); - After verification, driver control is sent to the chassis for execution, and the cockpit service drives the HMI and vehicle comfort components; - An independent MCU continuously monitors and triggers MRC in case of anomalies.

[0074] Step S6: Data Closure (Optional) - Record takeover events and user feedback in shadow mode, write them to data storage, and use them for model iteration; - High-quality event data is uploaded to the cloud after being de-identified and participates in the next round of training.

[0075] Through the above process, this invention achieves end-to-end integrated cockpit-driver decision-making, which integrates the "driving brain" and the "cockpit brain" into "an all-knowing brain".

[0076] Based on the aforementioned hardware and software system design, the core technical points of this invention can be summarized as follows, forming a complete technical system: Technical Point 1: VLA Enhances the Layered Collaborative Architecture of Real-Time AD - The bottom layer is the real-time autonomous driving system (end-to-end perception + control such as BEV / OCC), outputting 50Hz-level driving control; the upper layer is the VLA cognitive enhancement layer, outputting upper-level planning decisions and cockpit services; - VLA does not replace real-time AD, but enhances the existing vehicle's intelligent driving capabilities through complex scenario solutions and intelligent planning decisions; - Multi-source data is encoded in multiple modalities and then jointly inferred by VLA to output policy parameters and cockpit service commands that drive real-time AD, breaking down the traditional cockpit-driving domain barriers.

[0077] Technical Point 2: Multimodal Coding and Spatiotemporal Joint Perception - Visual, linguistic, and audio coding modules are processed in parallel, outputting a unified token representation that is injected into the VLA cognitive center; - A unified ViT encoder simultaneously processes forward-facing driving video and in-cabin occupant video, using a cross-view attention mechanism with the driver's gaze as the query and road targets as the key / value, to achieve online inference of "whether the driver has noticed a target"; - Supports active safety with gaze guidance and joint understanding of semantics inside and outside the cabin.

[0078] Technical Point 3: Dual-Domain Action Decoding and Cockpit-Driver Interaction - The driving-level control (50Hz) is output by the real-time AD system (BEV + regulatory control), and the VLA outputs upper-level decisions (mode, strategy parameters, complex scenario solutions) to the regulatory control, which then transforms them into executable control; - Cockpit services (5Hz) are directly output by the VLA using LLM Text + JSON; - Under a unified scenario understanding, the driving and cockpit outputs are coordinated to achieve semantic-level cockpit-driver interaction (e.g., "traffic jam + motion sickness" → VLA outputs deceleration strategy + ventilation + music, and the regulatory control executes the underlying control).

[0079] Technical Point 4: External Perception Empowers the Cockpit and Internal Perception Empowers Driving - External camera semantics (construction, scenery, parking locks, etc.) drive the cabin experience, including AR-HUD, intelligent capture, and parking prompts; - Internal DMS / OMS semantics (fatigue, mood, occupant activity) drive adaptive driving style, predictive control of passenger comfort, and emergency health takeover; - Cross-domain strategy orchestration is achieved through a unified semantic space of VLA.

[0080] Technical Point 5: Topology-based Access and Zero-Copy Hardware Architecture - Access is grouped by communication method: CAN carries navigation and positioning, chassis and environmental sensors; Ethernet carries high-bandwidth perception data such as internal and external vision and radar; - The gateway and time synchronization module in the domain controller establish a unified time base, and the external and internal videos are aligned under a unified timestamp, supporting cross-view attention and event-level data recording; - Raw sensor data is directly supplied to unified video memory (optional) via PCIe Switch, and VLA simultaneously processes intelligent driving and cockpit branches, sharing intermediate features.

[0081] Technical Point 6: Functional Safety Module and Three-Level Safety Fence - The functional safety module serves as a unified verification gateway between application services and vehicle execution; both driver and cockpit outputs must be verified by it before being issued. - The model layer (Constitutional AI), rule layer (RSS), and hardware layer (independent MCU) form a three-level safety fence. - Driver 50Hz control is independently handled by real-time AD, without relying on VLA inference speed. VLA meets the latency requirements of upper-level decision-making (1-5Hz) and cockpit services (5Hz) through technologies such as model distillation, speculative decoding, and token caching.

[0082] Technical Point 7: Four-Dimensional Spatiotemporal Data Closed Loop - Constructing a VLA training data structure containing exterior_video, interior_video, language, and action; - Shadow mode acquisition of user takeover and language feedback, simulation backfeeding to generate corner cases, supporting end-to-end data flywheel and continuous learning.

[0083] Technical Point 8: Privacy and Compliance of In-Vehicle Data Processing - In-vehicle videos are anonymized at the edge, with only non-sensitive tokens such as gestures extracted and uploaded; - External data is updated through federated learning to update the model, and the original video does not leave the vehicle, meeting privacy and regulatory requirements.

[0084] In summary, the VLA-based cockpit-driver integrated intelligent driving system proposed in this invention adopts a layered architecture of "real-time AD base layer (end-to-end system such as BEV / OCC) + VLA cognitive enhancement layer." Through multimodal coding (visual / language / audio), a unified cognitive center, and a functional safety module, it achieves an end-to-end pipeline from data input → multi-source synchronization → multimodal coding → VLA cognition → dual-domain application → functional safety → vehicle execution. VLA does not replace existing autonomous driving; rather, it elevates the existing vehicle's intelligent driving capabilities through complex scenario solutions and intelligent upper-level planning and decision-making. It evolves the vehicle from a dual-domain architecture of "driver brain + cockpit brain" into a collaborative intelligent agent integrating perception, planning, and cockpit-driver interaction, providing a feasible and scalable technical path for the next generation of intelligent vehicles.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0089] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A cockpit-driver integrated intelligent driving system based on a VLA model, characterized in that, include: The cockpit-driver integrated domain controller, the external sensor group connected to the cockpit-driver integrated domain controller, the internal sensor group, and the vehicle bus network; The cockpit-rider integrated domain controller is configured to execute: The multimodal coding module is used to perform time synchronization and multimodal coding on multi-source data from the external sensor group, the internal sensor group and the vehicle bus network to obtain a unified feature representation; The VLA cognitive center is used to receive the unified feature representation and perform multimodal joint reasoning to generate upper-level decision-making instructions on the driver's side and service control instructions on the cockpit side. The functional safety module is used to perform safety verification on the driver-side upper-level decision-making commands and the cockpit-side service control commands; The driver-side upper-level decision-making instructions are sent to the real-time autonomous driving system after verification to modulate its control strategy. The cockpit-side service control commands are sent to the cockpit actuators after verification.

2. The cockpit-driver integrated intelligent driving system based on the VLA model according to claim 1, characterized in that, The multimodal coding module includes: A unified visual coding unit is used to encode the road scene video stream captured by the external camera and the driver monitoring video stream captured by the internal camera to generate a visual token sequence; The language encoding unit is used to encode language data in text form and generate a sequence of language tokens; The audio encoding unit is used to encode the audio data collected by the in-cabin microphone array and generate audio feature tokens.

3. The cockpit-driver integrated intelligent driving system based on the VLA model according to claim 2, characterized in that, The unified visual coding unit is also configured as follows: Based on the driver monitoring video stream, estimate the driver's gaze direction vector; Using the gaze direction vector as the query vector and the road target features obtained by encoding the road scene video stream as the key and value vectors, cross-view attention calculation is performed to obtain the driver's attention confidence for a specific road target.

4. The cockpit-driver integrated intelligent driving system based on a VLA model according to claim 1, characterized in that, The driver-side upper-level decision-making instructions include at least one of the following: Driving mode commands, following distance parameters, speed limit parameters, and strategies for navigating complex scenarios.

5. The cockpit-driver integrated intelligent driving system based on a VLA model according to claim 1, characterized in that, The functional safety module is configured to perform a three-level safety fence, including ConstitutionalAI constraint verification at the model layer, Responsibility-Sensitive Safety RSS verification at the rule layer, and Independent Monitoring Controller verification at the hardware layer.

6. The cockpit-driver integrated intelligent driving system based on a VLA model according to claim 1, characterized in that, The external sensor group and the internal sensor group are connected to the integrated cockpit-driving domain controller via an Ethernet interface; The vehicle bus network is a CAN bus, used to transmit the vehicle status data; The integrated cockpit-pilot domain controller includes a time synchronization unit, which uses PTP or GPS absolute time as the system time reference to stamp data accessed through different interfaces with a unified timestamp.

7. The cockpit-driver integrated intelligent driving system based on a VLA model according to claim 1, characterized in that, The VLA cognitive center is configured as follows: The generated upper-level decision-making instructions on the driver's side are sent to the planning and control module of the real-time autonomous driving system in the form of strategy parameters, which are used to modulate the underlying vehicle control instructions generated by the module. The generated cockpit-side service control commands are sent directly to the corresponding cockpit actuators in a format that includes natural language text and structured data.

8. The cockpit-driver integrated intelligent driving system based on the VLA model according to claim 7, characterized in that, The VLA cognitive center is further configured as follows: Based on the encoding results of the external visual data by the multimodal coding module, the semantic information of the external environment is identified, and the corresponding cockpit-side service control commands are generated based on the semantic information. as well as, Based on the encoding results of the in-cabin visual data by the multimodal coding module, the status information of the in-cabin personnel is identified, and the corresponding upper-level decision-making instructions on the driver's side are generated based on the status information.

9. The cockpit-driver integrated intelligent driving system based on the VLA model according to claim 1, characterized in that, It also includes an end-to-end data closed-loop module, used to iterate the VLA cognitive center and / or the real-time autonomous driving system by collecting multimodal driving data and generating training data; The multimodal driving data includes at least: external video data, internal video data, voice data, and control sequence data containing vehicle control actions and cockpit control actions; The end-to-end data closed-loop module is configured to perform at least one of the following operations: Shadow mode data acquisition: After detecting that the driver has taken over the vehicle, the system automatically records multimodal driving data before and after the takeover, as well as the driver's feedback information, as data for supervision and fine-tuning. Simulation refeedback: Reconstruct driving scenarios based on collected multimodal driving data and generate training data for rare scenarios.

10. The cockpit-driver integrated intelligent driving system based on a VLA model according to claim 1, characterized in that, It also includes a privacy compliance processing module, which is configured as follows: Before the multimodal coding module encodes the in-cabin visual data, the data is desensitized in real time to remove or blur biometric information that can identify an individual, and only the non-sensitive feature data obtained after desensitization is used for subsequent processing or uploading. And / or, The model updates the VLA cognitive center and / or the real-time autonomous driving system model in the cloud by using federated learning methods and model update parameters obtained after processing external visual data on the vehicle side, without uploading the original video data.

Images