An automobile intelligent cockpit perception system, method and vehicle

Abstract

This invention provides an intelligent cockpit perception system, method, and vehicle for automobiles. The system includes a multimodal perception module, a data fusion processing module, a decision control module, a feedback optimization module, a privacy protection module, and a storage module. The multimodal perception module collects raw data on occupant status, environmental parameters, and cockpit equipment status. The data fusion processing module performs noise reduction and normalization on the raw data, dynamically determines weighting coefficients based on the real-time perception accuracy of the sensors, and performs adaptive weighted fusion. The decision control module generates cockpit equipment control commands based on the fused perception results, preset decision rules, and occupant preferences. The feedback optimization module adjusts the sensor perception frequency, weighting coefficients, and fusion algorithm parameters based on equipment operation feedback and occupant operation feedback. The privacy protection module implements differential privacy protection and access control for sensitive data. The storage module stores data from each stage to form a closed loop of perception, fusion, decision-making, execution, feedback, and optimization.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to an intelligent cockpit perception system, method, and vehicle for automobiles. Background Technology

[0002] With the development of automotive intelligence and connectivity, intelligent cockpits are gradually evolving from passive human-machine interaction spaces to comprehensive systems capable of sensing occupant status, cabin environment, and equipment operating status, and proactively providing services. Existing driver monitoring systems typically rely on cameras to identify eyelid closure, gaze direction, or head posture to determine whether the driver is fatigued or distracted; existing cabin environment sensing systems typically collect parameters such as temperature and humidity; and the status of cabin equipment is mainly managed separately by the controllers of each device.

[0003] However, relying solely on visual information is easily affected by lighting conditions, occlusion, and individual driver differences, making it difficult to accurately assess the driver's condition by combining it with physiological signals such as heart rate and EEG signals. Existing environmental perception systems typically have limited dimensions, failing to adequately detect detailed parameters such as particulate matter concentration, harmful gas levels, noise frequency, and light intensity, thus failing to comprehensively reflect the cabin environment.

[0004] Furthermore, data generated by different sensors and cockpit devices are typically processed independently by different control units, lacking a unified preprocessing and fusion mechanism. The different dimensions, sampling frequencies, and noise characteristics of multi-source data can easily lead to inconsistent perception results and a high rate of misjudgment if used directly. Adopting a complex fusion model may exceed the computing power and storage capacity of the onboard computing platform. Occupant physiological data and facial feature data also involve privacy and security concerns; direct transmission or storage poses a risk of sensitive data leakage.

[0005] Existing systems typically lack an effective feedback loop between perception results and cockpit device control. This makes it difficult for the system to dynamically adjust sensor parameters, fusion parameters, and decision rules based on device execution results and subsequent occupant actions, thus hindering the formation of a continuously optimizing closed loop. Therefore, a new automotive intelligent cockpit perception solution is needed that balances perception dimensions, fusion accuracy, onboard computing power, privacy and security, and active control capabilities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide an intelligent cockpit perception system, method, and vehicle for automobiles, enabling multi-dimensional perception of occupant status, environmental parameters, and equipment status, and forming a closed loop through adaptive fusion, decision control, feedback optimization, and privacy protection.

[0007] To achieve the above-mentioned technical effects, the present invention adopts the following technical solution:

[0008] According to a first aspect of the present invention, an intelligent cockpit perception system for automobiles is provided, comprising a multimodal perception module, a data fusion processing module, a decision control module, a feedback optimization module, a privacy protection module, and a storage module;

[0009] The multimodal perception module is used to synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin, and send the raw data to the data fusion processing module.

[0010] The data fusion processing module is used to preprocess the raw data, dynamically determine the weight coefficients based on the real-time sensing accuracy of each sensor, perform adaptive weighted fusion on the preprocessed raw data, and output the fused sensing result.

[0011] The decision control module is used to generate control commands based on the fused perception results, preset decision rules and occupant preference data, and send the control commands to the cabin electrical equipment;

[0012] The feedback optimization module is used to collect equipment operation feedback data and occupant operation feedback data after the cockpit electrical equipment executes the control command, and send the feedback data to the data fusion processing module so that the data fusion processing module can adjust the sensing parameters and fusion parameters.

[0013] The privacy protection module is used to perform privacy protection processing on sensitive data in the original data, the fusion perception results and the feedback data.

[0014] The storage module is used to store the raw data, the fusion perception results, the control commands, and the feedback data, thereby forming a closed-loop perception system of perception, fusion, decision-making, execution, feedback, and optimization.

[0015] Preferably, the multimodal sensing module includes an occupant status sensing unit, an environmental parameter sensing unit, and an equipment status sensing unit; the occupant status sensing unit includes a visual sensor, a physiological electrophysiological sensor, and a millimeter-wave sensor, used to collect at least one of occupant facial features, limb movements, heart rate, electroencephalogram (EEG) signals, sitting posture, and body distance; the environmental parameter sensing unit includes a temperature sensor, a humidity sensor, an air quality sensor, a noise sensor, and a light sensor, used to collect at least one of temperature, humidity, particulate matter concentration, harmful gas content, noise frequency, and light intensity; the equipment status sensing unit communicates with the cabin electrical equipment through a Controller Area Network (CAN) bus interface and / or a Local Interconnect Network (LIN) bus interface.

[0016] Preferably, the preprocessing includes denoising the original data using a Kalman filter algorithm, and mapping the original data of different dimensions to a normalized value using a maximum-minimum normalization method. The interval; the adaptive weighted fusion obtains the fusion sensing result according to the following formula:

[0017] ;

[0018] in, The fusion perception result, For the number of sensors, For the first The weighting coefficients of each sensor For the first Preprocessed data from each sensor.

[0019] Preferred, the first Real-time sensing accuracy of individual sensors Determine according to the following formula:

[0020]

[0021] in, For the first Standard reference data for each sensor, For the first The absolute value of the deviation between the preprocessed data of each sensor and the standard reference data; the weighting coefficients are normalized according to the following formula so that the sum of the weighting coefficients of each sensor is 1:

[0022] .

[0023] Preferably, the data fusion processing module includes a model compression unit, which uses at least one of quantization processing and pruning processing to compress the algorithm model used for data fusion, so as to reduce the computing power and storage resources required by the vehicle computing platform to execute the algorithm model.

[0024] Preferably, the control commands include at least one of the following: warning commands, environmental adjustment commands, equipment control commands, and reminder commands; when the fusion perception results indicate that the driver is fatigued or distracted, the cabin environment parameters exceed a preset threshold, the occupant's sitting posture is improper, or the distance between the occupant and the cabin equipment is less than a preset distance, the decision control module controls at least one of the following to change the operating state: audio system, ambient lighting, air conditioning, air purifier, seats, windows, and display devices.

[0025] Preferably, the equipment operation feedback data includes the actual operating parameters of the cabin electrical equipment after executing the control command, and the occupant operation feedback data includes the operation records of the occupant manually adjusting the cabin electrical equipment; the data fusion processing module adjusts at least one of the sensor sensing frequency, the weighting coefficient, and the fusion algorithm parameters according to the feedback data, and the decision control module corrects the occupant preference data and / or the preset decision rules according to the occupant operation feedback data.

[0026] Preferably, the privacy protection module employs a differential privacy protection algorithm to perturb the occupant's physiological data and / or facial feature data according to the following formula:

[0027]

[0028] in, Sensitive data processed for privacy protection For raw, sensitive data, ε represents the privacy protection budget. The disturbance coefficient is... The storage module includes an in-vehicle embedded multimedia card eMMC storage chip, and the storage period of the storage module is not less than 30 days. The storage module is equipped with Laplace perturbation noise and restricts access to sensitive data through data access permission control.

[0029] According to a second aspect of the present invention, a method for sensing in an intelligent cockpit of an automobile is provided, employing the intelligent cockpit sensing system of the automobile as described above, the method comprising:

[0030] S1: Synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin;

[0031] S2: The original data is denoised and normalized, the weight coefficients are dynamically determined according to the real-time sensing accuracy of each sensor, and the pre-processed original data is adaptively weighted and fused to obtain the fused sensing result.

[0032] S3: Generate control commands based on the fused perception results, preset decision rules, and occupant preference data, and control the cabin electrical equipment to execute the control commands;

[0033] S4: Collect equipment operation feedback data and occupant operation feedback data of the cabin electrical equipment;

[0034] S5: Adjust at least one of the sensor sensing frequency, the weighting coefficient, and the fusion algorithm parameters according to the feedback data, and perform privacy protection processing on the original data, the fusion sensing result, and the sensitive data in the feedback data;

[0035] S6: Store the original data, the fused sensing results, the control commands, and the feedback data, and repeat steps S1 to S5 to form a closed-loop sensing control.

[0036] According to a second aspect of the present invention, a vehicle is provided, including the automotive intelligent cockpit perception system as described above.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention provides an intelligent cockpit perception system, method and vehicle for automobiles, which synchronously collects data such as vision, physiological electricity, millimeter wave, air quality, noise, light and equipment operation status through occupant status perception unit, environmental parameter perception unit and equipment status perception unit, which can reduce the perception blind spot of a single sensor.

[0039] 2. The intelligent cockpit perception system, method and vehicle provided by the present invention perform noise reduction and normalization processing on multi-source data, and dynamically allocate weight coefficients according to the real-time perception accuracy of the sensors, which is beneficial to improving the consistency and reliability of multi-source data fusion.

[0040] 3. The present invention provides an intelligent cockpit perception system, method and vehicle, which adjusts the perception frequency, weight coefficient, fusion algorithm parameters and decision rules through equipment operation feedback data and occupant operation feedback data to form a closed loop of perception, fusion, decision, execution, feedback and optimization.

[0041] 4. The present invention provides an intelligent cockpit perception system, method and vehicle for automobiles, which reduces the computational and storage resource consumption of the algorithm model by quantization, pruning and other model compression methods, and protects the sensitive data of occupants by differential privacy and access control, and is suitable for deployment in vehicle computing platforms.

[0042] 5. The present invention provides an intelligent cockpit perception system, method and vehicle that can proactively generate warning, environmental adjustment, equipment control or reminder commands based on driver status, passenger posture and cockpit environment, thereby improving cockpit safety, comfort and proactive service capabilities. Attached Figure Description

[0043] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0044] Figure 1 This is an architecture diagram of the automotive intelligent cockpit perception system in Example 1;

[0045] Figure 2 This is a flowchart of the intelligent cockpit perception method for automobiles in Example 2. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0048] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, all directional indications (such as up, down, left, right, front, back, bottom, etc.) in this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indication will also change accordingly. Furthermore, descriptions involving "first," "second," etc., in this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0049] Example 1

[0050] This embodiment provides an intelligent cockpit perception system for automobiles, including a multimodal perception module, a data fusion processing module, a decision control module, a feedback optimization module, a privacy protection module, and a storage module;

[0051] The multimodal perception module is used to synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin, and then send the raw data to the data fusion processing module.

[0052] The data fusion processing module is used to preprocess the raw data and dynamically determine the weight coefficients based on the real-time sensing accuracy of each sensor. It then performs adaptive weighted fusion on the preprocessed raw data and outputs the fused sensing results.

[0053] The decision control module is used to generate control commands based on the fusion perception results, preset decision rules and occupant preference data, and send the control commands to the cabin electrical equipment;

[0054] The feedback optimization module is used to collect equipment operation feedback data and crew operation feedback data after the cockpit electrical equipment executes control commands, and send the feedback data to the data fusion processing module so that the data fusion processing module can adjust the sensing parameters and fusion parameters.

[0055] The privacy protection module is used to perform privacy protection processing on sensitive data in raw data, fused perception results, and feedback data;

[0056] The storage module is used to store raw data, fused sensing results, control commands, and feedback data, thereby forming a closed-loop sensing system of perception, fusion, decision-making, execution, feedback, and optimization.

[0057] Furthermore, the multimodal perception module includes an occupant status perception unit, an environmental parameter perception unit, and an equipment status perception unit. The occupant status perception unit includes a visual sensor, a physiological electrical sensor, and a millimeter-wave sensor, used to collect at least one of the following: occupant facial features, limb movements, heart rate, electroencephalogram (EEG) signals, sitting posture, and body distance. The environmental parameter perception unit includes a temperature sensor, a humidity sensor, an air quality sensor, a noise sensor, and a light sensor, used to collect at least one of the following: temperature, humidity, particulate matter concentration, harmful gas content, noise frequency, and light intensity. The equipment status perception unit communicates with the cabin electrical equipment through a CAN bus interface of the controller local area network and / or a LIN bus interface of the local interconnect network.

[0058] Furthermore, preprocessing includes using the Kalman filter algorithm to denoise the original data, and using the maximum-minimum normalization method to map the original data of different dimensions to... Interval; Adaptive weighted fusion obtains the fusion perception result according to the following formula:

[0059] ;

[0060] in, To integrate the perception results, For the number of sensors, For the first The weighting coefficients of each sensor For the first Preprocessed data from each sensor.

[0061] Furthermore, the first Real-time sensing accuracy of individual sensors Determine according to the following formula:

[0062]

[0063] in, For the first Standard reference data for each sensor, For the first The absolute value of the deviation between the preprocessed data of each sensor and the standard reference data; the weighting coefficients are normalized according to the following formula so that the sum of the weighting coefficients of each sensor is 1:

[0064] .

[0065] Furthermore, the data fusion processing module includes a model compression unit, which uses at least one of quantization processing and pruning processing to compress the algorithm model used for data fusion, thereby reducing the computing power and storage resources required by the on-board computing platform to execute the algorithm model.

[0066] Furthermore, the control commands include at least one of the following: warning commands, environmental adjustment commands, equipment control commands, and reminder commands; when the fusion perception results indicate that the driver is fatigued or distracted, the cabin environment parameters exceed the preset threshold, the occupant's sitting posture is improper, or the distance between the occupant and the cabin equipment is less than the preset distance, the decision control module controls at least one of the following to change the operating state: audio system, ambient lighting, air conditioning, air purifier, seats, windows, and display devices.

[0067] Furthermore, the equipment operation feedback data includes the actual operating parameters of the cabin electrical equipment after executing control commands, and the occupant operation feedback data includes the operation records of the occupant manually adjusting the cabin electrical equipment; the data fusion processing module adjusts at least one of the sensor sensing frequency, weight coefficient and fusion algorithm parameters according to the feedback data, and the decision control module corrects the occupant preference data and / or preset decision rules according to the occupant operation feedback data.

[0068] Furthermore, the privacy protection module employs a differential privacy protection algorithm to perturb occupant physiological data and / or facial feature data according to the following formula:

[0069]

[0070] in, Sensitive data processed for privacy protection For raw, sensitive data, ε represents the privacy protection budget. The disturbance coefficient is... The Laplace perturbation noise is controlled, and access to sensitive data is restricted through data access permission control; the storage module includes an in-vehicle embedded multimedia card eMMC storage chip, and the storage cycle of the storage module is not less than 30 days.

[0071] The following is a detailed description of the invention, designed for commercial vehicle (such as trucks and buses) cabin scenarios, to accommodate the characteristics of long driving hours and variable passenger numbers in commercial vehicles:

[0072] The system hardware configuration includes:

[0073] Multimodal perception module: The high-definition vision sensor uses a 2K automotive camera, installed in the center of the cabin ceiling, with a sampling frequency of 25fps, covering the entire cabin area and enabling simultaneous perception of the driver and multiple occupants; the physiological electrical sensor uses wear-resistant electrode sensors integrated into the steering wheel grip and driver's seat cushion, with a sampling frequency of 80Hz, suitable for commercial vehicle drivers who hold the steering wheel for extended periods and maintain a fixed sitting posture; the millimeter-wave sensor uses a 77GHz automotive millimeter-wave radar, installed on the front and rear sides of the cabin, with a detection distance of 0.3-5m, capable of sensing the status of rear-seat occupants; the temperature sensor uses a K-type thermocouple sensor, with a measurement range of -50℃ to 120℃ and an accuracy of ±0.2℃. Adapted to extreme operating conditions in commercial vehicles; the humidity sensor uses a resistive humidity sensor with a measurement range of 0-100%RH and an accuracy of ±3%RH; the air quality sensor adds a dust sensor to detect PM10 concentration, adapting to dusty operating environments in commercial vehicles; the noise sensor uses an industrial-grade electret microphone with a measurement range of 20-130dB, filtering high-frequency external noise such as engine noise; the light sensor uses an anti-strong light photoelectric sensor with a measurement range of 0-20000lux, adapting to outdoor operating scenarios in commercial vehicles; the equipment status sensing unit uses dual interfaces of CAN bus and LIN bus to establish communication with equipment such as air conditioning, heating, and vehicle radio in commercial vehicles to collect equipment operating parameters.

[0074] Data fusion processing module: Adopting an in-vehicle industrial-grade heterogeneous computing platform, integrating CPU, GPU, and NPU, with a computing power of 60 TOPS, it supports stable operation in high temperature and high vibration environments. It has built-in Kalman filter algorithm, adaptive weighted fusion algorithm, and differential privacy protection algorithm, and adds a noise filtering subunit, which can filter external interference noise such as commercial vehicle engine and road conditions.

[0075] Decision control module: It adopts an industrial-grade MCU controller with a main frequency of 1.2GHz, stores preset decision rules and multiple sets of passenger preference data (adapted to drivers and passengers), supports real-time communication with commercial vehicle on-board terminals, has a response time of ≤0.4s, and adds long-term driving warning rules.

[0076] Feedback optimization module: It adopts wear-resistant pressure sensors and voice acquisition unit. The pressure sensors are integrated into the steering wheel, driver's seat and rear seats to collect occupant operation pressure data and sitting posture change data; the voice acquisition unit adopts a 6-microphone array with noise suppression function to collect occupant voice feedback data with a sampling frequency of 20kHz.

[0077] Privacy protection module: integrates an industrial-grade encryption chip, supports AES-256 encryption algorithm and differential privacy algorithm, privacy protection budget ε is set to 0.4, perturbation coefficient λ is set to 0.6, and adds a function to periodically clear sensitive data to avoid privacy risks caused by long-term data storage.

[0078] Storage module: It adopts automotive industrial-grade eMMC storage chip with a capacity of 128GB, a read / write speed of ≥120MB / s, supports stable data storage under high temperature and high vibration environments, the storage period is set to 60 days, and supports data synchronization with the commercial vehicle back-end management system.

[0079] After the system starts up, each sensing unit of the multimodal perception module starts up synchronously. The high-definition vision sensor collects the driver's facial image, body movements and the status of the rear passengers. The physiological electrophysiological sensor collects the driver's heart rate and EEG signals. The millimeter-wave sensor collects the sitting posture and body distance parameters of the driver and the rear passengers. The environmental parameter sensing unit collects environmental parameters such as cabin temperature, humidity, PM2.5, PM10 concentration, harmful gas content, noise, and light intensity. The equipment status sensing unit collects cabin electrical equipment operating parameters such as air conditioning operating power, heating status, and vehicle radio volume through dual interfaces of CAN bus and LIN bus. Subsequently, the multimodal perception module transmits all the collected raw data to the data fusion processing module. This module first removes noise interference from the data using the Kalman filter algorithm, then filters external interference noise such as that from the commercial vehicle engine and road conditions through the noise filtering subunit. Then, it uses the min-max normalization method to uniformly map the raw data of different dimensions and magnitudes to the same interval. Then, it dynamically allocates weight coefficients according to the real-time perception accuracy of each sensor to perform fusion processing on the multi-source data and outputs high-precision fusion perception results. At the same time, it compresses the fusion algorithm through quantization and pruning techniques to reduce computing power consumption and adapt to the computing power requirements of the vehicle-mounted industrial-grade computing platform.

[0080] The data fusion processing module transmits the fused perception results to the decision control module. The decision control module combines preset decision rules and multiple sets of occupant preference data to generate corresponding control commands and send them to various electrical devices in the cabin. When the fused perception results indicate that the driver has been driving continuously for more than 4 hours (the long-term driving warning threshold for commercial vehicles), and their heart rate and eyelid closure frequency have increased, indicating fatigue, the decision control module generates warning and auxiliary control commands. This includes controlling the in-vehicle radio to play a sharp warning tone, switching the ambient lighting to red, adjusting the air conditioning fan speed to level 4 and the temperature to 22°C, and triggering a seat vibration warning to remind the driver to stop and rest. When the fused perception results indicate that rear passengers are in an improper sitting posture or are too close to the windows, a reminder command is generated, controlling the rear seats to vibrate as a reminder, and adjusting the window raising and lowering speed to avoid safety hazards. When the fused perception results indicate that the PM10 concentration in the cabin exceeds the standard, an environmental adjustment command is generated, controlling the air purifier to start high-speed mode and closing the windows to prevent dust from entering.

[0081] During the execution of control commands by the cockpit equipment, the feedback optimization module simultaneously collects equipment operation feedback data and occupant operation feedback data. This feedback data is transmitted to the data fusion processing module for dynamically adjusting sensor sensing frequency, weighting coefficients, and fusion algorithm parameters, thereby achieving dynamic optimization of system performance. In this process, the privacy protection module encrypts sensitive information such as driver physiological data and occupant facial feature data, and restricts access to sensitive data through data access permission controls. Expired sensitive data is also periodically cleared according to a preset cycle to ensure data security and compliance. The storage module synchronously stores all raw data, fusion sensing results, control commands, feedback data, and other data from each stage. It supports data synchronization with the commercial vehicle back-end management system, providing data support for subsequent system optimization, fault diagnosis, and driver behavior analysis, forming a complete closed-loop workflow adapted to the usage scenarios of commercial vehicles.

[0082] The following is a detailed description of the invention, designed for mid-to-low-end passenger cars and economy vehicles:

[0083] The system hardware configuration includes:

[0084] Multimodal perception module: The high-definition vision sensor uses a 720P automotive camera, installed in the rearview mirror inside the cabin, with a sampling frequency of 20fps, mainly collecting the driver's facial features and body movements; the physiological electrical sensor uses a simple electrode sensor, integrated only in the steering wheel grip, with a sampling frequency of 60Hz, collecting the driver's heart rate data; the millimeter-wave sensor uses a 24GHz automotive millimeter-wave radar, installed on the top of the cabin, with a detection range of 0.5-2m, collecting the driver's sitting posture data; the temperature sensor uses an NTC thermistor sensor, with a measurement range of -30°C. The temperature range is ℃-80℃, with an accuracy of ±0.3℃; the humidity sensor uses a simple capacitive humidity sensor, with a measurement range of 10-90%RH and an accuracy of ±4%RH; the air quality sensor uses a simple PM2.5 sensor, which can detect PM2.5 concentration; the noise sensor uses a common electret microphone, with a measurement range of 40-110dB; the light sensor uses a simple photodiode sensor, with a measurement range of 100-8000 lux; the equipment status sensing unit uses a CAN bus interface and only collects the core operating parameters of the air conditioner, audio equipment, and display equipment.

[0085] Data fusion processing module: It adopts an in-vehicle entry-level heterogeneous computing platform, integrating CPU and NPU, with a computing power of 20 TOPS. It supports basic model compression and parallel processing of multi-source data. It has built-in simplified Kalman filter algorithm, adaptive weighted fusion algorithm and differential privacy protection algorithm to simplify the data preprocessing process and reduce computing power consumption.

[0086] Decision control module: It adopts an entry-level vehicle-mounted MCU controller with a main frequency of 800MHz. It stores preset basic decision rules and individual occupant preference data, supports real-time communication with core electrical equipment in the cockpit, has a response time of ≤0.6s, simplifies control command types, and focuses on core functions.

[0087] Feedback optimization module: It adopts a simple pressure sensor and a single-microphone voice acquisition unit. The pressure sensor is only integrated into the steering wheel to collect driver operation pressure data; the voice acquisition unit uses a single microphone to collect simple voice feedback data from the driver, with a sampling frequency of 12kHz.

[0088] Privacy protection module: integrates a simple encryption chip, supports AES-128 encryption algorithm and simplified differential privacy algorithm, privacy protection budget ε is set to 0.2, perturbation coefficient λ is set to 0.4, simplifying the data encryption process and reducing computing power consumption while ensuring privacy and security.

[0089] Storage module: It adopts an entry-level automotive eMMC storage chip with a capacity of 32GB, a read / write speed of ≥80MB / s, supports data cyclic storage, and sets the storage period to 30 days. It only stores core data to reduce storage pressure.

[0090] After system startup, each sensing unit of the multimodal perception module starts synchronously. High-definition visual sensors collect driver facial features and body movements, physiological electrical sensors collect driver heart rate data, millimeter-wave sensors collect driver posture data, environmental parameter sensing units collect core environmental parameters such as cabin temperature, humidity, PM2.5 concentration, noise, and light intensity, and equipment status sensing units collect core operating parameters such as air conditioning operating power, audio volume, and display brightness via the CAN bus interface. Subsequently, the multimodal perception module transmits all collected raw data to the data fusion processing module. This module first removes major noise interference from the data using a simplified Kalman filter algorithm, then uses a min-max normalization method to map raw data of different dimensions and magnitudes to the same interval. Next, it dynamically allocates weight coefficients based on the real-time sensing accuracy of each sensor, fuses the multi-source data, and outputs a high-precision fusion perception result that meets basic requirements. Simultaneously, quantization and pruning techniques are used to compress the fusion algorithm, significantly reducing computational power consumption and adapting to the computational power requirements of entry-level automotive computing platforms.

[0091] The data fusion processing module transmits the fused perception results to the decision control module. The decision control module combines preset basic decision rules and individual occupant preference data to generate core control commands and send them to the core electrical equipment in the cockpit. When the fused perception results indicate that the driver is fatigued or distracted, a warning command is generated, controlling the audio system to play a warning tone and the ambient lighting to switch to a warning color. When the fused perception results indicate that the cabin environmental parameters exceed preset thresholds, an environmental adjustment command is generated, controlling the air conditioning to adjust the temperature and humidity and activating the air purifier. When the fused perception results indicate that the driver's posture is improper, a reminder command is generated, controlling the seat to vibrate as a reminder.

[0092] During the execution of control commands by the cockpit equipment, the feedback optimization module simultaneously collects equipment operation feedback data and driver operation feedback data. This feedback data is transmitted to the data fusion processing module, which dynamically adjusts sensor sensing frequencies, weighting coefficients, and fusion algorithm parameters to achieve dynamic optimization of system performance. In this process, the privacy protection module uses a simplified differential privacy algorithm to encrypt sensitive information such as driver heart rate and facial features, and restricts access to sensitive data through data access control to ensure data security and compliance. The storage module simultaneously stores core raw data, fusion sensing results, control commands, feedback data, and other core data from each stage, supporting data querying and traceability. This provides fundamental data support for subsequent system optimization and troubleshooting, meeting the usage needs of low-to-mid-range passenger cars and economy vehicles while reducing cost and computing power consumption.

[0093] The intelligent cockpit perception system provided in this embodiment synchronously collects data such as visual, physiological, millimeter-wave, air quality, noise, light, and equipment operating status through occupant status perception units, environmental parameter perception units, and equipment status perception units. This reduces the blind spots of single sensors. It performs noise reduction and normalization on multi-source data and dynamically allocates weight coefficients based on the real-time perception accuracy of sensors, which helps improve the consistency and reliability of multi-source data fusion. It adjusts the perception frequency, weight coefficients, fusion algorithm parameters, and decision rules based on equipment operation feedback data and occupant operation feedback data, forming a closed loop of perception, fusion, decision-making, execution, feedback, and optimization. It reduces the computational and storage resource consumption of the algorithm model through model compression methods such as quantization and pruning, and protects sensitive occupant data through differential privacy and access control, making it suitable for deployment in in-vehicle computing platforms. It can proactively generate warnings, environmental adjustments, equipment control, or reminder commands based on driver status, occupant posture, and cockpit environment, improving cockpit safety, comfort, and proactive service capabilities.

[0094] Example 2

[0095] This embodiment provides a vehicle intelligent cockpit perception method, employing the vehicle intelligent cockpit perception system as described in Embodiment 1, the method comprising:

[0096] S1: Synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin;

[0097] S2: Denoise and normalize the raw data, dynamically determine the weight coefficients based on the real-time sensing accuracy of each sensor, and perform adaptive weighted fusion on the preprocessed raw data to obtain the fused sensing result.

[0098] S3: Generate control commands based on the fusion perception results, preset decision rules and occupant preference data, and control the cabin electrical equipment to execute the control commands;

[0099] S4: Collect equipment operation feedback data and crew operation feedback data of cabin electrical equipment;

[0100] S5: Adjust at least one of the sensor sensing frequency, weight coefficient and fusion algorithm parameters based on the feedback data, and perform privacy protection processing on sensitive data in the original data, fusion sensing results and feedback data;

[0101] S6: Store the raw data, fuse the sensing results, control commands and feedback data, and repeat steps S1 to S5 to form a closed-loop sensing control.

[0102] Example 3

[0103] This embodiment provides a vehicle including the intelligent cockpit perception system as described in Embodiment 1.

[0104] The specific embodiments of the present invention have been described above. Based on the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention.

Claims

1. A vehicle intelligent cockpit perception system, characterized in that, It includes a multimodal perception module, a data fusion and processing module, a decision control module, a feedback optimization module, a privacy protection module, and a storage module; The multimodal perception module is used to synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin, and send the raw data to the data fusion processing module. The data fusion processing module is used to preprocess the raw data, dynamically determine the weight coefficients based on the real-time sensing accuracy of each sensor, perform adaptive weighted fusion on the preprocessed raw data, and output the fused sensing result. The decision control module is used to generate control commands based on the fused perception results, preset decision rules and occupant preference data, and send the control commands to the cabin electrical equipment; The feedback optimization module is used to collect equipment operation feedback data and occupant operation feedback data after the cockpit electrical equipment executes the control command, and send the feedback data to the data fusion processing module so that the data fusion processing module can adjust the sensing parameters and fusion parameters. The privacy protection module is used to perform privacy protection processing on sensitive data in the original data, the fusion perception results and the feedback data. The storage module is used to store the raw data, the fusion perception results, the control commands, and the feedback data, thereby forming a closed-loop perception system of perception, fusion, decision-making, execution, feedback, and optimization.

2. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The multimodal sensing module includes an occupant status sensing unit, an environmental parameter sensing unit, and an equipment status sensing unit. The occupant status sensing unit includes a visual sensor, a physiological electrophysiological sensor, and a millimeter-wave sensor, used to collect at least one of the following: occupant facial features, limb movements, heart rate, electroencephalogram (EEG) signals, sitting posture, and body distance. The environmental parameter sensing unit includes a temperature sensor, a humidity sensor, an air quality sensor, a noise sensor, and a light sensor, used to collect at least one of the following: temperature, humidity, particulate matter concentration, harmful gas content, noise frequency, and light intensity. The equipment status sensing unit communicates with the cabin electrical equipment through a Controller Area Network (CAN) bus interface and / or a Local Interconnect Network (LIN) bus interface.

3. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The preprocessing includes denoising the original data using a Kalman filter algorithm, and mapping the original data of different dimensions to a normalized value using a maximum-minimum normalization method. The interval; the adaptive weighted fusion obtains the fusion sensing result according to the following formula: ； in, The fusion perception result, For the number of sensors, For the first The weighting coefficients of each sensor For the first Preprocessed data from each sensor.

4. The automotive intelligent cockpit perception system according to claim 3, characterized in that, No. Real-time sensing accuracy of individual sensors Determine according to the following formula: in, For the first Standard reference data for each sensor, For the first The absolute value of the deviation between the preprocessed data of each sensor and the standard reference data; the weighting coefficients are normalized according to the following formula so that the sum of the weighting coefficients of each sensor is 1: 。 5. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The data fusion processing module includes a model compression unit, which uses at least one of quantization processing and pruning processing to compress the algorithm model used for data fusion, so as to reduce the computing power and storage resources required by the vehicle computing platform to execute the algorithm model.

6. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The control commands include at least one of the following: warning commands, environmental adjustment commands, equipment control commands, and reminder commands; when the fusion perception results indicate that the driver is fatigued or distracted, the cabin environment parameters exceed a preset threshold, the occupant's sitting posture is improper, or the distance between the occupant and the cabin equipment is less than a preset distance, the decision control module controls at least one of the following to change the operating status: audio system, ambient lighting, air conditioning, air purifier, seats, windows, and display devices.

7. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The equipment operation feedback data includes the actual operating parameters of the cabin electrical equipment after executing the control command, and the occupant operation feedback data includes the operation records of the occupant manually adjusting the cabin electrical equipment; the data fusion processing module adjusts at least one of the sensor sensing frequency, the weighting coefficient and the fusion algorithm parameters according to the feedback data, and the decision control module corrects the occupant preference data and / or the preset decision rules according to the occupant operation feedback data.

8. The automotive intelligent cockpit perception system according to claim 1, characterized in that, The privacy protection module employs a differential privacy protection algorithm to perturb occupant physiological data and / or facial feature data according to the following formula: in, Sensitive data processed for privacy protection For raw, sensitive data, ε represents the privacy protection budget. The disturbance coefficient is... The storage module includes an in-vehicle embedded multimedia card eMMC storage chip, and the storage period of the storage module is not less than 30 days. The storage module is equipped with Laplace perturbation noise and restricts access to sensitive data through data access permission control.

9. A method for sensing in an intelligent cockpit of an automobile, employing the intelligent cockpit sensing system as described in any one of claims 1-8, characterized in that, The method includes: S1: Synchronously collect raw data on occupant status, environmental parameters, and cabin equipment status within the cabin; S2: The original data is denoised and normalized, the weight coefficients are dynamically determined according to the real-time sensing accuracy of each sensor, and the pre-processed original data is adaptively weighted and fused to obtain the fused sensing result. S3: Generate control commands based on the fused perception results, preset decision rules, and occupant preference data, and control the cabin electrical equipment to execute the control commands; S4: Collect equipment operation feedback data and occupant operation feedback data of the cabin electrical equipment; S5: Adjust at least one of the sensor sensing frequency, the weighting coefficient, and the fusion algorithm parameters according to the feedback data, and perform privacy protection processing on the original data, the fusion sensing result, and the sensitive data in the feedback data; S6: Store the original data, the fused sensing results, the control commands, and the feedback data, and repeat steps S1 to S5 to form a closed-loop sensing control.

10. A vehicle, characterized in that, Including the automotive intelligent cockpit perception system as described in any one of claims 1-8.

Images