Multi-modal perception enhancement method for automobile cabin end side model

Abstract

The invention provides an automobile cabin end side model multi-modal perception enhancement method, which comprises the following steps: S1) deploying an RGB camera in a cabin, and collecting facial biological characteristic data, hand interaction behavior data and cabin overall environment data of a driver; s2) adjusting image illumination and contrast by adopting an algorithm, dynamically focusing a key area through a target detection frame to process a shielding problem, and eliminating static redundant information; s3) constructing a neural network, extracting facial biological features and the like in parallel, and outputting low-dimensional feature vectors; s4) time sequence association is established, key area feature weights are enhanced through a space attention mechanism, and a logic relationship among different features is explicitly modeled; s5) aggregating the weighted feature maps by using feature attention pooling, and retaining a core feature channel in combination with a channel pruning technology to realize model compression; s6) dynamically adjusting the calculation priority and weight distribution of each feature extraction branch; and S7) the end side uploads the sample to the cloud side, and the cloud side generates an update package and pushes the update package to the end side to complete model iteration.

Description

Technical Field

[0001] This application relates to a perception enhancement method, specifically a multimodal perception enhancement method for an end-side model of an automotive cockpit. Background Technology

[0002] With the advancement of automotive intelligence, the intelligent cockpit has become the core carrier of human-vehicle interaction. Among them, vision-based perception technologies, especially the monitoring of driver status and passenger behavior, are crucial for improving driving safety and the cockpit interaction experience.

[0003] Currently, the mainstream cockpit perception solutions in the industry are mainly divided into two categories:

[0004] The first category is multi-sensor fusion solutions. These solutions combine various types of sensors, such as RGB cameras, infrared cameras, and millimeter-wave radar, leveraging the complementary nature of different sensors in information acquisition to improve the accuracy and robustness of the perception system. For example, infrared cameras are used for facial recognition in low-light environments, or depth cameras are used to enhance the accuracy of gesture recognition. However, this approach has significant drawbacks: First, the introduction of multiple sensors directly leads to a substantial increase in hardware costs, typically 30% to 50% higher than pure vision solutions; second, the system becomes more complex, presenting significant challenges in wiring, power supply, and time synchronization and spatial calibration between multiple sensors, increasing the difficulty of engineering implementation and subsequent maintenance; finally, the complex system architecture hinders large-scale deployment in cost-sensitive vehicle models.

[0005] The second category is the pure RGB camera solution. This solution has low hardware cost and is easy to integrate, making it a feasible path to reduce system complexity. However, existing pure RGB solutions have significant drawbacks: First, their perception performance is highly susceptible to environmental factors. When lighting conditions change drastically (such as strong light suppression at tunnel entrances or low-light environments at night) or physical obstructions occur (such as the driver wearing a mask, sunglasses, or their hands being obstructed by the steering wheel), the model's perception accuracy drops sharply, exhibiting insufficient robustness. Second, under the constraint of limited computing resources on the cockpit-side devices, the feature dimensions that can be extracted from a single RGB modality are relatively limited, making it difficult to simultaneously and effectively complete the complex perception requirements of multiple tasks and scenarios, such as driver fatigue monitoring, fine gesture interaction recognition, and judgment of the status of objects or people in the cockpit. Traditional end-to-end processing models typically involve high computational demands and require high computing power from the edge chip, making it difficult to achieve efficient, low-latency inference. Summary of the Invention

[0006] This invention aims to solve three core problems existing in current automotive cockpit edge sensing solutions based on pure RGB cameras:

[0007] (1) Poor environmental adaptability and insufficient perception robustness under complex lighting and physical shading conditions;

[0008] (2) The feature dimensions that can be extracted from single-modal RGB data are limited, making it difficult to meet the complex perception needs of multiple tasks and multiple scenarios;

[0009] (3) With limited computing resources on the edge, the model has a large amount of computation and is difficult to deploy, making it difficult to achieve efficient and low-latency real-time inference.

[0010] The purpose of this invention is to provide a low-cost, high-efficiency, and highly robust multimodal perception enhancement method for automotive cockpit end-side models. This method does not rely on additional sensors such as infrared cameras, millimeter-wave radar, or TOF cameras, but can achieve accurate and real-time perception of multiple scenes and targets in the cockpit using only an RGB camera.

[0011] This invention provides a multimodal perception enhancement method for an end-side model of a vehicle cockpit, comprising the following steps:

[0012] S1) Multi-dimensional data acquisition based on distributed RGB camera array: Deploy a distributed array of 2 or 3 RGB cameras in the cockpit to simultaneously collect driver facial biometric data, hand interaction behavior data and overall cockpit environment data;

[0013] S2) End-side adaptive data preprocessing: The Retinex algorithm is used to adjust the image illumination and contrast, the key areas are dynamically focused through the target detection box to handle the occlusion problem, and the inter-frame difference method is used to filter moving targets to remove static redundant information.

[0014] S3) Multi-dimensional feature extraction: Construct a multi-branch lightweight convolutional neural network to extract facial biometric features, interactive behavior features and cabin environment features in parallel, and output a low-dimensional feature vector.

[0015] S4) Spatiotemporal attention feature association: Establish temporal association by aligning features of consecutive frames through temporal attention mechanism, strengthen the feature weight of key regions through spatial attention mechanism, and explicitly model the logical relationship between different features;

[0016] S5) Lightweight feature fusion: Feature attention pooling is used to aggregate weighted feature maps, and channel pruning is combined to retain core feature channels to achieve model compression;

[0017] S6) Scene-driven dynamic perception: The scene recognition module determines the core application scenario in real time and dynamically adjusts the calculation priority and weight allocation of each feature extraction branch.

[0018] S7) Edge-Cloud Collaborative Closed-Loop Iteration: Low-confidence, difficult samples are uploaded from the device to the cloud. The cloud optimizes the model through transfer learning and generates a lightweight incremental update package, which is then pushed to the device to complete the model iteration.

[0019] In step S1), the deployment positions of the RGB camera include: directly above the steering wheel or the upper edge of the dashboard, in front of the center console, near the rearview mirror or the center of the roof, and visual data of the corresponding areas are collected in a targeted manner.

[0020] In step S2), the key areas focused by the target detection box include the driver's eyes, mouth, hands, and central control interaction area.

[0021] In step S3), the multi-branch lightweight convolutional neural network is constructed based on the MobileNet series of networks.

[0022] In step S4), when performing fatigue monitoring, the spatial attention mechanism increases the weight of eye region features to over 0.6.

[0023] In step S5), the channel pruning technique retains the top 70% of feature channels in terms of contribution, reducing the number of model parameters by more than 40%.

[0024] In step S6), the core application scenarios include fatigue monitoring, gesture interaction, and cockpit safety detection.

[0025] In step S7), the low-confidence difficult samples include severely occluded facial images, blurry gesture images, and data from uncommon interaction scenarios.

[0026] The multi-branch lightweight convolutional neural network is deployed based on the TensorFlow LiteMicro framework, and the preprocessing algorithm is optimized using the ARMNEON instruction set.

[0027] The end-side inference process in steps S1) to S6) is executed on the cockpit domain controller equipped with an NPU, wherein the NPU computing power is not less than 1.5 TOPS and the end-to-end inference latency is less than 60ms.

[0028] Beneficial effects of the present invention

[0029] Compared with the prior art, the present invention has the following significant advantages:

[0030] 1. Significantly reduced hardware costs and system complexity: This solution relies solely on inexpensive, general-purpose RGB cameras, reducing hardware costs by 30% to 50% compared to multi-sensor fusion solutions using infrared, millimeter-wave radar, and other sensors. Simultaneously, it eliminates the need for wiring, power supply, and complex calibration processes for multiple sensors, greatly simplifying system integration and engineering implementation.

[0031] 2. Significantly improved perception robustness and accuracy: Through adaptive preprocessing algorithms (such as Retinex) and spatiotemporal attention mechanisms, the performance bottleneck of pure RGB solutions under complex lighting and occlusion conditions is effectively overcome. In complex lighting environments (strong light / dark light), perception accuracy can be improved by 20% to 35%, achieving robustness close to that of multi-sensor solutions.

[0032] 3. Achieve efficient edge deployment and real-time response: Employing a lightweight model architecture of "deconstruction-association-compression" combined with techniques such as channel pruning, the model maintains high accuracy while possessing extremely low computational complexity. Edge inference latency can be controlled within 60 milliseconds (ms), enabling smooth operation on mainstream cockpit MCU / SoC chips without relying on expensive high-performance computing platforms.

[0033] 4. Enhanced Scene Adaptability and Functional Scalability: The scene-driven dynamic perception mechanism allows limited computing resources to be allocated on demand, precisely targeting the most critical perception tasks and maximizing computational efficiency. Simultaneously, the edge-cloud collaborative iterative mechanism enables the model to continuously learn and evolve, adapting to new driving scenarios and user behaviors, demonstrating excellent scalability.

[0034] Figure Labels

[0035] RGB camera array-1;

[0036] End-side adaptive prediction processing module-2;

[0037] Multidimensional feature extraction module - 3;

[0038] Spatiotemporal attention association module -4;

[0039] Scene recognition module -5;

[0040] End-to-Cloud Collaboration Optimization Module - 6. Attached Figure Description

[0041] Figure 1 This is a flowchart of a multimodal perception enhancement method for an end-side model of an automobile cockpit according to the present invention;

[0042] Figure 2 This is a schematic diagram of the modules and operation flow in one embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0044] Figure 1This is a flowchart of a multimodal perception enhancement method for an end-side model of an automobile cockpit according to the present invention, as shown below. Figure 1 The present invention provides a multimodal perception enhancement method for an end-side model of an automotive cockpit, comprising the following steps:

[0045] S1) Multi-dimensional data acquisition based on distributed RGB camera array: Deploy a distributed array of 2 or 3 RGB cameras in the cockpit to simultaneously collect driver facial biometric data, hand interaction behavior data and overall cockpit environment data;

[0046] S2) End-side adaptive data preprocessing: The Retinex algorithm is used to adjust the image illumination and contrast, the key areas are dynamically focused through the target detection box to handle the occlusion problem, and the inter-frame difference method is used to filter moving targets to remove static redundant information.

[0047] S3) Multi-dimensional feature extraction: Construct a multi-branch lightweight convolutional neural network to extract facial biometric features, interactive behavior features and cabin environment features in parallel, and output a low-dimensional feature vector.

[0048] S4) Spatiotemporal attention feature association: Establish temporal association by aligning features of consecutive frames through temporal attention mechanism, strengthen the feature weight of key regions through spatial attention mechanism, and explicitly model the logical relationship between different features;

[0049] S5) Lightweight feature fusion: Feature attention pooling is used to aggregate weighted feature maps, and channel pruning is combined to retain core feature channels to achieve model compression;

[0050] S6) Scene-driven dynamic perception: The scene recognition module determines the core application scenario in real time and dynamically adjusts the calculation priority and weight allocation of each feature extraction branch.

[0051] S7) Edge-Cloud Collaborative Closed-Loop Iteration: Low-confidence, difficult samples are uploaded from the device to the cloud. The cloud optimizes the model through transfer learning and generates a lightweight incremental update package, which is then pushed to the device to complete the model iteration.

[0052] In step S1), the deployment positions of the RGB camera include: directly above the steering wheel or the upper edge of the dashboard, in front of the center console, near the rearview mirror or the center of the roof, and visual data of the corresponding areas are collected in a targeted manner.

[0053] In step S2), the key areas focused by the target detection box include the driver's eyes, mouth, hands, and central control interaction area.

[0054] In step S3), the multi-branch lightweight convolutional neural network is constructed based on the MobileNet series of networks.

[0055] In step S4), when performing fatigue monitoring, the spatial attention mechanism increases the weight of eye region features to over 0.6.

[0056] In step S5), the channel pruning technique retains the top 70% of feature channels in terms of contribution, reducing the number of model parameters by more than 40%.

[0057] In step S6), the core application scenarios include fatigue monitoring, gesture interaction, and cockpit safety detection.

[0058] In step S7), the low-confidence difficult samples include severely occluded facial images, blurry gesture images, and data from uncommon interaction scenarios.

[0059] The multi-branch lightweight convolutional neural network is deployed based on the TensorFlow LiteMicro framework, and the preprocessing algorithm is optimized using the ARMNEON instruction set.

[0060] The end-side inference process in steps S1) to S6) is executed on the cockpit domain controller equipped with an NPU, wherein the NPU computing power is not less than 1.5 TOPS and the end-to-end inference latency is less than 60ms.

[0061] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so as to fully disclose the specific implementation details of the present invention and enable those skilled in the art to understand and implement the present invention.

[0062] This embodiment provides a multimodal perception enhancement method for an end-side model of an automotive cockpit. Its hardware deployment and data flow can be referenced. Figure 2 The diagram shows the modules and their operation process.

[0063] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0064] 1. Single-source multi-dimensional data acquisition and adaptive preprocessing based on distributed RGB camera 1:

[0065] Data Acquisition: A distributed camera array consisting of 2 to 3 RGB cameras is deployed at different locations within the cockpit. For example, one camera faces the driver's face to collect biometric data such as facial expressions, posture, and gaze; another camera faces the central control or interaction area to collect interactive behavior data such as hand movements and gestures; and a third optional camera collects data from the entire cockpit space to obtain information on passenger status or items left behind. This array simultaneously collects multi-dimensional visual data, using algorithmic processing to replace the hardware combination of multiple types of sensors.

[0066] Edge-side adaptive preprocessing module 2: Performs real-time preprocessing of acquired image data on the edge to enhance environmental adaptability. To address illumination variations, it employs the Retinex algorithm or similar illumination equalization algorithms to dynamically adjust image contrast and brightness, restoring image details in both bright and dark environments. For partial occlusion, it pre-locates key regions (such as the driver's eyes, mouth, and hands) using a lightweight object detection model (e.g., YOLOv-fastest) and dynamically focuses on these regions for data processing. Simultaneously, it uses inter-frame differencing to filter data frames containing motion information, effectively removing redundant information such as static backgrounds and reducing the computational burden on subsequent models.

[0067] 2. Construct an RGB single-modal cross-dimensional feature fusion model with a "deconstruction-association-compression" paradigm:

[0068] Adaptive preprocessing: Adaptive preprocessing is performed using a multi-dimensional feature extraction module 3, and a multi-branch lightweight convolutional neural network (CNN) architecture is designed, for example, based on an improvement of the MobileNet series. Each branch is responsible for extracting semantic features of a specific dimension in parallel from different preprocessed data streams. Specifically, a "facial biometrics" extraction branch (for analyzing eyelid opening and closing, yawning frequency, etc.), an "interaction behavior features" extraction branch (for recognizing gestures, key presses, etc.), and a "cabin environment features" extraction branch (for recognizing object positions, personnel states, etc.) are established, and each branch outputs a low-dimensional feature vector.

[0069] Feature Association (Spatiotemporal Attention Mechanism): A spatiotemporal attention module 4 is introduced to perform correlation analysis and weight recalibration on the feature vectors extracted from the multi-branch model. The temporal attention mechanism is responsible for aligning and associating features between consecutive frames. For example, it establishes a temporal association between the feature "hand raised" and the feature "hand approaching the central control screen" in a subsequent frame. The spatial attention mechanism is responsible for strengthening the importance of different feature regions within the current frame. For example, when performing fatigue monitoring tasks, the weight assigned to the eye region features is increased to above 0.6, higher than other facial regions. Through this mechanism, the logical relationship between different semantic features under a single RGB modality is explicitly established.

[0070] Feature compression (lightweight fusion): To adapt to low-computing-power chips on the edge, a technique of "feature attention pooling + channel pruning" is used to lightweight the model. First, attention pooling is used to efficiently aggregate the weighted feature maps; then, dynamic pruning is performed based on the actual contribution of feature channels in model inference, for example, only the top 70% of feature channels in terms of contribution are retained, thereby significantly reducing the number of model parameters and computational cost (e.g., by more than 40%) while ensuring perceptual accuracy.

[0071] 3. Implement scenario-driven dynamic perception enhancement and edge-cloud collaborative iteration:

[0072] Scene-driven dynamic perception: The system pre-defines core application scenarios such as "fatigue monitoring," "gesture interaction," and "cabin safety" (e.g., detection of children or pets left behind in the rear seats). A lightweight scene recognition module 5 determines the most likely cabin scenario in real time and dynamically adjusts the priority and attention weight of the feature extraction branch accordingly. For example, when a "fatigue monitoring" scenario is detected, the system prioritizes and allocates more computing resources to the "facial biometrics" extraction branch.

[0073] Edge-Cloud Collaborative Closed-Loop Iteration: Establish an edge-cloud collaborative model optimization module 6. During operation, the edge model encrypts and packages "difficult sample data" with low recognition confidence (such as severely occluded faces, blurry or uncommon gestures), and uploads them to the cloud server when network connectivity allows. The cloud utilizes more powerful computing power and massive amounts of data to perform targeted optimization training on specific branches of the model (such as feature extraction branches) through methods such as transfer learning. After training, a lightweight incremental update package is generated and pushed to the edge device, realizing "hot updates" or "fine-tuning" of the edge model, thereby enabling the model's perception capabilities to continuously learn and evolve. The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0074] The embodiments described above are merely further illustrations of the present invention and are not intended to limit the present invention in any other way. The present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding modifications and changes based on the present invention, but all such modifications and changes should fall within the protection scope of the present invention.

Claims

1. A method for enhancing multimodal perception of an end-side model of a vehicle cockpit, characterized in that, Includes the following steps: S1) Multi-dimensional data acquisition based on distributed RGB camera array: Deploy a distributed array of 2 or 3 RGB cameras in the cockpit to simultaneously collect driver facial biometric data, hand interaction behavior data and overall cockpit environment data; S2) End-side adaptive data preprocessing: The Retinex algorithm is used to adjust the image illumination and contrast, the key areas are dynamically focused through the target detection box to handle the occlusion problem, and the inter-frame difference method is used to filter moving targets to remove static redundant information. S3) Multi-dimensional feature extraction: Construct a multi-branch lightweight convolutional neural network to extract facial biometric features, interactive behavior features and cabin environment features in parallel, and output a low-dimensional feature vector. S4) Spatiotemporal attention feature association: Establish temporal association by aligning features of consecutive frames through temporal attention mechanism, strengthen the feature weight of key regions through spatial attention mechanism, and explicitly model the logical relationship between different features; S5) Lightweight feature fusion: Feature attention pooling is used to aggregate weighted feature maps, and channel pruning is combined to retain core feature channels to achieve model compression; S6) Scene-driven dynamic perception: The scene recognition module determines the core application scenario in real time and dynamically adjusts the calculation priority and weight allocation of each feature extraction branch. S7) Edge-Cloud Collaborative Closed-Loop Iteration: Low-confidence, difficult samples are uploaded from the device to the cloud. The cloud optimizes the model through transfer learning and generates a lightweight incremental update package, which is then pushed to the device to complete the model iteration.

2. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S1), the deployment positions of the RGB camera include: directly above the steering wheel or the upper edge of the dashboard, in front of the center console, near the rearview mirror or the center of the roof, and visual data of the corresponding areas are collected in a targeted manner.

3. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S2), the key areas focused by the target detection box include the driver's eyes, mouth, hands, and central control interaction area.

4. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S3), the multi-branch lightweight convolutional neural network is constructed based on the MobileNet series of networks.

5. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S4), when performing fatigue monitoring, the spatial attention mechanism increases the weight of eye region features to over 0.

6.

6. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S5), the channel pruning technique retains the top 70% of feature channels in terms of contribution, reducing the number of model parameters by more than 40%.

7. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S6), the core application scenarios include fatigue monitoring, gesture interaction, and cockpit safety detection.

8. The multimodal perception enhancement method for the end-side model of a vehicle cockpit according to claim 1, characterized in that, In step S7), the low-confidence difficult samples include severely occluded facial images, blurry gesture images, and data from uncommon interaction scenarios.

9. The multimodal perception enhancement method for the end-side model of an automotive cockpit according to any one of claims 1-8, characterized in that, The multi-branch lightweight convolutional neural network is deployed based on the TensorFlow LiteMicro framework, and the preprocessing algorithm is optimized using the ARMNEON instruction set.

10. The multimodal perception enhancement method for the end-side model of an automotive cockpit according to any one of claims 1-8, characterized in that, The end-side inference process in steps S1) to S6) is executed on the cockpit domain controller equipped with an NPU, wherein the NPU computing power is not less than 1.5 TOPS and the end-to-end inference latency is less than 60ms.

Images