A multi-modal lightweight perception method and device for distraction and fatigue driving

Abstract

The present application relates to a kind of multi-modal light weight perception methods for distraction and fatigue driving, comprising: multi-modal data acquisition, constitute multi-source heterogeneous original data set;Establish tree analysis model;Calculate sample set characteristic vector variance;Generate high-quality final data set S*;Light weight perception model is constructed;Light weight perception model is split into tasks, and intermediate feature and inference result are obtained;Dynamic adjustment early warning threshold, output driver fatigue and distraction state and trigger hierarchical early warning.The present application is collected by multi-modal data fusion, combines RGB vision, infrared thermal imaging, vehicle CAN bus multi-source information, improves the robustness of fatigue and distraction state recognition;Eliminate invalid data, improve the information gain of data set and model training efficiency;Combined with knowledge distillation, pruning, quantization technology, the model parameter quantity and the great reduction of calculation complexity are realized, adapt to vehicle embedded hardware deployment, meet the low power consumption, high frame rate real-time inference demand.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving safety technology, and in particular to a multimodal lightweight perception method and device for distracted and fatigued driving. Background Technology

[0002] Driver distraction and fatigue are among the core causes of traffic accidents, and accurate, real-time driver state perception is a crucial element for the safety redundancy of intelligent driving systems. Currently, driver distraction and fatigue perception technologies mostly rely on data acquisition from a single visual modality, identifying driver status solely through facial features. However, real-world in-vehicle scenarios present various complex conditions, such as strong glare, low-light conditions at night, drivers wearing sunglasses / masks obscuring their vision, and interference from complex interior decorations. A single modality is easily affected by environmental and scene interference, leading to feature extraction failure and an inability to accurately identify fatigue and distraction. Furthermore, existing technologies often directly collect raw data for model training without systematically constructing and validating the dataset, easily introducing a large number of invalid samples, increasing model training costs and generalization errors.

[0003] Furthermore, some studies have attempted to improve perception robustness by introducing multimodal data and combining infrared sensors, vehicle sensors, and other sources of information. However, they have not built lightweight architectures adapted to vehicle-mounted embedded hardware for multi-source data, resulting in large model parameters and high computational complexity, making it impossible to achieve low-power, high-frame-rate real-time inference on vehicle-side devices. Existing dataset construction methods mostly rely on manual annotation and simple data augmentation, lacking a systematic dataset validity evaluation mechanism. This makes it difficult to select effective samples with practical value for model training, easily generating a large amount of duplicate, invalid, or low-information-gain data. At the same time, the lack of lightweight model design tailored to the characteristics of actual vehicle hardware makes it difficult for trained models to meet the power consumption, frame rate, and size requirements of vehicle-side deployment, limiting the practical application of the technology. Summary of the Invention

[0004] To address the technical problems of poor robustness of single modalities, numerous invalid samples in datasets, and difficulty in real-time deployment of models on vehicle-mounted devices in existing technologies, the primary objective of this invention is to provide a multimodal lightweight perception method for fatigue and distraction state recognition that improves the robustness of fatigue and distraction state recognition, enhances the information gain of datasets and model training efficiency, and adopts an edge-edge collaborative deployment architecture. The edge is responsible for lightweight feature extraction, while the edge is responsible for complex fusion and temporal modeling, achieving a balance between inference efficiency and resource utilization.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal lightweight perception method for distracted and fatigued driving, the method comprising the following sequential steps:

[0006] (1) Collect multimodal data, and combine the multimodal data with publicly available driving behavior datasets to form a multi-source heterogeneous original dataset;

[0007] (2) Decompose the constituent factors of the multi-source heterogeneous original dataset into hierarchical structures, establish a tree-like analysis model, and describe the hierarchical relationship of each factor in the form of a set;

[0008] (3) Discretize the constituent factors of the multi-source heterogeneous original dataset, calculate the variance of the feature vector of the sample set, select valid samples with variance greater than the preset threshold, and remove invalid data.

[0009] (4) The constituent factors of the effective samples are randomly combined to form the initial population sample set. Based on the genetic model, selection, crossover and mutation operations are performed iteratively to generate a high-quality final dataset S*.

[0010] (5) Use the high-quality final dataset S* to train the original multimodal perception model, perform neural network architecture search, and use knowledge distillation, structured pruning, and 8-bit integer quantization combined compression techniques to construct a lightweight perception model;

[0011] (6) The lightweight perception model is divided into tasks, and collaborative reasoning is achieved through end-edge data interaction and synchronization mechanism to obtain intermediate features and reasoning results;

[0012] (7) Based on intermediate features and reasoning results, dynamically adjust the warning threshold, output the driver's fatigue and distraction status and trigger graded warnings.

[0013] Step (1) specifically refers to: the multimodal data includes driver facial and behavioral visual information collected by the vehicle-mounted RGB camera, physiological feature information in low-light and occluded scenarios collected by the infrared thermal imaging sensor, and vehicle speed, steering, and pedal operation vehicle status information collected by the vehicle-mounted CAN bus. The collected multimodal data is synchronized in time, registered in space, and normalized with the publicly available driving behavior dataset to construct a multi-source heterogeneous original dataset covering complex working conditions.

[0014] Step (2) specifically refers to: decomposing the influencing factors of the multi-source heterogeneous original dataset into first-level categories and second-level sub-factors. The first-level categories include driving scenarios, environmental conditions, driver characteristics, and vehicle status. The second-level sub-factors of the driving scenarios include urban roads, highways, rural roads, tunnels, and underground parking garages. The second-level sub-factors of the environmental conditions include light intensity, weather conditions, temperature, and occlusion. The second-level sub-factors of the driver characteristics include facial appearance, behavioral actions, and physiological characteristics. The second-level sub-factors of the vehicle status include vehicle speed, steering angle, pedal operation, and CAN bus signal. Using the four first-level categories as root nodes and the second-level sub-factors as leaf nodes, a tree-like analysis model with a scene-environment-person-vehicle hierarchical structure is formed.

[0015] Step (3) specifically refers to: discretizing the scene, environment, personnel, and vehicle factors of the multi-source heterogeneous original dataset, constructing sample feature vectors, calculating the variance of each sample feature vector, retaining high information gain effective samples with variance greater than a preset threshold, and eliminating redundant, invalid, and noisy samples with variance less than the threshold.

[0016] Step (4) specifically refers to: using the constituent factors of the effective samples as gene codes, randomly generating the initial population sample set, iteratively performing selection, crossover, and mutation operations based on the genetic model, retaining high-fitness samples through selection operations, combining advantageous factors through crossover operations, increasing sample diversity through mutation operations, and generating a high-quality final dataset S* with rich information and balanced distribution after iterative optimization.

[0017] Step (5) specifically refers to: training the original multimodal perception model with the high-quality final dataset S* as the training sample, conducting neural network architecture search, improving the original multimodal perception model, and obtaining the improved model, namely the lightweight perception model; the original multimodal perception model adopts a lightweight LSTM temporal modeling layer, and on the basis of the lightweight LSTM temporal modeling layer, a multimodal feature extraction layer, a multi-head self-attention fusion layer, and an adaptive threshold decision layer are introduced to jointly form the lightweight perception model;

[0018] The lightweight perception model consists of a multimodal feature extraction layer, a multi-head self-attention fusion layer, a lightweight LSTM temporal modeling layer, and an adaptive threshold decision layer connected sequentially. The multimodal feature extraction layer uses a lightweight ShuffleNetV2 backbone with EMA attention to achieve parallel extraction of RGB visual features, infrared features, and vehicle CAN status features. The multi-head self-attention fusion layer employs a multi-head self-attention mechanism, where each attention head focuses on different input feature parts, generating multiple contextual representations. The lightweight LSTM temporal modeling layer uses an LSTM network. The adaptive threshold decision layer uses a numerical decision mechanism to perform adaptive threshold judgment.

[0019] Step (6) specifically refers to: deploying the computationally inefficient multimodal feature extraction layer and adaptive threshold decision layer in the lightweight perception model on end-side devices such as vehicle cameras and domain controllers, and deploying the computationally intensive multi-head self-attention fusion layer and lightweight LSTM temporal modeling layer on roadside or vehicle edge computing nodes to establish a low-latency end-to-edge communication link and realize the collaborative operation of feature uploading and inference result distribution.

[0020] Step (7) specifically refers to: using a multi-head self-attention mechanism to adaptively weighted fuse multimodal data to obtain multimodal global fusion features, modeling the temporal change law of driving state through a lightweight LSTM, the temporal change law of driving state refers to the continuous change law of driver fatigue level or distraction state type over time, dynamically adjusting the warning threshold in combination with current illumination, vehicle speed, and road condition information, outputting graded state results and triggering audio-visual graded warnings; the graded state results include fatigue level and distraction state type: the fatigue level includes normal, mild fatigue, moderate fatigue, and severe fatigue; the distraction state type includes normal, visual distraction, cognitive distraction, operational distraction, and dangerous distraction.

[0021] Another object of the present invention is to provide an electronic device comprising:

[0022] Processor; and

[0023] A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the multimodal lightweight perception method for distracted and fatigued driving as described above.

[0024] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the multimodal lightweight perception method for distracted and fatigued driving as described above.

[0025] As can be seen from the above technical solution, the beneficial effects of the present invention are as follows: First, the present invention, through multimodal data fusion acquisition, combines RGB vision, infrared thermal imaging, and multi-source information from the vehicle CAN bus, overcoming the perception limitations of a single modality under complex working conditions and improving the robustness of fatigue and distraction state recognition; Second, the present invention, through stepwise decomposition of constituent factors and feature vector variance screening of effective samples, and then generating a high-quality dataset based on a genetic model, eliminates invalid data, thereby improving the information gain of the dataset and the model training efficiency; Third, the present invention adopts a hardware-adaptive lightweight model design, combined with knowledge distillation, pruning, and quantization techniques, to significantly reduce the number of model parameters and computational complexity, adapting to vehicle embedded hardware deployment and meeting the requirements of low power consumption and high frame rate real-time inference; Fourth, the present invention adopts an end-edge collaborative deployment architecture, with the end side responsible for lightweight feature extraction and the edge responsible for complex fusion and temporal modeling, achieving a balance between inference efficiency and resource utilization; Fifth, the present invention, through multimodal attention fusion and temporal modeling, accurately captures the instantaneous state changes of the driver, and combined with dynamic warning threshold adjustment, achieves graded and accurate warnings, improving vehicle safety protection capabilities. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0027] like Figure 1 As shown, a multimodal lightweight perception method for distracted and fatigued driving is proposed, which includes the following sequential steps:

[0028] (1) Collect multimodal data, and combine the multimodal data with publicly available driving behavior datasets to form a multi-source heterogeneous original dataset;

[0029] (2) Decompose the constituent factors of the multi-source heterogeneous original dataset into hierarchical structures, establish a tree-like analysis model, and describe the hierarchical relationship of each factor in the form of a set;

[0030] (3) Discretize the constituent factors of the multi-source heterogeneous original dataset, calculate the variance of the feature vector of the sample set, select valid samples with variance greater than the preset threshold, and remove invalid data.

[0031] (4) The constituent factors of the effective samples are randomly combined to form the initial population sample set. Based on the genetic model, selection, crossover and mutation operations are performed iteratively to generate a high-quality final dataset S*.

[0032] (5) Use the high-quality final dataset S* to train the original multimodal perception model, perform neural network architecture search, and use knowledge distillation, structured pruning, and 8-bit integer quantization combined compression techniques to construct a lightweight perception model;

[0033] (6) The lightweight perception model is divided into tasks, and collaborative reasoning is achieved through end-edge data interaction and synchronization mechanism to obtain intermediate features and reasoning results;

[0034] (7) Based on intermediate features and reasoning results, dynamically adjust the warning threshold, output the driver's fatigue and distraction status and trigger graded warnings.

[0035] Step (1) specifically refers to: the multimodal data includes driver facial and behavioral visual information collected by the vehicle-mounted RGB camera, physiological feature information in low-light and occluded scenarios collected by the infrared thermal imaging sensor, and vehicle speed, steering, and pedal operation vehicle status information collected by the vehicle-mounted CAN bus. The collected multimodal data is synchronized in time, registered in space, and normalized with the publicly available driving behavior dataset to construct a multi-source heterogeneous original dataset covering complex working conditions.

[0036] Step (2) specifically refers to: decomposing the influencing factors of the multi-source heterogeneous original dataset into first-level categories and second-level sub-factors. The first-level categories include driving scenarios, environmental conditions, driver characteristics, and vehicle status. The second-level sub-factors of the driving scenarios include urban roads, highways, rural roads, tunnels, and underground parking garages. The second-level sub-factors of the environmental conditions include light intensity, weather conditions, temperature, and occlusion. The second-level sub-factors of the driver characteristics include facial features, behavioral actions, and physiological characteristics. The second-level sub-factors of the vehicle status include vehicle speed, steering angle, pedal operation, and CAN bus signal. Using the four first-level categories as root nodes and the second-level sub-factors as leaf nodes, a tree-like analysis model with a scenario-environment-person-vehicle hierarchical structure is formed, and the hierarchical relationship of each constituent factor is described in set form.

[0037] Step (3) specifically refers to: discretizing the scene, environment, personnel, and vehicle factors of the multi-source heterogeneous original dataset, constructing sample feature vectors, calculating the variance of each sample feature vector, retaining high information gain effective samples with variance greater than a preset threshold, and eliminating redundant, invalid, and noisy samples with variance less than the threshold.

[0038] Step (4) specifically refers to: using the constituent factors of the effective samples as gene codes, randomly generating the initial population sample set, iteratively performing selection, crossover, and mutation operations based on the genetic model, retaining high-fitness samples through selection operations, combining advantageous factors through crossover operations, increasing sample diversity through mutation operations, and generating a high-quality final dataset S* with rich information and balanced distribution after iterative optimization.

[0039] Step (5) specifically refers to: training the original multimodal perception model with the high-quality final dataset S* as the training sample, conducting neural network architecture search, improving the original multimodal perception model, and obtaining the improved model, namely the lightweight perception model; the original multimodal perception model adopts a lightweight LSTM temporal modeling layer, and on the basis of the lightweight LSTM temporal modeling layer, a multimodal feature extraction layer, a multi-head self-attention fusion layer, and an adaptive threshold decision layer are introduced to jointly form the lightweight perception model;

[0040] The lightweight perception model consists of a multimodal feature extraction layer, a multi-head self-attention fusion layer, a lightweight LSTM temporal modeling layer, and an adaptive threshold decision layer connected sequentially. The multimodal feature extraction layer uses a lightweight ShuffleNetV2 backbone with EMA attention to achieve parallel extraction of RGB visual features, infrared features, and vehicle CAN status features. The multi-head self-attention fusion layer employs a multi-head self-attention mechanism, where each attention head focuses on different input feature parts, generating multiple contextual representations. The lightweight LSTM temporal modeling layer uses an LSTM network. The adaptive threshold decision layer uses a numerical decision mechanism to perform adaptive threshold judgment.

[0041] Step (6) specifically refers to: deploying the computationally inefficient multimodal feature extraction layer and adaptive threshold decision layer in the lightweight perception model on end-side devices such as vehicle cameras and domain controllers, and deploying the computationally intensive multi-head self-attention fusion layer and lightweight LSTM temporal modeling layer on roadside or vehicle edge computing nodes to establish a low-latency end-to-edge communication link and realize the collaborative operation of feature uploading and inference result distribution.

[0042] Step (7) specifically refers to: using a multi-head self-attention mechanism to adaptively weighted fuse multimodal data to obtain multimodal global fusion features, modeling the temporal change law of driving state through a lightweight LSTM, the temporal change law of driving state refers to the continuous change law of driver fatigue level or distraction state type over time, dynamically adjusting the warning threshold in combination with current illumination, vehicle speed, and road condition information, outputting graded state results and triggering audio-visual graded warnings; the graded state results include fatigue level and distraction state type: the fatigue level includes normal, mild fatigue, moderate fatigue, and severe fatigue; the distraction state type includes normal, visual distraction, cognitive distraction, operational distraction, and dangerous distraction.

[0043] In summary, this invention overcomes the limitations of single-modal perception under complex conditions by acquiring multimodal data through fusion, combining RGB vision, infrared thermal imaging, and multi-source information from the vehicle's CAN bus, thus improving the robustness of fatigue and distraction state recognition. This invention improves the information gain of the dataset and model training efficiency by progressively decomposing constituent factors, filtering effective samples using feature vector variance, and generating a high-quality dataset based on a genetic model to eliminate invalid data. This invention employs a hardware-adaptive lightweight model design, combined with knowledge distillation, pruning, and quantization techniques, significantly reducing the number of model parameters and computational complexity, adapting to vehicle-mounted embedded hardware deployment, and meeting the requirements for low-power, high-frame-rate real-time inference. This invention adopts an end-edge collaborative deployment architecture, with the edge responsible for lightweight feature extraction and the edge responsible for complex fusion and temporal modeling, achieving a balance between inference efficiency and resource utilization. This invention accurately captures instantaneous changes in the driver's state through multimodal attention fusion and temporal modeling, combined with dynamic warning threshold adjustment, to achieve graded and accurate warnings and improve vehicle safety protection capabilities.

[0044] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A multimodal lightweight perception method for distracted and fatigued driving, characterized in that: The method includes the following steps in sequence: (1) Collect multimodal data, and combine the multimodal data with publicly available driving behavior datasets to form a multi-source heterogeneous original dataset; (2) Decompose the constituent factors of the multi-source heterogeneous original dataset into hierarchical structures, establish a tree-like analysis model, and describe the hierarchical relationship of each factor in the form of a set; (3) Discretize the constituent factors of the multi-source heterogeneous original dataset, calculate the variance of the feature vector of the sample set, select valid samples with variance greater than the preset threshold, and remove invalid data. (4) The constituent factors of the effective samples are randomly combined to form the initial population sample set. Based on the genetic model, selection, crossover and mutation operations are performed iteratively to generate a high-quality final dataset S*. (5) Use the high-quality final dataset S* to train the original multimodal perception model, perform neural network architecture search, and use knowledge distillation, structured pruning, and 8-bit integer quantization combined compression techniques to construct a lightweight perception model; (6) The lightweight perception model is divided into tasks, and collaborative reasoning is achieved through end-edge data interaction and synchronization mechanism to obtain intermediate features and reasoning results; (7) Based on intermediate features and reasoning results, dynamically adjust the warning threshold, output the driver's fatigue and distraction status and trigger graded warnings.

2. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (1) specifically refers to: the multimodal data includes driver facial and behavioral visual information collected by the vehicle-mounted RGB camera, physiological feature information in low-light and occluded scenarios collected by the infrared thermal imaging sensor, and vehicle speed, steering, and pedal operation vehicle status information collected by the vehicle-mounted CAN bus. The collected multimodal data is synchronized in time, registered in space, and normalized with the publicly available driving behavior dataset to construct a multi-source heterogeneous original dataset covering complex working conditions.

3. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (2) specifically refers to: decomposing the influencing factors of the multi-source heterogeneous original dataset into first-level categories and second-level sub-factors. The first-level categories include driving scenarios, environmental conditions, driver characteristics, and vehicle status. The second-level sub-factors of the driving scenarios include urban roads, highways, rural roads, tunnels, and underground parking garages. The second-level sub-factors of the environmental conditions include light intensity, weather conditions, temperature, and occlusion. The second-level sub-factors of the driver characteristics include facial appearance, behavioral actions, and physiological characteristics. The second-level sub-factors of the vehicle status include vehicle speed, steering angle, pedal operation, and CAN bus signal. Using the four first-level categories as root nodes and the second-level sub-factors as leaf nodes, a tree-like analysis model with a scene-environment-person-vehicle hierarchical structure is formed.

4. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (3) specifically refers to: discretizing the scene, environment, personnel, and vehicle factors of the multi-source heterogeneous original dataset, constructing sample feature vectors, calculating the variance of each sample feature vector, retaining high information gain effective samples with variance greater than a preset threshold, and eliminating redundant, invalid, and noisy samples with variance less than the threshold.

5. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (4) specifically refers to: using the constituent factors of the effective samples as gene codes, randomly generating the initial population sample set, iteratively performing selection, crossover, and mutation operations based on the genetic model, retaining high-fitness samples through selection operations, combining advantageous factors through crossover operations, increasing sample diversity through mutation operations, and generating a high-quality final dataset S* with rich information and balanced distribution after iterative optimization.

6. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (5) specifically refers to: training the original multimodal perception model with the high-quality final dataset S* as the training sample, conducting neural network architecture search, improving the original multimodal perception model, and obtaining the improved model, namely the lightweight perception model; the original multimodal perception model adopts a lightweight LSTM temporal modeling layer, and on the basis of the lightweight LSTM temporal modeling layer, a multimodal feature extraction layer, a multi-head self-attention fusion layer, and an adaptive threshold decision layer are introduced to jointly form the lightweight perception model; The lightweight perception model consists of a multimodal feature extraction layer, a multi-head self-attention fusion layer, a lightweight LSTM temporal modeling layer, and an adaptive threshold decision layer connected sequentially. The multimodal feature extraction layer uses a lightweight ShuffleNetV2 backbone with EMA attention to achieve parallel extraction of RGB visual features, infrared features, and vehicle CAN status features. The multi-head self-attention fusion layer employs a multi-head self-attention mechanism, where each attention head focuses on different input feature parts, generating multiple contextual representations. The lightweight LSTM temporal modeling layer uses an LSTM network. The adaptive threshold decision layer uses a numerical decision mechanism to perform adaptive threshold judgment.

7. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (6) specifically refers to: deploying the computationally inefficient multimodal feature extraction layer and adaptive threshold decision layer in the lightweight perception model on end-side devices such as vehicle cameras and domain controllers, and deploying the computationally intensive multi-head self-attention fusion layer and lightweight LSTM temporal modeling layer on roadside or vehicle edge computing nodes to establish a low-latency end-to-edge communication link and realize the collaborative operation of feature uploading and inference result distribution.

8. The multimodal lightweight perception method for distracted and fatigued driving according to claim 1, characterized in that: Step (7) specifically refers to: using a multi-head self-attention mechanism to adaptively weighted fuse multimodal data to obtain multimodal global fusion features, modeling the temporal change law of driving state through a lightweight LSTM, the temporal change law of driving state refers to the continuous change law of driver fatigue level or distraction state type over time, dynamically adjusting the warning threshold in combination with current illumination, vehicle speed, and road condition information, outputting graded state results and triggering audio-visual graded warnings; the graded state results include fatigue level and distraction state type: the fatigue level includes normal, mild fatigue, moderate fatigue, and severe fatigue; the distraction state type includes normal, visual distraction, cognitive distraction, operational distraction, and dangerous distraction.

9. An electronic device, comprising: processor; as well as A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the multimodal lightweight perception method for distracted and fatigued driving as described in any one of claims 1-8.

10. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the multimodal lightweight perception method for distracted and fatigued driving as described in any one of claims 1-8.

Images