Driver state detection method and system based on multi-dimensional data

Abstract

This invention belongs to the field of driver state detection, specifically disclosing a method and system for driver state detection based on multidimensional data. The method acquires binocular image data of the driver's area using a binocular vision camera, and constructs a multidimensional data acquisition system by combining pressure and angle sensors from locations such as the seat and steering wheel. A binocular vision image analysis algorithm is used to establish a visual spatial model of the driver's body posture. A multidimensional data human posture calculation model converts the sensor data into physical parameters, which are then fused and matched to construct a complete body posture representation model. This model is combined with preset rules to detect real-time driver states such as fatigue and distraction. The system comprises six units, including binocular image acquisition and sensing, and multidimensional data acquisition and sensing, all working collaboratively. This invention utilizes multi-source data and an innovative model to achieve accurate and real-time detection of driver state.

Description

Technical Field

[0001] This invention relates to the field of driver state detection, and more particularly to a method and system for driver state detection based on multidimensional data. Background Technology

[0002] With the continuous increase in car ownership and the development of intelligent transportation technology, driver status detection is crucial for ensuring driving safety. Traditional driver status detection often relies on single-modal data, such as visual information acquired solely through cameras or limited sensor data, which is insufficient to comprehensively and accurately reflect the driver's true state. Single-modal visual information is easily affected by environmental factors such as lighting and occlusion. In low-light scenarios such as nighttime or tunnels, or when the driver is wearing sunglasses or their face is partially obscured, the detection accuracy drops significantly. On the other hand, relying solely on limited sensor data cannot cover multi-dimensional information such as the driver's body posture and operational behavior, resulting in blind spots and failing to meet the safety requirements of complex driving scenarios.

[0003] Existing technologies also have significant shortcomings in data processing and model building. On the one hand, they lack the ability to fuse and process multi-source data. Different types of data differ in format, temporal scale, and spatial scale. Traditional methods lack effective data processing and fusion mechanisms, failing to fully explore the potential correlations between data, resulting in low information utilization and poor reliability and stability of detection results. On the other hand, the detection models used are poorly adaptable. Faced with drivers with different driving habits and physical characteristics, as well as changing driving environments, it is difficult to dynamically adjust model parameters and detection strategies, making it impossible to achieve accurate and real-time driver status detection and effectively prevent traffic accidents caused by driver fatigue, distraction, and other conditions. Summary of the Invention

[0004] In order to overcome the shortcomings and deficiencies of the existing technology, the present invention provides a driver state detection method and system based on multidimensional data.

[0005] The technical solution adopted in this invention is a driver state detection method based on multi-dimensional data, comprising the following steps: Step S1: Continuously collect binocular image data of the driver's driving area using a binocular vision camera. The binocular vision camera is stably installed at a specific position on the vehicle according to preset installation parameters and angles to obtain complete image information including the driver's whole body and the driving operation area. Step S2: Construct a multi-dimensional data acquisition system by using pressure sensors and angle sensors installed on the seat and steering wheel to collect multi-dimensional body state related data, such as driver body contact pressure data, steering wheel grip force data, and steering wheel rotation angle data. Step S3: Using a binocular vision image analysis algorithm, feature extraction and analysis are performed on the collected binocular image data to identify the position coordinates of key points of various parts of the driver's body in three-dimensional space, and a visual space model of the driver's body posture is established. Step S4: Based on the multidimensional body state data acquired by the multidimensional data acquisition system, the multidimensional data human posture calculation model is used, combined with mechanical principles and ergonomics knowledge, to convert pressure and angle data into physical parameters that reflect the driver's body posture. Step S5: The visual space model obtained from the binocular vision image analysis is fused and matched with the physical parameters output by the multidimensional data human posture calculation model to construct a complete driver body posture representation model. Step S6: Based on the complete driver body posture representation model and the preset driver state judgment rules, analyze and detect the driver's real-time state, including fatigue, distraction, and abnormal action state, and output the detection results.

[0006] Furthermore, in step S3, a binocular vision feature matching optimization model is used when utilizing the binocular vision image parsing algorithm:

[0007] in, Indicating the first image in a binocular image The feature point and the first The matching degree of each feature point; For feature points With feature points The feature similarity parameters are calculated using image feature descriptors; For feature points With feature points Distance parameters in three-dimensional space; For feature points With feature points The time-series correlation parameter reflects the changing trend of feature points in consecutive frame images; Adjust the parameters of the model and optimize the settings according to different lighting conditions and driving scenarios.

[0008] Furthermore, in step S4, a multi-dimensional data fusion and transformation model is adopted when using the multi-dimensional data human pose calculation model:

[0009] in, For the converted first Physical parameters of body posture; For the first The pressure sensor data is at the first eigenvalues ​​of dimension; For the first The angle sensor data at the first... eigenvalues ​​of dimension; For pressure sensor data The weighting coefficients are determined by the sensor accuracy and the importance of its reflection of body posture. For angle sensor data The weighting coefficients are obtained by evaluating the correlation between the sensor installation location and the measurement parameters. For the number of pressure sensors, This represents the number of angle sensors.

[0010] Furthermore, in step S5, an attitude model fusion calibration model is introduced during the fusion matching process:

[0011] in, To integrate the calibrated driver body posture representation model; A visual space model obtained from binocular vision image analysis; The physical parameter model output by the multidimensional data human pose calculation model; To integrate the calibration coefficients, they are dynamically adjusted based on the reliability assessment of visual data and multidimensional data under different driving scenarios.

[0012] Furthermore, in step S6, a fatigue state quantification assessment model is used when detecting the driver's fatigue state:

[0013] in, Quantifying driver fatigue levels; The number of body posture parameters related to fatigue state; For the first Fatigue-related body posture parameters include head tilt angle, eyelid closure time, and degree of body relaxation. For the first The weighting coefficients of the influence of individual body posture parameters on fatigue state were obtained through training with a large amount of fatigue driving data.

[0014] Furthermore, in step S6, a distraction behavior recognition model is used when detecting a driver's distracted state:

[0015] in, Values ​​for judging the driver's distracted state; The number of body posture and operational behavior parameters related to distraction behavior; For the first Several parameters related to distraction include the angle of head deviating from the driving direction, the time of hands off the steering wheel, and the time of eyes leaving the road ahead. For the first The weighting coefficients of the influence of each parameter on the distraction state are determined by training based on data from different distraction behavior scenarios.

[0016] Furthermore, in step S1, a dynamic adjustment model for image acquisition parameters is used when acquiring binocular image data:

[0017] in, This is a set of dynamically acquired parameters from a binocular vision camera. for The resolution parameters of the camera are dynamically adjusted according to the ambient light intensity and the distance between the driver and the camera. for The frame rate parameters of the real-time camera are set according to the driver's movement range and the real-time detection requirements; for The camera's exposure parameters are automatically adjusted based on changes in ambient light.

[0018] Furthermore, throughout the detection process, the binocular vision image parsing algorithm and the multidimensional data human pose calculation model are jointly optimized using a dynamic weighted model co-evolutionary optimization framework.

[0019] in, To optimize the objective function; The length of the time window; These are dynamic weighting coefficients, determined by an assessment of the current scenario complexity, data reliability, and model stability. for Time feature points In the Differences before and after layer feature update; for Time of the first The data source in the first Differences before and after dimensional data processing; for Time of the first The driver state parameters are in the first Gradient changes in each direction; The number of driver status parameters; Calculate the number of directions for the gradient; by minimizing this objective function, combined with an adaptive learning rate adjustment mechanism:

[0020] in, for Learning rate at any given moment The initial learning rate, The decay coefficient is used for dynamic adjustment and co-evolution of model parameters.

[0021] Furthermore, in step S2, a spatiotemporal correlation sensor data compensation and fusion model is used when collecting multidimensional body state data:

[0022] in, After compensation Time of the first The sensor at the first Dimensional data; Original collection Time of the first The sensor at the first Dimensional data; for Time sensor With sensors The spatial correlation weight is determined by the correlation between the sensor's physical location and the data. for The time decay weight of historical data reflects the reference value of data at different times; for Time of the first Measurements of environmental disturbance factors, including temperature and vibration; For the first Historical average values ​​of environmental disturbance factors; for Time of the first Sensitivity coefficient of dimensional data to environmental disturbance; for Time of the first The model assesses the influence weights of various environmental disturbance factors and employs a two-dimensional spatiotemporal correlation analysis combined with a Kalman filter prediction mechanism.

[0023] in, for Time based Predicted value at time, Here is the state transition matrix. To control the input matrix, To control the input vector.

[0024] A driver state detection system based on multidimensional data, the system includes: A binocular image acquisition and sensing unit is installed at a specific location on the vehicle according to preset parameters to continuously acquire binocular image data of the driver's driving area and transmit the data to the data processing unit. The multi-dimensional data acquisition and sensing unit is integrated with the seat and steering wheel components. Through pressure sensors and angle sensors, it collects multi-dimensional body state-related data such as the pressure data of the driver's body contacting the seat and the steering wheel grip force, and transmits the data to the data processing unit. The binocular vision image analysis and processing unit is connected to the data processing unit. It receives binocular image data, uses binocular vision image analysis algorithms to extract and analyze features, and constructs a visual space model of the driver's body posture. The multidimensional data human posture calculation and processing unit is connected to the data processing unit. It receives multidimensional body state data and uses the multidimensional data human posture calculation model to convert the data into physical parameters of body posture. The data fusion and model building unit is connected to the binocular vision image analysis and processing unit and the multi-dimensional data human posture calculation and processing unit to fuse and match the visual space model and physical parameters to build a complete driver body posture representation model. The driver state analysis and judgment unit is connected to the data fusion and model building unit. Based on the complete driver body posture representation model, it analyzes and detects the driver's real-time state in combination with preset rules and outputs the detection results.

[0025] Beneficial Effects: This invention proposes a driver state detection method and system based on multidimensional data. It acquires binocular image data of the driver's location using a binocular vision camera, and simultaneously utilizes pressure and angle sensors located on the seat, steering wheel, etc., to construct a multidimensional data acquisition system, obtaining multidimensional body state-related data. This changes the traditional single-data source model, avoiding detection blind spots caused by environmental interference or data gaps, and comprehensively covering driver posture and operational behavior information. This invention uses a binocular vision image analysis algorithm to extract image features and establish a visual space model. It uses a multidimensional data human posture calculation model to convert sensor data into physical parameters. Multiple innovative models achieve deep data fusion and calibration, such as a binocular vision feature matching optimization model to improve image feature matching accuracy, and a multidimensional data fusion and transformation model to achieve complementary advantages of multi-source data, effectively solving the problem of weak multi-source data fusion capabilities in traditional technologies. Furthermore, a model co-evolution model jointly optimizes the core algorithm and model, dynamically adjusting parameters according to different driving scenarios and data characteristics, enhancing the system's adaptability to different drivers and complex environments, overcoming the poor adaptability of traditional models. Ultimately, this invention can accurately and in real time detect driver fatigue, distraction, and other states, output reliable detection results, and significantly improve driving safety. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method steps of the present invention; Figure 2 This is a diagram showing the system unit composition of the present invention. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. The following describes the application in further detail with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, the driver state detection method based on multidimensional data includes the following steps: Step S1: Continuously collect binocular image data of the driver's driving area using a binocular vision camera. The binocular vision camera is stably installed at a specific position on the vehicle according to preset installation parameters and angles to obtain complete image information including the driver's whole body and the driving operation area. Specifically, step S1 utilizes a binocular vision camera to continuously acquire binocular image data of the driver's driving area, serving as the fundamental source of visual information for the entire detection method. The binocular vision camera must be strictly fixed to the vehicle according to specific installation parameters and angles, typically near the rearview mirror or above the dashboard. The distance between the two cameras is generally between 80-120 mm to simulate the parallax effect of human eyes, thereby acquiring image data with stereoscopic information. Its shooting angle must cover the driver's entire body and main driving operation areas, with a horizontal angle of at least 120 degrees and a vertical angle of at least 90 degrees, ensuring complete capture of the driver's body movements, facial expressions, and other visual features. The image acquisition resolution is generally set to 1920×1080 pixels or higher, with a frame rate maintained at 30-60 frames per second to ensure image clarity and dynamic capture capabilities, providing high-quality raw data for subsequent image analysis. The acquisition effect directly impacts the accuracy of driver state feature extraction.

[0029] In terms of implementation, the binocular vision cameras are connected to the vehicle-mounted data processing module via a dedicated data transmission cable. Upon vehicle startup, they automatically enter working mode, continuously acquiring binocular image data according to preset parameters and transmitting it to the data processing unit in real time in a specific data format (such as RGB or YUV). To ensure the stability and reliability of the cameras, rigorous position calibration and parameter calibration are required during installation to ensure that the parallelism error of the optical axes of the left and right cameras is controlled within a minimal range. Simultaneously, the system monitors the camera's operating status in real time. When changes in lighting conditions are detected, it automatically adjusts the camera's exposure, gain, and other parameters to ensure clear and usable image data is acquired under different lighting conditions, providing a stable data source for subsequent image-based driver status analysis.

[0030] Step S2: Construct a multi-dimensional data acquisition system by using pressure sensors, angle sensors and other devices installed in the seat, steering wheel and other locations to collect multi-dimensional body state related data such as driver body contact pressure data, steering wheel grip force data, and steering wheel rotation angle data. Specifically, step S2 constructs a multi-dimensional data acquisition system aimed at acquiring driver physical state data from multiple physical dimensions, overcoming the limitations of single visual data. This system uses a pressure sensor array evenly distributed across the seat surface, typically with 8-16 high-precision pressure sensors per seat. This allows for precise sensing of the pressure distribution between the driver's body and the seat, with a pressure measurement accuracy of up to 0.1 Newtons, enabling analysis of driver posture changes, weight shifts, and other information. An angle sensor and a grip force sensor are integrated into the steering wheel. The angle sensor monitors the steering wheel rotation angle in real time, with a measurement range typically ±360 degrees and an accuracy of 0.5 degrees. The grip force sensor detects the driver's grip force on the steering wheel, with a range of 0-100 Newtons and an accuracy of 0.5 Newtons, thus acquiring the driver's driving force and steering intentions. These different types of sensors work together to provide rich data support for driver state detection from both mechanical and operational behavior dimensions, making the detection results more comprehensive and accurate.

[0031] During implementation, various sensors are connected to the data acquisition module via dedicated signal conditioning circuits. The analog signals acquired by the sensors are converted from analog to digital and transmitted to the data processing unit in digital form. To ensure data synchronization, all sensors are connected to the same clock synchronization system, ensuring consistency of different data types across time. Simultaneously, to prevent electromagnetic interference to sensor data, shielded cables are used during sensor wiring, and the data acquisition module undergoes electromagnetic shielding. The system also periodically calibrates and maintains the sensors, adjusting their zero-point and gain parameters based on usage time and performance degradation to ensure the accuracy and stability of the acquired data, providing a reliable data foundation for subsequent multi-dimensional driver status analysis.

[0032] Step S3: Using a binocular vision image analysis algorithm, feature extraction and analysis are performed on the collected binocular image data to identify the position coordinates of key points of various parts of the driver's body in three-dimensional space, and a visual space model of the driver's body posture is established. Specifically, step S3 employs a binocular vision image analysis algorithm to deeply process the acquired binocular image data, a crucial step in extracting the visual features of the driver's body posture. This algorithm first performs stereo matching on the binocular images, converting two-dimensional image information into three-dimensional spatial coordinates by calculating the disparity of corresponding feature points in the left and right images. During feature point extraction, a high-performance feature extraction operator is used to accurately identify key points in various parts of the driver's body, such as joints in the head, shoulders, elbows, and hands, achieving sub-pixel accuracy. By analyzing and modeling the three-dimensional coordinates of these key points, a visual spatial model of the driver's body posture is constructed. This model accurately describes the driver's limb positions, angles, and posture changes in the driving space, providing an intuitive visual representation for subsequent data fusion with other dimensions and driver state analysis.

[0033] In implementation, the binocular vision image analysis algorithm runs on an onboard high-performance computing unit equipped with a multi-core processor and a dedicated graphics processing unit (GPU) to meet the computational demands of processing large amounts of image data. Before processing the images, the algorithm performs preprocessing operations such as denoising and enhancement to improve image quality. During feature point matching, a fast matching algorithm combined with local optimization strategies is used to significantly improve processing speed while ensuring matching accuracy, ensuring real-time processing of image data captured by the camera. As the driver's body posture changes, the algorithm updates the visual space model in real time, ensuring that the model can dynamically reflect the driver's current posture and provide timely and accurate visual feature information for driver state detection.

[0034] Step S4: Based on the multidimensional body state data acquired by the multidimensional data acquisition system, the multidimensional data human posture calculation model is used, combined with mechanical principles and ergonomics knowledge, to convert data such as pressure and angle into physical parameters that reflect the driver's body posture. Specifically, step S4 employs a multi-dimensional data-driven human posture calculation model to transform pressure, angle, and other sensor data acquired by the multi-dimensional data acquisition system into physical parameters reflecting the driver's body posture, thus achieving the conversion from raw sensor signals to human posture information. For seat pressure sensor data, by analyzing the changing patterns of pressure distribution and combining ergonomic principles, the pressure values ​​of various parts of the driver's body on the seat are calculated, thereby inferring posture parameters such as sitting angle and body tilt. Steering wheel angle sensor and grip force sensor data are used to calculate parameters such as the driver's arm movement angle and grip force variation trend, which reflect the driver's steering intentions and body exertion. This model establishes a mathematical mapping relationship between sensor data and human posture parameters, unifying different types and dimensions of sensor data into physical quantities usable for posture analysis, providing standardized data support for subsequent fusion with visual information and driver state judgment.

[0035] During implementation, the multidimensional human posture calculation model is integrated into the vehicle's data processing unit as a software algorithm. Before running, the algorithm requires model parameter calibration based on parameters such as seat structure and steering wheel size for different vehicle models to ensure accurate adaptation. During data processing, filtering algorithms are used to denoise and smooth the raw data collected by sensors, eliminating random noise and interference signals. Simultaneously, to improve computational efficiency, the algorithm employs parallel computing technology to process data from different types of sensors synchronously. As the driver's driving behavior changes, the model updates the calculation results in real time, dynamically reflecting changes in the driver's body posture and providing accurate multidimensional posture information for driver state detection.

[0036] Step S5: The visual space model obtained from the binocular vision image analysis is fused and matched with the physical parameters output by the multidimensional data human posture calculation model to construct a complete driver body posture representation model. Specifically, step S5 fuses and matches the visual spatial model obtained from binocular vision image analysis with the physical parameters output from the multi-dimensional data human posture calculation model to construct a complete driver body posture representation model. This is the core step in achieving multi-dimensional information integration. During the fusion process, firstly, a spatial mapping relationship is established between the visual spatial model and the physical parameters, unifying the three-dimensional coordinate information based on the image with the physical posture parameters calculated from sensor data into the same coordinate system. Then, a data association algorithm is used to find the correspondence between visual features and physical parameters. For example, the arm position detected in the image is matched with the arm movement angle calculated by the steering wheel angle sensor, ensuring that data from different sources accurately corresponds to the same part and state of the driver's body. Finally, a fusion algorithm is used to weight and fuse the matched data, assigning appropriate weights based on the reliability and importance of different data in different scenarios, generating a complete body posture representation model containing both visual and physical information. This model can comprehensively and accurately describe the driver's body posture and movement state.

[0037] During implementation, the data fusion and model building process is completed by a dedicated fusion algorithm module in the onboard data processing unit. This module adopts a hierarchical fusion architecture. First, visual features and physical parameters are initially fused at the feature layer to extract representative joint features. Then, a comprehensive decision is made at the decision layer based on the joint features to construct the final body posture representation model. To improve the accuracy and stability of the fusion, the algorithm module automatically adjusts the fusion weight parameters according to different driving scenarios (such as urban road driving and highway driving). Simultaneously, the system performs consistency checks on the fused data. When significant differences are detected between different data sources, a data correction and compensation mechanism is activated to ensure that the constructed driver body posture representation model can realistically and reliably reflect the driver's actual state, providing an accurate model foundation for subsequent state detection.

[0038] Step S6: Based on the complete driver body posture representation model and the preset driver state judgment rules, analyze and detect the driver's real-time state, including fatigue, distraction, abnormal actions, etc., and output the detection results.

[0039] Specifically, step S6, based on a complete driver body posture representation model and pre-defined driver state judgment rules, analyzes and detects the driver's real-time state, representing the final goal of the entire detection method. The pre-defined judgment rules are based on extensive driving behavior data and safe driving research findings, covering criteria for various states including fatigue, distraction, and abnormal movements. For fatigue detection, parameters such as head tilt angle, eyelid closure time, and body relaxation level in the body posture representation model are analyzed and compared with fatigue thresholds to determine if the driver is fatigued. For distraction detection, parameters such as head deviation from the driving direction, time spent with hands off the steering wheel, and time spent looking away from the road are used, combined with pre-defined distraction behavior patterns, to identify whether the driver is exhibiting distracted behavior. When abnormal movements are detected, such as sudden, violent body swaying or abnormal limb postures, the system immediately issues a warning. Finally, the detection results are output as digital signals to the vehicle alarm device or vehicle control system, providing timely assurance for driving safety.

[0040] During implementation, the driver state detection algorithm runs on the core processor of the onboard data processing unit. This processor possesses high-speed data processing and analysis capabilities, enabling it to process body posture representation model data and make state judgments in real time. To improve the accuracy and reliability of detection, the algorithm employs a multi-level judgment mechanism to perform secondary verification and confirmation of the initial detection results. Simultaneously, the system personalizes the parameters in the judgment rules based on different drivers' physical characteristics and driving habits, achieving adaptive state detection. When an abnormal state is detected, the system will issue warnings to the driver through various means such as sound and lights, according to a preset alarm strategy. If necessary, it will also coordinate with the vehicle's active safety system to take measures such as deceleration and automatic distance maintenance, effectively preventing traffic accidents caused by abnormal driver states and ensuring driving safety.

[0041] Preferably, in step S3, a binocular vision feature matching optimization model is used when utilizing the binocular vision image parsing algorithm:

[0042] in, Indicating the first image in a binocular image The feature point and the first The matching degree of each feature point; For feature points With feature points The feature similarity parameters are calculated using image feature descriptors; For feature points With feature points Distance parameters in three-dimensional space; For feature points With feature points The time-series correlation parameter reflects the changing trend of feature points in consecutive frame images; Adjust the parameters of the model and optimize the settings according to different lighting conditions and driving scenarios.

[0043] Specifically, in step S3, when applying the binocular vision image analysis algorithm, a binocular vision feature matching optimization model is used to improve image analysis accuracy by calculating the feature point matching degree. The feature similarity parameter measures the degree of similarity between feature points in visual features, the spatial distance parameter reflects the positional relationship of feature points in three-dimensional space, and the time series correlation parameter considers the changing trend of feature points in consecutive frames. Model adjustment parameters are optimized according to different lighting conditions and driving scenarios. By comprehensively considering these factors, accurate feature point matching is achieved, providing more accurate basic data for subsequently constructing a visual spatial model of the driver's body posture, and improving the algorithm's adaptability to complex environments and the reliability of feature extraction.

[0044] Preferably, in step S4, a multi-dimensional data fusion and transformation model is used when calculating the human pose using multi-dimensional data:

[0045] in, For the converted first Physical parameters of body posture; For the first The pressure sensor data is at the first eigenvalues ​​of dimension; For the first The angle sensor data at the first... eigenvalues ​​of dimension; For pressure sensor data The weighting coefficients are determined by the sensor accuracy and the importance of its reflection of body posture. For angle sensor data The weighting coefficients are obtained by evaluating the correlation between the sensor installation location and the measurement parameters. For the number of pressure sensors, This represents the number of angle sensors.

[0046] Specifically, in step S4, when applying the multidimensional data human posture calculation model, a multidimensional data fusion and transformation model is used to weight and fuse data from different types of sensors. The weighting coefficients for pressure sensor data are determined by the sensor's accuracy and its importance in reflecting body posture, while the weighting coefficients for angle sensor data are obtained based on the sensor's installation location and the correlation of measurement parameters. This model scientifically integrates multidimensional data from different sensors, fully leveraging the advantages of each sensor and effectively reducing the limitations of single-sensor data. It achieves accurate conversion from multidimensional data to physical parameters of body posture, providing strong support for subsequent comprehensive and accurate analysis of the driver's body posture.

[0047] Preferably, in step S5, an attitude model fusion calibration model is introduced during the fusion matching process:

[0048] in, To integrate the calibrated driver body posture representation model; A visual space model obtained from binocular vision image analysis; The physical parameter model output by the multidimensional data human pose calculation model; To integrate the calibration coefficients, they are dynamically adjusted based on the reliability assessment of visual data and multidimensional data under different driving scenarios.

[0049] Specifically, in step S5, the fusion matching process introduces a posture model fusion calibration model. This model dynamically adjusts the fusion calibration coefficients to achieve accurate fusion of the visual spatial model obtained from binocular visual image analysis and the physical parameter model output from the multi-dimensional data human posture calculation model. The fusion calibration coefficients are dynamically adjusted based on the reliability assessment of visual and multi-dimensional data under different driving scenarios. In scenarios with high visual data reliability, the weight of the visual spatial model is increased; in scenarios with more reliable sensor data, the proportion of the physical parameter model is increased. This dynamic fusion calibration mechanism ensures that the most accurate driver body posture representation model is obtained under various driving scenarios, improving the system's adaptability to complex and changing environments and the accuracy of posture representation.

[0050] Preferably, in step S6, a fatigue state quantification assessment model is used when detecting driver fatigue state:

[0051] in, Quantifying driver fatigue levels; The number of body posture parameters related to fatigue state; For the first Fatigue-related body posture parameters include head tilt angle, eyelid closure time, and degree of body relaxation. For the first The weighting coefficients of the influence of individual body posture parameters on fatigue state were obtained through training with a large amount of fatigue driving data.

[0052] Specifically, in step S6, when detecting driver fatigue, a fatigue state quantification assessment model is used. This model quantifies the driver's fatigue level by weighting multiple fatigue-related body posture parameters. These parameters include head tilt angle, eyelid closure time, and degree of relaxation. Their weighting coefficients are obtained through training with a large amount of fatigue driving data, reflecting the importance of each parameter in determining fatigue status. This model integrates multiple disparate posture parameters into a comprehensive fatigue state quantification value, making fatigue detection more objective and accurate, enabling timely detection of driver fatigue, and providing effective protection for driving safety.

[0053] Preferably, in step S6, a distraction behavior recognition model is used when detecting a driver's distracted state:

[0054] in, Values ​​for judging the driver's distracted state; The number of body posture and operational behavior parameters related to distraction behavior; For the first Several parameters related to distraction include the angle at which the head deviates from the driving direction, the time the hands are off the steering wheel, and the time the eyes are off the road ahead. For the first The weighting coefficients of the influence of each parameter on the distraction state are determined by training based on data from different distraction behavior scenarios.

[0055] Specifically, in step S6, when detecting the driver's distracted state, the distraction behavior recognition model uses multiple parameters related to distraction behavior to accurately identify the driver's distracted state. These parameters, such as the angle at which the head deviates from the driving direction, the time the hands are off the steering wheel, and the time the gaze is off the road ahead, reflect the driver's attention span from different dimensions. The weight coefficients of each parameter are determined based on training data from different distraction behavior scenarios to ensure that the model can accurately capture various distraction behavior characteristics. Through comprehensive analysis of these parameters, the model can quickly and accurately identify the driver's distracted behavior, issue timely warnings, and effectively prevent traffic accidents caused by distraction.

[0056] Preferably, in step S1, a dynamic adjustment model for image acquisition parameters is used when acquiring binocular image data.

[0057] in, This is a set of dynamically acquired parameters from a binocular vision camera. for The resolution parameters of the camera are dynamically adjusted according to the ambient light intensity and the distance between the driver and the camera. for The frame rate parameters of the real-time camera are set according to the driver's movement range and the real-time detection requirements; for The camera's exposure parameters are automatically adjusted based on changes in ambient light.

[0058] Specifically, in step S1, when acquiring binocular image data, a dynamic adjustment model for image acquisition parameters is used. This model dynamically adjusts the camera's resolution, frame rate, and exposure parameters based on factors such as ambient light intensity, the distance between the driver and the camera, the driver's movement amplitude, and real-time detection requirements. In well-lit scenarios with minimal driver movement, the resolution and frame rate are appropriately reduced to decrease data processing volume. In dimly lit scenarios with vigorous driver movement, the resolution and frame rate are increased to ensure clear capture of driver movement details. This dynamic adjustment mechanism enables the camera to acquire high-quality image data in various driving scenarios, providing a solid data foundation for subsequent image analysis and driver status detection, while also optimizing the system's resource utilization efficiency.

[0059] Preferably, throughout the detection process, the binocular vision image parsing algorithm and the multidimensional data human pose calculation model are jointly optimized using a dynamic weighted model co-evolutionary optimization framework.

[0060] in, To optimize the objective function; The length of the time window; These are dynamic weighting coefficients, determined by an assessment of the current scenario complexity, data reliability, and model stability. for Time feature points In the Differences before and after layer feature update; for Time of the first The data source in the first Differences before and after dimensional data processing; for Time of the first The driver state parameters are in the first Gradient changes in each direction; The number of driver status parameters; The number of directions for the gradient is calculated. This is achieved by minimizing the objective function, combined with an adaptive learning rate adjustment mechanism.

[0061] in, for Learning rate at any given moment The initial learning rate, The attenuation coefficient enables dynamic adjustment and co-evolution of model parameters, thereby improving the generalization ability and robustness of the detection system.

[0062] Specifically, throughout the detection process, a dynamic weighted model co-evolutionary optimization framework is employed. This framework jointly optimizes the binocular vision image parsing algorithm and the multi-dimensional data human pose calculation model by optimizing the objective function. It introduces dynamic weight coefficients, determining the weights of each component based on the current scene complexity, data reliability, and model stability assessment. This allows the system to flexibly adjust the optimization focus according to actual conditions. Simultaneously, an adaptive learning rate adjustment mechanism dynamically adjusts the learning rate as training progresses, ensuring rapid model convergence and maintaining good generalization ability. Through this joint optimization approach, the performance and collaborative capabilities of the two core models are continuously improved, enabling the entire detection system to maintain high accuracy and stability in various driving scenarios.

[0063] Preferably, when collecting multidimensional body state data in step S2, a spatiotemporal correlation sensor data compensation and fusion model is used:

[0064] in, After compensation Time of the first The sensor at the first Dimensional data; Original collection Time of the first The sensor at the first Dimensional data; for Time sensor With sensors The spatial correlation weight is determined by the correlation between the sensor's physical location and the data. for The time decay weight of historical data reflects the reference value of data at different times; for Time of the first Measurement values ​​of environmental interference factors, such as temperature and vibration; For the first Historical average values ​​of environmental disturbance factors; for Time of the first Sensitivity coefficient of dimensional data to environmental disturbance; for Time of the first The model assigns weights to the influence of each environmental disturbance factor. It employs a two-dimensional spatiotemporal correlation analysis combined with a Kalman filter prediction mechanism.

[0065] in, for Time based Predicted value at time, Here is the state transition matrix. To control the input matrix, To control the input vector, effectively reduce random noise and systematic errors in sensor data, and improve data quality and stability.

[0066] Specifically, in step S2, when collecting multidimensional body state data, a spatiotemporal correlation sensor data compensation and fusion model is used to compensate and fuse the original sensor data through spatiotemporal two-dimensional correlation analysis. Spatial correlation weights reflect the physical location relationships and data correlations between different sensors, time decay weights reflect the reference value of data at different times, and environmental interference factor weights consider the degree of interference from the external environment on the sensor data. This model, combined with a Kalman filter prediction mechanism, performs real-time prediction and correction of sensor data, effectively reducing random noise and systematic errors in the sensor data, improving data quality and stability, and providing a more reliable data source for subsequent driver state analysis based on multidimensional data.

[0067] like Figure 2 As shown, the driver state detection system based on multidimensional data includes: A binocular image acquisition and sensing unit is installed at a specific location on the vehicle according to preset parameters to continuously acquire binocular image data of the driver's driving area and transmit the data to the data processing unit. The multi-dimensional data acquisition and sensing unit is integrated with components such as seats and steering wheels. Through devices such as pressure sensors and angle sensors, it collects multi-dimensional body state-related data such as driver body contact pressure data and steering wheel grip force data, and transmits the data to the data processing unit. The binocular vision image analysis and processing unit is connected to the data processing unit. It receives binocular image data, uses binocular vision image analysis algorithms to extract and analyze features, and constructs a visual space model of the driver's body posture. The multidimensional data human posture calculation and processing unit is connected to the data processing unit. It receives multidimensional body state data and uses the multidimensional data human posture calculation model to convert the data into physical parameters of body posture. The data fusion and model building unit is connected to the binocular vision image analysis and processing unit and the multi-dimensional data human posture calculation and processing unit to fuse and match the visual space model and physical parameters to build a complete driver body posture representation model. The driver state analysis and judgment unit, connected to the data fusion and model building unit, analyzes and detects the driver's real-time state based on a complete driver body posture representation model and pre-defined rules, and outputs the detection results. Data interaction and command transmission between units occur via a high-speed data transmission channel.

[0068] This invention proposes a driver state detection method and system based on multi-dimensional data. On one hand, by acquiring binocular image data of the driver's area using a binocular vision camera, visual information such as facial expressions and body posture can be obtained. Even in complex environments such as changing light or occlusion, the stereoscopic imaging characteristics of binocular vision ensure the integrity of the information. On the other hand, by using pressure sensors and angle sensors installed on the seat, steering wheel, and other locations, data such as body contact pressure with the seat, steering wheel grip strength, and rotation angle are collected, supplementing information from the dimensions of mechanics and operational behavior. This multi-source data acquisition method, compared to traditional single-modal detection, eliminates detection blind spots, comprehensively covers driver state information, and lays the foundation for accurate detection.

[0069] To address the shortcomings of traditional technologies in data processing and model adaptability, this method and system have undergone in-depth optimization. In the data processing stage, a binocular vision image analysis algorithm and a multi-dimensional data human posture calculation model are employed, along with several innovative models. These include a binocular vision feature matching optimization model to improve image feature matching accuracy, a multi-dimensional data fusion and transformation model to achieve complementary advantages from multiple data sources, and a posture model fusion calibration model to ensure accurate fusion of data from different modalities. This effectively solves the problem of heterogeneous multi-source data and improves data utilization. In terms of model construction, a model co-evolution model is used to jointly optimize the core algorithm and model, dynamically adjusting parameters based on different driving scenarios and individual driver differences, enabling the system to adapt to varying driving environments and different driver characteristics. Ultimately, the system can accurately detect driver fatigue, distraction, and other states, providing reliable protection for driving safety and significantly improving driving safety and intelligence.

[0070] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A driver state detection method based on multidimensional data, characterized by, Includes the following steps: Step S1: Continuously collect binocular image data of the driver's driving area using a binocular vision camera. The binocular vision camera is stably installed at a specific position on the vehicle according to preset installation parameters and angles to obtain complete image information including the driver's whole body and the driving operation area. Step S2: Construct a multi-dimensional data acquisition system by using pressure sensors and angle sensors installed on the seat and steering wheel to collect multi-dimensional body state related data, such as driver body contact pressure data, steering wheel grip force data, and steering wheel rotation angle data. Step S3: Using a binocular vision image analysis algorithm, feature extraction and analysis are performed on the collected binocular image data to identify the position coordinates of key points of various parts of the driver's body in three-dimensional space, and a visual space model of the driver's body posture is established. Step S4: Based on the multidimensional body state data acquired by the multidimensional data acquisition system, the multidimensional data human posture calculation model is used, combined with mechanical principles and ergonomics knowledge, to convert pressure and angle data into physical parameters that reflect the driver's body posture. Step S5: The visual space model obtained from the binocular vision image analysis is fused and matched with the physical parameters output by the multidimensional data human posture calculation model to construct a complete driver body posture representation model. Step S6: Based on the complete driver body posture representation model and the preset driver state judgment rules, analyze and detect the driver's real-time state, including fatigue, distraction, and abnormal action state, and output the detection results. Throughout the detection process, the binocular vision image parsing algorithm and the multidimensional data human pose calculation model are jointly optimized using a dynamic weighted model co-evolutionary optimization framework. in, To optimize the objective function; The length of the time window; These are dynamic weighting coefficients, determined by an assessment of the current scenario complexity, data reliability, and model stability. for Time feature points In the Differences before and after layer feature update; for Time of the first The data source in the first Differences before and after dimensional data processing; for Time of the first The driver state parameters are in the first Gradient changes in each direction; The number of driver status parameters; Calculate the number of directions for the gradient; by minimizing this objective function, combined with an adaptive learning rate adjustment mechanism: in, for Learning rate at any given moment The initial learning rate, The decay coefficient is used for dynamic adjustment and co-evolution of model parameters.

2. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S3, when using the binocular vision image parsing algorithm, a binocular vision feature matching optimization model is employed: in, Indicating the first image in a binocular image The feature point and the first The matching degree of each feature point; For feature points With feature points The feature similarity parameters are calculated using image feature descriptors; For feature points With feature points Distance parameters in three-dimensional space; For feature points With feature points The time-series correlation parameter reflects the changing trend of feature points in consecutive frame images; Adjust the parameters of the model and optimize the settings according to different lighting conditions and driving scenarios.

3. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S4, a multi-dimensional data fusion and transformation model is used when calculating the human pose using multi-dimensional data: in, For the converted first Physical parameters of body posture; For the first The pressure sensor data is at the first eigenvalues ​​of dimension; For the first The angle sensor data at the first... eigenvalues ​​of dimension; For pressure sensor data The weighting coefficients are determined by the sensor accuracy and the importance of its reflection of body posture. For angle sensor data The weighting coefficients are obtained by evaluating the correlation between the sensor installation location and the measurement parameters. For the number of pressure sensors, This represents the number of angle sensors.

4. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S5, an attitude model fusion calibration model is introduced during the fusion matching process: in, To integrate the calibrated driver body posture representation model; A visual space model obtained from binocular vision image analysis; The physical parameter model output by the multidimensional data human pose calculation model; To integrate the calibration coefficients, they are dynamically adjusted based on the reliability assessment of visual data and multidimensional data under different driving scenarios.

5. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S6, a fatigue state quantification assessment model is used when detecting driver fatigue state: in, Quantifying driver fatigue levels; The number of body posture parameters related to fatigue state; For the first Fatigue-related body posture parameters include head tilt angle, eyelid closure time, and degree of body relaxation. For the first The weighting coefficients of the influence of individual body posture parameters on fatigue state were obtained through training with a large amount of fatigue driving data.

6. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S6, a distraction behavior recognition model is used when detecting a driver's distracted state. in, Values ​​for judging the driver's distracted state; The number of body posture and operational behavior parameters related to distraction behavior; For the first Several parameters related to distraction include the angle of head deviating from the driving direction, the time of hands off the steering wheel, and the time of eyes leaving the road ahead. For the first The weighting coefficients of the influence of each parameter on the distraction state are determined by training based on data from different distraction behavior scenarios.

7. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S1, a model for dynamically adjusting image acquisition parameters is used when acquiring binocular image data. in, This is a set of dynamically acquired parameters from a binocular vision camera. for The resolution parameters of the camera are dynamically adjusted according to the ambient light intensity and the distance between the driver and the camera. for The frame rate parameters of the real-time camera are set according to the driver's movement range and the real-time detection requirements; for The camera's exposure parameters are automatically adjusted based on changes in ambient light.

8. The driver state detection method based on multidimensional data according to claim 1, characterized in that, In step S2, a spatiotemporal correlation sensor data compensation and fusion model is used when collecting multidimensional body state data: in, After compensation Time of the first The sensor at the first Dimensional data; Original collection Time of the first The sensor at the first Dimensional data; for Time sensor With sensors The spatial correlation weight is determined by the correlation between the sensor's physical location and the data. for The time decay weight of historical data reflects the reference value of data at different times; for Time of the first Measurements of environmental disturbance factors, including temperature and vibration; For the first Historical average values ​​of environmental disturbance factors; for Time of the first Sensitivity coefficient of dimensional data to environmental disturbance; for Time of the first The model assesses the influence weights of various environmental disturbance factors and employs a two-dimensional spatiotemporal correlation analysis combined with a Kalman filter prediction mechanism. in, for Time based Predicted value at time, Here is the state transition matrix. To control the input matrix, To control the input vector.

9. A driver state detection system based on multidimensional data, used to implement the driver state detection method based on multidimensional data as described in any one of claims 1-8, characterized in that, The system includes: A binocular image acquisition and sensing unit is installed at a specific location on the vehicle according to preset parameters to continuously acquire binocular image data of the driver's driving area and transmit the data to the data processing unit. The multi-dimensional data acquisition and sensing unit is integrated with the seat and steering wheel components. Through pressure sensors and angle sensors, it collects multi-dimensional body state-related data such as the pressure data of the driver's body contacting the seat and the steering wheel grip force, and transmits the data to the data processing unit. The binocular vision image analysis and processing unit is connected to the data processing unit. It receives binocular image data, uses binocular vision image analysis algorithms to extract and analyze features, and constructs a visual space model of the driver's body posture. The multidimensional data human posture calculation and processing unit is connected to the data processing unit. It receives multidimensional body state data and uses the multidimensional data human posture calculation model to convert the data into physical parameters of body posture. The data fusion and model building unit is connected to the binocular vision image analysis and processing unit and the multi-dimensional data human posture calculation and processing unit to fuse and match the visual space model and physical parameters to build a complete driver body posture representation model. The driver state analysis and judgment unit is connected to the data fusion and model building unit. Based on the complete driver body posture representation model, it analyzes and detects the driver's real-time state in combination with preset rules and outputs the detection results.

Images