Driver response time detection method for real vehicle scene and vehicle control method

Abstract

The invention belongs to the field of road vehicle control systems, and relates to a driver response time detection method and a vehicle control method.The driver response time detection method comprises the steps that firstly, front view data and eye movement data of a driver are collected based on double cameras, and vehicle state data are obtained; converting the sight line direction and eyeball center coordinates under the infrared camera coordinate system to a scene camera coordinate system based on a coordinate conversion algorithm, and constructing a ray; taking an intersection point of the ray and the imaging plane of the scene camera as a fixation point, and projecting the fixation point to a two-dimensional pixel coordinate system for imaging of the scene camera; environment target information is extracted; driver danger perception time and operation response time are determined by combining fixation points, environment target information and the like, and time difference between the two is calculated to obtain driver response time in real time. According to the method, the danger sensing time can be automatically and accurately extracted in a real vehicle environment, and the response time of a driver is obtained in real time.

Description

Technical Field

[0001] This invention belongs to the field of road vehicle control systems and relates to the calculation or prediction of driving parameters. Specifically, it relates to a real-time detection method for driver reaction time in real vehicle scenarios and a multi-scenario personalized vehicle control method. Background Technology

[0002] With the continuous growth of car ownership, road traffic volume has surged, leading to frequent safety risks such as rear-end collisions, side scrapes, and pedestrian accidents, which seriously threaten life and property safety. In the traffic system composed of people, vehicles, and roads, the driver's emergency reaction time, that is, the time taken from the driver's perception of a dangerous stimulus to taking an action (focusing on the driver's proactive reaction behavior), is the core factor determining the vehicle's hazard avoidance effectiveness. Its accurate measurement is crucial for the design of driver assistance systems and the optimization of traffic safety.

[0003] Existing driver reaction time measurement technologies have significant shortcomings, making it difficult to balance accuracy and scenario adaptability. In non-real-vehicle scenarios, such as the method in Chinese patent CN 110236575 A that uses an eye tracker combined with a driving simulator, although basic measurements can be completed, this solution relies on a virtual environment constructed by the simulator, which differs significantly from key factors in real driving, such as traffic flow and road condition changes. As a result, its measurement results cannot be transferred to real-vehicle scenarios, and it is essentially unsuitable for reaction time assessment in real driving environments. In real-vehicle scenarios, although Chinese patent CN 107126223 A has achieved real-vehicle measurement, it still has the following problems: (1) Although it uses an eye tracker to capture changes in the driver's eye focus in seven visual areas—directly in front, left window, right window, left rearview mirror, right rearview mirror, rearview mirror, and dashboard—to determine the perception state, it can only capture the moment when the eye's gaze area changes and cannot determine the moment when the driver perceives danger; (2) The eye tracker requires the driver to wear it and undergo complex pre-adjustment, which not only takes a long time to prepare and has a high usage threshold, but also easily causes discomfort to the driver during the wearing process, restricts head movement, and thus interferes with the stability of eye movement data acquisition. Therefore, it is necessary to provide a new method for detecting driver reaction time to achieve real-time detection in real-vehicle scenarios. Summary of the Invention

[0004] To address the issues of inaccurate driver reaction time measurements and reliance on specialized equipment such as eye trackers, this invention provides a driver reaction time detection method for real-world vehicle scenarios. This method replaces dedicated eye trackers with a dual-camera vision solution to collect gaze points and achieve coordinate mapping, thereby enabling automated and accurate extraction of the moment of hazard perception. This is then compared with the moment of operational reaction to obtain the driver's reaction time in real time, achieving accurate measurement of the driver's reaction time.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for detecting driver reaction time in real-world vehicle scenarios, comprising the following steps: Step S1. Multi-source data acquisition: The driver's forward field of view data is acquired through the scene camera, the driver's eye movement data is acquired through the infrared camera, and the vehicle status data of the vehicle is acquired simultaneously. All acquired data is bound to the unified timestamp of the vehicle system. Step S2. Perform distortion correction processing on the infrared and scene camera images; use the cornea-pupil offset to characterize the rotational offset of the eyeball on the infrared imaging plane, thereby constructing the initial gaze vector; obtain the head pose and solve for the rotation matrix of the head relative to the infrared camera. With translation vector Based on the attitude matrix ( , Obtain the coordinates of the eyeball center in the infrared camera coordinate system. Simultaneously through rotation matrix The initial gaze vector is corrected to obtain the gaze direction in the infrared camera coordinate system. Then, based on the coordinate transformation algorithm, the line of sight and eye center coordinates in the infrared camera coordinate system are transformed to the scene camera coordinate system, and a ray is constructed. ; to ray The intersection point with the scene camera's imaging plane is taken as the gaze point, projected onto the scene camera's two-dimensional pixel coordinate system, and the gaze point coordinates are smoothed to output the gaze point coordinates in the scene camera image. And associate with timestamp; Step S3. Environmental target information extraction: Use target detection algorithms to identify vehicles ahead, vehicles in adjacent lanes, and pedestrians who have entered the lane. Simultaneously extract the braking status of vehicles ahead, the accurate location and speed of the area where the brake lights are located, the speed of adjacent vehicles, the angle and lateral distance between them and the vehicle, and the position and lateral intrusion speed of pedestrians. Step S4. Combining the data obtained in steps S1 to S3, determine the moment when the driver perceives the danger and the moment when the driver reacts, and calculate the time difference between the two to obtain the driver's reaction time in real time.

[0006] As a preferred embodiment of the present invention, the scene camera is fixed at the upper middle position inside the windshield of the vehicle, with the lens facing the road ahead. Its collection range covers the vehicles, lane lines, pedestrians and adjacent lane environment in front of the driver's normal field of vision. It is used to collect continuous video stream data of the road ahead in real time, and each frame of the image is associated with a unified timestamp of the vehicle system. The infrared camera is installed in the center above the dashboard, with the lens facing the driver's face and focusing on the eye area. It is used to collect images of the driver's eye movements in real time, record the trajectory of eye movement and the direction of gaze. Each frame of eye movement image is also associated with a unified timestamp of the vehicle system. The vehicle status data includes the vehicle speed and brake pedal pressure, and all vehicle status data is bound to a unified timestamp of the vehicle system.

[0007] As a preferred embodiment of the present invention, in step S2, the distortion coefficients obtained by calibration are used to perform distortion correction processing on the infrared and scene camera images; when calculating the corneal-pupil offset, the coordinates of the corneal reflection center and the geometric center coordinates of the pupil are calculated first, and then the difference between the two is used to characterize the rotational offset of the eyeball on the infrared imaging plane, that is, the corneal-pupil offset.

[0008] As a preferred embodiment of the present invention, the rotation matrix of the head relative to the infrared camera in step S2... With translation vector The method for determining these features is as follows: A set of key points is detected in the facial image captured by the infrared camera. These key points include the corners of the eyes, the tip of the nose, and the corners of the mouth. This is then combined with the intrinsic parameters of the infrared camera. The PnP algorithm is used to establish a mapping relationship between points in a 3D face model and their corresponding 2D image points, thereby solving for the rotation matrix of the head relative to the infrared camera. With translation vector .

[0009] As a preferred embodiment of the present invention, in step S2, the line of sight and eye center coordinates in the infrared camera coordinate system are transformed to the scene camera coordinate system based on a coordinate transformation algorithm, and a ray is constructed. The specific method is as follows: By comparing the external parameters of the infrared camera and the scene camera The rotation matrix in the image represents the line-of-sight direction in the infrared camera coordinate system. View direction vector transformed into scene camera coordinate system ; By comparing the external parameters of the infrared camera and the scene camera The center point of the eye in the infrared camera coordinate system Transform to the scene camera coordinate system to determine the eye reference point in the scene camera coordinate system. ; In the scene camera coordinate system, with the eye reference point Starting from the line of sight vector Constructing ray The expression is: ; in, These are the ray parameters.

[0010] As a preferred embodiment of the present invention, in step S2, when smoothing the gaze point coordinates, a first-order low-pass filtering algorithm is used to smooth the gaze point coordinates, and the filtering coefficients are adjusted accordingly. To make dynamic adjustments, the expression is: ; in, Based on the basic filter coefficients, take ; The magnitude of the head angular velocity is determined by rotating the matrix across consecutive frames. The rate of change was calculated. The rate of change of gaze vectors between adjacent frames; , This is the adjustment coefficient; Adjustment coefficient , The expression is: ; in, , 3 seconds before the current frame , 95th percentile.

[0011] As a preferred embodiment of the present invention, in step S3, after extracting the vehicle parameters of adjacent lanes, it is simultaneously determined whether a vehicle in an adjacent lane has "cut into" the current lane. When the lateral distance is satisfied... , Change rate of angle in 3 consecutive frames And speed When a vehicle in the adjacent lane enters the lane, it is marked as "cutting in." Based on the extracted pedestrian parameters, it is determined whether a pedestrian has entered the lane, and for pedestrians who have "entered the lane," it is determined whether they are in a "dangerous" state. If the overlap between the pedestrian bounding box and the lane area is significant... If the pedestrian enters the lane, it is determined that "the pedestrian has entered this lane"; otherwise, it is determined that "the pedestrian has not entered"; if the pedestrian "has entered this lane", the lateral intrusion speed is... If it is, then it is marked as "dangerous".

[0012] As a preferred embodiment of the present invention, step S4 includes the following steps: Step S41. Driver's moment of hazard perception Acquisition: ①Sensing the hazard of a vehicle braking ahead: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The coordinates of the driver's gaze point in this frame match the coordinates of the brake light area at the rear of the vehicle ahead. The vehicle in front of this frame is braking; ② Danger perception of vehicles cutting into adjacent lanes: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze coordinates in this frame match the bounding box coordinates of adjacent vehicles. The vehicle status in the adjacent lane of this frame is "cutting in"; ③ Pedestrian intrusion danger perception: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze point coordinates match the pedestrian bounding box coordinates in this frame; The pedestrian's dangerous situation in this frame is "dangerous"; Step S42. Operation reaction time Acquisition: When an upward trend is detected in the brake pedal pressure signal, this moment is recorded as a candidate moment. If the pressure signal shows a sustained and significant increase within a preset time, exceeding a set threshold, it is confirmed that the driver has performed an emergency braking operation, and the previously recorded timestamp is determined as the reaction moment. If the condition for continuous increase is not met, the candidate moment will not be considered as the operational response moment. Step S43. Timestamp Alignment Calibration: Step S44. Reaction time calculate: Calculating the moment of danger perception With the reaction time of the operation The difference is the driver's reaction time. .

[0013] The present invention also provides a vehicle control method for real-world vehicle scenarios, which formulates personalized vehicle control strategies based on different driver reaction times to improve road traffic safety.

[0014] A vehicle control method for real-world vehicle scenarios, comprising the following steps: Step A1. Calculate the collision time under different dangerous stimulus scenarios based on the identified dangerous stimulus scenarios; Step A2. Calculate the time required for the driver to complete the collision avoidance based on the driver's reaction time obtained above; Step A3. For each identified hazardous stimulus scenario, calculate its safety level based on the collision time and the time required for the driver to complete the collision avoidance. When more than two hazardous stimulus scenarios are detected, calculate the overall safety level of all hazardous stimulus scenarios. The expression is: ; in, It is a collection of current dangerous and stimulating scenarios. Safety level in the corresponding scenario; For the dynamic weighting coefficients of the corresponding scenario, satisfying The expression is: ; in, It is a very small positive number. The calculated collision time for the corresponding scene; Step A4. Based on the overall safety level Based on different calculation results, implement differentiated security intervention strategies adapted to each security state.

[0015] As a preferred embodiment of the present invention, in step A4, if Without any intervention, the driver completes the operation based on their own judgment, and the system only continuously monitors the vehicle status and the driver's reaction; like This triggers an audio-visual warning, including a continuously flashing red warning light on the dashboard and a high-frequency warning sound from the vehicle's speakers, prompting the driver to take timely braking or evasive action. like The vehicle's automatic collision avoidance mechanism is activated, automatically applying maximum braking force for emergency deceleration. At the same time, it combines the steering assist system to fine-tune the direction to avoid collisions and simultaneously activates the hazard lights to warn surrounding vehicles.

[0016] The advantages and beneficial effects of this invention are as follows: 1. This invention proposes a technical solution that can acquire driver reaction time in real-time under real vehicle conditions, so as to calculate the total time required for the driver to complete the avoidance based on the driver's perception and operation status, and combine it with the time to collision (TTC) to calculate the degree of safety, thereby realizing graded safety intervention decision-making and vehicle control, forming a complete closed loop from hazard perception, operation reaction to control intervention, effectively improving road traffic safety and system intelligence level.

[0017] 2. This invention proposes a dual-camera solution using an infrared camera and a scene camera. The dual-camera eye-tracking solution can effectively replace expensive and complex eye trackers, enabling contactless and low-cost tracking of driver eye movements. This solution avoids the drawbacks of eye trackers, such as cumbersome operation and interference with driving, while capturing eye and head movement features in real time through stereo vision. Combined with algorithms, it automatically extracts the moment of danger perception, reducing human intervention errors. At the same time, it is adapted to real vehicle scenarios, providing a feasible path for efficient and accurate measurement of driver reaction time.

[0018] 3. This invention proposes a gaze point mapping method based on corneal-pupil offset and dual-camera fusion. It uses an infrared camera to construct an initial gaze vector and maps it uniformly to the scene camera imaging plane through extrinsic parameter calibration and posture correction. Compared with traditional methods that rely solely on corneal-pupil geometry or monocular imaging, this method can better eliminate the deviation caused by changes in head posture and achieve real-time capture of the driver's gaze point in real road scene images.

[0019] 4. This invention proposes an adaptive adjustment method for filter coefficients based on head angular velocity and gaze vector change rate, replacing the traditional fixed coefficient low-pass filter. This achieves a dynamic balance between gaze point smoothing in "stable gaze" and "rapid saccade" scenarios, significantly improving the stability and real-time performance of gaze point output, and enhancing the feasibility and practicality of in-vehicle applications.

[0020] 5. This invention proposes using whether the gaze point falls within the target bounding box as the criterion for determining whether a driver perceives a hazard, thereby achieving automatic identification of the moment of hazard perception, replacing manual annotation, and completing the transformation from "regional level" to "target level", significantly improving the accuracy and real-time performance of the judgment.

[0021] 6. In the safety intervention decision-making process, this invention calculates the total time required for the driver to complete collision avoidance by combining the driver's reaction time and the vehicle's braking response time. Based on the collision time and the time required for the driver to complete collision avoidance, the safety level is calculated for different dangerous stimulus scenarios. A graded safety intervention decision-making mechanism is formed based on the safety level in a single scenario.

[0022] 7. This invention proposes a comprehensive overall safety calculation method covering single-scenario and multi-scenario superposition. In a single scenario, the available time sufficiency is quantified by the ratio of collision time to collision avoidance time. In multiple scenarios, the overall safety is calculated by dynamically allocating weights based on the collision time of each scenario. This replaces the traditional method of directly comparing collision avoidance time and collision time for a single scenario, and achieves a unified quantitative assessment of the safety status of single-scenario and multi-hazard superposition scenarios, providing a precise basis for subsequent differentiated safety intervention decisions. Attached Figure Description

[0023] Figure 1 Flowchart of the driver reaction time detection method for real-world vehicle scenarios of this invention; Figure 2 This is a mapping diagram of the gaze point coordinates in this invention; Figure 3 This is a flowchart of the hazard identification process in this invention; Figure 4 This is a flowchart of the vehicle control method for real-world vehicle scenarios according to the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions and advantages of the present invention, the present application will be described in detail below with reference to the accompanying drawings, but this is not intended to limit the scope of protection of the present invention.

[0025] Example 1:

[0026] like Figures 1 to 3 As shown, this embodiment provides a driver reaction time detection method for real-world vehicle scenarios. This method uses a dual-camera vision solution to replace a dedicated eye tracker to collect gaze points and achieve coordinate mapping, thereby enabling automated and accurate extraction of the moment of danger perception. This is then compared with the moment of operational reaction to obtain the driver's reaction time in real time. The specific steps are as follows: Step S1. Multi-source data acquisition: The driver's forward field of vision data is acquired through a scene camera (also known as a scene camera or scene camera) facing the driver's field of vision, and the driver's eye movement data is acquired through an infrared camera (also known as an infrared camera or infrared camera) facing the driver. The vehicle status data such as vehicle speed and brake pedal pressure are acquired simultaneously. All acquired data is bound to a unified timestamp of the vehicle system. Specifically, in this embodiment, the scene camera is fixed at the upper middle position inside the windshield of the vehicle, with the lens facing the road ahead. Its collection range covers the vehicles, lane lines, pedestrians and adjacent lane environment in front of the driver's normal field of vision. It is used to collect continuous video stream data of the road ahead in real time, and each frame of the image is associated with a unified timestamp of the vehicle system.

[0027] The infrared camera is installed in the center above the dashboard, with the lens facing the driver's face and focusing on the eye area. It is used to collect images of the driver's eye movements in real time, record the trajectory of eye movement and the direction of gaze. Each frame of eye movement image is also associated with a unified timestamp of the vehicle system.

[0028] In this embodiment, a vehicle speed sensor is connected via the vehicle's CAN bus interface to acquire real-time vehicle speed data; a high-precision pressure sensor is integrated at the brake pedal mechanical linkage, and a microcontroller module converts the analog signal of the pedal force into a digital signal to synchronously collect brake pedal pressure data; all the above vehicle status data are bound to a unified timestamp of the vehicle system to maintain consistency with the time reference of the data collected by the camera.

[0029] Step S2. Perform distortion correction processing on the infrared and scene camera images; use the cornea-pupil offset to characterize the rotational offset of the eyeball on the infrared imaging plane, thereby constructing the initial gaze vector; obtain the head pose and solve for the rotation matrix of the head relative to the infrared camera. With translation vector Based on the attitude matrix ( , Obtain the coordinates of the eyeball center in the infrared camera coordinate system. Simultaneously through rotation matrix The initial gaze vector is corrected to obtain the gaze direction in the infrared camera coordinate system. Then, based on the coordinate transformation algorithm, the line of sight and eye center coordinates in the infrared camera coordinate system are transformed to the scene camera coordinate system, and a ray is constructed. ; to ray The intersection point with the scene camera's imaging plane is taken as the gaze point, projected onto the scene camera's two-dimensional pixel coordinate system, and the gaze point coordinates are smoothed to output the gaze point coordinates in the scene camera image. And associate it with a timestamp.

[0030] Specifically, in this embodiment, step S2 achieves accurate mapping of the driver's gaze point in the actual field of vision, and the specific process is as follows: Step S21. Dual camera (scene camera, infrared camera) parameter calibration: According to the pinhole camera model, the projection of a spatial point in the camera coordinate system can be expressed as:

[0031] in, This is the scaling factor for homogeneous coordinates; , These are the pixel coordinates in the camera's imaging plane; , , For the camera , Focal length along the axial direction, , Principal point coordinates; , For the camera's extrinsic matrix and translation vector; , , The coordinates of the point in space are the three-dimensional coordinates in the camera coordinate system.

[0032] By using Zhang Zhengyou's calibration method to calibrate the intrinsic and extrinsic parameters of infrared cameras and scene cameras, the intrinsic parameters of infrared cameras can be obtained. and distortion coefficient Scene camera internal parameters and distortion coefficient And the relative extrinsic parameters from the infrared camera to the scene camera. This provides a basis for subsequent coordinate transformations; the expression is:

[0033] in, It is a rotation matrix; It is a translation vector.

[0034] In addition, in order to realize the conversion between pixels and actual physical distance, the pixel-to-actual distance conversion coefficient needs to be solved simultaneously during the calibration process. Select a calibration object or road lane line of known actual size on the scene camera's imaging plane, whose physical length is... The pixel spacing in the image plane is Then we have:

[0035] This coefficient remains approximately constant when the camera mounting height and pitch angle are fixed, and can subsequently be used to directly convert pixel distance into actual lateral or longitudinal distance in the vehicle coordinate system.

[0036] Step S22. Image distortion correction; Because the lens exhibits radial and tangential distortion, the calibrated distortion coefficients are used to perform distortion correction processing on the infrared and scene camera images. The distortion correction formula is as follows:

[0037] in, These are the original pixel coordinates of the spatial point in the camera's imaging plane before distortion correction; For the main point and focal length Normalized pixel coordinates; Normalized distortion coordinates To the main point The radial distance; The coordinates are after distortion correction. , , , , The distortion coefficient is denoted as .

[0038] Step S23. Extraction of corneal reflection center and pupil center: To obtain the corneal reflection point, a combination of high and low thresholding is used. First, a high threshold is used to quickly locate the approximate region of the spot in the grayscale image. Then, a lower threshold is used to extract the region of interest (ROI) within this region. Finally, the coordinates of the corneal reflection center are accurately calculated using the grayscale weighted centroid formula. The expression is:

[0039] in, Coordinates within the ROI The pixel grayscale value at that location.

[0040] For pupil center extraction, the eye region is first binarized under multiple different thresholds, and the corresponding ratio of "pupil area / enclosing rectangle area" is calculated. When the ratio The threshold value at which the segmentation result is maximized is considered optimal, and this threshold is used to obtain the pupil region. Subsequently, a gradient operator is used to extract feature points of the pupil edge, and an ellipse fitting method is used to determine the geometric center coordinates of the pupil. .

[0041] This yields the corneal-pupil offset. The expression is:

[0042] The corneal reflection center is basically stable on the imaging plane and can be used as a reference point; the pupil center shifts with eye movement, so the difference between the two can effectively characterize the amount of eye movement shift on the infrared imaging plane.

[0043] Step S24. Head pose estimation: To eliminate the impact of driver head movements on gaze point calculation, head posture needs to be acquired simultaneously. A set of key points (corners of the eyes, tip of the nose, corners of the mouth, etc.) can be detected in facial images captured by infrared cameras. These points are then combined with the intrinsic parameters of the infrared camera. The PnP algorithm is used to establish a mapping relationship between points in a 3D face model and their corresponding 2D image points, thereby solving for the rotation matrix of the head relative to the infrared camera. With translation vector This not only provides a basis for subsequent line-of-sight direction calculation and attitude compensation, but also allows for the calculation of rotation matrices in consecutive frames. The head angular velocity is obtained by calculating the rate of change. This provides support for adaptive adjustment of the filter coefficients; simultaneously, since the fixed position of the eyeball center in the facial coordinate system is predefined in the 3D face model, the pose matrix can be used ( , Map this location point to the infrared camera coordinate system to obtain the coordinates of the eye center. This point serves as the starting point for constructing the gaze ray in the scene camera coordinate system, providing a benchmark for gaze projection calculation.

[0044] Step S25. Constructing the gaze vector and transforming the coordinate system: Based on the aforementioned features and pose information, an initial gaze vector is first constructed. The expression is:

[0045] The line of sight in the infrared camera coordinate system is obtained by correcting the head pose rotation matrix. :

[0046] Subsequently, the relative external parameters of the infrared camera (infrared camera) and the scene camera (scene camera) were combined. The rotation matrix in the image is used to transform the view direction vector into the scene camera coordinate system. :

[0047] Step S26. Projection of the gaze point: Relative extrinsic parameters obtained using dual-camera calibration The center point of the eye in the infrared camera coordinate system Transform to the scene camera coordinate system; eye reference point in the scene camera coordinate system. The expression is:

[0048] In the scene camera coordinate system, with the eye reference point Starting from the line of sight vector Constructing ray The expression is:

[0049] in, The ray parameter is used to characterize the scale factor in space when extending from the center of the eye along the line of sight. This parameter does not need to be measured directly, but is naturally determined by the intersection conditions of the ray and the scene camera imaging plane under imaging geometric constraints.

[0050] The ray intersects the imaging plane of the scene camera, and the intersection point is obtained. The intersection point is used as the focal point, and the intrinsic parameter matrix of the scene camera is used. The intersection point is projected from the scene camera's 3D coordinate system to the scene camera's 2D pixel coordinate system, expressed as:

[0051] in, Let these be the initial pixel coordinates of the gaze point in the scene image. This is the scaling factor for homogeneous coordinates.

[0052] Step S27. Fixation Smoothing and Output: A first-order low-pass filter algorithm is used to smooth the gaze point coordinates to reduce the impact of high-frequency jitter. The expression is:

[0053] in, The optimized gaze point coordinates from the previous frame. These are the filter coefficients.

[0054] Furthermore, to overcome the shortcomings of traditional fixed filter coefficients in balancing stability and real-time performance, this invention proposes an adaptive modeling method for filter coefficients based on the driver's dynamic state: When the driver's head rotation angular velocity is large, the line of sight changes rapidly, at which point increasing... To reduce filtering lag; when the rate of change of the gaze vector is large (rapid saccades), it also improves This is to ensure the timeliness of gaze tracking.

[0055] Taking all the above factors into account, the filter coefficients Dynamically adjusted to:

[0056] in, Based on the basic filter coefficients, take ; This refers to the magnitude of the head's angular velocity. The rate of change of the gaze vector (view direction vector) between adjacent frames; , This is the adjustment coefficient.

[0057] Adjustment coefficient , The expression is:

[0058] in, , 3 seconds before the current frame , Choosing the 95th percentile strikes a balance between robustness and sensitivity. Compared to the mean or median, the 95th percentile avoids interference from outliers and, compared to the upper quartiles, better reflects the high-amplitude dynamic characteristics of drivers under rapid gaze shifts or abrupt head movements, providing a reference close to the true extreme values. Modeling with this can more scientifically characterize extreme but common fluctuations in drivers, thereby improving the rationality and applicability of adaptive adjustment of the filter coefficients.

[0059] Step S3. Environmental target information extraction: Based on the forward field of view data collected in step S1, the target detection algorithm is used to identify vehicles in front (including the rear of the vehicle and the taillight area), vehicles in adjacent lanes, and pedestrians who have entered the lane. Key information such as the speed of the vehicles in front, the angle between the adjacent vehicles and the vehicle, and the position of the pedestrians are extracted simultaneously. Specifically, step S3 includes the following steps: Step S31. Lane line detection and lane area definition: The algorithm identifies the pixel coordinates of the left and right boundary lines of the current lane using a lane line detection algorithm, and then fits and generates the equations for the left and right boundaries. (Left line...) Right line Construct the pixel mask for the lane region as The expression is:

[0060] The fitting parameters of the left and right lane lines are output synchronously for subsequent adjacent lane definition and target position determination.

[0061] Step S32. Multi-target detection and tracking: The YOLOv11 algorithm is used to detect targets in each frame of the foreground view, identifying three types of targets and outputting the bounding box pixel coordinates: Vehicle ahead: Label "car_front", bounding box coordinates ( , , , ),in, , The horizontal and vertical pixel coordinates of the top-left corner vertex of the bounding box of the preceding vehicle. , These are the horizontal and vertical pixel coordinates of the bottom right corner vertex of the front vehicle's bounding box. Vehicles in adjacent lanes: Label "car_adjacent", bounding box coordinates ( , , , ),in, , These are the horizontal and vertical pixel coordinates of the top-left corner vertex of the bounding box of vehicles in adjacent lanes. , The horizontal and vertical pixel coordinates of the bottom right corner vertex of the vehicle bounding box in the adjacent lane; Pedestrian: Label "person", bounding box coordinates ( , , , ),in, , These are the horizontal and vertical pixel coordinates of the top-left vertex of the pedestrian bounding box. , These are the horizontal and vertical pixel coordinates of the bottom right corner vertex of the pedestrian bounding box.

[0062] The SORT algorithm is used to perform multi-target tracking for three types of targets: Kalman filtering is combined to predict the motion trajectory of each target, and the Hungarian algorithm is used to associate the detection boxes of consecutive frames with the existing trajectory. A unique ID is assigned to each target to ensure continuous tracking in the video sequence. The bounding box coordinates of all targets are associated with the timestamp of the current frame. .

[0063] Step S33. Extract features of the vehicle ahead and determine the braking status of the vehicle ahead based on the extracted features: For the target labeled "car_front", extract its tail features and motion parameters: An RGB image of the rear area of ​​the vehicle in front is extracted, converted to the HSV color space, and an HSV threshold is set to extract the brake light area. The image is then binarized, and a closing operation is performed on the segmented binarized image. The rate of change in the proportion of white pixels in the brake light area in consecutive frames is calculated. The working status of the brake light is obtained by comparing the changes in the proportion of white pixels in consecutive frames. Simultaneously, the brake light is accurately located within the bounding box of the rear of the vehicle in front, and the precise location of the brake light area is identified using a deep learning model (e.g., the NanoDet model). , , , ),in, , The horizontal and vertical pixel coordinates of the top-left vertex of the brake light bounding box. , These are the horizontal and vertical pixel coordinates of the bottom right corner vertex of the brake light bounding box. Based on the longitudinal coordinates of the vehicle bounding box center in two consecutive frames , By combining the camera's intrinsic parameters with the actual longitudinal distance changes, the speed of the vehicle ahead can be calculated. The expression is:

[0064] in, is the pixel-to-actual distance conversion coefficient.

[0065] Step S34. Extract vehicle parameters from adjacent lanes. Based on the extracted vehicle parameters from adjacent lanes, determine whether a vehicle in an adjacent lane has "cut into" this lane. For the target labeled "car_adjacent", extract the following key parameters: Based on the bounding box center coordinates of two consecutive frames , The velocity is obtained after calculating the actual displacement, and its expression is:

[0066] Establish a polar coordinate system with the center of the vehicle's front as the origin, and the lateral distance between the centers of the bounding boxes of adjacent vehicles is... Longitudinal distance is Then the included angle The expression is:

[0067] Calculate the shortest pixel distance between the bounding boxes of adjacent vehicles and the boundary line of the current lane, and convert it to the actual distance, which is the lateral distance. ; When the horizontal distance is satisfied , Change rate of angle in 3 consecutive frames And speed At that time, vehicles in the adjacent lane are marked as "cutting in".

[0068] Step S35. Pedestrian parameter extraction: Based on the extracted pedestrian parameters, determine whether a pedestrian has entered the lane, and for pedestrians who have "entered the lane", determine whether they are in a "dangerous" state. pedestrian bounding box pixel set The expression is: ; Calculate the overlap ratio with this lane area. The expression is:

[0069] like If the pedestrian enters the lane, it is determined that "the pedestrian has entered the lane"; otherwise, it is determined that "the pedestrian has not entered the lane".

[0070] For pedestrians who have "entered this lane", the lateral coordinates of the bounding box center are used as the basis for the calculation. , Calculate the lateral intrusion velocity The expression is:

[0071] like If it is, then it is marked as "dangerous".

[0072] Step S4. Combining the gaze point mapping results obtained in Step S2, the environmental target information extracted in Step S3, and the brake pedal pressure data in Step S1, determine the moment of driver's hazard perception and the moment of operation reaction, and calculate the time difference between the two to obtain the driver's reaction time in real time.

[0073] Specifically, step S4 includes the following steps: Step S41. Driver's moment of hazard perception Acquisition: ①Sensing the hazard of a vehicle braking ahead: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The coordinates of the driver's gaze point in this frame match the coordinates of the brake light area at the rear of the vehicle ahead, i.e. ; The braking status of the vehicle in front in this frame is "braking" (determined by the working status of the brake lights of the vehicle in front; if the brake lights are on, the vehicle in front is braking; otherwise, it is not braking).

[0074] ② Danger perception of vehicles cutting into adjacent lanes: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze coordinates in this frame match the bounding box coordinates of the adjacent vehicles, i.e. ; The vehicle status in the adjacent lane of this frame is "cutting in".

[0075] ③ Pedestrian intrusion danger perception: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze coordinates in this frame match the pedestrian bounding box coordinates, i.e. ; The pedestrian's dangerous condition in this frame is "dangerous".

[0076] Step S42. Operation reaction time Acquisition: The microcontroller system acquires the analog pressure signal from the high-precision pressure sensor integrated in the brake pedal in real time, converts it into a digital signal, and transmits it to the processing unit. The processing unit monitors the received digital pressure signal in real time. When an upward trend is detected in the pressure signal, this moment is recorded as a candidate moment. If the pressure signal shows a continuous and significant increase within a preset time (short time) and exceeds a set threshold, it is confirmed that the driver has performed an emergency braking operation, and the previously recorded timestamp is determined as the reaction moment. If the condition for continuous increase is not met, the candidate moment will not be used as the operational response moment.

[0077] Step S43. Timestamp Alignment Calibration: The video data captured by the camera and the pressure analog signal data collected by the pressure sensor are time-stamped and calibrated using the unified system clock of the onboard computing platform during data reception. The onboard computing unit then reads the hazard perception moments marked in the camera video stream. Timestamps, and the timing of the action response transmitted by the brake pedal pressure sensor. Timestamps directly extract timestamp values ​​under a unified benchmark, eliminating time benchmark differences between devices and ensuring... and They are in the same time coordinate system.

[0078] Step S44. Reaction time calculate: Calculating the moment of danger perception With the reaction time of the operation The difference is the driver's reaction time. The calculation formula is:

[0079] If the calculation result is positive, it indicates that the operation response is after the danger perception and conforms to the normal response logic, and the result is output; if the result is negative, it is judged as a time synchronization anomaly, and the result is not recorded.

[0080] Example 2: This embodiment develops a personalized vehicle control method based on the different driver reaction times obtained in Embodiment 1. The method calculates the time required for the driver to complete collision avoidance based on the real-time driver reaction time, calculates the time to collision (TTC) in combination with the identified dangerous stimulus scenarios, calculates the safety level based on the time required for the driver to complete collision avoidance and the collision time, dynamically judges the risk level, and formulates and executes differentiated safety intervention strategies.

[0081] In this embodiment, the collision time is calculated for the three types of dangerous stimulus scenarios identified in step S3 above, as follows: ① TTC scenario involving the vehicle ahead braking: Get the speed of this vehicle Speed ​​of the vehicle in front Workshop distance The collision time in this scenario is calculated using a relative motion model. The expression is:

[0082] ② TTC scenario where vehicles cut into adjacent lanes: Obtain the lateral distance between the vehicle in the adjacent lane and the vehicle itself. Lateral relative velocity Longitudinal distance and longitudinal relative velocity Calculate the collision time in this scenario. The expression is:

[0083] ③ Pedestrian intrusion scene TTC: Obtain the longitudinal speed of this vehicle Pedestrian lateral intrusion speed Lateral distance between pedestrian and lane boundary Longitudinal distance between the vehicle and the pedestrian's current position Calculate the collision time in this scenario. The expression is:

[0084] In this embodiment, in order to make reasonable differentiated strategy divisions, it is necessary to comprehensively consider the time required for the driver to complete collision avoidance and the collision time of the corresponding scenario. The expression for the time required for the driver to complete collision avoidance is as follows:

[0085] in, This refers to the driver's reaction time. For the vehicle braking response time, The time is 1.0s. The time is 2.1s. The time is 3.0s.

[0086] Furthermore, in order to overcome the potential intervention inaccuracies that may arise from single-scenario decision-making in complex traffic environments, it is necessary to determine the time required for the driver to complete the hazard avoidance maneuver. and collision time in each individual scene Based on this, this embodiment introduces a safety degree calculation method that covers single and multiple scenarios, enabling it to perform collaborative safety status assessment when any one, two, or three of the three scenarios—braking of the vehicle in front, cutting in from a neighboring vehicle, and pedestrian intrusion—occur simultaneously.

[0087] Specifically, for each dangerous stimulus scenario identified in step S3, its safety level is calculated. The calculation formula is:

[0088] in, The calculated collision time for the corresponding scene. These correspond to the braking of the vehicle ahead, the cutting in by a vehicle in the adjacent lane, and the intrusion of a pedestrian, respectively.

[0089] security It directly represents the sufficiency of available time relative to required time, when When, it indicates that there is ample available time and the situation is safe; when At that time, the available time is only slightly more than the required time, with limited margin, and the situation is critical; when When the time is insufficient, it indicates that there is not enough time available and the situation is dangerous.

[0090] When the system simultaneously detects two or three dangerous stimulus scenarios, it calculates the overall safety level across all dangerous stimulus scenarios. Specifically, this is achieved through the following weighted fusion formula:

[0091] in, A collection of current dangerous and stimulating scenarios; For the dynamic weighting coefficients of the corresponding scenario, satisfying .

[0092] The smaller the scene collision time value, the greater the weight of that scene in the overall safety assessment. Therefore, the dynamic weighting coefficient for multiple scenes... It is determined by the following formula based on the degree of urgency:

[0093] in, It is a very small positive number, such as 0.01, used to avoid when Numerical instability problem when approaching zero.

[0094] Overall security across single and multiple scenarios Based on different calculation results, implement differentiated security intervention strategies adapted to each security state: Specifically, if Without any intervention, the driver completes the operation based on their own judgment, and the system only continuously monitors the vehicle status and the driver's reaction; like This triggers an audio-visual warning, including a continuously flashing red warning light on the dashboard and a high-frequency warning sound from the vehicle's speakers, prompting the driver to take timely braking or evasive action. like The vehicle's automatic collision avoidance mechanism is activated, automatically applying maximum braking force for emergency deceleration. At the same time, it combines the steering assist system to fine-tune the direction to avoid collisions and simultaneously activates the hazard lights to warn surrounding vehicles.

[0095] In this embodiment, the security level of a single scenario The above method enables a unified quantitative assessment of the safety status in both single-scenario and multi-hazard scenarios.

[0096] The present invention also provides an electronic device, comprising: one or more processors and a memory; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the vehicle control method described above.

[0097] The present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the vehicle control method described above.

[0098] Those skilled in the art will understand that all or part of the functions of the various methods / modules in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the above functions can be implemented by executing the program with a computer. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented.

[0099] In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the programs can also be stored in storage media such as servers, other computers, disks, optical discs, flash drives, or portable hard drives. They can be downloaded or copied to the memory of the local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection described in the claims.

Claims

1. A method for detecting driver reaction time in real-world vehicle scenarios, characterized in that, The method includes the following steps: Step S1. Multi-source data acquisition: The driver's forward field of view data is acquired through the scene camera, the driver's eye movement data is acquired through the infrared camera, and the vehicle status data of the vehicle is acquired simultaneously. All acquired data is bound to the unified timestamp of the vehicle system. Step S2. Perform distortion correction processing on the infrared and scene camera images; use the cornea-pupil offset to characterize the rotational offset of the eyeball on the infrared imaging plane, thereby constructing the initial gaze vector; obtain the head pose and solve for the rotation matrix of the head relative to the infrared camera. With translation vector Based on the attitude matrix ( , Obtain the coordinates of the eyeball center in the infrared camera coordinate system. Simultaneously through rotation matrix The initial gaze vector is corrected to obtain the gaze direction in the infrared camera coordinate system. Then, based on the coordinate transformation algorithm, the line of sight and eye center coordinates in the infrared camera coordinate system are transformed to the scene camera coordinate system, and a ray is constructed. ; to ray The intersection point with the scene camera's imaging plane is taken as the gaze point, projected onto the scene camera's two-dimensional pixel coordinate system, and the gaze point coordinates are smoothed to output the gaze point coordinates in the scene camera image. And associate with timestamp; Step S3. Environmental target information extraction: Use target detection algorithms to identify vehicles ahead, vehicles in adjacent lanes, and pedestrians who have entered the lane. Simultaneously extract the braking status of vehicles ahead, the accurate location and speed of the area where the brake lights are located, the speed of adjacent vehicles, the angle and lateral distance between them and the vehicle, and the position and lateral intrusion speed of pedestrians. Step S4. Combining the data obtained in steps S1 to S3, determine the moment when the driver perceives the danger and the moment when the driver reacts, and calculate the time difference between the two to obtain the driver's reaction time in real time.

2. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, The scene camera is fixed at the upper middle position inside the windshield of the vehicle, with the lens facing the road ahead. Its collection range covers the vehicles, lane lines, pedestrians and adjacent lane environment in front of the driver's normal field of vision. It is used to collect continuous video stream data of the road ahead in real time, and each frame of the image is associated with a unified timestamp of the vehicle system. The infrared camera is installed in the center above the dashboard, with the lens facing the driver's face and focusing on the eye area. It is used to collect images of the driver's eye movements in real time, record the trajectory of eye movement and the direction of gaze. Each frame of eye movement image is also associated with a unified timestamp of the vehicle system. The vehicle status data includes the vehicle speed and brake pedal pressure, and all vehicle status data is bound to a unified timestamp of the vehicle system.

3. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, In step S2, the distortion coefficients obtained from the calibration are used to perform distortion correction processing on the infrared and scene camera images. When calculating the corneal-pupil offset, the coordinates of the corneal reflection center and the geometric center coordinates of the pupil are calculated first, and then the difference between the two is used to characterize the rotational offset of the eyeball on the infrared imaging plane, i.e., the corneal-pupil offset.

4. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, The rotation matrix of the head relative to the infrared camera in step S2 With translation vector The method for determining these features is as follows: A set of key points is detected in the facial image captured by the infrared camera. These key points include the corners of the eyes, the tip of the nose, and the corners of the mouth. This is then combined with the intrinsic parameters of the infrared camera. The PnP algorithm is used to establish a mapping relationship between points in a 3D face model and their corresponding 2D image points, thereby solving for the rotation matrix of the head relative to the infrared camera. With translation vector .

5. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, In step S2, based on a coordinate transformation algorithm, the line of sight and eye center coordinates in the infrared camera coordinate system are transformed to the scene camera coordinate system, and a ray is constructed. The specific method is as follows: By comparing the external parameters of the infrared camera and the scene camera The rotation matrix in the image represents the line-of-sight direction in the infrared camera coordinate system. View direction vector transformed into scene camera coordinate system ; By comparing the external parameters of the infrared camera and the scene camera The center point of the eye in the infrared camera coordinate system Transform to the scene camera coordinate system to determine the eye reference point in the scene camera coordinate system. ; In the scene camera coordinate system, with the eye reference point Starting from the line of sight vector Constructing ray The expression is: ； in, These are the ray parameters.

6. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, In step S2, when smoothing the gaze point coordinates, a first-order low-pass filter algorithm is used to smooth the gaze point coordinates, and the filter coefficients are adjusted accordingly. To make dynamic adjustments, the expression is: ； in, Based on the basic filter coefficients, take ; The magnitude of the head angular velocity is determined by rotating the matrix across consecutive frames. The rate of change was calculated. The rate of change of gaze vectors between adjacent frames; , This is the adjustment coefficient; Adjustment coefficient , The expression is: ； in, , 3 seconds before the current frame , 95th percentile.

7. The driver reaction time detection method for real-vehicle scenarios according to claim 1, characterized in that, In step S3, after extracting the vehicle parameters of adjacent lanes, it is simultaneously determined whether a vehicle in an adjacent lane has "cut into" this lane. When the lateral distance is satisfied... , Change rate of angle in 3 consecutive frames And speed When a vehicle in the adjacent lane enters the lane, it is marked as "cutting in." Based on the extracted pedestrian parameters, it is determined whether a pedestrian has entered the lane, and for pedestrians who have "entered the lane," it is determined whether they are in a "dangerous" state. If the overlap between the pedestrian bounding box and the lane area is significant... If the pedestrian enters the lane, it is determined that "the pedestrian has entered this lane"; otherwise, it is determined that "the pedestrian has not entered"; if the pedestrian has entered the lane, the lateral intrusion speed is considered to be... If it is, then it is marked as "dangerous".

8. The driver reaction time detection method for real-vehicle scenarios according to claim 7, characterized in that, Step S4 includes the following steps: Step S41. Driver's moment of hazard perception Acquisition: ①Sensing the hazard of a vehicle braking ahead: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The coordinates of the driver's gaze point in this frame match the coordinates of the brake light area at the rear of the vehicle ahead. The vehicle in front of this frame is braking; ② Danger perception of vehicles cutting into adjacent lanes: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze coordinates in this frame match the bounding box coordinates of adjacent vehicles. The vehicle status in the adjacent lane of this frame is "cutting in"; ③ Pedestrian intrusion danger perception: At timestamp If both of the following conditions are met in a frame, the current timestamp is recorded and used as the moment of danger perception. : The driver's gaze point coordinates match the pedestrian bounding box coordinates in this frame; The pedestrian's dangerous situation in this frame is "dangerous"; Step S42. Operation reaction time Acquisition: When an upward trend is detected in the brake pedal pressure signal, this moment is recorded as a candidate moment. If the pressure signal shows a sustained and significant increase within a preset time, exceeding a set threshold, it is confirmed that the driver has performed an emergency braking operation, and the previously recorded timestamp is determined as the reaction moment. If the condition for continuous increase is not met, the candidate moment will not be considered as the operational response moment. Step S43. Timestamp Alignment Calibration: Step S44. Reaction time calculate: Calculate the moment of danger perception With the reaction time of the operation The difference is the driver's reaction time. .

9. A vehicle control method for real-world vehicle scenarios, characterized in that, The method includes the following steps: Step A1. Calculate the collision time under different dangerous stimulus scenarios based on the identified dangerous stimulus scenarios; Step A2. Calculate the time required for the driver to complete collision avoidance based on the driver reaction time obtained by the driver reaction time detection method for real vehicle scenarios as described in any one of claims 1 to 8; Step A3. For each identified hazardous stimulus scenario, calculate its safety level based on the collision time and the time required for the driver to complete the collision avoidance. When more than two hazardous stimulus scenarios are detected, calculate the overall safety level of all hazardous stimulus scenarios. The expression is: ； in, It is a collection of current dangerous and stimulating scenarios. Safety level in the corresponding scenario; For the dynamic weighting coefficients of the corresponding scenario, satisfying The expression is: ； in, It is a very small positive number. The calculated collision time for the corresponding scene; Step A4. Based on the overall safety level Based on different calculation results, implement differentiated security intervention strategies adapted to each security state.

10. The vehicle control method for real-world vehicle scenarios according to claim 9, characterized in that, In step A4, if Without any intervention, the driver completes the operation based on their own judgment, and the system only continuously monitors the vehicle status and the driver's reaction; like This triggers an audio-visual warning, including a continuously flashing red warning light on the dashboard and a high-frequency warning sound from the vehicle's speakers, prompting the driver to take timely braking or evasive action. like The vehicle's automatic collision avoidance mechanism is activated, automatically applying maximum braking force for emergency deceleration. At the same time, it combines the steering assist system to fine-tune the direction to avoid collisions and simultaneously activates the hazard lights to warn surrounding vehicles.

Images