Foot type mobile platform driving fatigue characterization and detection method based on driver behavior characteristics and attention level
By using surface electromyography signals and visual information, combined with action recognition algorithms and survival analysis models, accurate monitoring of driver fatigue status is achieved, and the problem of quantitative modeling of driver fatigue on foot mobile platform is solved, and driving safety is improved.
Patent Information
- Application Number
- CN202510011774.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-05
- Publication Date
- 2025-04-15
AI Technical Summary
The multi-dimensional movement ability of the foot-type mobile platform depends on the driver's participation in the entire operation, resulting in cumbersome operation, increasing the driver's cognitive difficulty and cognitive burden, which can easily lead to the driver's physical and mental fatigue, and thus causing safety accidents.
By using the surface electromyography signal of the driver's upper arm, combining visual and velocity information, an action recognition algorithm based on adaptive moving average and separable convolution is established to identify driving behavior and reaction time, and combined with survival analysis model, the driver's driving fatigue degree is estimated.
Accurate monitoring and real-time detection of driver fatigue status is achieved, reducing the driver's cognitive burden, improving driving safety, and avoiding safety accidents caused by fatigue.
Smart Images
Figure CN119832529A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of legged robot driving operation, and in particular relates to a machine learning research method for evaluating a driver through the driver's behavior and attention level. Background Art
[0002] The discontinuity of the landing point of the legged robot gives it a stronger ability to adapt to terrain than traditional wheeled and tracked robots, which to a certain extent reflects the development prospects of mobile robots in complex terrain environments with changing geometric and physical characteristics. The excellent stability and strong load capacity of the foot-type mobile platform enable it to be competent for tasks such as material transportation and disaster relief in complex environments. However, the multi-dimensional movement ability of the foot-type mobile platform depends on the driver's full participation in the operation. The cumbersome operation process increases the cognitive difficulty and cognitive burden of the driver when performing tasks, which can easily lead to physical and mental fatigue of the driver, and even cause safety accidents. Therefore, how to perceive the driver's fatigue state during driving and incorporate it into human-machine decision-making has become an important issue that needs to be solved in this field.
[0003] Driver fatigue is an information fusion problem that is difficult to detect directly and is constrained by multiple factors. There are two main fatigue detection technology routes commonly used at present. One is to detect the EEG activity of the subject in monotonous behavior, and directly estimate the fatigue state of the subject by reconstructing and combining the signals of α, β, and γ waves in the cerebral cortex; the other is to detect the regional feature information of the eyes, mouth, and head of the subject in repetitive movements, and indirectly estimate the fatigue state of the subject by analyzing the frequency of physiological activities such as blinking and yawning of the subject, combined with questionnaire questions. These two technical routes have made great progress, but there are congenital deficiencies that are difficult to avoid. The former requires complex information acquisition and processing units, and the EEG signal is extremely sensitive to environmental noise, and the actual driving process is difficult to meet the low-noise conditions required; the latter relies too much on the personal experience of the subject, ignoring the additional driving fatigue caused by the driver's experience and attention.
[0004] Surface electromyography, as a stable medium with rich information expression, has been maturely applied in the fields of action recognition and motion control. The present invention uses the surface electromyography signal of the driver's upper arm to accurately monitor the activation and generation process of driving behavior. Combined with visual and speed information, the degree of concentration of the driver's attention during driving behavior is estimated according to the changes in the driver's field of vision during driving. According to the survival analysis model, by establishing a mapping relationship between covariates and reaction time, the basic probability distribution of behavior transformation is obtained in reverse, thereby estimating the driver's driving fatigue. Summary of the invention
[0005] The purpose of the present invention is to provide a method for characterizing and detecting driving fatigue of a foot-type mobile platform based on driver behavior characteristics and attention level, so as to solve the problem of quantitative modeling of driving fatigue of a foot-type mobile platform driver.
[0006] The solution adopted by the present invention to solve the above problems is: a method for characterizing and detecting driving fatigue of a foot-type mobile platform based on driver behavior characteristics and attention level, and the specific implementation process of the method is as follows:
[0007] Step 1: Driver behavior change and reaction time detection based on behavior characteristics:
[0008] In the preparation stage, the subjects performed various driving behaviors under the instructions of the researchers. The EMG sensors were used to collect the EMG signals of the subjects' upper arms, and the EMG signals were marked by timestamps to construct training sets and test sets. The training was based on the action recognition algorithm based on adaptive moving average and separable convolution. This algorithm used the adaptive moving average method to change the time window length to avoid the impact of the non-uniformity of the driving behavior duration on the classification performance of the algorithm. Separable convolution was used to reduce the number of parameters and calculations of the model and speed up the operation. Using the action recognition algorithm, the driving behavior corresponding to the EMG signal was identified during the driving process. The driving behavior was segmented with the threshold of the adjacent driving behavior window period exceeding 10 time windows, and continuous driving behavior was obtained. In the continuous driving behavior, the window period was recorded as the reaction time, and the behavior changes before and after the reaction time were recorded.
[0009] Step 2: Driver attention level detection combining speed information and visual information:
[0010] A depth camera is arranged in the cockpit and the camera is calibrated to record the driver's facial image stream during driving. The human eye image is segmented according to the detection results of the Haar cascade classifier, and the coordinates of the eye centroid relative to the camera coordinate system are calculated. The pupil of the eye is detected using Hough transform, the two-dimensional coordinates of the pupil circumference are calculated, and the pupil-eye centroid line equation is established. The coordinates of the intersection of the cockpit windshield and the pupil-eye centroid line relative to the camera coordinate system are calculated, and the driver's field of view is estimated in combination with the body speed and the driver's angular resolution, thereby detecting the driver's attention level.
[0011] Step 3: Characterization of response fatigue based on survival analysis model:
[0012] During driving, according to steps 1 and 2, the set of behavior change and reaction time and the set of attention level and reaction time are obtained. The behavior change and attention level are used as covariates to construct the Cox proportional hazard model. The partial likelihood function is written by reaction time and the regression coefficient optimization equation is established. The regression coefficient is iterated using the Newton-Raphson method. By establishing the mapping relationship between covariates and reaction time, the basic probability distribution of behavior change is obtained in reverse, thereby estimating the driver's driving fatigue.
[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention comprehensively considers the impact of the decision-making load and attention level on the reaction time caused by the driver's driving fatigue and behavior changes. Through the action recognition algorithm based on adaptive moving average and separable convolution, the event flow of the driver's entire process from encountering problems, decision-making problems to taking actions is accurately recorded. By integrating speed information and visual information, starting from the driver's receptive field, real-time detection of the driver's attention level is achieved. By monitoring different physiological characteristics and establishing analysis models, the interference of behavior changes and attention levels in establishing the mapping relationship between driving fatigue and reaction time is avoided to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic diagram of the technical route of the present invention.
[0015] Figure 2 It is a residual module with adaptive moving average and separable convolution.
[0016] Figure 3 Action recognition algorithm based on adaptive moving average and separable convolution DETAILED DESCRIPTION
[0017] The implementation modes of the present invention are further described in detail below with reference to the accompanying drawings and examples.
[0018] One embodiment of the present invention is a method for characterizing and detecting driving fatigue of a foot-based mobile platform based on driver behavior characteristics and attention level. The specific implementation of the method is as follows:
[0019] Step 1: Driver behavior change and reaction time detection based on behavioral characteristics:
[0020] The structure of the action recognition algorithm is as follows: in terms of signal preprocessing, a sixth-order Butterworth filter is used to bandpass filter the signal matrix; in terms of algorithm composition, the algorithm is as follows Figure 2 The residual module of adaptive moving average and separable convolution shown in the figure is composed of: In the module, the initial time window length is set to 200ms, and the Z-Score of each channel of the electromyographic signal in the window is calculated when the window is divided. The window mean of a channel of the signal is set to The sequence length, mean, and standard deviation of a channel are N s , μ, σ, then Dynamically adjust the window size to minimize the Z-Score of each channel; set the batch size to B, the number of EMG signal channels to C, and the time window length to L w , then the EMG signal is reconstructed as B×C×L w signal matrix; assuming the module width is n, the reconstructed signal matrix undergoes two-way processing: (1) The reconstructed signal matrix is passed through n 1×1 convolution kernels to map the signal correlation across EMG channels. Then, n separable convolution kernels with three different kernel sizes and padding sizes are used to extract the time domain features of short time window and long time window sequences, and finally input into the concatenation layer; (2) The module limits the size of the feature map through maximum pooling and passes through n 1×1 convolution kernels to the concatenation layer. After two-way processing, the original signal matrix is decomposed into combined data of local features under different receptive fields. The specific structure of the algorithm Figure 3 As shown: (1) Feature extraction: The signal matrix is concatenated with the original signal matrix after 128 1×1 convolution kernels and batch normalization through 16-width and 32-width residual modules to form a new signal matrix, and then nonlinear mapping is performed through the RELU activation function; the signal matrix is concatenated with the original signal matrix after 512 1×1 convolution kernels and batch normalization through 64-width and 128-width residual modules to form a new signal matrix, and then nonlinear mapping is performed through the RELU activation function; (2) Classification: The signal matrix undergoes adaptive mean pooling and its dimension is reduced to B×512×50. The signal matrix is downsampled through a convolutional layer, batch normalization, and RELU activation function to obtain the prediction result. Loss function: The multi-classification task uses the cross entropy function to calculate the loss, i.e. Among them, Tp is the algorithm classification result, and Lb is the real label. Using the action recognition algorithm, the driving behavior corresponding to the electromyographic signal is identified during driving, and the driving behavior is segmented by the threshold of the gap period of adjacent driving behaviors exceeding 10 time windows to obtain continuous driving behavior. In the continuous driving behavior, the gap period is recorded as the reaction time, and the behavior changes before and after the reaction time are recorded.
[0021] Step 2: Driver attention level detection combining speed information and visual information:
[0022] A depth camera is placed in the cockpit, and a checkerboard image is generated according to Zhang Zhengyou's calibration method, which is mounted vertically on the seat pillow. Assume that a pixel point on the checkerboard image is relative to the camera coordinate system P = [X0, Y0, Z0, 1] T The point projected onto the image plane is p = [u0, v0, 1] T , the perspective projection model of the camera can be expressed as Among them, s is the scaling factor and A is the camera intrinsic parameter matrix [M|t] is the camera extrinsic parameter, M is the rotation matrix, t is the translation vector, and H is the homography matrix. Represent the column vector of M as [m1,m2,m3] and substitute it into the equation to get The above equation is expressed in the form of a homography matrix, that is, H = A[m1, m2, t']. According to the mapping relationship between several pixel points on the chessboard and the image points, the homography matrix H is solved using the least squares method.
[0023] Change the seat backrest angle multiple times and generate a new homography matrix H according to the above steps to solve the intrinsic parameter matrix A. According to the intrinsic parameter matrix A, restore the corresponding extrinsic parameter from each homography matrix H. Since m1 and m2 meet the orthogonality condition, restore m3 using orthogonality to obtain the translation vector t and the camera extrinsic parameter solution.
[0024] During driving, the human eye image in the driver's facial image stream is segmented according to the detection results of the Haar cascade classifier. The Haar feature is a simple feature composed of white and black rectangular areas. The eigenvalue calculation method is: Where I(i) is the gray value of the image at pixel i. In order to quickly calculate the pixel sum of the prediction box and thus obtain the feature value of the prediction box, the integral map is used to simplify the calculation steps. AdaBoost algorithm is used to screen features and obtain strong Haar features. For m positive samples and l negative samples, the initial weight is Train the weak classifiers iteratively multiple times and select the best weak classifier h t (x) and the corresponding threshold, so that the weighted error rate ε t Minimum, that is Calculate the weak classifier coefficient α t Update sample weight w t+1 And normalized After T rounds of iterations, the strong classifier D(x) is obtained Among them, λ is the threshold parameter, which is
[0025] Record the coordinates of the prediction box on the camera image, and solve the spatial point position of the prediction box centroid relative to the camera coordinate system according to the homography matrix, that is, the coordinates of the eye centroid in the camera coordinate system are (X c ,Y c ,Z c ).
[0026] Segment the human eye image and use Hough transform to detect the pupil of the eye. For each edge point in the image, it may exist in the pupil circle. Suppose it satisfies Create a three-dimensional accumulator group Acc(x e ,y e ,r e ), for each edge point in the image, for each possible radius r p The value range is traversed, for each radius r p , calculate the possible center of the circle (x e ,y e ) Where δ is the angle from 0 to 2π; for the calculated Acc(x e ,y e ,r e ), add one to the corresponding accumulator unit count. Find the unit with the largest cumulative number in the accumulator, and its peak value corresponds to (x e ,y e ,r e ) represents the parameters of the circle supported by the most edge points.
[0027] Assuming that the human eye is a standard sphere, the equation of the pupil-eye center of mass line is Assume that the cockpit windshield inclination angle is θ, the windshield length is L, the windshield width is W, and the camera is located at the midpoint of the upper edge of the cockpit windshield. Then the equation of the cockpit windshield plane relative to the camera coordinate system is: The coordinates of the intersection of the cockpit windshield and the pupil-eye centroid relative to the camera coordinate system are: According to the data, the horizontal field of view of the human eye at rest is -60°~60°, and the vertical field of view is -70°~60°. The mapping relationship between the body speed v and the driver's field of view is: Wherein, φ is the visual angular resolution of the human eye, which satisfies φ=1 / V with the standard visual acuity V. Combine the intersection coordinates to get the field of view Calculate the IoU between the field of view and the cockpit windshield, that is, At this point, the driver's attention level can be measured based on IoU.
[0028] Step 3: Driving fatigue characterization based on survival analysis model:
[0029] During driving, the behavior change and reaction time set are obtained according to step 1, and the attention level and reaction time set are obtained according to step 2. With behavior change and attention level as covariate factors, a Cox proportional hazard model is constructed, which can be expressed as Among them, β is the regression coefficient of the covariate, g0(t) represents the basic possibility of behavioral change, and g(t) represents the instantaneous possibility of behavioral change. Write the partial likelihood function according to the reaction time and optimize the regression coefficient β, that is, Among them, N d is the total number of consecutive driving behaviors that have occurred, F i is the covariate vector of the ith continuous driving behavior, R(t i ) is the time duration t i The continuous driving behavior set that undergoes behavior transformation after the behavior transformation is iterated using the Newton-Raphson method, that is, Calculate the cumulative benchmark occurrence function estimate, the formula is Then the basis for the behavior change may be the difference of the cumulative benchmark occurrence function estimate, that is, Estimated based on the current cumulative benchmark occurrence function Calculate the basic cumulative probability S0(t i ) Calculate the underlying probability distribution function for behavior transformation f0(t i )=h0(t i )·S0(t i ) At this time, the driving fatigue can be estimated based on the basic probability distribution function of the behavior transformation. Among them, T i-1 The actual time when the last behavior change occurred.
Claims
1. A method for characterizing and detecting driving fatigue of a foot-based mobile platform based on driver behavior characteristics and attention level, characterized by: A method for characterizing and detecting driving fatigue of a foot-based mobile platform based on driver behavior characteristics and attention level includes the following steps: Step 1: Driver behavior change and reaction time detection based on action features: In the preparation stage, the subjects performed various driving behaviors under the instructions of the researchers, and the electromyographic sensors were used to collect the electromyographic signals of the subjects' upper arms. The electromyographic signals were marked by timestamps to construct training sets and test sets; Training an action recognition algorithm based on adaptive moving average and separable convolution. This algorithm uses the adaptive moving average method to change the time window length to avoid the impact of the non-uniformity of driving behavior duration on the classification performance of the algorithm. Separable convolution is used to reduce the number of model parameters and calculations, speeding up the operation speed. Using the action recognition algorithm, the driving behavior corresponding to the electromyographic signal is identified during the driving process. The driving behavior is segmented using the threshold of the adjacent driving behavior window period exceeding 10 time windows to obtain continuous driving behavior. In the continuous driving behavior, the window period is recorded as the reaction time, and the behavior changes before and after the reaction time are recorded. Step 2: Detection of driver attention level by combining speed information and visual information: a depth camera is arranged in the cockpit and the camera is calibrated, and the driver's facial image stream is recorded during driving; the human eye image is segmented based on the detection results of the Haar cascade classifier, and the coordinates of the eye center of mass relative to the camera coordinate system are calculated; The pupil of the eye is detected using Hough transform, the two-dimensional coordinates of the pupil circumference are calculated, and the pupil-eye centroid line equation is established; Calculate the coordinates of the intersection of the cockpit windshield and the line connecting the pupil and the eye center of mass relative to the camera coordinate system, and estimate the driver's field of view in combination with the aircraft speed and the driver's angular resolution, so as to detect the driver's attention level; Step 3: Characterization of driving fatigue based on survival analysis model: Using behavioral change and attention level as covariate factors, a Cox proportional hazard model was constructed; Write partial likelihood functions based on reaction time, establish regression coefficient optimization equations, and use Newton-Raphson method to iterate regression coefficients; A mapping relationship between covariates and the estimated reaction time of continuous driving behaviors is established within the continuous driving behaviors, and the basic probability distribution of the behavior transformation is obtained in reverse, thereby estimating the driver's driving fatigue level.