Abnormal driving detection method and application thereof in abnormal driving training system

Abstract

The invention relates to the field of locomotive abnormal driving detection, in particular to an abnormal driving detection method and application thereof in an abnormal driving training system. The method comprises the following steps: acquiring an operation table and a control image, acquiring each historical moment when a driver needs to operate through the operation table, and calculating the fatigue of the driver at the moment when the driver needs to operate through an operation track and a front view area of a hand; the front view vector of the driver's glasses is calculated through the pupil position of the driver's eyes, then the front view curve is calculated, the front view area is calculated according to the phenomenon that the visual area of the human eyes is conical, and the fatigue is used for reflecting the hand-eye coordination degree of the driver; the factor is used as an important factor for reflecting whether the driver is in the abnormal driving condition of fatigue driving or not; the problem that whether the driver is in fatigue driving or not cannot be judged from whether the response of the driver is timely or not when whether the driver is fatigue or not is detected through images at present is solved.

Description

Technical Field

[0001] The invention relates to the field of abnormal locomotive driving detection, in particular to an abnormal driving detection method and its application in an abnormal driving training system. Background Art

[0002] During the normal operation of the locomotive, it will face many abnormal driving scenarios. The staff are required to detect and identify the driving status of the locomotive during the driving process. After that, the staff on the locomotive will properly handle the dangerous factors in the abnormal driving scenarios to ensure the normal driving safety of the locomotive. In addition, for some possible abnormal driving scenarios, training needs to be organized to improve the handling ability of the locomotive staff to improve the ability of the locomotive staff to deal with such abnormal driving problems.

[0003] Monitoring the driver's fatigue status has always been one of the important conditions for ensuring the safe driving of locomotives. Fatigue driving is usually manifested as slow reaction or no reaction. There are currently many methods for detecting fatigue driving through images. For example, by identifying the reduction in the size of eyelid opening and closing, specific gesture errors or non-standard gestures, and the driver's body not moving during the action, all of which can be used as a method to judge whether the driver is fatigued. However, in the early stage of fatigue driving, when the driver's fatigue level is low, it is difficult to find abnormalities in the size of the eyelid opening and closing, the completion of specific gestures, and the frequency and amplitude of body movements. Therefore, these methods are difficult to judge whether the driver is fatigued from the aspect of whether the driver's reaction is timely, and have certain limitations. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an abnormal driving detection method and its application in an abnormal driving training system, which solves the problem that when currently detecting whether a driver is fatigued through images, there is a lack of means to judge whether the driver is driving fatigued based on whether his reaction is timely.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for detecting abnormal driving, the method comprising the following steps:

[0007] S1. Establish an operation table of historical operation points and corresponding operation times according to the driving process, and obtain the control time including the driver's driving control, the corresponding control items and the control image of the locomotive cab;

[0008] S2, recognizing the driver's hand according to the control image, and generating an operation trajectory of the hand;

[0009] S3, calculating the face facing vector of the driver's face at any time according to the control image, calculating its front view vector according to the sight direction, and calculating the front view area of ​​the driver on the console according to the front view vector;

[0010] S4, the control image is calculated according to the minimum distance between the operation trajectory and the corresponding control button at any time, and whether the control image at that time is an operation image is determined according to whether the minimum distance is less than a distance threshold, and a standard image is extracted;

[0011] If so, extract the corresponding control image to generate the operation image, and extract the standard image based on the operation image;

[0012] If not, the corresponding control image is not extracted;

[0013] S5. Mark the time corresponding to any operation time and the standard image in the operation table as the operation time, and calculate the driver's fatigue level according to the front view area and the standard image corresponding to any operation time.

[0014] Preferably, in step S2, the following steps are specifically included:

[0015] S21, extracting a contour image of the driver's hand according to the control image;

[0016] S22, calculating the sharp angle of any point on the contour line of the driver's hand on the contour image; the calculation formula of the sharp angle is:

[0017]

[0018] In the above formula, cosδ(0) represents the cosine value of the sharp angle at point 0 on the hand contour line, P 0-K' and P K-0 Represents the vector formed by the 0 point and the Kth and K'th pixel points on the hand contour lines on both sides thereof.

[0019] S23, judging whether any point on the contour line of the driver's hand is a fingertip point according to whether the curvature of the point is greater than a curvature threshold;

[0020] If so, the point is the fingertip point;

[0021] If not, the point is a contour point;

[0022] S24. Generate the operation trajectory of the driver's hand according to the coordinates of the fingertip points at each moment.

[0023] Preferably, in step S3, the following steps are specifically included:

[0024] S31, acquiring the coordinates of the driver's eyes and mouth according to the control image;

[0025] S32, calculating the base plane of the driver's head at any time; the expression of the base plane is:

[0026]

[0027] In the above formula, x, y and z represent the x-axis, y-axis and z-axis coordinates of the point set on the base plane, respectively; x1, x2 and x3 represent the x-axis coordinates of the driver's eyes and mouth, respectively; y1, y2 and y3 represent the y-axis coordinates of the driver's eyes and mouth, respectively; z1, z2 and z3 represent the z-axis coordinates of the driver's eyes and mouth, respectively;

[0028] S33, calculating the face facing vector of the driver's face according to the base plane; the expression of the face facing vector is:

[0029] S=A1i+A2j+A3k

[0030] in,

[0031] A1=y1z2-y1z3-y2z1+y2z3+y3z1-y3z2

[0032] A2=-x1z2+x1z3+x2z1-x2z3-x3z1+x3z2

[0033] A3=x1y2-x1y3-x2y2+x2y3+x3y1-x3y2

[0034] In the above formula, S represents the face facing vector, A1, A2 and A3 are the first coefficient, the second coefficient and the third coefficient respectively, x1, x2 and x3 represent the x-axis coordinates of the driver's eyes and mouth respectively, y1, y2 and y3 represent the y-axis coordinates of the driver's eyes and mouth respectively, z1, z2 and z3 represent the z-axis coordinates of the driver's eyes and mouth respectively;

[0035] S34, acquiring binocular images of the driver according to the control image;

[0036] S35, calculating the offset angle of the facial facing vector according to the binocular images, and correcting the face facing vector to obtain the face facing vector;

[0037] S36, constructing an emmetropia curve according to the emmetropia vector and the eye coordinates of the driver, and calculating the emmetropia area of ​​the driver.

[0038] Preferably, in step S35, the following steps are specifically included:

[0039] S351, calculating the first vector of the pupils of the two eyes according to the binocular images; the expression of the first vector is:

[0040] S1=(x′1-x′2, y′1-y′2, z′1-z′2)

[0041] In the above formula, S1 represents the first vector, x'1, y'1 and z'1 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of one eye, and x'2, y'2 and z'2 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of the other eye.

[0042] S352, calculating the lateral offset and longitudinal offset of the pupil of the eye;

[0043] The calculation formula for the lateral offset is:

[0044]

[0045] The calculation formula for the longitudinal offset is:

[0046]

[0047] In the above formula, Δm1 and Δm2 are the lateral offset and the longitudinal offset respectively, x'1, y'1 and z'1 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of one eye, and x'2, y'2 and z'2 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of the other eye;

[0048] S353, calculating a lateral offset angle and a longitudinal offset angle according to the lateral offset amount and the longitudinal offset amount;

[0049]

[0050] In the above formula, θ1 and θ2 are the lateral offset angle and the longitudinal offset angle respectively. and are the lateral and longitudinal offsets respectively, and R is the radius of the eyeball;

[0051] S354, correcting the facial facing vector according to the lateral offset angle and the longitudinal offset angle to obtain the face facing vector.

[0052] Preferably, in step S354, the following steps are specifically included:

[0053] S3541, establishing an eye space coordinate system with the facial facing vector as the y-axis, the first vector as the x-axis, and the direction perpendicular to the facing vector and the first vector as the z-axis;

[0054] S3542, calculating the first projection vector of the orthostatic vector on the xy plane of the eye space coordinate system according to the lateral offset angle and the facial orthostatic vector; the expression of the first projection vector is:

[0055]

[0056] In the above formula, cosθ1 represents the cosine of the angle between the first projection vector and the first vector, represents the first projection vector of the orthographic vector on the xy plane of the eye space coordinate system, S1 is the first vector;

[0057] S3543, calculating the second projection vector of the orthogonal vector on the yz plane of the eye space coordinate system according to the longitudinal offset angle and the facial orthogonal vector; the expression of the second projection vector is:

[0058]

[0059] In the above formula, cosθ2 represents the cosine of the angle between the second projection vector and the face-facing vector. S represents the second projection vector of the normal view vector on the xy plane of the eye space coordinate system. 0 is the face facing vector;

[0060] S3544, calculating the orthographic vector according to the first projection vector and the second projection vector; the expression of the orthographic vector is:

[0061]

[0062] In the above formula, S2 represents the normal view vector, represents the x-axis coordinate of the first projection vector, represents the y-axis coordinate of the first projection vector, Represents the z-axis coordinate of the second projection vector.

[0063] Preferably, in step S36, the following steps are specifically included:

[0064] S361, constructing an emmetropia curve according to the emmetropia vector and eye coordinates of both eyes of the driver;

[0065] The expression of the positive view curve is:

[0066]

[0067] In the above formula, A4, A5, A6 and A7 represent the fourth, fifth, sixth and seventh coefficients respectively, x, y and z are the x-axis, y-axis and z-axis coordinates of the point on the emmetropia curve respectively, x1, y1 and z1 are the x-axis, y-axis and z-axis coordinates of one eye respectively, and x2, y2 and z2 are the x-axis, y-axis and z-axis coordinates of the other eye respectively;

[0068] S362. A cone-shaped emmetropia area is established with the center point of the driver's eyes as the vertex and the emmetropia curve as the central axis. The expression of the emmetropia area is:

[0069]

[0070] In the above formula, x, y, and z are the coordinates of the x-axis, y-axis, and z-axis of the point set in the normal viewing area, respectively. i ,y i and z i The coordinates on the positive curve are is the x-axis, y-axis, and z-axis coordinates of any point that is the starting point and satisfies the normal view vector constraint, x1, y1, and z1 are the x-axis, y-axis, and z-axis coordinates of one eye, x2, y2, and z2 are the x-axis, y-axis, and z-axis coordinates of the other eye, and θ is the line of sight expansion angle.

[0071] Preferably, in step S4, the following steps are specifically included:

[0072] S41, calculating the minimum distance between the hand and the corresponding operation key at any time in the operation trajectory according to the control image; the calculation formula of the minimum distance is: D min =MIN[d(pos α ,pos2)]

[0073] In the above formula, D min Indicates the minimum distance between the hand and the corresponding operation button at any time, MIN(·) means taking the minimum value, d(pos α ,pos2) represents the calculation of the minimum distance between the αth hand position and the corresponding operation key;

[0074] S42, setting a distance threshold, and judging whether the control image at the moment is an operation image according to whether the minimum distance is less than the distance threshold;

[0075] If yes, the control image at that moment is extracted and marked as the operation image;

[0076] If not, the corresponding control image is not extracted.

[0077] S43, selecting an operation image with the smallest minimum distance value from a plurality of operation images adjacent to each other at the same time, and marking it as a standard image.

[0078] Preferably, in step S5, the following steps are specifically included:

[0079] S51, marking the time corresponding to any operation time and the standard image in the operation table as the operation time;

[0080] S52, marking the moment only in the standard image as the first operation moment, and marking the remaining operation moments as the second operation moments;

[0081] S54, calculating the second fatigue degree at any second operation moment according to the normal vision area; the calculation formula of the second fatigue degree at any moment is:

[0082]

[0083] In the above formula, represents the second fatigue level at any second operation time t, is the set of all time points of any operation time closest to the second operation time point t in the operation table, is the time of the standard image corresponding to any second operation time t, F 2 and F 3 The second and third fatigue degrees, Indicates the coordinates of the corresponding operation button, ZM t A set of points representing the front view area;

[0084] S55. Assign a value to the first fatigue degree at the first operation moment, and calculate the fatigue degree of the driver at any moment; the calculation formula of the fatigue degree of the driver at any moment is:

[0085]

[0086] In the above formula, is the first fatigue degree at any first operation time t, represents the second fatigue degree at any second operation time ti, n represents the total number of driving moments of the driver, and h represents the number of times the driver is fatigued, that is, The total number of occurrences of the time and the first operation time.

[0087] The present invention also provides an application method of the abnormal driving detection method in an abnormal driving training system, the method comprising the following steps:

[0088] S11, producing several driving scene videos of drivers driving motorcycles;

[0089] S12, calculating and displaying the fatigue value of the driver at any time in each driving scene video;

[0090] S13, setting a fatigue threshold, and calculating the abnormal risk factor of each driver according to the fatigue value and growth rate;

[0091] S14. Divide the abnormal risk factor into several attention levels and set several corresponding treatment measures;

[0092] S15. According to the different attention levels of the drivers, corresponding handling measures are shown to the trainees through driving scene videos.

[0093] Preferably, in step S13, the calculation formula of the abnormal risk coefficient is:

[0094]

[0095] In the above formula, DL t represents the abnormal risk factor of the driver at time t, FL t represents the driver's first fatigue level at time t, FL represents the basic fatigue level, and FL t-z represents the fatigue degree at the zth moment before the tth moment, and ΔFL represents the fatigue coefficient.

[0096] Compared with the prior art, the present invention provides an abnormal driving detection method and its application in an abnormal driving training system, which has the following beneficial effects:

[0097] 1. The present invention obtains the operation table and the control image, obtains the historical moments when the driver needs to perform operations through the operation table, and calculates the driver's fatigue level at the moment when the operation is required through the hand operation trajectory and the front view area, so as to reflect the driver's hand-eye coordination degree, which is an important factor reflecting whether the driver is performing fatigue driving, an abnormal driving situation.

[0098] 2. The present invention first calculates the driver's facial facing vector, and then further calculates the driver's eye view vector and view area through the driver's pupil position, calculates the direction of the view vector by calculating the lateral offset angle and the longitudinal offset angle, and by means of vector projection, further calculates the view curve through the driver's eye position and the view vector, and sets the sight expansion angle according to the cone-shaped visual area of ​​the human eye to simulate the area seen by the human eye, thereby better showing the view area of ​​the human eye.

[0099] 3. The present invention extracts a standard image, so that when the driver's hand reaches out to the corresponding key and leaves the corresponding key, it can only extract the most representative moment, at which the locomotive driver's hand is most likely to contact the key. This can prevent the driver from judging that the hand and the key may be in contact at multiple moments in the operation image during the process of pressing a key, and extract a most representative standard image to avoid subsequent repeated calculations, thereby improving the calculation accuracy of fatigue.

[0100] 4. The present invention calculates whether there is overlap between the moments in the standard image and the corresponding operation table, so as to calculate the driver's fatigue level at any moment in different ways, so as to reflect the hand-eye coordination degree of the driver when operating during driving, which is an important factor in determining whether the driver is fatigued during driving.

[0101] 5. The present invention further calculates the abnormal risk factor based on the calculated fatigue level, and sets a number of attention levels and corresponding treatment measures, thereby helping trainees to handle different abnormal risk factors of drivers according to different preset treatment measures, so that trainees can better handle such abnormal driving problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0103] Figure 1 It is a flow chart of the abnormal driving detection method of the present invention;

[0104] Figure 2 is a schematic diagram of the eye space coordinate system of the present invention;

[0105] Figure 3 The present invention is a flow chart of an application method of the abnormal driving detection method in an abnormal driving training system. DETAILED DESCRIPTION

[0106] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0107] Those skilled in the art can understand that all or part of the steps in the following embodiments can be completed by instructing the relevant hardware through a program, so the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0108] When the locomotive is in motion, when the driver receives an automatic reminder from inside the locomotive or is in a corresponding driving scenario, he will operate the locomotive according to the operation time and operation sequence of the corresponding scenario, such as the operation sequence of slowing down when speeding, the operation sequence when the locomotive is stopped, and the operation sequence when the locomotive is started, etc. When the driver is fatigued, taking deceleration as an example, the driver may experience several situations after hearing the speeding notification. One is that the driver does not respond to the deceleration operation, the second is slow response, which is manifested as the speed continuing to rise for a certain period of time, and the third is operation error, such as when the speed needs to be decelerated, the operation becomes acceleration, or the speed reduction operation process is wrong, etc. For this reason, the present invention provides an abnormal driving detection method, which determines whether the driver is fatigued driving by detecting the timeliness and correctness of the locomotive driver's reaction process when a specific operation is required. Specifically, it is mainly determined whether the driver is fatigued driving by detecting the degree of coordination between the driver's sight and action, thereby judging the driver's fatigue state at an early stage to reduce the probability of safety hazards. Please refer to Figure 1-Figure 3 , the detection method comprises the following steps:

[0109] S1. Establish an operation table of historical operation points and corresponding operation times according to the driving process, and obtain the control time of the driver's driving control, the corresponding control items and the control image of the locomotive cab; the operation table is the operation rules formulated in advance at a specific time, and also includes abnormal driving records and corresponding times of sudden situations such as speeding during driving. The control image is the full video image of the driver's control of the locomotive.

[0110] S2, identifying the driver's hand according to the control image, and generating the operation trajectory of the hand; there are a large number of operation buttons on the locomotive, such as the reversing handle, the main handle, the CIR button, the self valve and the single valve, so it is necessary to obtain the operation trajectory of the driver's hand to determine whether the driver's hand falls on the corresponding button within a specific time when a certain operation needs to be performed. In step S2, the following steps are specifically included:

[0111] S21, extracting a contour image of the driver's hand according to the control image;

[0112] S22, calculating the sharp angle of any point on the contour line of the driver's hand on the contour image; the calculation formula of the sharp angle is:

[0113]

[0114] In the above formula, cosδ(0) represents the cosine value of the sharp angle at point 0 on the hand contour line, P 0-K' and P K-0 Represents the vector formed by the 0 point and the Kth and K'th pixel points on the hand contour lines on both sides thereof.

[0115] S23, judging whether any point on the contour line of the driver's hand is a fingertip point according to whether the curvature of the point is greater than a curvature threshold;

[0116] If so, the point is the fingertip point;

[0117] If not, the point is a contour point;

[0118] S24, generating an operation trajectory of the driver's hand according to the coordinates of the fingertip points at each time;

[0119] S3, calculating the face facing vector of the driver's face at any time according to the control image, calculating the front view vector according to the sight line direction, and calculating the front view area of ​​the driver on the console according to the front view vector; in order to calculate the position of the driver's sight line, so as to facilitate determining whether the driver's sight line falls on the corresponding button when the driver needs to perform a certain action, so as to serve as an indicator for determining whether the driver is driving fatigued, in step S3, the following steps are specifically included:

[0120] S31, acquiring the coordinates of the driver's eyes and mouth according to the control image;

[0121] S32, calculating the base plane of the driver's head at any time; the expression of the base plane is:

[0122]

[0123] In the above formula, x, y and z represent the x-axis, y-axis and z-axis coordinates of the point set on the base plane, respectively; x1, x2 and x3 represent the x-axis coordinates of the driver's eyes and mouth, respectively; y1, y2 and y3 represent the y-axis coordinates of the driver's eyes and mouth, respectively; z1, z2 and z3 represent the z-axis coordinates of the driver's eyes and mouth, respectively;

[0124] S33, calculating the face facing vector of the driver's face according to the base plane; the expression of the face facing vector is:

[0125] S=A1i+A2j+A3k

[0126] in,

[0127] A1=y1z2-y1z3-y2z1+y2z3+y3z1-y3z2

[0128] A2=-x1z2+x1z3+x2z1-x2z3-x3z1+x3z2

[0129] A3=x1y2-x1y3-x2y2+x2y3+x3y1-x3y2

[0130] In the above formula, S represents the face facing vector, A1, A2 and A3 are the first coefficient, the second coefficient and the third coefficient respectively, x1, x2 and x3 represent the x-axis coordinates of the driver's eyes and mouth respectively, y1, y2 and y3 represent the y-axis coordinates of the driver's eyes and mouth respectively, z1, z2 and z3 represent the z-axis coordinates of the driver's eyes and mouth respectively;

[0131] S34, acquiring binocular images of the driver according to the control image;

[0132] S35, calculating the offset angle of the facial facing vector according to the binocular image, and correcting the front view vector to obtain the front view vector; after determining the facial facing vector of the driver through the eyes and mouth of the driver's head, since the driver's eyes can rotate, in order to further calculate the driver's line of sight direction, the facial facing vector is used as a basis, combined with the relative position of the driver's eye through hole in the eye socket, the driver's line of sight direction is further calculated. In step S35, the following steps are specifically included:

[0133] S351, calculating the first vector of the pupils of the two eyes according to the binocular images; the expression of the first vector is:

[0134] S1=(x′1-x′2, y′1-y′2, z′1-z′2)

[0135] In the above formula, S1 represents the first vector, x'1, y'1 and z'1 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of one eye, and x'2, y'2 and z'2 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of the other eye.

[0136] S352, calculating the lateral offset and longitudinal offset of the pupil of the eye;

[0137] The calculation formula for the lateral offset is:

[0138]

[0139] The calculation formula for the longitudinal offset is:

[0140]

[0141] In the above formula, Δm1 and Δm2 are the lateral offset and the longitudinal offset respectively, x'1, y'1 and z'1 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of one eye, and x'2, y'2 and z'2 represent the coordinates of the x-axis, y-axis and z-axis of the pupil of the other eye;

[0142] S353, calculating a lateral offset angle and a longitudinal offset angle according to the lateral offset amount and the longitudinal offset amount;

[0143]

[0144] In the above formula, θ1 and θ2 are the lateral offset angle and the longitudinal offset angle respectively. and are the lateral and longitudinal offsets respectively, and R is the radius of the eyeball;

[0145] S354, correcting the face facing vector according to the lateral offset angle and the longitudinal offset angle to obtain the front view vector; in three-dimensional space, there are infinite vectors that satisfy the lateral offset angle and the longitudinal offset angle with the face facing vector, so a large number of vectors need to be corrected to obtain a determined front view vector. In step S354, the following steps are specifically included:

[0146] S3541, establishing an eye space coordinate system with the facial facing vector as the y-axis, the first vector as the x-axis, and the direction perpendicular to the facing vector and the first vector as the z-axis;

[0147] S3542, calculating the first projection vector of the orthostatic vector on the xy plane of the eye space coordinate system according to the lateral offset angle and the facial orthostatic vector; the expression of the first projection vector is:

[0148]

[0149] In the above formula, cosθ1 represents the cosine of the angle between the first projection vector and the first vector, represents the first projection vector of the orthographic vector on the xy plane of the eye space coordinate system, S1 is the first vector;

[0150] S3543, calculating the second projection vector of the orthogonal vector on the yz plane of the eye space coordinate system according to the longitudinal offset angle and the facial orthogonal vector; the expression of the second projection vector is:

[0151]

[0152] In the above formula, cosθ2 represents the cosine of the angle between the second projection vector and the face-facing vector. S represents the second projection vector of the normal view vector on the xy plane of the eye space coordinate system. 0 is the face facing vector;

[0153] S3544, calculating the orthographic vector according to the first projection vector and the second projection vector; the expression of the orthographic vector is:

[0154]

[0155] In the above formula, S2 represents the normal view vector, represents the x-axis coordinate of the first projection vector, represents the y-axis coordinate of the first projection vector, Represents the z-axis coordinate of the second projection vector.

[0156] S36, constructing an emmetropia curve according to the emmetropia vector and the eye coordinates of the driver, and calculating the emmetropia area of ​​the driver; after calculating the emmetropia vector, it is necessary to further calculate the emmetropia area that the driver's line of sight can see according to the position coordinates of the driver's face eyes, so as to facilitate the judgment of whether the emmetropia area of ​​the driver's line of sight contains the coordinates of the button to be operated at any time. In step S36, the following steps are specifically included:

[0157] S361, constructing an emmetropia curve according to the emmetropia vector and eye coordinates of both eyes of the driver;

[0158] The expression of the positive view curve is:

[0159]

[0160] In the above formula, A4, A5, A6 and A7 represent the fourth, fifth, sixth and seventh coefficients respectively, x, y and z are the x-axis, y-axis and z-axis coordinates of the point on the emmetropia curve respectively, x1, y1 and z1 are the x-axis, y-axis and z-axis coordinates of one eye respectively, and x2, y2 and z2 are the x-axis, y-axis and z-axis coordinates of the other eye respectively;

[0161] S362. A cone-shaped emmetropia area is established with the center point of the driver's eyes as the vertex and the emmetropia curve as the central axis. The expression of the emmetropia area is:

[0162]

[0163] In the above formula, x, y, and z are the coordinates of the x-axis, y-axis, and z-axis of the point set in the normal viewing area, respectively. i ,y i and z i The coordinates on the positive curve are is the x-axis, y-axis and z-axis coordinates of any point that is the starting point and satisfies the orthographic vector constraint, x1, y1 and z1 are the x-axis, y-axis and z-axis coordinates of one eye respectively, x2, y2 and z2 are the x-axis, y-axis and z-axis coordinates of the other eye respectively, θ is the line of sight expansion angle, which is set according to the cone-shaped visual area of ​​the human eye to simulate the area seen by the human eye, so as to better show the orthographic area of ​​the human eye.

[0164] S4, the control image is calculated according to the minimum distance between the operation trajectory and the corresponding control button at any time, and whether the control image at that time is an operation image is determined according to whether the minimum distance is less than a distance threshold, and a standard image is extracted;

[0165] If so, extract the corresponding control image to generate the operation image, and extract the standard image based on the operation image;

[0166] If not, the corresponding control image is not extracted; there are several moments in the control image when the distance between the hand and the button in the locomotive cab is short, that is, there may be contact moments. At this time, it is necessary to extract this section of the control image to obtain a standard image. The standard image is a time period at this time. Therefore, the image of the moment when the driver's hand is most likely to contact the button in the standard image is extracted, that is, the standard image, to facilitate subsequent further calculations. In step S4, the following steps are specifically included:

[0167] S41, calculating the minimum distance between the hand and the corresponding operation key at any time in the operation trajectory according to the control image; the calculation formula of the minimum distance is: D min =MIN[d(pos α ,pos2)]

[0168] In the above formula, D min Indicates the minimum distance between the hand and the corresponding operation button at any time, MIN(·) means taking the minimum value, d(pos α ,pos2) represents the calculation of the minimum distance between the αth hand position and the corresponding operation key;

[0169] S42, setting a distance threshold, and judging whether the control image at the moment is an operation image according to whether the minimum distance is less than the distance threshold;

[0170] If yes, the control image at that moment is extracted and marked as the operation image;

[0171] If not, the corresponding control image is not extracted.

[0172] S43, selecting an operation image with the smallest minimum distance value from a plurality of operation images adjacent to each other at the same time, and marking it as a standard image.

[0173] S5, marking the time corresponding to any operation time and the standard image in the operation table as the operation time, and calculating the driver's fatigue according to the front view area and the standard image corresponding to any operation time; at this time, according to the temporal overlap between the operation time and the standard image, the fatigue data is calculated to reflect the degree of coordination between the driver's hands and vision during driving, so as to facilitate the subsequent judgment of whether the driver is fatigued. In step S5, the following steps are specifically included:

[0174] S51, marking the time corresponding to any operation time and the standard image in the operation table as the operation time;

[0175] S52, marking the moment only in the standard image as the first operation moment, and marking the remaining operation moments as the second operation moments;

[0176] S54, calculating the second fatigue degree at any second operation moment according to the normal vision area; the calculation formula of the second fatigue degree at any moment is:

[0177]

[0178] In the above formula, represents the second fatigue level at any second operation time t, is the set of all time points of any operation time closest to the second operation time point t in the operation table, is the time of the standard image corresponding to any second operation time t, F 2 and F 3 The second and third fatigue degrees, Indicates the coordinates of the corresponding operation button, ZM t A set of points representing the front view area;

[0179] S55. Assign a value to the first fatigue degree at the first operation moment, and calculate the fatigue degree of the driver at any moment; the calculation formula of the fatigue degree of the driver at any moment is:

[0180]

[0181] In the above formula, FL t Indicates the driver's fatigue level at any time. is the first fatigue degree at any first operation time t, represents the second fatigue degree at any second operation time ti, n represents the total number of driving moments of the driver, and h represents the number of times the driver is fatigued, that is, The total number of occurrences of the time and the first operation time.

[0182] The present invention first obtains the operation table and control image of the locomotive during driving, first obtains the operation trajectory of the driver's fingertips through the control image, then calculates the driver's facial facing vector through the control image, and further calculates the driver's frontal viewing area based on the facial facing vector and the rotation of his eyeballs, and then segments the control image to obtain a standard image, and finally calculates the fatigue degree at that moment and the fatigue degree at any moment according to the time of the standard image and the driver's frontal viewing area near any corresponding moment, so as to serve as an important criterion for judging whether the driver is experiencing fatigue driving.

[0183] In this embodiment, an application method of applying the abnormal driving detection method to an abnormal driving training system is also provided. The measurement of fatigue is to reflect the fatigue level of the driver. However, it is inaccurate to judge whether the locomotive driver is driving fatigued only by setting a threshold. Since each person has different abilities to resist fatigue, the speed at which fatigue increases for different people is different. When the detection personnel detect that the locomotive driver's fatigue reaches a critical threshold and take corresponding measures to deal with it within a fixed time, the locomotive driver's fatigue may reach a higher value, thereby generating an extremely high safety risk of abnormal driving. In order to avoid this situation, a method for calculating an abnormal risk factor based on fatigue to further judge whether there is a risk of fatigue driving is provided. The method comprises the following steps:

[0184] S11, producing several driving scene videos of drivers driving motorcycles;

[0185] S12, calculating and displaying the fatigue value of the driver at any time in each driving scene video;

[0186] S13. Set a fatigue threshold, and calculate the abnormal risk factor of each driver according to the value and growth rate of fatigue. The calculation formula of the abnormal risk factor is:

[0187]

[0188] In the above formula, DL t represents the abnormal risk factor of the driver at time t, FL t represents the driver's first fatigue level at time t, FL represents the basic fatigue level, and FL t-z represents the fatigue degree at the zth moment before the tth moment, and ΔFL represents the fatigue coefficient;

[0189] S14, dividing the abnormal risk factor into several attention levels, and setting several corresponding treatment measures; several attention levels are several different ranges of abnormal risk factors;

[0190] S15. According to the different attention levels of the drivers, corresponding handling measures are shown to the trainees through driving scene videos.

[0191] By further calculating the abnormal risk coefficient as a measure of the degree of risk of fatigue driving, when the locomotive driver's fatigue increases rapidly, the value of the abnormal risk coefficient is higher. At this time, according to the set processing measures corresponding to the abnormal risk coefficient, the abnormal driving training system can prompt the detection personnel to take corresponding processing measures according to the abnormal risk coefficient of the locomotive driver, so that the detection personnel can better handle such situations when encountering them during normal driving, thereby ensuring the normal driving of the locomotive driver and the safety of the locomotive.

[0192] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting abnormal driving. It is characterized in that The detection method comprises the following steps: S1. Establish an operation table of historical operation points and corresponding operation times according to the driving process, and obtain the control time including the driver's driving control, the corresponding control items and the control image of the locomotive cab; S2, recognizing the driver's hand according to the control image, and generating an operation trajectory of the hand; S3, calculating the face facing vector of the driver's face at any time according to the control image, calculating its front view vector according to the sight direction, and calculating the front view area of ​​the driver on the console according to the front view vector; S4, the control image is calculated according to the minimum distance between the operation trajectory and the corresponding control button at any time, and whether the control image at that time is an operation image is determined according to whether the minimum distance is less than a distance threshold, and a standard image is extracted; If so, extract the corresponding control image to generate the operation image, and extract the standard image based on the operation image; If not, the corresponding control image is not extracted; S5. Mark the time corresponding to any operation time and the standard image in the operation table as the operation time, and calculate the driver's fatigue level according to the front view area and the standard image corresponding to any operation time.

2. The detection method according to claim 1, It is characterized in that In step S2, the following steps are specifically included: S21, extracting a contour image of the driver's hand according to the control image; S22, calculating the sharp angle of any point on the contour line of the driver's hand on the contour image; the calculation formula of the sharp angle is: In the above formula, cosδ(0) represents the cosine value of the sharp angle at point 0 on the hand contour line, P 0-K' and P K-0 Represents the vector formed by the 0 point and the Kth and K'th pixel points on the hand contour lines on both sides thereof. S23, judging whether any point on the contour line of the driver's hand is a fingertip point according to whether the curvature of the point is greater than a curvature threshold; If so, the point is the fingertip point; If not, the point is a contour point; S24. Generate the operation trajectory of the driver's hand according to the coordinates of the fingertip points at each moment.

3. The detection method according to claim 1, It is characterized in that In step S3, the following steps are specifically included: S31, acquiring the coordinates of the driver's eyes and mouth according to the control image; S32, calculating the base plane of the driver's head at any time; the expression of the base plane is: In the above formula, x, y, and z represent the x-axis, y-axis, and z-axis coordinates of the point set on the base plane, respectively. 1 、x 2 and x 3 Represents the x-axis coordinates of the driver's eyes and mouth, y 1 ,y 2 and 3 The y-axis coordinates of the driver's eyes and mouth, 1 、z 2 and z 3 Represent the z-axis coordinates of the driver's eyes and mouth respectively; S33, calculating the face facing vector of the driver's face according to the base plane; the expression of the face facing vector is: S=A 1 i+A 2 j+A 3 k in, A 1 =y 1 z 2 -y 1 z 3 -y 2 z 1 +y 2 z 3 +y 3 z 1 -y 3 z 2 A 2 =-x 1 z 2 +x 1 z 3 +x 2 z 1 -x 2 z 3 -x 3 z 1 +x 3 z 2 TO 3 =x 1 and 2 -x 1 and 3 -x 2 and 2 +x 2 and 3 +x 3 and 1 -x 3 and 2 In the above formula, S represents the face facing vector, A 1 , A 2 and A 3 are the first, second and third coefficients respectively, x 1 、x 2 and x 3 Represents the x-axis coordinates of the driver's eyes and mouth, y 1 ,y 2 and 3 The y-axis coordinates of the driver's eyes and mouth, 1 、z 2 and z 3 Represent the z-axis coordinates of the driver's eyes and mouth respectively; S34, acquiring binocular images of the driver according to the control image; S35, calculating the offset angle of the facial facing vector according to the binocular images, and correcting the face facing vector to obtain the face facing vector; S36, constructing an emmetropia curve according to the emmetropia vector and the eye coordinates of the driver, and calculating the emmetropia area of ​​the driver.

4. The detection method according to claim 3, It is characterized in that In step S35, the following steps are specifically included: S351, calculating the first vector of the pupils of the two eyes according to the binocular images; the expression of the first vector is: S 1 =(x′ 1 -x′ 2 ,y′ 1 -y′ 2 ,z′ 1 -z′ 2 ) In the above formula, S 1 Represents the first vector, x' 1 ,y' 1 and z' 1 represents the x-axis, y-axis, and z-axis coordinates of one of the pupils of the eye, x' 2 ,y' 2 and z' 2 represents the x-axis, y-axis, and z-axis coordinates of the other eye's pupil; S352, calculating the lateral offset and longitudinal offset of the pupil of the eye; The calculation formula for the lateral offset is: The calculation formula for the longitudinal offset is: In the above formula, Δm 1 and Δm 2 are the horizontal and vertical offsets respectively, x'1, y'1 and z'1 represent the x-axis, y-axis and z-axis coordinates of the pupil of one eye, x' 2 ,y' 2 and z' 2 represents the x-axis, y-axis, and z-axis coordinates of the other eye's pupil; S353, calculating a lateral offset angle and a longitudinal offset angle according to the lateral offset amount and the longitudinal offset amount; In the above formula, θ 1 and θ 2 are the lateral offset angle and the longitudinal offset angle respectively, and are the lateral and longitudinal offsets respectively, and R is the radius of the eyeball; S354, correcting the facial facing vector according to the lateral offset angle and the longitudinal offset angle to obtain the face facing vector.

5. The detection method according to claim 4, It is characterized in that In step S354, the following steps are specifically included: S3541, establishing an eye space coordinate system with the facial facing vector as the y-axis, the first vector as the x-axis, and the direction perpendicular to the facing vector and the first vector as the z-axis; S3542, calculating the first projection vector of the orthogonal vector on the xy plane of the eye space coordinate system according to the lateral offset angle and the facial orthogonal vector; the expression of the first projection vector is: In the above formula, cosθ 1 represents the cosine of the angle between the first projection vector and the first vector, S represents the first projection vector of the normal view vector on the xy plane of the eye space coordinate system. 1 is the first vector; S3543, calculating the second projection vector of the orthogonal vector on the yz plane of the eye space coordinate system according to the longitudinal offset angle and the facial orthogonal vector; the expression of the second projection vector is: In the above formula, cosθ 2 represents the cosine of the angle between the second projection vector and the face facing vector, S represents the second projection vector of the normal view vector on the xy plane of the eye space coordinate system. 0 is the face facing vector; S3544, calculating the orthographic vector according to the first projection vector and the second projection vector; the expression of the orthographic vector is: In the above formula, S 2 represents the normal view vector, represents the x-axis coordinate of the first projection vector, represents the y-axis coordinate of the first projection vector, Represents the z-axis coordinate of the second projection vector.

6. The detection method according to claim 3, It is characterized in that In step S36, the following steps are specifically included: S361, constructing an emmetropia curve according to the emmetropia vector and eye coordinates of both eyes of the driver; The expression of the positive view curve is: In the above formula, A 4 , A 5 , A 6 and A 7 denote the fourth, fifth, sixth and seventh coefficients respectively, x, y and z are the x-axis, y-axis and z-axis coordinates of the point on the normal view curve respectively, x 1 ,y 1 and z 1 are the x-axis, y-axis, and z-axis coordinates of one eye respectively. 2 ,y 2 and z 2 are the x-axis, y-axis, and z-axis coordinates of the other eye respectively; S362. A cone-shaped emmetropia area is established with the center point of the driver's eyes as the vertex and the emmetropia curve as the central axis. The expression of the emmetropia area is: In the above formula, x, y, and z are the coordinates of the x-axis, y-axis, and z-axis of the point set in the normal viewing area, respectively. i ,y i and z i The coordinates on the positive curve are The x-axis, y-axis, and z-axis coordinates of any point that is the starting point and satisfies the normal view vector constraint. 1 ,y 1 and z 1 are the x-axis, y-axis, and z-axis coordinates of one eye respectively. 2 ,y 2 and z 2 are the x-axis, y-axis, and z-axis coordinates of the other eye, and θ is the sight expansion angle.

7. The detection method according to claim 1, It is characterized in that In step S4, the following steps are specifically included: S41, calculating the minimum distance between the hand and the corresponding operation key at any time in the operation trajectory according to the control image; the calculation formula of the minimum distance is: D min =MIN[d(pos α ,pos 2 )] In the above formula, D min Indicates the minimum distance between the hand and the corresponding operation button at any time, MIN(·) means taking the minimum value, d(pos α ,pos 2 ) represents the calculation of the minimum distance between the αth hand position and the corresponding operation key; S42, setting a distance threshold, and judging whether the control image at the moment is an operation image according to whether the minimum distance is less than the distance threshold; If yes, the control image at that moment is extracted and marked as the operation image; If not, the corresponding control image is not extracted. S43, selecting an operation image with the smallest minimum distance value from a plurality of operation images adjacent to each other at the same time, and marking it as a standard image.

8. The detection method according to claim 1, It is characterized in that In step S5, the following steps are specifically included: S51, marking the time corresponding to any operation time and the standard image in the operation table as the operation time; S52, marking the moment only in the standard image as the first operation moment, and marking the remaining operation moments as the second operation moments; S54, calculating a second fatigue degree at any second operation time according to the front-viewing area; The calculation formula of the second fatigue degree at any time is: In the above formula, represents the second fatigue level at any second operation time t, is the set of all time points of any operation time closest to the second operation time point t in the operation table, is the time of the standard image corresponding to any second operation time t, F 2 and F 3 The second and third fatigue degrees, Indicates the coordinates of the corresponding operation button, ZM t A set of points representing the front view area; S55. Assign a value to the first fatigue degree at the first operation moment, and calculate the fatigue degree of the driver at any moment; the calculation formula of the fatigue degree of the driver at any moment is: In the above formula, FL t Indicates the driver's fatigue level at any time. is the first fatigue degree at any first operation time t, represents the second fatigue degree at any second operation time ti, n represents the total number of driving moments of the driver, and h represents the number of times the driver is fatigued, that is, The total number of occurrences of the time and the first operation time.

9. The method for applying the abnormal driving detection method according to any one of claims 1 to 8 in an abnormal driving training system, It is characterized in that The method comprises the following steps: S11, producing several driving scene videos of drivers driving motorcycles; S12, calculating and displaying the fatigue value of the driver at any time in each driving scene video; S13, setting a fatigue threshold, and calculating the abnormal risk factor of each driver according to the fatigue value and growth rate; S14. Divide the abnormal risk factor into several attention levels and set several corresponding treatment measures; S15. According to the different attention levels of the drivers, corresponding handling measures are shown to the trainees through driving scene videos.

10. The detection method according to claim 9, It is characterized in that In step S13, the calculation formula of the abnormal risk coefficient is: In the above formula, DL t represents the abnormal risk factor of the driver at time t, FL t represents the driver's first fatigue level at time t, FL represents the basic fatigue level, and FL t-z represents the fatigue degree at the zth moment before the tth moment, and ΔFL represents the fatigue coefficient.

Images