Driver on-duty status monitoring and early warning system and method

Abstract

The present invention relates to the technical field of driver status monitoring, and in particular to a driver duty status monitoring and early warning system and method, comprising an early warning host installed above a cab instrument panel, a main camera for collecting a driver's face video, a panoramic camera for recording a panoramic view of the cab and the instrument panel, a road condition camera for collecting road conditions, a data acquisition unit for collecting data from the early warning host, a first analysis unit connected to the data acquisition unit, extracting the latest frame image in the face video, and judging whether an early warning is needed according to the face image confidence obtained by a target detection model, and a second analysis unit determining whether an early warning is issued according to the driver's eye closing time, eye aspect ratio and sight track evaluation value. The present invention overcomes the problem that the quality of the face video collected in the prior art is poor, and the accuracy of the analysis of the face data is low, which makes it impossible to accurately judge whether the driver is in a fatigue driving condition.

Description

Technical Field

[0001] The present invention relates to the technical field of driver status monitoring, and in particular to a driver duty status monitoring and early warning system and method. Background Art

[0002] In modern traffic systems, driver fatigue or distracted driving is one of the main causes of traffic accidents. In order to reduce accidents caused by such driving behaviors, various driver status monitoring systems have been developed. The purpose of these systems is to identify potential fatigue or distracted states by analyzing the physiological and behavioral characteristics of the driver, and to warn the driver when necessary to prompt him to take action to avoid possible dangers.

[0003] Traditional driver monitoring systems may include simple behavior monitoring, however, these systems often have certain limitations, such as sensitivity to ambient light resulting in poor image acquisition quality, susceptibility to individual driver differences, or challenges in real-time data processing.

[0004] Chinese Patent Publication No.: CN116933205A discloses a high-speed rail driver duty status early warning system based on multi-source information fusion, including a presentation layer, a business layer, a data layer, a transmission layer, a data acquisition layer and specific steps for detecting the duty status of the high-speed rail driver, characterized in that: the data acquisition layer includes an infrared camera, an EEG headband and an EEG chest patch; the infrared camera and the EEG headband are installed on the surface of the high-speed rail driver, and the EEG chest patch is set on one side of the high-speed rail driver;

[0005] It can be seen that the prior art has the following problems: since it is impossible to accurately judge the quality of the collected facial video, the accuracy of judging the driver's duty status is low; and since the accuracy of the analysis of facial data is low, it is impossible to accurately judge whether the driver is fatigued driving. Summary of the invention

[0006] To this end, the present invention provides a driver duty status monitoring and early warning system and method to overcome the problem that the quality of facial videos collected in the prior art is poor and the accuracy of facial data analysis is low, resulting in an inability to accurately determine whether the driver is fatigued.

[0007] To achieve the above object, the present invention provides a driver duty status monitoring and early warning system, comprising:

[0008] The early warning host is installed above the dashboard of the cab through a mounting bracket;

[0009] The warning host includes a main camera for collecting the driver's face video, a panoramic camera for collecting panoramic audio and video records of the driver's cab and instrument panel, a voice reminder alarm for alarming, and a road condition camera for collecting track road videos;

[0010] A data acquisition unit, connected to the early warning host, for acquiring data collected by the early warning host;

[0011] A first analysis unit, which is connected to the data acquisition unit, is used to extract the latest frame of the face image from the face video acquired by the data acquisition unit, obtain the confidence of the driver's face image according to the target detection model, and determine whether to issue an early warning according to the confidence;

[0012] or, calculating a confidence evaluation value of the confidence level, and determining whether to issue an early warning according to the confidence evaluation value;

[0013] A second analysis unit, which is connected to the data acquisition unit and the first analysis unit respectively, and is used to determine whether to issue a warning according to the driver's eye closing time;

[0014] Or, obtaining the number of frames in which the driver's eye aspect ratio is under a preset condition within a preset historical time period, and determining whether to issue a warning based on the number of frames;

[0015] Or, calculating the driver's sight trajectory evaluation value, and determining whether to issue a warning according to the sight trajectory evaluation value;

[0016] an early warning correction unit, connected to the second analysis unit, for determining a correction to a second preset number of frames based on a relative difference between the sight track evaluation value and a preset sight track evaluation value;

[0017] A control execution unit is connected to the first analysis unit, the second analysis unit and the early warning host respectively, and is used to control the alarm of the early warning host according to the analysis results of the first analysis unit and the second analysis unit.

[0018] Further, the first analysis unit determines to issue a warning based on a comparison result in which the confidence of the facial image is less than a first preset confidence, and determines to calculate a confidence evaluation value of the confidence based on a comparison result in which the confidence of the facial image is greater than or equal to the first preset confidence and less than or equal to a second preset confidence.

[0019] Furthermore, the first analysis unit calculates the confidence evaluation value of the confidence according to the following formula:

[0020] Among them, P represents the confidence evaluation value, Di represents the confidence of the face image at the i-th moment within the historical preset time length T, Dd represents the average confidence of the face images at all moments within the historical preset time length T, Dd0 represents the preset confidence average value, and N represents the number of times the confidence is obtained within the historical preset time length T.

[0021] Further, the first analysis unit determines to issue an early warning based on the confidence evaluation value being less than or equal to a preset confidence evaluation value.

[0022] Furthermore, the second analysis unit determines to issue a warning based on the comparison result that the driver's eyes closing time is greater than the preset eyes closing time, and determines to obtain the number of frames in which the driver's eye aspect ratio is under preset conditions within the historical preset time period based on the comparison result that the driver's eyes closing time is less than or equal to the preset eyes closing time.

[0023] Furthermore, the second analysis unit determines to issue a warning based on the comparison result that the frame number is greater than a second preset frame number, and determines to calculate the driver's line of sight trajectory evaluation value based on the comparison result that the frame number is greater than or equal to the first preset frame number and less than or equal to the second preset frame number.

[0024] Furthermore, the second analysis unit calculates the driver's sight trajectory evaluation value according to the following formula: Wherein, G represents the eye trajectory evaluation value, represents the difference between the horizontal coordinate of the i-th point of the driver's real-time sight trajectory and the i-th point of the driver's preset sight trajectory, represents the difference between the ordinate of the i-th point of the driver's real-time sight trajectory and the ordinate of the i-th point of the driver's preset sight trajectory, n represents the number of sampling points for sampling the driver's real-time sight trajectory, K represents the driver's pupil diameter,

[0025] Further, the second analysis unit determines to issue an early warning based on the comparison result that the sight track evaluation value is less than or equal to a preset sight track evaluation value and determines that the early warning correction unit performs a secondary determination on the sight track evaluation value.

[0026] Further, the early warning correction unit determines to correct the second preset number of frames based on the comparison result that the relative difference is greater than the preset relative difference;

[0027] The second preset frame number is corrected according to the following correction coefficient: Wherein, X represents the correction coefficient, and ΔG represents the relative difference.

[0028] On the other hand, the present invention also provides a method for a driver duty status monitoring and early warning system, comprising:

[0029] The first analysis unit extracts the latest frame of the face image from the face video acquired by the data acquisition unit, obtains the confidence of the driver's face image according to the target detection model, and determines whether to issue an early warning according to the confidence;

[0030] When the first analysis unit determines that the confidence level is greater than or equal to a first preset confidence level and less than or equal to a second preset confidence level, a confidence evaluation value of the confidence level is calculated, and whether to issue an early warning is determined according to the confidence evaluation value;

[0031] When the first analysis unit determines not to issue a warning, the second analysis unit obtains a video of the driver's eye activities within a preset time period according to history, and determines whether to issue a warning according to the driver's eye closing time;

[0032] When the eye-closing duration is less than or equal to the preset eye-closing duration, the second analysis unit obtains the number of frames in which the driver's eye aspect ratio is under the preset condition within the historical preset duration, and determines whether to issue an early warning based on the comparison result between the number of frames and the preset number of frames;

[0033] When the frame number is greater than or equal to the first preset frame number and less than or equal to the second preset frame number, the second analysis unit calculates the driver's sight track evaluation value, and determines whether to issue a warning according to the sight track evaluation value;

[0034] When the sight track evaluation value is less than or equal to a preset sight track evaluation value, the early warning correction unit performs a secondary judgment on the sight track evaluation value to determine whether to correct the second preset number of frames according to a relative difference between the sight track evaluation value and the preset sight track evaluation value.

[0035] Compared with the prior art, the beneficial effect of the present invention is that the present invention can accurately judge whether the quality of the facial image is qualified by comparing the confidence of the facial image with the preset confidence, thereby judging whether there is any abnormal light or the driver deliberately blocks the camera to avoid the camera.

[0036] Furthermore, the present invention calculates a confidence evaluation value through the confidence of the facial image at the i-th moment within the historical preset time length, the average confidence of the facial images at all moments within the historical preset time length, the preset confidence average value and the number of times the confidence is obtained within the historical preset time length, thereby accurately evaluating the quality of the driver's facial image within the historical preset time length.

[0037] Furthermore, the present invention accurately screens out whether poor quality of facial images frequently occurs within a preset historical period of time by comparing the confidence evaluation value with a preset confidence evaluation value, thereby determining whether abnormal lighting frequently occurs within the preset historical period of time or whether the driver deliberately blocks the camera to avoid the camera.

[0038] Furthermore, the present invention can quickly determine whether the driver is driving fatigued by comparing the driver's eyes closed time with the preset eyes closed time, and issue a timely warning to avoid danger. Without issuing a warning, it can further obtain the number of frames in which the driver's eye aspect ratio is under preset conditions within the historical preset time period to further accurately determine whether to issue a warning.

[0039] Furthermore, the present invention determines whether to issue a warning by comparing the number of frames in which the driver's eye aspect ratio is under preset conditions with the preset number of frames within a historical preset time period, accurately screens out situations where the driver is in a fatigued state but has not completely closed his eyes, and issues a timely warning to avoid danger, and calculates the driver's line of sight trajectory evaluation value for subsequent judgment when it is impossible to make an accurate judgment (i.e., the number of frames is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames) when the driver's eye aspect ratio is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames).

[0040] Furthermore, the present invention uses the characteristic that the pupils of people will contract when they are tired, and combines the coincidence of the driver's real-time sight track with the preset sight track to accurately calculate the sight track evaluation value, so as to more accurately evaluate the driver's mental state.

[0041] Furthermore, the present invention accurately determines whether the driver is driving fatigued by comparing the gaze trajectory evaluation value with a preset gaze trajectory evaluation value to avoid the occurrence of danger, and determines to perform a secondary judgment on the gaze trajectory evaluation value when it is determined to issue an early warning to determine whether the system sensitivity needs to be adjusted.

[0042] Furthermore, the present invention determines whether to correct the second preset frame number by comparing the relative difference with the preset relative difference, so as to adjust the warning sensitivity of the system to ensure more accurate, sensitive and quick detection of the driver's fatigue driving condition. According to the formula, a reasonable correction coefficient can be more accurately calculated based on the line of sight trajectory evaluation value. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an isometric view of a structural diagram of a driver duty status monitoring and early warning system according to an embodiment of the present invention;

[0044] Figure 2 It is a rear view of the structural diagram of the driver duty status monitoring and early warning system according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of a driver duty status monitoring and early warning system according to an embodiment of the present invention;

[0046] Figure 4 A flowchart of a method for applying a driver duty status monitoring and early warning system according to an embodiment of the present invention;

[0047] In the figure, 1-early warning host, 2-mounting bracket, 3-main camera, 4-panoramic camera, 5-voice reminder alarm, 6-road condition camera. DETAILED DESCRIPTION

[0048] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0050] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0051] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] See also Figure 1-4 As shown, Figure 1 It is an isometric view of a structural diagram of a driver duty status monitoring and early warning system according to an embodiment of the present invention; Figure 2 It is a rear view of the structural diagram of the driver duty status monitoring and early warning system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a driver duty status monitoring and early warning system according to an embodiment of the present invention; Figure 4 The present invention is a flowchart of a method for applying a driver duty status monitoring and warning system according to an embodiment of the present invention.

[0053] The driver duty status monitoring and early warning system of the embodiment of the present invention includes:

[0054] An early warning host 1 is arranged above the dashboard of the cab via a mounting bracket 2;

[0055] The warning host includes a main camera 3 for collecting the driver's face video, a panoramic camera 4 for collecting panoramic audio and video records of the driver's cab and the instrument panel, a voice reminder alarm 5 for alarming, and a road condition camera 6 for collecting track road videos;

[0056] A data acquisition unit, connected to the early warning host, for acquiring data collected by the early warning host;

[0057] a first analysis unit, which is connected to the data acquisition unit, and is used to extract the latest frame of a face image from the face video acquired by the data acquisition unit, obtain a confidence level D of the driver's face image according to the target detection model, and determine whether to issue an early warning according to the confidence level D;

[0058] Or, calculating a confidence evaluation value P of the confidence D, and determining whether to issue an early warning according to the confidence evaluation value P;

[0059] A second analysis unit, which is connected to the data acquisition unit and the first analysis unit respectively, and is used to determine whether to issue a warning according to the driver's eye closing time B;

[0060] Or, obtaining the number of frames Z in which the driver's eye aspect ratio H is under a preset condition within a historical preset time period T, and determining whether to issue a warning based on the number of frames Z;

[0061] Or, calculating the driver's sight trajectory evaluation value G, and determining whether to issue a warning according to the sight trajectory evaluation value G;

[0062] an early warning correction unit, connected to the second analysis unit, for determining a correction to the second preset frame number Z2 based on a relative difference ΔG between the sight trajectory evaluation value G and a preset sight trajectory evaluation value G0;

[0063] A control execution unit is connected to the first analysis unit, the second analysis unit and the early warning host respectively, and is used to control the alarm of the early warning host according to the analysis results of the first analysis unit and the second analysis unit.

[0064] In the embodiment of the present invention, the target detection model is preferably YOLOv5, and the hyperparameters are set as follows:

[0065] 1. Data enhancement: enhance the brightness, saturation, contrast, angle and scale of training image materials in proportion;

[0066] 2.AnchorBox: Recalculate the AnchorBox through the clustering algorithm to meet the actual detection scenario;

[0067] 3. Input image: According to the imaging quality and effect of the main camera and the performance requirements, adjust the input image size and use 416 as the input size;

[0068] 4.NMS: This model only detects faces and uses full-category NMS processing to increase recognition accuracy;

[0069] 5. Learning rate: The cosine annealing function is used to adjust the learning rate. The initial value of the learning rate is set to 0.005, and the default training batch is 300 batches.

[0070] The target detection dataset is composed of two parts, one is an open source dataset, and the other is a dataset of driver frontal face images collected in the actual operating environment. The proportions of the two datasets are 60% and 40% respectively. The overall dataset is preprocessed, and the processing steps are as follows:

[0071] 1. Adjust the image size: adjust the image to 416*416. Use interpolation method (bilinear interpolation) to scale;

[0072] 2. Image normalization: Normalize the image pixel values ​​and scale the pixel values ​​to a smaller range, between [-1, 1];

[0073] 3. Color change: The actual application scenario is infrared images, which convert all images into grayscale images;

[0074] 4. Cropping and padding: Crop or pad the image to fit the input size of the model;

[0075] 5. Random cropping: Random cropping generates images containing different parts of the target;

[0076] 6. Data augmentation: Increase the diversity of data through random transformations, including random rotation, flipping, cropping, and scaling.

[0077] Specifically, the first analysis unit determines whether to issue an early warning according to a comparison result between the confidence D of the face image and a preset confidence, and the first analysis unit is provided with a first preset confidence DY1 and a second preset confidence DY2;

[0078] If D<DY1, the first analysis unit determines to issue an early warning;

[0079] If DY1≤D≤DY2, the first analysis unit determines to calculate the confidence evaluation value P of the confidence D;

[0080] If D>DY2, the first analysis unit determines not to issue an early warning;

[0081] In the embodiment of the present invention, the first preset reliability DY1 is set to 0.4, and the second preset reliability DY2 is set to 0.8. Those skilled in the art may adjust the first preset reliability DY1 and the second preset reliability DY2 according to specific circumstances.

[0082] Specifically, the present invention accurately determines whether the quality of the facial image is qualified by comparing the confidence of the facial image with a preset confidence, thereby determining whether there is abnormal light or the driver deliberately blocks the camera to avoid the camera.

[0083] Specifically, the first analysis unit calculates the confidence evaluation value P of the confidence D according to the following formula: Among them, Di represents the confidence of the face image at the i-th moment within the historical preset time length T, Dd represents the average confidence of the face images at all moments within the historical preset time length T, Dd0 represents the preset confidence average value, and N represents the number of times the confidence is obtained within the historical preset time length T.

[0084] In the embodiment of the present invention, the preset reliability average value Dd0 is 0.85, and the historical preset time length T is 10 seconds. Those skilled in the art may adjust the preset reliability average value Dd0 and the historical preset time length T according to specific circumstances.

[0085] Specifically, the present invention calculates a confidence evaluation value through the confidence of the facial image at the i-th moment within the historical preset time length, the average confidence of the facial image at all moments within the historical preset time length, the preset confidence average value and the number of times the confidence is obtained within the historical preset time length, so as to accurately evaluate the quality of the driver's facial image within the historical preset time length.

[0086] Specifically, the first analysis unit determines whether to issue an early warning according to a comparison result between the confidence evaluation value P and a preset confidence evaluation value P0 under the condition that the confidence evaluation value P is calculated;

[0087] If P≤P0, the first analysis unit determines to issue an early warning;

[0088] If P>P0, the first analysis unit determines not to issue an early warning;

[0089] In the embodiment of the present invention, the preset reliability evaluation value P0 is set to 0.6. Those skilled in the art may adjust the preset reliability evaluation value P0 according to specific circumstances.

[0090] Specifically, the present invention accurately screens out whether poor quality of facial images frequently occurs within a preset historical period of time by comparing the confidence evaluation value with a preset confidence evaluation value, thereby judging whether abnormal lighting frequently occurs within the preset historical period of time or whether the driver deliberately blocks the camera to avoid the camera.

[0091] Specifically, the second analysis unit obtains the eye activity video of the driver within the historical preset time T under the condition that the first analysis unit determines that no warning is issued, and determines whether to issue a warning according to the comparison result of the driver's eye closing time B and the preset eye closing time B0;

[0092] If B≤B0, the second analysis unit determines to obtain the number of frames Z in which the driver's eye aspect ratio H is under a preset condition within a preset historical time period T;

[0093] If B>B0, the second analysis unit determines to issue an early warning;

[0094] In the embodiment of the present invention, the preset condition is the eye aspect ratio H∈[0.2,0.3], and the preset eye closure duration B0 is 10% of the historical preset duration T. Those skilled in the art may adjust the preset eye closure duration B0 according to specific circumstances.

[0095] Specifically, the present invention can quickly determine whether the driver is driving fatigued by comparing the driver's eyes closed time with the preset eyes closed time, and issue a timely warning to avoid danger. Without issuing a warning, it can further obtain the number of frames in which the driver's eye aspect ratio is under preset conditions within the historical preset time period to further accurately determine whether to issue a warning.

[0096] Specifically, when the second analysis unit completes obtaining the number of frames Z in which the driver's eye aspect ratio H is under the preset condition within the historical preset time T, it determines whether to issue an early warning according to the comparison result between the number of frames Z and the preset number of frames, and the second analysis unit is provided with a first preset number of frames Z1 and a second preset number of frames Z2;

[0097] If Z<Z1, the second analysis unit determines not to issue an early warning;

[0098] If Z1≤Z≤Z2, the second analysis unit determines to calculate the driver's sight trajectory evaluation value G;

[0099] If Z>Z2, the second analysis unit determines to issue an early warning;

[0100] In an embodiment of the present invention, the first preset frame number Z1 takes 30% of the total frame number of the eye activity video, and the second preset frame number Z2 takes 60% of the total frame number of the eye activity video. Those skilled in the art can adjust the first preset frame number Z1 and the second preset frame number Z2 according to specific circumstances.

[0101] Specifically, the present invention determines whether to issue a warning by comparing the number of frames in which the driver's eye aspect ratio is under preset conditions with the preset number of frames within a historical preset time period, accurately screens out situations where the driver is in a fatigued state but has not completely closed his eyes, and issues a timely warning to avoid danger. In the event that an accurate judgment cannot be made (i.e., the number of frames is greater than or equal to the first preset number of frames and less than or equal to the second preset number of frames), the present invention calculates the driver's line of sight trajectory evaluation value for subsequent judgment.

[0102] Specifically, the second analysis unit calculates the driver's sight trajectory evaluation value G according to the following formula: Wherein, G represents the eye trajectory evaluation value, represents the difference between the horizontal coordinate of the i-th point of the driver's real-time sight trajectory and the i-th point of the driver's preset sight trajectory, represents the difference between the ordinate of the i-th point of the driver's real-time sight trajectory and the ordinate of the i-th point of the driver's preset sight trajectory, n represents the number of sampling points for sampling the driver's real-time sight trajectory, K represents the driver's pupil diameter, .

[0103] In the embodiment of the present invention, the preset sight line trajectory is the sight line trajectory of the driver when he correctly checks the instrument at a specific driving section and a specific driving time.

[0104] Specifically, the present invention uses the characteristic that a person's pupils will contract when they are fatigued, and combines the overlap between the driver's real-time sight track and the preset sight track to accurately calculate the sight track evaluation value, thereby more accurately evaluating the driver's mental state.

[0105] Specifically, the second analysis unit determines whether to issue a warning based on a comparison result between the sight trajectory evaluation value G and a preset sight trajectory evaluation value G0 when calculating the sight trajectory evaluation value G of the driver.

[0106] If G≤G0, the second analysis unit determines to issue an early warning and determines that the early warning correction unit performs a secondary determination on the sight trajectory evaluation value G;

[0107] If G>G0, the second analysis unit determines not to issue an early warning;

[0108] In the embodiment of the present invention, the preset sight track evaluation value G0 is set to 0.6. Those skilled in the art may adjust the preset sight track evaluation value G0 according to specific circumstances.

[0109] Specifically, the present invention accurately determines whether the driver is driving fatigued by comparing the gaze trajectory evaluation value with the preset gaze trajectory evaluation value to avoid the occurrence of danger, and determines to make a secondary judgment on the gaze trajectory evaluation value when it is determined to issue an early warning to determine whether the system sensitivity needs to be adjusted.

[0110] Specifically, the second analysis unit determines that the early warning correction unit performs a secondary determination on the sight trajectory evaluation value G, the early warning correction unit calculates the relative difference △G between the sight trajectory evaluation value G and the preset sight trajectory evaluation value G0, and determines whether to correct the second preset frame number Z2 according to the comparison result of the relative difference △G and the preset relative difference △G0, and sets △G=(G0-G) / G0;

[0111] If △G≤△G0, the early warning correction unit determines not to correct the second preset frame number Z2;

[0112] If △G>△G0, the early warning correction unit determines to correct the second preset frame number Z2;

[0113] In the embodiment of the present invention, the preset relative difference △G0 is set to 0.5. The preset relative difference △G0 is obtained when the sight trajectory evaluation value G is 0.3. Those skilled in the art can adjust the preset relative difference △G0 according to specific circumstances.

[0114] Specifically, the early warning correction unit corrects the second preset frame number Z2 according to the following correction coefficient X when determining the correction condition for the second preset frame number Z2, setting The modified second preset frame number is set to Zz2=Z2×X.

[0115] Specifically, the present invention determines whether to correct the second preset frame number by comparing the relative difference with the preset relative difference, so as to adjust the warning sensitivity of the system to ensure more accurate, sensitive and quick detection of the driver's fatigue driving condition. According to the formula, a reasonable correction coefficient can be more accurately calculated based on the line of sight trajectory evaluation value.

[0116] The method of the present invention applied to a driver duty status monitoring and early warning system comprises:

[0117] Step S1, the first analysis unit extracts the latest frame of the face image from the face video acquired by the data acquisition unit, obtains the confidence D of the driver's face image according to the target detection model, and determines whether to issue an early warning according to the confidence D;

[0118] Step S2, when the first analysis unit determines that the confidence is greater than or equal to the first preset confidence and less than or equal to the second preset confidence, calculate the confidence evaluation value P of the confidence D, and determine whether to issue an early warning according to the confidence evaluation value P;

[0119] Step S3, when the first analysis unit determines not to issue a warning, the second analysis unit obtains the driver's eye activity video within a historical preset time T, and determines whether to issue a warning according to the driver's eye closing time B;

[0120] Step S4, when the eye-closing duration B is less than or equal to the preset eye-closing duration B0, the second analysis unit obtains the number of frames Z in which the driver's eye aspect ratio H is under the preset condition within the historical preset duration T, and determines whether to issue an early warning based on the comparison result of the number of frames Z with the preset number of frames;

[0121] Step S5, when the frame number Z is greater than or equal to the first preset frame number Z1 and less than or equal to the second preset frame number Z2, the second analysis unit calculates the driver's sight trajectory evaluation value G, and determines whether to issue a warning according to the sight trajectory evaluation value G;

[0122] Step S6, when the sight trajectory evaluation value G is less than or equal to the preset sight trajectory evaluation value G0, the early warning correction unit performs a secondary judgment on the sight trajectory evaluation value G to determine whether to correct the second preset frame number Z2 according to the relative difference ΔG between the sight trajectory evaluation value G and the preset sight trajectory evaluation value G0.

[0123] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A driver duty status monitoring and early warning system, characterized in that: include: The early warning host is installed above the dashboard of the cab through a mounting bracket; The warning host includes a main camera for collecting the driver's face video, a panoramic camera for collecting panoramic audio and video records of the driver's cab and instrument panel, a voice reminder alarm for alarming, and a road condition camera for collecting track road videos; A data acquisition unit, connected to the early warning host, for acquiring data collected by the early warning host; a first analysis unit, connected to the data acquisition unit, for extracting a face image of the latest frame from the face video acquired by the data acquisition unit, obtaining a confidence level of the driver's face image according to the target detection model, and determining to issue an early warning based on a comparison result that the confidence level of the face image is less than a first preset confidence level; Or, based on the comparison result that the confidence of the face image is greater than or equal to a first preset confidence and less than or equal to a second preset confidence, a confidence evaluation value of the confidence is calculated, and whether to issue an early warning is determined according to the confidence evaluation value; A second analysis unit, which is connected to the data acquisition unit and the first analysis unit respectively, and is used to determine whether to issue a warning according to the driver's eye closing time, and to determine whether to issue a warning based on a comparison result that the driver's eye closing time is greater than a preset eye closing time; Or, based on the comparison result that the driver's eye closing time is less than or equal to the preset eye closing time, determine the number of frames in which the driver's eye aspect ratio is under a preset condition within the historical preset time, and based on the comparison result that the frame number is greater than a second preset frame number, determine to issue an early warning; Or, based on the comparison result that the frame number is greater than or equal to the first preset frame number and less than or equal to the second preset frame number, the driver's sight track evaluation value is determined and calculated; The second preset frame number is 60% of the total frame number of the eye activity video; an early warning correction unit, connected to the second analysis unit, for determining a correction to a second preset number of frames based on a relative difference between the sight track evaluation value and a preset sight track evaluation value; A control execution unit is connected to the first analysis unit, the second analysis unit and the early warning host respectively, and is used to control the alarm of the early warning host according to the analysis results of the first analysis unit and the second analysis unit.

2. The driver duty status monitoring and early warning system according to claim 1 is characterized in that: The first analysis unit calculates the confidence evaluation value of the confidence according to the following formula: ， Among them, P represents the confidence evaluation value, Dj represents the confidence of the face image at the jth moment within the historical preset time T, Dd represents the average confidence of the face images at all moments within the historical preset time T, Dd0 represents the preset confidence average value, and N represents the number of times the confidence is obtained within the historical preset time T.

3. The driver duty status monitoring and early warning system according to claim 2 is characterized in that: The first analysis unit determines to issue an early warning based on the confidence evaluation value being less than or equal to a preset confidence evaluation value.

4. The driver duty status monitoring and early warning system according to claim 3 is characterized in that: The second analysis unit calculates the driver's sight trajectory evaluation value according to the following formula: ， Wherein, G represents the eye trajectory evaluation value, represents the difference between the horizontal coordinate of the i-th point of the driver's real-time sight trajectory and the i-th point of the driver's preset sight trajectory, represents the difference between the ordinate of the i-th point of the driver's real-time sight trajectory and the ordinate of the i-th point of the driver's preset sight trajectory, n represents the number of sampling points for sampling the driver's real-time sight trajectory, K represents the driver's pupil diameter, .

5. The driver duty status monitoring and early warning system according to claim 4 is characterized in that: The second analysis unit determines to issue an early warning based on a comparison result that the sight track evaluation value is less than or equal to a preset sight track evaluation value and determines that the early warning correction unit performs a secondary determination on the sight track evaluation value.

6. The driver duty status monitoring and early warning system according to claim 5 is characterized in that: The early warning correction unit determines to correct the second preset number of frames based on the comparison result that the relative difference is greater than the preset relative difference; The second preset frame number is corrected according to the following correction coefficient: , wherein X represents the correction coefficient and ΔG represents the relative difference.

7. A method for use in the driver duty status monitoring and early warning system according to any one of claims 1 to 6, characterized in that: The first analysis unit extracts the latest frame of the face image from the face video acquired by the data acquisition unit, obtains the confidence of the driver's face image according to the target detection model, and determines whether to issue an early warning according to the confidence; When the first analysis unit determines that the confidence level is greater than or equal to a first preset confidence level and less than or equal to a second preset confidence level, a confidence evaluation value of the confidence level is calculated, and whether to issue an early warning is determined according to the confidence evaluation value; When the first analysis unit determines not to issue a warning, the second analysis unit obtains a video of the driver's eye activities within a preset time period according to history, and determines whether to issue a warning according to the driver's eye closing time; When the eye-closing duration is less than or equal to the preset eye-closing duration, the second analysis unit obtains the number of frames in which the driver's eye aspect ratio is under the preset condition within the historical preset duration, and determines whether to issue an early warning based on the comparison result between the number of frames and the preset number of frames; When the frame number is greater than or equal to the first preset frame number and less than or equal to the second preset frame number, the second analysis unit calculates the driver's sight track evaluation value, and determines whether to issue a warning according to the sight track evaluation value; When the sight track evaluation value is less than or equal to a preset sight track evaluation value, the early warning correction unit performs a secondary judgment on the sight track evaluation value to determine whether to correct the second preset number of frames according to a relative difference between the sight track evaluation value and the preset sight track evaluation value.

Images