Driver fatigue prediction method and device, electronic equipment and storage medium

Abstract

The application provides a driver fatigue prediction method and device, electronic equipment and storage medium, which obtains a monitoring video; obtains facial state features and a fatigue score according to the monitoring video; trains a fatigue prediction regression model by taking the facial state features as an input vector and the fatigue score as a target result; extracts facial state features of a to-be-detected driver from a to-be-detected image or video, inputs the facial state features of the to-be-detected driver into the trained fatigue prediction regression model, obtains a fatigue score corresponding to each time in a future period of time, and then can infer the time required to reach different fatigue levels according to the current time and the time corresponding to different fatigue levels, so as to achieve the purpose of predicting the fatigue condition of the driver in advance, help early warning, avoid vehicle accidents, and the same model can obtain corresponding prediction time according to different fatigue level requirements, and has high expansibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of driver monitoring, and in particular to a driver fatigue prediction method and device, an electronic device and a storage medium. BACKGROUND

[0002] A driver monitoring system (DMS) refers to a system that monitors the behavior state of a driver during driving and gives a warning and reminder function. Since fatigue driving accounts for a major part of traffic accidents, fatigue detection is an important monitoring content of the DMS system. Fatigue detection mainly includes two forms. One is a non-invasive form, that is, the visual image captured by the camera placed in front of the driver is analyzed to detect the fatigue of the driver through the face, head and body information of the driver. The other is an invasive form, that is, the fatigue of the driver is detected through the contact between the driver and the wearable device, the electrode, and the like. When the fatigue of the driver is detected by the above two detection forms, the vehicle is often at the edge of a traffic accident, the detection result has a lag, and there is a driving safety hazard. In related technologies, an artificial neural network model is used to predict the time required to reach a preset fatigue threshold. However, since the fatigue threshold is determined according to different requirements, the model needs to be trained multiple times for different user requirements, and the model has poor expansion capability. SUMMARY

[0003] The present application provides a driver fatigue prediction method, device, electronic device and storage medium to solve the defects of the traditional fatigue detection method, such as lag, driving safety hazard and poor fatigue prediction model expansion capability.

[0004] The present application provides a driver fatigue prediction method, which comprises:

[0005] obtaining a monitoring video;

[0006] obtaining a face state feature and a fatigue score of the driver from the monitoring video;

[0007] training a fatigue prediction regression model by taking the face state feature as an input vector and the fatigue score as a target result;

[0008] extracting the face state feature of the to-be-detected driver from a to-be-detected image or video, inputting the face state feature of the to-be-detected driver into the trained fatigue prediction regression model, and obtaining a fatigue score corresponding to each time in a future period of time.

[0009] The driver fatigue prediction method provided by the application comprises the following steps:

[0010] The face key point information corresponding to the single frame image is extracted from the monitoring video, and the face key point information comprises the ratio of the upper and lower eye distances and the left and right eye corner distances and the ratio of the upper and lower mouth distances and the left and right mouth corner distances.

[0011] The face key point information corresponding to the multiple frames of images is statistically analyzed to obtain the first face state feature.

[0012] The monitoring video is input into the video classification model to obtain the second face state feature and the fatigue score.

[0013] The first face state feature and the second face state feature are taken as the face state feature of the driver.

[0014] The driver fatigue prediction method provided by the application comprises the following steps:

[0015] The closed eye event information, the mouth opening event information, the correlation event information and the first statistical information;

[0016] The closed eye event information comprises the number of closed eye events and the time length of the closed eye events, and whether the closed eye event occurs is determined according to whether the ratio of the upper and lower eye distances and the left and right eye corner distances is lower than a preset closed eye threshold value;

[0017] The mouth opening event information comprises the number of mouth opening events and the time length of the mouth opening events, and whether the mouth opening event occurs is determined according to whether the ratio of the upper and lower mouth distances and the left and right mouth corner distances is higher than a preset closed mouth threshold value;

[0018] The correlation event information comprises the proportional relationship between the number of closed eye events and the number of mouth opening events;

[0019] The first statistical information comprises the mean value, the variance, the maximum value and the minimum value corresponding to the closed eye event information, the mouth opening event information and the correlation event information.

[0020] The driver fatigue prediction method provided by the application comprises the following steps:

[0021] The closed eye score, the yawning score, the downward looking score and the second statistical information, and the second statistical information comprises the mean value, the variance, the maximum value and the minimum value corresponding to the closed eye score, the yawning score and the downward looking score.

[0022] The driver fatigue prediction method provided by the application further comprises the following steps:

[0023] Using time as the independent variable, construct the first Gaussian distribution function based on the yawning score at each moment;

[0024] Construct a second Gaussian distribution function based on the maximum value of the Gaussian distribution function obtained at all times;

[0025] The yawning score of the second Gaussian distribution function at each time step is multiplied by the fatigue score at the corresponding time step to obtain the updated fatigue score.

[0026] The fatigue prediction regression model is trained using the updated fatigue score as the target result.

[0027] According to the driver fatigue prediction method provided by the present invention, the fatigue prediction regression model includes an objective function, which is constructed based on a gradient boosting decision tree model and a loss function.

[0028] The gradient boosting decision tree model includes a regularization term corresponding to the regression tree.

[0029] According to a driver fatigue prediction method provided by the present invention, training the fatigue prediction regression model includes:

[0030] The fatigue prediction regression model is incrementally trained using an enhancement method, specifically including:

[0031] During the t-th training round, the calculation is performed based on the input vector x before the t-th round. i The sum of the obtained contribution values, and the sum of the values ​​obtained in the t-th round based on the input vector x i The contribution value obtained;

[0032] Based on the input vector x before round t i The sum of the obtained contribution values ​​and the sum of the values ​​obtained in the t-th round based on the input vector x i The summation function between the obtained contribution values This serves as the new objective function until the training termination condition is met, where... For round t, based on the input vector x i The sum of the contribution values ​​obtained, h t (x i ) is the number of rounds t based on the input vector x i The contribution value obtained.

[0033] The present invention also provides a driver fatigue prediction device, comprising:

[0034] The acquisition module is used to acquire surveillance video.

[0035] The calculation module is used to obtain the driver's facial features and fatigue score based on the monitoring video;

[0036] a training module configured to train a fatigue prediction regression model by taking the face state features as input vectors and the fatigue scores as target results;

[0037] a prediction module configured to extract face state features of a to-be-detected driver from a to-be-detected image or video, input the face state features of the to-be-detected driver into the trained fatigue prediction regression model, and obtain fatigue scores corresponding to each time point in a future period of time.

[0038] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the driver fatigue prediction method according to any one of the above embodiments when executing the program.

[0039] The present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the driver fatigue prediction method according to any one of the above embodiments.

[0040] The driver fatigue prediction method, device, electronic device, and storage medium provided by the present application can obtain a monitoring video, obtain face state features and fatigue scores from the monitoring video, train a fatigue prediction regression model by taking the face state features as input vectors and the fatigue scores as target results, extract face state features of a to-be-detected driver from a to-be-detected image or video, input the face state features of the to-be-detected driver into the trained fatigue prediction regression model, and obtain fatigue scores corresponding to each time point in a future period of time, so as to infer the time required to reach different fatigue levels according to the current time point and the time points corresponding to different fatigue levels, thereby achieving the purpose of predicting the fatigue condition of the driver in advance, helping to give an early warning, avoiding vehicle accidents, and achieving high expansibility since the same model can obtain corresponding prediction time according to different fatigue level requirements. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0042] Figure 1 is one of the flowcharts of the driver fatigue prediction method provided by the present application;

[0043] Figure 2 is another flowchart of the driver fatigue prediction method provided by the present application;

[0044] Figure 3 is a structural schematic diagram of a driver fatigue prediction device provided by the present application;

[0045] Figure 4 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] Figure 1 A flow chart of a driver fatigue prediction method provided by an embodiment of the present application is shown in Figure 1 The driver fatigue prediction method provided by the embodiment of the present application includes:

[0048] Step 101, acquiring a monitoring video;

[0049] In the embodiment of the present application, the monitoring video is used to monitor a face and eye, mouth and other regions to obtain face key point information and time sequence information of the video.

[0050] Step 102, obtaining face state features and a fatigue score according to the monitoring video;

[0051] In the embodiment of the present application, the face state features and the fatigue score of the driver are obtained according to the monitoring video, including:

[0052] Step 1021, extracting face key point information corresponding to a single frame image from the monitoring video, the face key point information including a ratio of an upper and lower eye distance to a left and right eye corner distance and a ratio of an upper and lower mouth distance to a left and right mouth corner distance;

[0053] Step 1022, statistically analyzing the face key point information corresponding to multiple frame images to obtain first face state features;

[0054] In the embodiment of the present application, the first face state features include:

[0055] eye closing event information, mouth opening event information, associated event information and first statistical information;

[0056] The eye closing event information includes a number of eye closing events and a time length of the eye closing events, and whether the eye closing events occur is determined according to whether the ratio of the upper and lower eye distance to the left and right eye corner distance is lower than a preset eye closing threshold;

[0057] The mouth opening event information includes a number of mouth opening events and a time length of the mouth opening events, and whether the mouth opening event occurs is determined according to whether a ratio of the upper-lower mouth distance to the left-right mouth corner distance exceeds a preset mouth closing threshold value;

[0058] The correlation event information includes a proportional relationship between a number of the eye closing events and a number of the mouth opening events.

[0059] The first statistical information includes mean values, variances, maximum values and minimum values corresponding to the eye closing event information, the mouth opening event information and the correlation event information.

[0060] In step 1023, the monitored video is input into the video classification model to obtain second facial state features and a fatigue score.

[0061] In the embodiment of the present application, the video classification model is, for example, a video classification model based on a video action classification network. The second facial state features and the fatigue score are obtained through the video classification model. For example, first frame images and second frame images are extracted from the monitored video in the cockpit in pairs, the target features located at the specified positions are extracted from the first frame images and the second frame images by using the face key point detection model, and face key point detection is performed to output face key point information. The second facial state features and the fatigue score are determined based on the face key point information.

[0062] The second facial state features include an eye closing score, a yawning score, a downward looking score and second statistical information. The second statistical information includes mean values, variances, maximum values and minimum values corresponding to the eye closing score, the yawning score and the downward looking score.

[0063] In step 1024, the first facial state features and the second facial state features are taken as the facial state features of the driver.

[0064] In step 103, the facial state features are taken as an input vector, and the fatigue prediction regression model is trained with the fatigue score as a target result.

[0065] In the embodiment of the present application, the first statistical information and the second statistical information are taken as the input vector to train the fatigue prediction regression model, which can improve the expression ability of the features and further improve the model training speed and the model output result.

[0066] In step 104, the facial state features of the to-be-detected driver are extracted from a to-be-detected image or video, and the facial state features of the to-be-detected driver are input into the trained fatigue prediction regression model to obtain the fatigue score corresponding to each time in a future period of time.

[0067] In the embodiment of the present application, the face state feature of the to-be-detected driver can be obtained by image processing or inputting the video classification model to the to-be-detected image or video, and the specific method and steps of the image processing or inputting the video classification model are the same as those in step 102, which will not be repeated here.

[0068] In the embodiment of the present application, the face state feature of the to-be-detected driver can be obtained by inputting the to-be-detected driver video collected in the vehicle into the video classification model, obtaining the face state feature of the to-be-detected driver at the current time according to the output of the video classification model, and inputting the face state feature of the to-be-detected driver at the current time into the trained fatigue prediction regression model to obtain the fatigue score corresponding to each time in the future period.

[0069] According to the current time and the fatigue score corresponding to each time, the time required to reach different fatigue levels can be inferred, so as to realize early warning.

[0070] The traditional fatigue detection method has a lagging detection result, and the invasive method based on physiological indicators is not only inconvenient to wear and requires professional guidance, but also has different results for different people and is not universal.

[0071] The traditional fatigue prediction model based on artificial neural network needs to specify the fatigue level threshold for predicting the time to reach the fatigue threshold and predicting the fatigue time, and needs to adjust an additional hyperparameter. Since the time to reach different fatigue levels is different, multiple models need to be trained.

[0072] Moreover, due to the multi-layer neuron weight and nonlinear activation function of the model, the logic behind the artificial neural network becomes difficult to study, and the limitations and lack of interpretability of the artificial neural network itself make it difficult to study and correct the decision.

[0073] In the embodiment of the present application, the fatigue prediction regression model includes but is not limited to Gradient Boosting Decision Trees (GBDT) model, time series prediction model such as Long Short-Term Memory network model, and Transfermer model, which has better interpretability than artificial neural network model.

[0074] The driver fatigue prediction method provided by the embodiment of the present application comprises the following steps: obtaining a monitoring video; obtaining facial state features and a fatigue score according to the monitoring video; training a fatigue prediction regression model by taking the facial state features as an input vector and the fatigue score as a target result; extracting facial state features of a to-be-detected driver from a to-be-detected image or video, and inputting the facial state features of the to-be-detected driver into the trained fatigue prediction regression model to obtain a fatigue score corresponding to each time in a future period of time, and then inferring the time required to reach different fatigue levels according to the current time and the time corresponding to different fatigue levels, so as to realize early prediction of the fatigue condition of the driver, early warning, and avoidance of vehicle accidents, and the same model can obtain corresponding prediction time according to different fatigue level requirements, and has high expansibility.

[0075] Based on any of the above embodiments, as shown in Figure 2 The driver fatigue prediction method provided by the embodiment of the present application comprises the following steps:

[0076] Step 201, taking time as an independent variable, and constructing a first Gaussian distribution function based on the yawn score of each time;

[0077]

[0078] Wherein, x is a different time identifier value, μ is the identifier value of the current time, σ = 0.4 * δ / y is inversely proportional to the yawn score of the current time, and δ = 64 is a relaxation coefficient.

[0079] Step 202, constructing a second Gaussian distribution function according to the maximum value of the Gaussian distribution function obtained at all times;

[0080]

[0081] Wherein, N represents the number of all time identifier values, i represents the i-th time, and j represents the j-th time identifier value.

[0082] Step 203, multiplying the yawn score of the second Gaussian distribution function at each time by the fatigue score corresponding to the time to obtain an updated fatigue score;

[0083] Step 204, training the fatigue prediction regression model by taking the updated fatigue score as a target result.

[0084] Since the detection of yawning is easier and more accurate, and the fatigue level before and after yawning is relatively greater, the fatigue before and after yawning is given a greater weight, the Gaussian distribution is constructed based on the yawn score, which is used as the weight of the existing fatigue score, and the updating of the fatigue score, i.e., the target result, can improve the model training speed and the accuracy of the model output.

[0085] In this embodiment of the invention, the fatigue prediction regression model includes an objective function, which is constructed based on a gradient boosting decision tree model and a loss function. The gradient boosting decision tree model includes a regularization term corresponding to the regression tree. The loss function is constructed based on the distance between the prediction result and the target result. The regularization term corresponding to the regression tree consists of multiple decision trees, and the conclusions of all trees are summed to obtain the final answer.

[0086] objective function

[0087] Where l(·,·) is the loss function, y i For the target result, For the prediction result, i is the label value corresponding to time, n is the total number of label values, the loss function is the error between the target result and the prediction result, K is the total number of regression trees, and h is the regularization term corresponding to the k-th regression tree. k For regression tree functions, Let be the function space for all regression trees.

[0088] In this embodiment of the invention, training the fatigue prediction regression model includes:

[0089] An enhancement method was used to incrementally train the fatigue prediction regression model.

[0090] Boosting methods are a family of algorithms that can boost weak learners into strong learners. The working mechanism is as follows: first, a base learner is trained from the initial training set. Then, the distribution of training samples is adjusted based on the performance of the base learner, so that the training samples that the base learner previously misjudged receive more attention in the future. Then, the next base learner is trained based on the adjusted sample distribution. This process is repeated until the number of base learners reaches a pre-specified value T. Finally, these T base learners are weighted and combined.

[0091] Incremental training involves starting from a constant and adding a new tree in each iteration. This method, using boosting techniques to incrementally train a fatigue prediction regression model, specifically includes:

[0092] During the t-th training round, the calculation is performed based on the input vector x before the t-th round. i The sum of the obtained contribution values, and the sum of the values ​​obtained in the t-th round based on the input vector x i The contribution value obtained;

[0093] Based on the input vector x before round t i The sum of the obtained contribution values ​​and the sum of the values ​​obtained in the t-th round based on the input vector x i The summation function between the obtained contribution values This serves as the new objective function until the training termination condition is met, where... is the contribution value obtained according to the input vector x i is the sum of the contribution values obtained t (x i is the contribution value obtained according to the input vector x i .

[0094] In the embodiment of the present application, the training end condition is that the training reaches a preset step number or the loss value is less than a preset value.

[0095] In the embodiment of the present application, the purpose of predicting fatigue is achieved by predicting the fatigue state in the future period of time, the prediction scheme is more direct, and no fatigue level threshold needs to be specified. The gradient boosting decision tree method has good interpretability; by fusing the yawning information, the accuracy of the fatigue score is improved; and the prediction model of the present application has good scalability, and the same training model can obtain different fatigue reaching times according to different fatigue degree requirements, meeting different user requirements.

[0096] The driver fatigue prediction device provided by the present application is described below, and the driver fatigue prediction device described below can be correspondingly referred to the driver fatigue prediction method described above.

[0097] Figure 3 The schematic diagram of the driver fatigue prediction device provided by the embodiment of the present application is shown in Figure 3 The driver fatigue prediction device provided by the embodiment of the present application comprises:

[0098] The acquisition module 301 is configured to acquire a monitoring video.

[0099] The calculation module 302 is configured to obtain a face state feature and a fatigue score of a driver according to the monitoring video.

[0100] The training module 303 is configured to train a fatigue prediction regression model by taking the face state feature as an input vector and taking the fatigue score as a target result.

[0101] The prediction module 304 is configured to extract a face state feature of a to-be-detected driver from a to-be-detected image or video, input the face state feature of the to-be-detected driver into the trained fatigue prediction regression model, and obtain a fatigue score corresponding to each time in a future period of time.

[0102] The driver fatigue prediction device provided by the embodiment of the present application obtains a monitoring video, obtains facial state features and a fatigue score according to the monitoring video, trains a fatigue prediction regression model by taking the facial state features as an input vector and the fatigue score as a target result, extracts facial state features of a to-be-detected driver from a to-be-detected image or video, inputs the facial state features of the to-be-detected driver into the trained fatigue prediction regression model, obtains a fatigue score corresponding to each time in a future period of time, and then infers a time required to reach different fatigue levels according to a current time and a time corresponding to different fatigue levels, so that the driver fatigue condition can be predicted in advance, early warning can be performed, and vehicle accidents can be avoided. In addition, the same model can obtain corresponding prediction time according to different fatigue level requirements, and has high expansibility.

[0103] Figure 4 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 4 The electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can communicate with each other through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute a driver fatigue prediction method, which includes obtaining a monitoring video, obtaining facial state features and a fatigue score according to the monitoring video, training a fatigue prediction regression model by taking the facial state features as an input vector and the fatigue score as a target result, extracting facial state features of a to-be-detected driver from a to-be-detected image or video, and inputting the facial state features of the to-be-detected driver into the trained fatigue prediction regression model to obtain a fatigue score corresponding to each time in a future period of time.

[0104] In addition, the logical instruction in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0105] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a driver fatigue prediction method provided by each of the above methods, the method comprising: obtaining a monitoring video; obtaining a face state feature and a fatigue score from the monitoring video; training a fatigue prediction regression model by taking the face state feature as an input vector and the fatigue score as a target result; extracting a face state feature of a to-be-detected driver from a to-be-detected image or video, and inputting the face state feature of the to-be-detected driver into the trained fatigue prediction regression model to obtain a fatigue score corresponding to each time point in a future period of time.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0107] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A driver fatigue prediction method characterized by, The method comprises the following steps: obtaining a monitoring video; obtaining a face state feature and a fatigue score of a driver according to the monitoring video; the step of obtaining the face state feature and the fatigue score of the driver according to the monitoring video comprises the following steps: extracting face key point information corresponding to a single frame image from the monitoring video, wherein the face key point information comprises a ratio of an upper and lower eye distance to a left and right eye corner distance and a ratio of an upper and lower mouth distance to a left and right mouth corner distance; performing statistical analysis on face key point information corresponding to multiple frames of images to obtain a first face state feature; inputting the monitoring video into a video classification model to obtain a second face state feature and a fatigue score; and taking the first face state feature and the second face state feature as the face state feature of the driver; taking the face state feature as an input vector and the fatigue score as a target result to train a fatigue prediction regression model; taking time as an independent variable, constructing a first Gaussian distribution function based on a yawning score at each time point, constructing a second Gaussian distribution function according to maximum values of Gaussian distribution functions corresponding to all time points, multiplying the yawning score at each time point by the fatigue score at the corresponding time point according to the second Gaussian distribution function to obtain an updated fatigue score, and taking the updated fatigue score as a target result to train the fatigue prediction regression model; extracting a face state feature of a to-be-detected driver from a to-be-detected image or video and inputting the face state feature of the to-be-detected driver into the trained fatigue prediction regression model to obtain a fatigue score corresponding to each time point in a future period of time; inferring a time required to reach different fatigue levels according to a current time and the fatigue score corresponding to each time point.

2. The driver fatigue prediction method according to claim 1, characterized by, The first face state feature comprises: eye closure event information, mouth opening event information, associated event information and first statistical information; the eye closure event information comprises a number of eye closure events and a time length of the eye closure events, and whether the eye closure event occurs is determined according to whether the ratio of the upper and lower eye distance to the left and right eye corner distance is lower than a preset eye closure threshold; the mouth opening event information comprises a number of mouth opening events and a time length of the mouth opening events, and whether the mouth opening event occurs is determined according to whether the ratio of the upper and lower mouth distance to the left and right mouth corner distance is higher than a preset mouth closure threshold; the associated event information comprises a proportional relationship between the number of eye closure events and the number of mouth opening events; the first statistical information comprises mean values, variances, maximum values and minimum values corresponding to the eye closure event information, the mouth opening event information and the associated event information.

3. The driver fatigue prediction method according to claim 2, characterized by, The second face state feature comprises: an eye closure score, a yawning score, a downward looking score and second statistical information, and the second statistical information comprises mean values, variances, maximum values and minimum values corresponding to the eye closure score, the yawning score and the downward looking score.

4. The driver fatigue prediction method according to claim 1, characterized by, The fatigue prediction regression model comprises a target function, and the target function is constructed according to a gradient boosting decision tree model and a loss function. The gradient boosting decision tree model comprises a regularization term corresponding to a regression tree.

5. The driver fatigue prediction method according to claim 4, characterized by, The training of the fatigue prediction regression model comprises the following steps: incrementally training the fatigue prediction regression model by using a boosting method, specifically comprising: When the tth round of training is performed, the contribution value obtained according to the input vector x i before the tth round, and the contribution value obtained according to the input vector x i after the tth round are calculated. the sum of the contribution values obtained from the input vector x i the sum of the contribution values obtained from the input vector x i the sum function between the contribution values obtained as a new objective function, until the training end condition is satisfied, wherein the sum of the contribution values obtained from the input vector x i h t (x i ) the contribution value obtained from the input vector x i t 6. A driver fatigue prediction device characterized by comprising: The method comprises the following steps: an acquisition module configured to acquire a monitoring video; A calculation module is configured to obtain a face state feature and a fatigue score of the driver according to the monitoring video; The obtaining of the face state feature and the fatigue score of the driver according to the monitoring video comprises: extracting face key point information corresponding to single-frame images from the monitoring video, wherein the face key point information comprises a ratio of an upper and lower eye distance and a left and right eye corner distance and a ratio of an upper and lower mouth distance and a left and right mouth corner distance; performing statistical analysis on face key point information corresponding to multiple frames of images to obtain a first face state feature; inputting the monitoring video into a video classification model to obtain a second face state feature and a fatigue score; and taking the first face state feature and the second face state feature as the face state feature of the driver A training module is configured to take the face state feature as an input vector and the fatigue score as a target result to train a fatigue prediction regression model; take time as an independent variable, and construct a first Gaussian distribution function based on a yawning score at each time point; construct a second Gaussian distribution function according to maximum values of Gaussian distribution functions corresponding to all time points; multiply the yawning score at each time point by the fatigue score at the corresponding time point according to the second Gaussian distribution function to obtain an updated fatigue score; and take the updated fatigue score as a target result to train the fatigue prediction regression model; A prediction module is configured to extract a face state feature of a to-be-detected driver from a to-be-detected image or video, input the face state feature of the to-be-detected driver into the trained fatigue prediction regression model, obtain a fatigue score corresponding to each time point in a future period of time, and infer a time required to reach different fatigue levels according to a current time and the fatigue score corresponding to each time point.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the driver fatigue prediction method of any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the driver fatigue prediction method of any one of claims 1 to 5 when executed by the processor.

Images