Method and system for predicting a fatigue state of a driver

By acquiring and analyzing the fatigue parameter data of the driver, using time series prediction models to predict future fatigue status, it solves the problem of accidents caused by fatigue in the prior art that cannot be predicted and avoided in advance, and realizes effective fatigue warning for construction machinery operators and Safety management.

CN119014871APending Publication Date: 2024-11-26CATERPILLAR INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310609430.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to predict the fatigue status of drivers, especially construction machinery operators, and it is impossible to take measures in advance to avoid accidents.

Method used

By obtaining the driver's fatigue parameter data sequence, generating model input data, using the trained time series fatigue prediction model to predict future fatigue states, providing fatigue warning function.

Benefits of technology

Accurate prediction of driver fatigue status is achieved, operators are warned in advance, work safety is improved, and dangerous events caused by fatigue are avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119014871A_ABST
    Figure CN119014871A_ABST
Patent Text Reader

Abstract

The invention relates to a method and a system for predicting the fatigue state of a driver, in particular of an operator of a construction machine, comprising the following steps: acquiring a data sequence of one or more fatigue parameters of the driver during at least one current time period; based on the obtained data sequence, determining optimal data of the one or more fatigue parameters when the driver is in the most fatigue state in each of the at least one time period, so as to generate model input data in the corresponding time period; predicting model prediction data for the one or more fatigue parameters within a future desired time period after the at least one time period using a trained time series fatigue prediction model based on the generated model input data; based on the model prediction data, a fatigue state of the driver over a future desired time period is determined. The system comprises a sensor device and a control device, in particular for carrying out the method according to the invention.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method and a system for predicting a fatigue state of a driver, in particular an operator of a construction machine. Background Art

[0002] Various types of construction machinery are widely used in mining operations today, such as excavators, electric shovels, loaders, etc. Operators of construction machinery almost always sit in a small, closed cab and perform a large number of repetitive operations, which can easily lead to fatigue when working for a long time.

[0003] Currently, most fatigue driving products on the market are used to monitor the fatigue level of motor vehicle drivers in real time. The principle is to capture the driver's facial features or behavioral movements through a camera, and trigger an alarm when improper driving behavior is identified.

[0004] However, this is more of a response action that is forced to be made when the driver is already in a fatigued state, rather than predicting in advance whether and when the driver will become fatigued and taking proactive measures. Therefore, existing solutions cannot completely prevent accidents from happening. Summary of the invention

[0005] Therefore, the object of the present invention is to provide an improved method and system for predicting the fatigue state of a driver, especially an operator of a construction machine, by which the driver fatigue warning function can be implemented.

[0006] The object is achieved by a method and a system having the following features.

[0007] According to one aspect of the present invention, a method for predicting the fatigue state of a driver, especially an operator of an engineering machinery, is proposed, the method comprising the following steps: acquiring a data series of one or more fatigue parameters of the driver in at least one current time period; determining, based on the acquired data series, the optimal data of the one or more fatigue parameters when the driver is in the most fatigued state in each time period of the at least one time period, so as to generate model input data in the corresponding time period; based on the generated model input data, using a trained time series fatigue prediction model to predict the model prediction data of the one or more fatigue parameters in a future expected time period after the at least one time period; and determining the fatigue state of the driver in the future expected time period based on the model prediction data.

[0008] According to another aspect of the present invention, a system for predicting the fatigue state of a driver, especially an operator of construction machinery, is proposed, wherein the system is particularly designed to execute the method described above, wherein the system comprises a sensor device and a control device, the sensor device being configured to acquire a data sequence of one or more fatigue parameters of the driver in at least one current time period, and the control device being configured to: determine, based on the acquired data sequence, the optimal data of the one or more fatigue parameters when the driver is in the most fatigued state in each time period of the at least one time period, so as to generate model input data in the corresponding time period; predict, based on the generated model input data, the model prediction data of the one or more fatigue parameters in a future expected time period after the at least one time period using a trained time series fatigue prediction model; and determine the fatigue state of the driver in the future expected time period based on the model prediction data.

[0009] The method and system of the present invention can improve the prediction function of the existing anti-fatigue driving system. This can well assist the operation process of the driver, especially the construction machinery operator, and can be reminded in time before fatigue driving or operation occurs, so as to adjust the working state and avoid the occurrence of danger. In addition, the method and system of the present invention can accurately predict the fatigue state of the driver at a certain time in the future, and the model calculation is more stable and reliable.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be described in detail below with reference to embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0012] Figure 1 An exemplary embodiment of a system for predicting a fatigue state of an operator of a construction machine according to the present invention is schematically shown.

[0013] Figure 2 A flow chart of an exemplary embodiment of a method for predicting a fatigue state of an operator of a construction machine according to the present invention is schematically shown. DETAILED DESCRIPTION

[0014] Figure 1An exemplary embodiment of a system 100 for predicting the fatigue state of an operator at a future time based on the current state of an operator of an engineering machine 1 (e.g., an excavator, an electric shovel, a loader, etc.) according to the present invention is schematically shown. The system 100 includes a sensor device 2 installed in the engineering machine 1. The sensor device 2 is configured to detect the value or data of one or more fatigue parameters of the operator. The detection may be performed continuously or intermittently, for example, once every several seconds or milliseconds. The fatigue parameter reflects the fatigue state of the operator. For example, in general, when gradually fatigued, the operator's eye opening may become smaller, the head drooping angle increases, the eye closing time becomes longer, and it may also be accompanied by a decrease in the pressure of the hand holding the operating lever, a decrease in the speed of the operating lever, and the like. For example, the fatigue parameter can be selected from a group including biological parameters or operating parameters of the operator. The biological parameters may include, for example, the eye opening of the operator, the eye blinking frequency, the head posture, the pupil diameter, the eyeball rotation speed, and the eye closing time. The operating parameters may include, for example, the pressure of the operator holding the operating lever and the speed of the operating lever.

[0015] In this embodiment, three fatigue parameters are used, namely, the eye opening of the operator, the pressure of the operating lever, and the speed of the operating lever. These three fatigue parameters can most intuitively reflect the fatigue degree of the operator. To this end, the sensor device 2 includes three sensors, namely, a camera 30, a pressure sensor 40, and a speed sensor 50. The camera 30 is configured to detect the eye opening data D of the operator when working. The camera 30 can be installed in the cab of the construction machinery 1 in a known manner, for example, aimed at the face of the operator. The pressure sensor 40 is configured to detect the pressure value P of the operator holding the operating lever of the construction machinery 1. The speed sensor 50 is configured to detect the speed value V of the operator pushing the operating lever. The pressure sensor 40 and the speed sensor 50 can be installed on the operating lever.

[0016] The system 100 also includes a control device 6. The sensor device 2 can communicate with the control device 6 to transmit the data D, P, V of the three fatigue parameters detected by the camera 30, the pressure sensor 40 and the speed sensor 50 to the control device 6. The control device 6 is configured to generate model input data Ds, Ps, Vs based on the received data D, P, V of the three fatigue parameters. The control device 6 is also configured to use a trained time series fatigue prediction model to predict the model prediction data Dd, Pp, Vv of the three fatigue parameters at a future expected time based on the generated model input data Ds, Ps, Vs. The control device 6 is also configured to determine the fatigue state of the operator at the future expected time based on the model prediction data Dd, Pp, Vv. The control device 6 can be set in the engineering machinery 1, for example, designed as a controller of the engineering machinery 1, or can be set outside the engineering machinery 1, for example, in a background that communicates with the engineering machinery 1.

[0017] As the name implies, a time series prediction model is an algorithmic model that predicts output data at another time step in the future based on an input data or a series of input data at a starting time step or multiple time steps. Therefore, a time series fatigue prediction model may include two types. In the first type, a time series fatigue prediction model can calculate model prediction data at a future expected time step from a model input data at a starting time step, wherein the output data of the fatigue parameters at the next time step (second time step) is calculated from the model input data of the fatigue parameters at the starting time step (first time step), and the output data of the fatigue parameters at the third time step is then calculated using the intermediate output data of the second time step as new input data. This iteration continues until the model prediction data of the fatigue parameters at the future expected time step is obtained.

[0018] Alternatively, in the second type, the time series fatigue prediction model can also calculate the model prediction data at the expected time step in the future by using the multiple model input data of the multiple time steps at the beginning. For example, taking three time steps as an example, the output data of the next time step (the fourth time step) is calculated by the three model input data in the first three time steps (the first time step, the second time step and the third time step), and then the model input data of the second and third time steps and the intermediate output data of the fourth time step are used as new input data to calculate the output data of the fifth time step. This rotation iteration continues until the model prediction data at the expected time step in the future is obtained.

[0019] Here, one model input data actually contains the data of all fatigue parameters at the same time step. Therefore, in the case of multiple fatigue parameters, one model input data is a data set.

[0020] Each time step of the time series fatigue prediction model is called a time cycle. The duration of a time cycle can be, for example, 1 second, 10 seconds, 30 seconds, 1 minute, 2 minutes, 3 minutes, 4 minutes or 5 minutes, etc., preferably 1 minute.

[0021] The time series fatigue prediction model may include any suitable time series prediction model, typically a TCN (time domain convolutional neural network) model or an LSTM (long short-term memory network) prediction model, etc.

[0022] Combine the following Figure 2 The method of the present invention is described in detail with reference to the embodiments.

[0023] Figure 2 Schematically showing the Figure 1 A flowchart of an embodiment of a method according to the present invention is provided, in which a system is used to predict the future fatigue state of an operator. Before the method of the present invention is started, as a preparatory step S0, a time series fatigue prediction model is first established and trained using historical data of fatigue parameters. In this embodiment, three fatigue parameters are used, namely: the eye opening of the operator detected by the camera 30, the operating lever pressure detected by the pressure sensor 40, and the operating lever speed detected by the speed sensor 50. Therefore, historical data (D, P, V) at the same time are selected from the historical data of the eye opening of the operator, the operating lever pressure, and the operating lever speed to form a data group, and the time series formed by such multiple data groups is used to train the time series fatigue prediction model.

[0024] After a trained time series fatigue prediction model is provided or already exists, in step S1, the data sequence of the three fatigue parameters of the operator's eye opening, operating lever pressure and operating lever speed in at least one current time period is obtained. In other words, the operator's current fatigue state is obtained to predict how long the operator may be fatigued from the current time. The operator will gradually feel tired as time passes during, for example, an 8-hour work process, but during this process, he may also temporarily cheer up at any time due to emergencies, such as being startled by a phone ring or an alarm clock. Therefore, fatigue in the future time period is always predicted based on the current state, so that the prediction result is more accurate.

[0025] Here, the current at least one time period should be understood as at least one time period from the current moment, that is, one time period or multiple time periods. For example, when the duration of each time period is 1 minute, the current at least one time period is one minute or several minutes or even more than ten minutes or tens of minutes from the current moment. The total number N (N = 1, 2, 3, ...) of the at least one time period for obtaining the data sequence depends on the type of the time series prediction model, more precisely, it is equal to the number or number of groups of model input data of the time series prediction model. In other words, if the number of groups of model input data of the time series prediction model is four, that is, based on four groups of model input data in four time periods (one group of model input data in each time period) to predict the model prediction data in another time period in the future, it is necessary to obtain the data sequence within four time periods from the current moment; if the number of groups of model input data of the time series prediction model is only one, that is, based on a group of model input data in one time period to predict the model prediction data in another time period in the future, it is only necessary to obtain the data sequence within one time period from the current moment. The data sequence means that, for each fatigue parameter, a corresponding value of the fatigue parameter is collected and determined at multiple time points in each time period, so that the value of the fatigue parameter detected in each time period forms a data sequence of the fatigue parameter.

[0026] To obtain these data sequences, the time sequence of the eye angle opening data D, the operating lever pressure value P, and the operating lever speed value V can be detected in real time in at least one current time period by the sensor device 2, such as the camera 30, the pressure sensor 40, and the speed sensor 50, respectively. Here, the acquisition moments in the time sequence of the eye angle opening data D, the operating lever pressure value P, and the operating lever speed value V may not be completely the same.

[0027] In step S2, the control device 6 determines the optimal data of the three fatigue parameters when the operator is in the most fatigued state in each of the at least one time period based on the acquired data sequence of the three fatigue parameters, so as to generate the model input data in the corresponding time period. Here, for each time period i (i=1, 2, ..., N), based on the data sequence of the first fatigue parameter, such as the data D of the eye angle opening degree acquired in the time period i, the intelligent optimization algorithm can be applied to calculate the optimal solution Ds of the eye angle opening degree in the time period i. i As the first optimal data. Here, the intelligent optimization algorithm may include any suitable local optimization algorithm, such as the whale algorithm, particle swarm optimization algorithm, artificial bee colony algorithm, ant colony algorithm, artificial fish swarm algorithm, firefly algorithm and gray wolf algorithm. Then, the first optimal data, i.e., the optimal solution Ds is selected. iThe corresponding data of other fatigue parameters at the same time, such as the operating lever pressure data Ps i and speed data Vs i As other optimal data. Thus, the corresponding model input data (Ds i ,Ps i ,Vs i ), which is composed of the first optimal data Ds in the corresponding time period i i and other optimal data Ps i , Vs i composition.

[0028] In step S3, the control device 6 generates at least one set of model input data (Ds i ,Ps i ,Vs i ), where i = 1, 2, ..., N, the trained time series fatigue prediction model is used to predict the model prediction data (Dd, Pp, Vv) at a future expected time period (e.g., the minute 1 hour after the current moment), the model prediction data including the prediction data Dd, Pp, Vv of the three fatigue parameters respectively. The future expected time period is after the at least one current time period in time.

[0029] In step S4, the control device 6 determines the fatigue state of the operator in the expected time period in the future based on the model prediction data (Dd, Pp, Vv). Here, the fatigue state of the operator in the expected time period in the future can be determined based on the comparison result of the first fatigue parameter, that is, the predicted data Dd of the eye angle opening and a predetermined threshold. For example, when the predicted data Dd is less than or equal to the threshold, it can be considered that the operator is in a fatigue state in the expected time period; conversely, if the predicted data Dd is greater than the threshold, it is considered that the operator is in a non-fatigue state in the expected time period. Here, the threshold is determined by the historical data of the eye angle opening.

[0030] In another embodiment not shown, the fatigue state of the operator in the expected time period in the future can also be determined based on the comparison results of the predicted data Dd, Pp, Vv of all three fatigue parameters with the predetermined reference ranges of the corresponding fatigue parameters. For example, when the predicted data Dd, Pp, Vv of the three fatigue parameters fall into their respective reference ranges [D1, D2], [P1, P2] and [V1, V2], that is, when D1≤Dd≤D2, P1≤Pp≤P2 and V1≤Vv≤V2 are satisfied at the same time, it is considered that the operator is in a fatigue state in the expected time period; on the contrary, if at least one of Dd, Pp, Vv falls outside the corresponding reference ranges [D1, D2], [P1, P2] and [V1, V2], it is considered that the operator is in a non-fatigue state in the expected time period. Here, the predetermined reference ranges [D1, D2], [P1, P2] and [V1, V2] of each fatigue parameter are determined by the historical data of the corresponding fatigue parameter.

[0031] Although three fatigue parameters are used as examples to illustrate the method and system of the present invention, another number of fatigue parameters is also feasible. For example, the number of fatigue parameters can be one, two or more than four. In the case of only one fatigue parameter, such as eye angle opening, the one fatigue parameter is the first fatigue parameter. Therefore, when determining the optimal data within the corresponding time period, based on the acquired data sequence of the first fatigue parameter, the optimal solution of the first fatigue parameter within the corresponding time period is calculated using an intelligent optimization algorithm, and the optimal solution is used as the model input data within the corresponding time period. After the model prediction data within the expected time period in the future is predicted by the model input data, it is also judged whether the operator is in a fatigue state based on the comparison of the first fatigue parameter with a predetermined threshold or reference range. The first fatigue parameter is preferably selected from biological parameters, such as eye angle opening, eye blinking frequency, head posture, pupil diameter, eyeball rotation speed and eye closure time.

[0032] The method and system of the present invention can also be used to predict fatigue driving of a motor vehicle driver traveling on the road. Here, fatigue parameters that best reflect the driver's fatigue state are selected, such as the driver's eye angle opening, eye blinking frequency, head posture, pupil diameter, eyeball rotation speed, eye closing time, driver's hand grip on the steering wheel, etc.

Claims

1. A method for predicting the fatigue state of a driver, in particular an operator of a construction machine, It is characterized in that The method comprises the following steps: Acquire a data sequence of one or more fatigue parameters of the driver in at least one current time period; Based on the acquired data sequence, determining optimal data of the one or more fatigue parameters when the driver is in a most fatigued state in each time period of the at least one time period, so as to generate model input data in the corresponding time period; Based on the generated model input data, using a trained time series fatigue prediction model to predict model prediction data of the one or more fatigue parameters in a future desired time period after the at least one time period; Based on the model prediction data, the fatigue state of the driver within the future expected time period is determined.

2. The method according to claim 1, It is characterized in that Determining the optimal data of the one or more fatigue parameters when the driver is in the most fatigued state in each time period comprises: based on the acquired data sequence of a first fatigue parameter among the one or more fatigue parameters, using an optimization algorithm to calculate the optimal solution of the first fatigue parameter in each time period as the first optimal data in the corresponding time period, In particular, in each time period, corresponding data of other fatigue parameters among the one or more fatigue parameters corresponding to the same time as the corresponding first optimal data are selected as other optimal data in the corresponding time period; The first optimal data in the corresponding time period is used as the model input data in the corresponding time period, or the first optimal data in the corresponding time period and other optimal data are used to form the model input data in the corresponding time period.

3. The method according to claim 2, It is characterized in that The optimization algorithm includes at least one of the following: whale algorithm, particle swarm optimization algorithm, artificial bee colony algorithm, ant colony algorithm, artificial fish swarm algorithm, firefly algorithm, and grey wolf algorithm.

4. The method according to claim 1, It is characterized in that Determining the fatigue state of the driver within the expected time period in the future based on the model prediction data includes: determining the fatigue state of the driver within the expected time period in the future based on a comparison result of the model prediction data of the first fatigue parameter among the one or more fatigue parameters with a predetermined threshold.

5. The method according to claim 4, It is characterized in that The predetermined threshold value is determined by historical data of the first fatigue parameter.

6. The method according to claim 1, It is characterized in that Determining the driver's fatigue state within the expected time period in the future based on the model prediction data includes: determining the driver's fatigue state within the expected time period in the future based on a comparison result of the model prediction data of the one or more fatigue parameters with a predetermined reference range of the corresponding fatigue parameters.

7. The method according to claim 6, It is characterized in that The predetermined reference ranges of the respective fatigue parameters are respectively determined by historical data of the corresponding fatigue parameters.

8. The method according to claim 1, It is characterized in that The number of the at least one time period is the same as the number of model input data.

9. The method according to claim 1, It is characterized in that Acquiring the data sequence of the one or more fatigue parameters in the at least one time period comprises: continuously or intermittently detecting sensor data of the one or more fatigue parameters in the at least one time period, in particular, detecting by a sensor device.

10. The method according to claim 9, It is characterized in that The sensor device includes a camera, a pressure sensor and a speed sensor.

11. The method according to claim 1, It is characterized in that The time series fatigue prediction model includes a TCN model or a LSTM model.

12. The method according to claim 1, It is characterized in that The time series fatigue prediction model is trained based on the time series of a data set consisting of historical data of various fatigue parameters at the same time or in the same time period.

13. The method according to claim 1, It is characterized in that The fatigue parameters are selected from a group including biological parameters and operational parameters of the driver, in particular, the biological parameters include the driver's eye opening, eye blinking frequency, head posture, pupil diameter, eye rotation speed and eye closure time, and the operational parameters include the pressure of the driver's hand holding the operating lever, the force applied by the driver to the steering wheel and the speed of the operating lever.

14. The method according to claim 13, It is characterized in that The one or more fatigue parameters include three fatigue parameters, wherein the first fatigue parameter is selected from the driver's biological parameters, especially the driver's eye angle opening, and the other two fatigue parameters are the pressure of the driver's hand holding the operating lever and the speed of the operating lever.

15. A system for predicting a fatigue state of a driver, in particular an operator of a construction machine, the system being in particular for carrying out the method according to any one of claims 1 to 14, in, The system comprises a sensor device and a control device, It is characterized in that The sensor device is configured to acquire a data sequence of one or more fatigue parameters of the driver within at least one current time period, The control device is configured to: determine the optimal data of the one or more fatigue parameters when the driver is in the most fatigued state in each time period of the at least one time period based on the acquired data sequence, so as to generate model input data in the corresponding time period; use a trained time series fatigue prediction model to predict the model prediction data of the one or more fatigue parameters in a future expected time period after the at least one time period based on the generated model input data; and determine the fatigue state of the driver in the future expected time period based on the model prediction data.