A method of monitoring a mine truck driver

Abstract

This invention discloses a method for monitoring mining truck drivers, comprising: Step 1, detecting the driver target; if the driver target is detected in the driver's seat, collecting vehicle speed, vehicle travel time, vehicle parking time, and the driver target's physical condition signals; Step 2, when v ≤ 10 km / h, determining that the vehicle has not started, no monitoring of the driver target is required; when v > 10 km / h, determining that the vehicle has started, and real-time monitoring of the driver target is performed: inputting the vehicle travel time and the driver target's physical condition signals into a CMAC model to obtain the driver target's fatigue level; when the fatigue level is level 1, issuing a voice reminder; when the fatigue level is level 2, issuing a voice reminder while simultaneously reporting to the regulatory department for real-time vehicle monitoring; when the fatigue level is level 3, if the driver still does not stop to rest after 5 voice reminders, the regulatory department will locate the vehicle's position and force the driver target to stop and rest for at least 30 minutes. This invention features improved monitoring accuracy.

Description

Technical Field

[0001] This invention relates to the field of driving safety technology, and more specifically, to a method for monitoring mining truck drivers. Background Technology

[0002] Open-pit mines are scattered throughout the country. The complex and diverse mining environments pose a great challenge to drivers' driving skills. In addition, the temperature in mining areas is extremely cold, with the lowest temperature reaching minus 42 degrees Celsius. Mining trucks are usually busy going back and forth between the loading and unloading areas in open-pit mines.

[0003] Mining truck drivers are forced to perform monotonous driving tasks while enduring physical and mental fatigue from bumpy roads. Technically, the speed and routes of mining trucks are generally fixed. In practice, mining operations typically run 24 hours a day, and drivers work in shifts, with overtime being commonplace. During long night shifts, the chances of drivers dozing off are extremely high. Due to the monotonous driving tasks, long overtime hours, bumpy roads, and poor lighting at night, drivers are prone to fatigue. This makes it difficult for drivers to react quickly and effectively in emergencies. Furthermore, because mining trucks are large, even minor mistakes by the driver can lead to major accidents. Crushing a normal-sized passenger car in the cab of a mining truck feels like running over a mound of dirt. There have been past accidents where mining trucks have veered off course, destroying houses and causing fatalities. Therefore, the loss of life and property caused by mining truck accidents is incalculable. Summary of the Invention

[0004] The purpose of this invention is to design and develop a method for monitoring mining truck drivers. By using a CMAC model to perform real-time fatigue detection on drivers, the method can accurately locate faces and precisely classify fatigue levels, thereby improving monitoring accuracy and reducing the occurrence of accidents.

[0005] The technical solution provided by this invention is as follows:

[0006] A method for monitoring mining truck drivers includes the following steps:

[0007] Step 1: Detect driver target. If driver target is detected in the driver's seat, collect vehicle speed, vehicle travel time, vehicle parking time and driver target's physical condition signals.

[0008] Step 2: When v ≤ 10km / h, it is determined that the vehicle has not started and there is no need to monitor the driver target.

[0009] When v > 10 km / h, the vehicle is considered to have started, and the driver's target is monitored in real time.

[0010] The vehicle driving time and the driver's physical condition signal are input into the CMAC model to obtain the driver's fatigue level;

[0011] The output of the CMAC model satisfies:

[0012]

[0013] In the formula, F(Z) ij The fatigue level of the driver is the target. Z is the network weight of the i-th input signal at time q. ij Let be the number of fused output signals of the i-th input signal in the j-th dimension. Let be the threshold of the i-th input signal at the s-th standard level, where s=1 indicates the driver target is in a normal state, s=2 indicates the driver target is in a first-level fatigue state, s=3 indicates the driver target is in a second-level fatigue state, and s=4 indicates the driver target is in a third-level fatigue state.

[0014] When the driver's fatigue level is Level 1, a voice reminder is given to the driver.

[0015] When the driver's fatigue level is level two, a voice reminder will be given to the driver while the regulatory authorities are notified to monitor the vehicle in real time.

[0016] If the driver's fatigue level is level three, and the driver still does not stop to rest after five voice reminders, the vehicle's location will be determined by the regulatory authorities, and the driver will be forced to stop and rest for at least 30 minutes.

[0017] Preferably, step one further includes:

[0018] After detecting the driver target, check whether the driver target is wearing a wristband and eye tracker. If v > 10km / h and the driver target is not wearing a wristband and eye tracker, issue a voice reminder to the driver target.

[0019] Preferably, the driver's physical condition signals include: eyelid closure frequency, pupil movement location, pupil size, heart rate signal, blood pressure signal, and body temperature signal.

[0020] Preferably, the number of fused output signals of the i-th input signal in the j-th dimension satisfies:

[0021]

[0022] In the formula, Y ij Let φ be the output signal of the i-th input signal in the j-th dimension. ij This is a cross signal.

[0023] Preferably, the cross signal satisfies:

[0024]

[0025] In the formula, Let be the variance of the output signal of the i-th input signal in the j-th dimension.

[0026] Preferably, the variance of the output signal of the i-th input signal in the j-th dimension satisfies:

[0027]

[0028] Preferably, the network weight update calculation formula for the i-th input signal is:

[0029]

[0030] In the formula, Let be the network weight of the i-th input signal at time q+1, μ be the learning rate, H be the update factor, and |X*| be the number of influencing factors contained in the set X* of signals that are simultaneously activated as 1 in a certain fatigue signal.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention presents a method for monitoring mining truck drivers. By combining a behavior monitoring device with an MTCNN model, the method accurately obtains the driver's facial features. Then, it uses a CMAC model to perform real-time fatigue detection on the driver, accurately locating the face and precisely classifying the fatigue level, thereby improving monitoring accuracy and reducing the occurrence of accidents. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the image pyramid structure described in this invention.

[0034] Figure 2 This is a schematic diagram of the P-Net network in the MTCNN model described in this invention.

[0035] Figure 3 This is a schematic diagram of the R-Net network in the MTCNN model described in this invention.

[0036] Figure 4 This is a schematic diagram of the O-Net network in the MTCNN model described in this invention. Detailed Implementation

[0037] The present invention will now be described in further detail so that those skilled in the art can implement it based on the description.

[0038] The present invention provides a method for monitoring mining truck drivers, comprising the following steps:

[0039] Step 1: Detect driver target. If driver target is detected in the driver's seat, collect vehicle speed, vehicle travel time, vehicle parking time and driver target's physical condition signals.

[0040] Specifically, a behavior monitoring device monitors the driver's seat in real time to determine if a driver is present and obtains the driver's real-time facial status. The monitoring device is positioned on the dashboard relative to the driver's seat. Since the distance between the monitoring device and the driver is not fixed, and faces vary in size, an image pyramid is constructed. Figure 1 As shown, the principle of the image pyramid is to scale a set of images according to a certain ratio, that is, to downsample the images in stages, with the resolution gradually decreasing until the termination condition is reached, so as to meet the compatibility of the model and detect faces of different sizes. In this embodiment, the downsampling scaling factor is 2. The parameters for each downsampling can be set. The smaller the parameter, the greater the density and the longer the detection time.

[0041] Since the driver cannot always be in the driver's seat, using the driver as the positioning target cannot guarantee the effect. Therefore, using the steering wheel as the positioning target keeps the positional relationship between the driver and the steering wheel relatively stable. Once the steering wheel can be positioned, the position of the driver can also be determined.

[0042] In this embodiment, the behavior monitoring device is a camera or a high-speed camera. If it is a camera, the video captured by the camera is composed of multiple frames of images. 30 frames are evenly extracted from each video at equal intervals, which is the optimal number for stable recognition. If it is a high-speed camera, the real-time scene in the driver's seat can be directly identified through the image.

[0043] In this embodiment, the driver's physical condition signals are monitored by an eye tracker and a wristband. The wristband monitors heart rate, blood pressure, and body temperature, while the eye tracker monitors eyelid closure frequency, pupil movement location, and pupil size. After detecting the driver, it checks whether the driver is wearing the wristband and eye tracker. If v > 10 km / h and the driver is not wearing the wristband and eye tracker, a voice reminder is given to the driver.

[0044] Using the steering wheel as the positioning target, center the steering wheel. The initial positioning box is defined within the width of the image. The height of the initial positioning box is expanded upwards to 4.8 times its original size, and the width of the positioning box is expanded to both sides to 1.2 times its original size. This is the complete positioning box of the driver target.

[0045] The complete bounding box image of the driver target is input into the MTCNN model, such as... Figure 2 , Figure 3 , Figure 4 As shown, the MTCNN model includes three cascaded CNN networks: P-Net, R-Net, and O-Net. The complete bounding box image of the driver target is input into the P-Net network, which filters out non-face objects and faces that are not the driver target. Keypoint localization and bounding box labeling are performed on the driver target's face. The output of P-Net is used as the input of R-Net. The convolutional neural network constructed by R-Net adds a fully connected layer compared to P-Net, thus making the data screening more rigorous and filtering out a large number of poor-performing candidate boxes. Then, non-maximum suppression-bounding box regression is used to filter the final candidate boxes. The output of R-Net is input into the O-Net network. O-Net is a more complex convolutional network that adds a convolutional layer compared to R-Net. Finally, the bounding box of the face and five feature points are output.

[0046] Before practical application, the MTCNN model is trained. In this embodiment, WIDER FACE is used to train the MTCNN model. P-Net preprocesses the driver's face image, locates the driver's face window in the image pyramid, and obtains candidate bounding boxes. Then, non-maximum suppression is used to merge the candidate bounding boxes to obtain the final candidate bounding boxes. The cross-entropy cost function is used as the face recognition loss function. The candidate bounding box output of P-Net is used as the input of R-Net to further process the face candidate bounding boxes detected by P-Net, removing candidate bounding boxes that do not contain faces, and using a box regression algorithm to correct tilted and offset face candidate bounding boxes. The loss function is L2 loss. The input of O-Net is the candidate bounding box output of R-Net. The face image is enlarged to generate the final face bounding box, and non-key points are discarded to obtain the most accurate facial key points. The loss function is L2 loss. The final total loss is equal to the sum of the individual losses multiplied by their respective weights, specifically:

[0047]

[0048] In the formula, j represents the task, and σ j Let the weight of the j-th task be... For sample type indicator;

[0049] The face recognition loss function satisfies:

[0050]

[0051] In the formula, For face recognition loss function, p is the label for the α-th driver target face sample. α To output the probability of the α-th driver's target face;

[0052] The face correction loss function satisfies:

[0053]

[0054] In the formula, For face correction loss function, To output the corrected bounding box after the α-th driver target face, The bounding box of the target face for the driver;

[0055] Facial landmark loss satisfies:

[0056]

[0057] In the formula, For loss of facial landmarks, To output the coordinates of each keypoint of the predicted driver's target face, The coordinates of the actual driver's facial landmarks.

[0058] In this embodiment, P-Net, R-Net, and O-Net σ1=1, σ2=0.5, σ3=0.5 in P-Net and R-Net, σ1=1, σ2=0.5, σ3=1 in O-Net.

[0059] After the MTCNN model is trained, it is evaluated by the ROC curve. When the coordinate trend on the ROC curve is close to (1,1), the model is closer to reality. The accuracy of the MTCNN model described in this invention reaches 99.20%.

[0060] After recognizing the face, the OpenPose model combined with the MobileNetV2 network is used to determine the key points of the driver's body. Then, the mobile phone is detected in the left and right ear areas of the driver. When the mobile phone is detected in the left and right ear areas of the driver, it is determined that the driver is making a phone call while driving.

[0061] Step 2: When the vehicle is stationary, it is reasonable for the driver target to take any action in the driver's seat. Moreover, during the period of stopping, the possibility of the driver target taking various other actions in the driver's seat is greatly increased, which will lead to a corresponding increase in the overall recognition rate and false alarm rate. Therefore, a speed threshold is set. When v≤10km / h, it is determined that the vehicle has not started and there is no need to monitor the driver target.

[0062] When v > 10 km / h, the vehicle is considered to have started, and the driver's target is monitored in real time.

[0063] Vehicle travel time, eyelid closure frequency, pupil movement location, pupil size, heart rate, blood pressure, and body temperature signals Xi = {x1, x2, x3, ..., xn} are input into the input module of the CMAC model. The input module outputs Yij = {y1j, y2j, y3j, ..., ynj}. Yij is then input into the fusion module. Since multiple signals are interleaved at different times, optimal segmentation is performed on the interleaved signals to avoid signal conflicts affecting monitoring accuracy, resulting in:

[0064]

[0065] In the formula, φ ij For cross signals, Let be the variance of the output signal of the i-th input signal in the j-th dimension;

[0066] The output of the fusion module is:

[0067]

[0068] In the formula, Z ij Y represents the number of fused output signals of the i-th input signal in the j-th dimension. ij Let i be the output signal of the i-th input signal in the j-th dimension.

[0069] The final output of the CMAC model is:

[0070]

[0071] In the formula, F(Z) ij The fatigue level of the driver is the target. Let be the network weight of the i-th input signal at time q. Let be the threshold of the i-th input signal at the s-th standard level, where s=1 indicates the driver target is in a normal state, s=2 indicates the driver target is in a first-level fatigue state, s=3 indicates the driver target is in a second-level fatigue state, and s=4 indicates the driver target is in a third-level fatigue state.

[0072] The formula for calculating the network weight update of the i-th input signal is as follows:

[0073]

[0074] In the formula, Let be the network weight of the i-th input signal at time q+1, μ be the learning rate, H be the update factor, and |X*| be the number of influencing factors contained in the set X* of signals that are simultaneously activated as 1 in a certain fatigue signal.

[0075] The thresholds for different standard levels are shown in Table 1:

[0076] Table 1. Thresholds for Multiple Signal Standard Levels for Driver Targets (Reference Range)

[0077]

[0078]

[0079] When the driver's fatigue level is normal, the driver is considered to be awake and no intervention is needed.

[0080] When the driver's fatigue level is Level 1, it is considered that the driver is slightly drowsy and needs to be reminded. Therefore, the central controller controls the vehicle broadcast to give the driver a voice reminder.

[0081] When the driver's fatigue level is level two, it is considered that the driver is drowsy. In this case, not only should the driver be given a voice reminder, but supervision should also be strengthened. Therefore, the central controller controls the vehicle broadcast to give the driver a voice reminder while reporting to the regulatory authorities to monitor the vehicle in real time.

[0082] When the driver's fatigue level is level three, it is considered that the driver is in a state of severe drowsiness. Not only is timely intervention required, but the driver must also be forced to rest. Therefore, if the driver still does not stop to rest after the central controller controls the vehicle broadcast to give the driver five voice reminders, the regulatory department will locate the vehicle and force the driver to stop and rest for at least 30 minutes.

[0083] Compared with the results of single signal monitoring, the mining truck driver monitoring method of the present invention integrates the judgment and output analysis model of multiple signals, and the accuracy of the obtained monitoring values ​​is much higher than that of single information monitoring. Furthermore, the average deviation between the monitoring values ​​obtained by the mining truck driver monitoring method of the present invention and the actual fatigue level trend of the driver does not exceed 0.1, while the average deviation between the results of single signal monitoring and the actual fatigue level trend of the driver reaches about 0.6. Therefore, the mining truck driver monitoring method of the present invention has higher accuracy.

[0084] This invention presents a method for monitoring mining truck drivers. By combining a behavior monitoring device with an MTCNN model, the method accurately obtains the driver's facial features. Then, it uses a CMAC model to perform real-time fatigue detection on the driver, accurately locating the face and precisely classifying the fatigue level, thereby improving monitoring accuracy and reducing the occurrence of accidents.

[0085] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and embodiments shown and described herein.

Claims

1. A method for monitoring mining truck drivers, characterized in that, Includes the following steps: Step 1: Detect driver target. If driver target is detected in the driver's seat, collect vehicle speed, vehicle travel time, vehicle parking time and driver target's physical condition signals. Step 2: When v ≤ 10km / h, it is determined that the vehicle has not started and there is no need to monitor the driver target. When v > 10 km / h, the vehicle is considered to have started, and the driver's target is monitored in real time. The vehicle driving time and the driver's physical condition signal are input into the CMAC model to obtain the driver's fatigue level; The output of the CMAC model satisfies: In the formula, F(Z) ij ξ represents the fatigue level of the driver. i q Z is the network weight of the i-th input signal at time q. ij Let be the number of fused output signals of the i-th input signal in the j-th dimension. Let be the threshold of the i-th input signal at the s-th standard level, where s=1 indicates the driver target is in a normal state, s=2 indicates the driver target is in a first-level fatigue state, s=3 indicates the driver target is in a second-level fatigue state, and s=4 indicates the driver target is in a third-level fatigue state. When the driver's fatigue level is Level 1, a voice reminder is given to the driver. When the driver's fatigue level is level two, a voice reminder will be given to the driver while the regulatory authorities are notified to monitor the vehicle in real time. If the driver's fatigue level is level three, and the driver still does not stop to rest after five voice reminders, the vehicle's location will be determined by the regulatory authorities, and the driver will be forced to stop and rest for at least 30 minutes.

2. The method for monitoring mining truck drivers as described in claim 1, characterized in that, Step one also includes: After detecting the driver target, check whether the driver target is wearing a wristband and eye tracker. If v > 10km / h and the driver target is not wearing a wristband and eye tracker, issue a voice reminder to the driver target.

3. The method for monitoring mining truck drivers as described in claim 2, characterized in that, The driver's physical condition signals include: eyelid closure frequency, pupil movement location, pupil size, heart rate signal, blood pressure signal, and body temperature signal.

4. The method for monitoring mining truck drivers as described in claim 3, characterized in that, The number of fused output signals of the i-th input signal in the j-th dimension satisfies: In the formula, Y ij Let φ be the output signal of the i-th input signal in the j-th dimension. ij This is a cross signal.

5. The method for monitoring mining truck drivers as described in claim 4, characterized in that, The cross signal satisfies: In the formula, Let be the variance of the output signal of the i-th input signal in the j-th dimension.

6. The method for monitoring mining truck drivers as described in claim 5, characterized in that, The variance of the output signal of the i-th input signal in the j-th dimension satisfies:

7. The method for monitoring mining truck drivers as described in claim 6, characterized in that, The formula for updating the network weights of the i-th input signal is as follows: In the formula, ξ i q+1 Here, μ represents the network weights for the i-th input signal at time q+1, μ is the learning rate, and H is the update factor. | X*| represents the number of influencing factors contained in the set of signals X* that are simultaneously activated as 1 in a certain fatigue signal.

Images