A driver fatigue monitoring method and system based on machine vision

Abstract

The present application relates to a kind of driver fatigue monitoring method based on machine vision, belong to electric digital data processing field, the method includes: after completing multilayer reconstruction, depth neural network is as intelligent state judgment model;Using intelligent state judgment model, according to the average speed corresponding to current time interval, the instant physiological information of setting driver and the custom screening data of multiple frame driving environment picture intelligent judgment setting driver fatigue grade in current time interval.The present application also relates to a kind of driver fatigue monitoring system based on machine vision.Through the present application, in view of the technical problem of weak driver fatigue monitoring data basis and not enough intelligent judgment mechanism in prior art, the intelligent state judgment model of custom structure design can be introduced to include speed, physiological information and mass video frame multiple depth mining visual information to execute the intelligent analysis of driver fatigue grade in time, thereby solve the above technical problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of electric digital data processing, and in particular to a driver fatigue monitoring method and system based on machine vision. BACKGROUND

[0002] With the rapid development of the mining industry and the continuous expansion of production scale, open-pit mine trucks have become one of the main tools for mine transportation. However, due to the long-time and high-intensity work of open-pit mine trucks, the drivers are prone to fatigue, leading to safety accidents, which has brought serious safety hazards and economic losses to mine operations. In the driving process of open-pit mine trucks, monitoring the fatigue state of the driver is of great significance for preventing open-pit mine accidents caused by fatigue driving. The fatigue state of the driver is affected by many factors, and the on-site analysis and judgment of the fatigue state of the driver are generally achieved by studying the relationship between various indicators and the fatigue level of the driver.

[0003] For example, the Chinese invention patent with publication number CN107977607A proposes a fatigue driving monitoring method based on machine vision. This method mainly uses video image processing technology to monitor the changes in the driver's eyes, yawning, and other conditions in real time, and comprehensively judges whether the driver is in a fatigue driving state. Compared with the detection method based on physiological signals, this method does not need to contact the driver's body and will not affect driving; compared with the detection method based on driving behavior, it has a lower misjudgment rate and more development potential. The method of the present application includes the following steps: face image preprocessing; obtaining a standard eye image; loading a classifier for feature classification and judging the opening and closing state of the standard eye image; and judging the fatigue driving state.

[0004] For example, the Chinese invention patent with publication number CN116935359A proposes a fatigue monitoring method based on machine vision, which includes collecting video images of the driver's face through the robot's own camera; cleaning the collected video images to obtain cleaned video images; segmenting the cleaned video images through the three-court five-eye rule to obtain eye video images, lip video images, and cheek video images; respectively performing real-time video image analysis through the eye video images, lip video images, and cheek video images; and prompting the driver when the recognition result is fatigue. The present application uses the robot's own video acquisition camera hardware and image processing technology to perform real-time video image analysis while performing real-time video acquisition, to determine whether the driver is in a fatigue driving state. This greatly improves the accuracy and reliability of the driver fatigue judgment, ensuring the safety of the driver's driving.

[0005] It can be seen that in the existing monitoring technology, the monitoring of the fatigue state of the driver is mostly based on a single index, or is limited to physiological parameters, or is limited to the analysis of limited frame images of the eyes, or is limited to the analysis of limited frame images of the eyes, lips and cheeks, without considering the intelligent judgment of the coordination of different types of multiple indexes, and without information mining of a large number of video pictures, resulting in a too weak data basis for the judgment of the fatigue state of the driver, and thus the judgment result is biased. SUMMARY

[0006] In order to solve the technical problems in the prior art, the present application provides a driver fatigue monitoring method and system based on machine vision, which introduces a plurality of different types of index data including the speed of the open-pit mine truck, the physiological information of the set driver and the eye visual information of the set driver to realize intelligent judgment of the fatigue level of the driver in a set time interval, and mines the eye information of the driver in a large number of video frames in the set time interval for intelligent judgment of the fatigue level, especially the imaging depth value and coordinate value of each constituent pixel point of the driver's eyes in each video frame, which ensures that the data basis for the judgment of the fatigue level is comprehensive and sufficient; on this basis, an intelligent state judgment model with customized structure design and a fatigue level time-sharing wireless reporting mechanism for open-pit mines are introduced, thereby ensuring the real-time and reliability of the fatigue judgment result of the open-pit mine truck driver, and effectively improving the work safety and health of the open-pit mine truck driver, reducing the risk of accidents, reducing production costs and economic losses, and improving the management efficiency and competitiveness of the mine.

[0007] According to a first aspect of the present application, a driver fatigue monitoring method based on machine vision is provided, the method comprising:

[0008] obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver driving the open-pit mine truck in the current time interval as the instant physiological information of the set driver;

[0009] photographing the set driver at a preset photographing frame rate to obtain a plurality of driving environment pictures in the current time interval, the plurality of driving environment pictures corresponding to a plurality of monitoring time points which are uniformly distributed on the time axis;

[0010] identifying each eye constituent pixel point in each driving environment picture based on the eye imaging characteristics of the set driver, and outputting each imaging depth value, each horizontal coordinate value and each vertical coordinate value corresponding to each eye constituent pixel point in each driving environment picture as the single customized screening data corresponding to the driving environment picture.

[0011] performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after the multi-layer reconstruction is completed as the intelligent state judgment model, wherein the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate;

[0012] The intelligent state judgment model is used to intelligently judge the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures.

[0013] According to a second aspect of the present application, a driver fatigue monitoring system based on machine vision is provided, which comprises a memory and one or more processors, the memory storing a computer program configured to be executed by the one or more processors to complete the following steps:

[0014] The average heart rate data, average hypertension value, average hypotension value, and average blood oxygen saturation of the set driver driving the open-pit mine truck in the current time interval are obtained as the instant physiological information of the set driver.

[0015] A preset shooting frame rate is used to shoot the set driver to obtain multiple frames of driving environment pictures in the current time interval, and the multiple monitoring time points corresponding to the multiple frames of driving environment pictures are uniformly distributed on a time axis.

[0016] Based on the eye imaging features of the set driver, each eye constituent pixel point in each frame of driving environment picture is identified, and each imaging depth value, horizontal coordinate value, and vertical coordinate value corresponding to each eye constituent pixel point in each frame of driving environment picture is output as the customized screening data corresponding to the frame of driving environment picture.

[0017] performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after the multi-layer reconstruction is completed as the intelligent state judgment model, wherein the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate;

[0018] The intelligent state judgment model is used to intelligently judge the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures.

[0019] According to a third aspect of the present application, there is provided a driver fatigue monitoring system based on machine vision, the system comprising:

[0020] a field sensor for obtaining average heart rate data, average high blood pressure value, average low blood pressure value and average blood oxygen saturation of a designated driver driving an open-pit mine truck in a current time interval as real-time physiological information of the designated driver;

[0021] a visual monitor for shooting the designated driver at a preset shooting frame rate to obtain a plurality of driving environment pictures in the current time interval, the plurality of driving environment pictures corresponding to a plurality of monitoring time points which are uniformly distributed on a time axis;

[0022] a content analysis device connected to the visual monitor, for identifying each eye constituent pixel point in each driving environment picture based on eye imaging features of the designated driver, and outputting each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye constituent pixel point in each driving environment picture as single customized screening data corresponding to the driving environment picture;

[0023] a multi-layer reconstruction device for performing a plurality of learning operations on a deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after the multi-layer reconstruction as an intelligent state judgment model, the number of reconstructed layers of the deep neural network being positively correlated with the preset shooting frame rate;

[0024] an intelligent judgment device connected to the field sensor, the content analysis device and the multi-layer reconstruction device, for intelligently judging the fatigue level of the designated driver in the current time interval based on the duration of the current time interval, the average speed of the open-pit mine truck in the current time interval, the real-time physiological information of the designated driver and the plurality of customized screening data corresponding to the plurality of driving environment pictures using the intelligent state judgment model.

[0025] Compared with the prior art, the present application has at least the following outstanding substantial features:

[0026] Firstly, in order to intelligently judge the fatigue level of the designated driver driving the open-pit mine truck in the current time interval, the imaging depth values and coordinate values of each constituent pixel point of the eyes of the designated driver in the plurality of continuous pictures are selected as sufficient and comprehensive basic information for intelligent judgment, thereby ensuring the stability and reliability of the intelligent judgment result;

[0027] Secondly, an intelligent state judgment model with a customized structure is designed for intelligently judging the fatigue level of the driver in the current time interval, the intelligent state judgment model is a deep neural network after multi-layer reconstruction, the number of reconstruction layers completed by the deep neural network is positively correlated with the shooting frame rate of the imaging device of the driver, so as to further ensure the stability and reliability of the intelligent judgment result.

[0028] Thirdly, in each learning operation performed on the deep neural network, the known fatigue level of the driver in a certain historical time interval is taken as the single output content of the deep neural network, the duration length of the certain historical time interval, the average speed of the open-pit mine truck in the certain historical time interval, each item of the instant physiological information of the driver corresponding to the certain historical time interval, and the multiple customized screening data corresponding to the multiple frames of driving environment pictures are taken as the multiple input contents of the deep neural network, and the learning operation performed on the deep neural network is completed, so as to ensure the learning effect of each learning operation performed on the deep neural network.

[0029] Finally, the intelligent state judgment model with a customized structure intelligently judges the fatigue level of the driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of the instant physiological information of the driver, and the multiple customized screening data corresponding to the multiple frames of driving environment pictures, the fatigue level of the driver in the current time interval is one of five fatigue levels: clear, mild fatigue, moderate fatigue, deep fatigue, and sleep, and the fatigue level of the driver in the current time interval intelligently judged is transmitted to the vehicle fleet monitoring server of the nearest open-pit mine through a wireless communication link, so as to realize intelligent analysis and real-time wireless reporting of the fatigue state of each driver of the open-pit mine truck fleet, and effectively reduce the probability of vehicle accidents caused by excessive fatigue. BRIEF DESCRIPTION OF DRAWINGS

[0030] The embodiments of the present application will be described below with reference to the accompanying drawings, in which:

[0031] Figure 1 The technical flowchart of the driver fatigue monitoring method and system based on machine vision according to the present application.

[0032] Figure 2 The step flowchart of the driver fatigue monitoring method based on machine vision according to embodiment 1 of the present application.

[0033] Figure 3 The step flowchart of the driver fatigue monitoring method based on machine vision according to embodiment 2 of the present application.

[0034] Figure 4 A step flow chart of a machine vision-based driver fatigue monitoring method according to Embodiment 3 of the present application is shown.

[0035] Figure 5 A step flow chart of a machine vision-based driver fatigue monitoring method according to Embodiment 4 of the present application is shown.

[0036] Figure 6 A structural schematic diagram of a machine vision-based driver fatigue monitoring system according to Embodiment 5 of the present application is shown.

[0037] Figure 7 A structural schematic diagram of a machine vision-based driver fatigue monitoring system according to Embodiment 6 of the present application is shown. DETAILED DESCRIPTION

[0038] The open-pit mine truck driver fatigue monitoring system development project has significant social significance. On the one hand, through the fatigue monitoring system, the accident rate can be reduced, and the economic loss caused by the accident can be reduced. According to statistics, a serious mine traffic accident can cause hundreds of thousands of yuan of loss, and the low investment of the fatigue monitoring system can significantly reduce the accident risk. On the other hand, the fatigue monitoring system can improve the production efficiency. Through real-time monitoring of the fatigue state of the driver, the enterprise can reasonably arrange the work and rest time, and avoid the reduction of work efficiency caused by fatigue driving.

[0039] In addition, through the fatigue monitoring system, the enterprise can prove that effective measures have been taken to prevent fatigue driving, thereby reducing the legal risk. Therefore, the market prospect of the research and development of the open-pit mine truck driver fatigue monitoring system is broad, and the economic benefit is significant. With the increasing attention of society to traffic safety and production efficiency, the open-pit mine truck driver fatigue monitoring system will receive more and more attention and demand.

[0040] As shown in Figure 1 , a technical flow chart of a machine vision-based driver fatigue monitoring method and system according to the present application is given, which is used for open-pit mine truck driver fatigue monitoring.

[0041] As shown in Figure 1 , the specific technical process of the present application is as follows:

[0042] Technical process A: intelligent judgment of the fatigue level of the driver driving the open-pit mine truck in the current time interval, targeted screening of various basic information, the various basic information involves different types of indicators, and deep information mining is performed on the mass video pictures;

[0043] Specifically, the different types of indicators involved include the truck speed of the open-pit mine, the physiological information of the setting driver, and the visual information of the setting driver's eyes in multiple consecutive frames of pictures;

[0044] Further specifically, the number of multiple consecutive frames is large, and the depth mining of the imaging depth value, horizontal coordinate value, and vertical coordinate value of each constituent pixel point of the setting driver's eyes is performed on each frame of picture;

[0045] In this way, the sufficient and comprehensive mining of the various basic data for intelligent judgment of the fatigue level ensures the stability and reliability of the intelligent judgment result;

[0046] Technical process B: intelligent judgment of the fatigue level of the setting driver driving the open-pit mine truck in the current time interval, involves an artificial intelligence model with customized structure, i.e., an intelligent state judgment model;

[0047] For example, the structure customization of the intelligent state judgment model mainly manifests in the following three aspects:

[0048] First: the intelligent state judgment model is a deep neural network after multi-layer reconstruction;

[0049] Second: the number of reconstruction layers completed by the deep neural network is positively correlated with the shooting frame rate of the imaging device of the setting driver;

[0050] Third: in each learning operation performed on the deep neural network, the known fatigue level of the setting driver in a certain historical time interval is taken as the single output content of the deep neural network, and the duration length of the certain historical time interval, the average speed of the open-pit mine truck in the certain historical time interval, the various instant physiological information of the setting driver corresponding to the certain historical time interval, and multiple customized screening data corresponding to multiple frames of driving environment pictures in the certain historical time interval are taken as multiple input contents of the deep neural network, thereby completing the current learning operation performed on the deep neural network, and ensuring the learning effect of each learning operation performed on the deep neural network;

[0051] Therefore, the customized structure design at each place further ensures the stability and reliability of the intelligent judgment result;

[0052] Technical process C: the intelligent state judgment model with customized structure design intelligently judges the fatigue level of the setting driver in the current time interval according to the sufficient and comprehensive mining of various basic data;

[0053] For example, the fatigue level of the setting driver in the current time interval is one of the five fatigue levels of wakefulness, mild fatigue, moderate fatigue, deep fatigue, and sleep;

[0054] Technical procedure D: transmitting the fatigue level of the set driver in the current time interval determined intelligently to the nearest open-pit mine truck fleet monitoring server through a wireless communication link;

[0055] In this way, the intelligent analysis and real-time wireless reporting of the fatigue state of each driver of the open-pit mine truck fleet is realized, and the probability of vehicle accidents caused by excessive fatigue is effectively reduced.

[0056] The key points of the present application are: the cooperative use of different types of indicators including the speed of the open-pit mine truck, the physiological information of the set driver, and the visual information of the set driver's eyes in multiple consecutive frames, the depth information mining of the massive video picture including the imaging depth value, the horizontal coordinate value and the vertical coordinate value of each constituent pixel point of the set driver's eyes, the multi-custom structure design of the intelligent state judgment model, and the real-time wireless uploading of the intelligent judgment result.

[0057] In the following, the driver fatigue monitoring method and system based on machine vision according to the present application will be described in the form of embodiments.

[0058] Embodiment 1

[0059] Figure 2 The step flow chart of the driver fatigue monitoring method based on machine vision according to Embodiment 1 of the present application is shown.

[0060] As shown in Figure 2 , the driver fatigue monitoring method based on machine vision includes the following specific steps:

[0061] Step S2001: obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver driving the open-pit mine truck in the current time interval as the instant physiological information of the set driver;

[0062] For example, obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver driving the open-pit mine truck in the current time interval as the instant physiological information of the set driver includes: using multiple different physiological sensors to respectively obtain the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver driving the open-pit mine truck in the current time interval;

[0063] Step S2002: taking pictures of the set driver at a preset shooting frame rate to obtain multiple frames of driving environment pictures in the current time interval, and the multiple monitoring time points corresponding to the multiple frames of driving environment pictures are evenly distributed on the time axis;

[0064] Specifically, the preset shooting frame rate is 60 frames per second.

[0065] Step S2003: Based on the eye imaging features of the setting driver, identify each eye constituent pixel point in each frame of driving environment picture, and output each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye constituent pixel point in each frame of driving environment picture as the single customized screening data corresponding to the frame of driving environment picture.

[0066] For example, based on the eye imaging features of the setting driver, identify each eye constituent pixel point in each frame of driving environment picture, and output each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye constituent pixel point in each frame of driving environment picture as the single customized screening data corresponding to the frame of driving environment picture, including: the total number of each eye constituent pixel point is less than the total number of pixel points of the driving environment picture.

[0067] Step S2004: Perform multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and output the deep neural network after completing the multi-layer reconstruction as an intelligent state judgment model, and the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate.

[0068] Specifically, perform multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and output the deep neural network after completing the multi-layer reconstruction as an intelligent state judgment model, and the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate, including: when the preset shooting frame rate is 60 frames per second, the number of reconstruction layers completed by the deep neural network is 300 layers, when the preset shooting frame rate is 50 frames per second, the number of reconstruction layers completed by the deep neural network is 200 layers, when the preset shooting frame rate is 30 frames per second, the number of reconstruction layers completed by the deep neural network is 150 layers, and so on.

[0069] Step S2005: Using the intelligent state judgment model, intelligently judge the fatigue level of the setting driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the setting driver and the multiple customized screening data corresponding to the multiple frames of driving environment picture.

[0070] For example, the test and simulation of the data processing process of intelligently judging the fatigue level of the set driver in the current time interval by the intelligent state judgment model according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures are completed by using the MATLAB toolbox.

[0071] In the method, intelligently judging the fatigue level of the set driver in the current time interval by the intelligent state judgment model according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures includes that the fatigue level of the set driver in the current time interval is one of five fatigue levels of wakefulness, mild fatigue, moderate fatigue, deep fatigue, and sleep.

[0072] In the method, intelligently judging the fatigue level of the set driver in the current time interval by the intelligent state judgment model according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures further includes that different binary values are used to identify the five fatigue levels of wakefulness, mild fatigue, moderate fatigue, deep fatigue, and sleep.

[0073] In the method, the multiple frames of driving environment pictures are obtained by photographing the set driver at a preset photographing frame rate, and the multiple pieces of monitoring time corresponding to the multiple frames of driving environment pictures are uniformly distributed on a time axis, and the resolution of the multiple frames of driving environment pictures is the same.

[0074] In the method, each eye-constituting pixel point in each frame of driving environment picture is identified based on the eye imaging feature of the set driver, and each imaging depth value, each horizontal coordinate value, and each vertical coordinate value corresponding to each eye-constituting pixel point in each frame of driving environment picture are output as a piece of customized screening data corresponding to the frame of driving environment picture, and the eye imaging feature of the set driver is a brightness value distribution range corresponding to the eyes of the set driver, and a pixel point in the frame of driving environment picture whose brightness value is within the brightness value distribution range corresponding to the eyes of the set driver is regarded as a single eye-constituting pixel point in the frame of driving environment picture.

[0075] For example, the pixels whose brightness values ​​in the driving environment frame are within the range of brightness values ​​corresponding to the set driver's eyes are used as individual eye-forming pixels in the driving environment frame, including: the brightness value of each pixel in the driving environment frame is between 0 and 255.

[0076] The process of performing multiple learning operations on the deep neural network to complete the multi-layer reconstruction of the deep neural network, and using the deep neural network after multi-layer reconstruction as the output of the intelligent state judgment model, with the number of reconstruction layers completed by the deep neural network being positively correlated with the preset shooting frame rate, also includes: in each learning operation performed on the deep neural network, using the known fatigue level of the set driver within a certain historical time interval as a single output of the deep neural network, and using the duration of the certain historical time interval, the average speed of the open-pit mine truck within the certain historical time interval, various real-time physiological information of the set driver corresponding to the certain historical time interval, and multiple sets of customized filtered data corresponding to the multiple frames of driving environment images corresponding to the certain historical time interval as multiple inputs of the deep neural network, thus completing the current learning operation performed on the deep neural network;

[0077] In addition, multiple learning operations are performed on the deep neural network to complete the multi-layer reconstruction of the deep neural network, and the deep neural network after multi-layer reconstruction is used as the output of the intelligent state judgment model. The positive correlation between the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate includes: using a content mapping function to represent the content mapping relationship between the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate.

[0078] Example 2

[0079] Figure 3 The following is a flowchart illustrating the steps of a machine vision-based driver fatigue monitoring method according to Embodiment 2 of the present invention.

[0080] like Figure 3 As shown, with Figure 2 Unlike the previous embodiment, before capturing multiple frames of the driving environment within the current time interval using a preset frame rate, and before the multiple monitoring times corresponding to each frame of the driving environment are evenly distributed on the time axis, i.e. before step S2002, the method further includes:

[0081] Step S3001: The user control interface is used to set the parameters of the preset shooting frame rate according to the input operation of the driver.

[0082] Specifically, the parameter setting of the preset shooting frame rate is achieved according to the input operation of the setting driver by using the user control interface, and the user control interface is implemented by using a programmable logic device.

[0083] Embodiment 3

[0084] Figure 4 A step flow chart of the machine vision-based driver fatigue monitoring method according to Embodiment 3 of the present application is shown.

[0085] As Figure 4 shown, unlike the embodiment in Figure 2 , after the fatigue level of the setting driver in the current time interval is intelligently judged by using the intelligent state judgment model according to the duration length of the current time interval, the average vehicle speed of the open-pit mine truck in the current time interval, each item of the instant physiological information of the setting driver, and multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures, that is, after step S2005, the method further includes:

[0086] Step S4001: transmitting the fatigue level of the setting driver in the current time interval intelligently judged to the vehicle team monitoring server of the nearest open-pit mine through a wireless communication link;

[0087] For example, transmitting the fatigue level of the setting driver in the current time interval intelligently judged to the vehicle team monitoring server of the nearest open-pit mine through a wireless communication link includes that the wireless communication link is a time division duplex communication link or a frequency division duplex communication link.

[0088] Embodiment 4

[0089] Figure 5 A step flow chart of the machine vision-based driver fatigue monitoring method according to Embodiment 4 of the present application is shown.

[0090] As Figure 5 shown, unlike the embodiment in Figure 2 , before the fatigue level of the setting driver in the current time interval is intelligently judged by using the intelligent state judgment model according to the duration length of the current time interval, the average vehicle speed of the open-pit mine truck in the current time interval, each item of the instant physiological information of the setting driver, and multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures, that is, before step S2005, the method further includes:

[0091] Step S5001: obtaining, by using a vehicle speed sensor, a plurality of vehicle speeds respectively corresponding to a plurality of time points in the current time interval, removing the maximum and minimum values of the plurality of vehicle speeds to obtain a plurality of remaining vehicle speeds, and performing mean value calculation on the plurality of remaining vehicle speeds to obtain the average vehicle speed of the open-pit mine truck in the current time interval;

[0092] Specifically, obtaining, by using a vehicle speed sensor, a plurality of vehicle speeds respectively corresponding to a plurality of time points in the current time interval, removing the maximum and minimum values of the plurality of vehicle speeds to obtain a plurality of remaining vehicle speeds, and performing mean value calculation on the plurality of remaining vehicle speeds to obtain the average vehicle speed of the open-pit mine truck in the current time interval includes that the vehicle speed sensor is connected with a rotating axle of the open-pit mine truck.

[0093] Next, the various method embodiments of the present application will be described in detail.

[0094] In the driver fatigue monitoring method based on machine vision according to the various method embodiments of the present application:

[0095] Identifying each eye constituent pixel point in each frame of driving environment picture based on the set eye imaging feature of the driver, and outputting each imaging depth value, each horizontal coordinate value and each vertical coordinate value corresponding to each eye constituent pixel point in each frame of driving environment picture as single customized screening data corresponding to the frame of driving environment picture further includes: in the frame of driving environment picture, taking a pixel point at the lower left corner of the frame of driving environment picture as the origin of a two-dimensional coordinate system, taking a pixel column at the leftmost side of the frame of driving environment picture as the vertical coordinate axis of the two-dimensional coordinate system in the positive direction, and taking a pixel row at the bottom of the frame of driving environment picture as the horizontal coordinate axis of the two-dimensional coordinate system in the positive direction, to establish the two-dimensional coordinate system of the frame of driving environment picture.

[0096] Specifically, in the frame of driving environment picture, taking a pixel point at the lower left corner of the frame of driving environment picture as the origin of a two-dimensional coordinate system, taking a pixel column at the leftmost side of the frame of driving environment picture as the vertical coordinate axis of the two-dimensional coordinate system in the positive direction, and taking a pixel row at the bottom of the frame of driving environment picture as the horizontal coordinate axis of the two-dimensional coordinate system in the positive direction, to establish the two-dimensional coordinate system of the frame of driving environment picture includes that the horizontal coordinate value and the vertical coordinate value of each pixel point in the frame of driving environment picture are both positive values.

[0097] And in the driver fatigue monitoring method based on machine vision according to the various method embodiments of the present application:

[0098] The method comprises the following steps: obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the setting driver driving the open-pit mine truck in the current time interval as the instant physiological information of the setting driver, and outputting the instant physiological information of the setting driver; uniformly dividing the current time interval to obtain the heart rate data corresponding to each time point; removing the maximum and minimum values of the heart rate data to obtain the remaining heart rate data; and performing the mean value calculation on the remaining heart rate data to obtain the average heart rate data of the setting driver driving the open-pit mine truck in the current time interval.

[0099] Specifically, the test and simulation of the data processing process of removing the maximum and minimum values of the heart rate data to obtain the remaining heart rate data, and performing the mean value calculation on the remaining heart rate data to obtain the average heart rate data of the setting driver driving the open-pit mine truck in the current time interval can be selected to be completed by using the numerical simulation mode.

[0100] The method further comprises the following steps: uniformly dividing the current time interval to obtain the high blood pressure value corresponding to each time point; removing the maximum and minimum values of the high blood pressure value to obtain the remaining high blood pressure value; and performing the mean value calculation on the remaining high blood pressure value to obtain the average high blood pressure value of the setting driver driving the open-pit mine truck in the current time interval.

[0101] The method further comprises the following steps: uniformly dividing the current time interval to obtain the low blood pressure value corresponding to each time point; removing the maximum and minimum values of the low blood pressure value to obtain the remaining low blood pressure value; and performing the mean value calculation on the remaining low blood pressure value to obtain the average low blood pressure value of the setting driver driving the open-pit mine truck in the current time interval.

[0102] and wherein the obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the designated driver driving the open-pit mine truck in the current time interval as the instant physiological information of the designated driver comprises: uniformly dividing the current time interval to obtain a plurality of blood oxygen saturations corresponding to respective time points; performing data processing on the plurality of blood oxygen saturations by removing maximum and minimum values to obtain a plurality of remaining blood oxygen saturations; and performing mean value calculation on the plurality of remaining blood oxygen saturations to obtain the average blood oxygen saturation of the designated driver driving the open-pit mine truck in the current time interval.

[0103] Embodiment 5

[0104] Figure 6 A structural schematic diagram of a machine vision-based driver fatigue monitoring system according to Embodiment 5 of the present application.

[0105] As shown in Figure 6 the machine vision-based driver fatigue monitoring system comprises a memory and one or more processors, the memory storing a computer program configured to be executed by the one or more processors to complete the following steps:

[0106] Step S2001: obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the designated driver driving the open-pit mine truck in the current time interval as the instant physiological information of the designated driver;

[0107] For example, the obtaining the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the designated driver driving the open-pit mine truck in the current time interval as the instant physiological information of the designated driver comprises: using a plurality of different physiological sensors to respectively obtain the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the designated driver driving the open-pit mine truck in the current time interval;

[0108] Step S2002: photographing the designated driver at a preset photographing frame rate to obtain a plurality of driving environment pictures in the current time interval, the plurality of driving environment pictures respectively corresponding to a plurality of monitoring time points uniformly distributed on a time axis;

[0109] Specifically, the photographing the designated driver at a preset photographing frame rate to obtain a plurality of driving environment pictures in the current time interval, the plurality of driving environment pictures respectively corresponding to a plurality of monitoring time points uniformly distributed on a time axis comprises: the preset photographing frame rate is 60 frames per second;

[0110] Step S2003: identifying each eye-constituted pixel point in each frame of driving environment picture based on the set eye imaging features of the driver, and outputting each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye-constituted pixel point in each frame of driving environment picture as single customized screening data corresponding to the frame of driving environment picture;

[0111] For example, identifying each eye-constituted pixel point in each frame of driving environment picture based on the set eye imaging features of the driver, and outputting each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye-constituted pixel point in each frame of driving environment picture as single customized screening data corresponding to the frame of driving environment picture includes that the total number of each eye-constituted pixel point is less than the total number of pixel points of the driving environment picture.

[0112] Step S2004: performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after completing the multi-layer reconstruction as the intelligent state judgment model, wherein the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate.

[0113] Specifically, performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after completing the multi-layer reconstruction as the intelligent state judgment model, wherein the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate includes that when the preset shooting frame rate is 60 frames per second, the number of reconstruction layers completed by the deep neural network is 300 layers, when the preset shooting frame rate is 50 frames per second, the number of reconstruction layers completed by the deep neural network is 200 layers, when the preset shooting frame rate is 30 frames per second, the number of reconstruction layers completed by the deep neural network is 150 layers, and so on.

[0114] Step S2005: intelligently judging the fatigue level of the set driver in the current time interval by the intelligent state judgment model according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the set driver and the multiple customized screening data corresponding to the multiple frames of driving environment pictures.

[0115] For example, the test and simulation of the data processing process of intelligently judging the fatigue level of the set driver in the current time interval by the intelligent state judgment model according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the set driver and the multiple customized screening data corresponding to the multiple frames of driving environment pictures is completed by using the MATLAB toolbox.

[0116] The fatigue level of the set driver in the current time interval is one of five fatigue levels of clear, mild fatigue, moderate fatigue, deep fatigue and sleep.

[0117] For example, the intelligent state judgment model is used to intelligently judge the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the set driver, and the multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures respectively. It also includes: using different binary values to identify the five fatigue levels of clear, mild fatigue, moderate fatigue, deep fatigue and sleep.

[0118] The multiple frames of driving environment pictures are obtained by photographing the set driver at a preset photographing frame rate, and the multiple monitoring time points corresponding to the multiple frames of driving environment pictures are uniformly distributed on the time axis. The resolution of the multiple frames of driving environment pictures is the same.

[0119] Based on the eye imaging characteristics of the set driver, each eye constituent pixel point in each frame of driving environment picture is identified, and each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye constituent pixel point in each frame of driving environment picture is output as the single piece of customized screening data corresponding to the frame of driving environment picture. The eye imaging characteristics of the set driver are the brightness value distribution range corresponding to the eyes of the set driver, and the pixel points in the frame of driving environment picture whose brightness values are within the brightness value distribution range corresponding to the eyes of the set driver are taken as single eye constituent pixel points in the frame of driving environment picture.

[0120] For example, the pixel points in the frame of driving environment picture whose brightness values are within the brightness value distribution range corresponding to the eyes of the set driver are taken as single eye constituent pixel points in the frame of driving environment picture. The brightness value of each pixel point in the frame of driving environment picture is between 0 and 255.

[0121] The process of performing multiple learning operations on the deep neural network to complete the multi-layer reconstruction of the deep neural network, and using the deep neural network after multi-layer reconstruction as the output of the intelligent state judgment model, with the number of reconstruction layers completed by the deep neural network being positively correlated with the preset shooting frame rate, also includes: in each learning operation performed on the deep neural network, using the known fatigue level of the set driver within a certain historical time interval as a single output of the deep neural network, and using the duration of the certain historical time interval, the average speed of the open-pit mine truck within the certain historical time interval, various real-time physiological information of the set driver corresponding to the certain historical time interval, and multiple sets of customized filtered data corresponding to the multiple frames of driving environment images corresponding to the certain historical time interval as multiple inputs of the deep neural network, thus completing the current learning operation performed on the deep neural network;

[0122] In addition, multiple learning operations are performed on the deep neural network to complete the multi-layer reconstruction of the deep neural network, and the deep neural network after multi-layer reconstruction is used as the output of the intelligent state judgment model. The positive correlation between the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate includes: using a content mapping function to represent the content mapping relationship between the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate.

[0123] like Figure 6 As shown, for example, N processors are given, where N is a natural number greater than or equal to 1.

[0124] Example 6

[0125] Figure 7 This is a schematic diagram of a machine vision-based driver fatigue monitoring system according to Embodiment 6 of the present invention.

[0126] like Figure 7 As shown, the machine vision-based driver fatigue monitoring system includes the following components:

[0127] The field sensor is used to acquire the average heart rate, average blood pressure, average blood pressure and average blood oxygen saturation of the designated driver of the open-pit mining truck within the current time interval, so as to output various real-time physiological information of the designated driver.

[0128] For example, acquiring the average heart rate, average hypertension, average diastolic blood pressure, and average blood oxygen saturation of a designated driver of an open-pit mining truck within the current time interval as various real-time physiological information outputs of the designated driver includes: using multiple different physiological sensors to acquire the average heart rate, average hypertension, average diastolic blood pressure, and average blood oxygen saturation of the designated driver of the open-pit mining truck within the current time interval respectively.

[0129] a visual monitor configured to capture a plurality of frames of driving environment pictures within a current time interval at a preset frame rate, the plurality of frames of driving environment pictures corresponding to a plurality of monitoring time points that are evenly distributed on a time axis;

[0130] Specifically, the plurality of frames of driving environment pictures within a current time interval are captured at a preset frame rate, and the plurality of frames of driving environment pictures correspond to a plurality of monitoring time points that are evenly distributed on a time axis, including that the preset frame rate is 60 frames per second;

[0131] a content analysis device connected to the visual monitor, configured to identify each eye-constituted pixel point in each frame of driving environment picture based on the eye imaging features of the set driver, and output each eye-constituted pixel point in each frame of driving environment picture, each imaging depth value, each horizontal coordinate value and each vertical coordinate value corresponding to the single customized screening data of the frame of driving environment picture;

[0132] Specifically, the plurality of frames of driving environment pictures within a current time interval are captured at a preset frame rate, and the plurality of frames of driving environment pictures correspond to a plurality of monitoring time points that are evenly distributed on a time axis, including that the preset frame rate is 60 frames per second;

[0133] a multi-layer reconstruction device configured to perform a plurality of learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and output the deep neural network after the multi-layer reconstruction as an intelligent state judgment model, and the number of reconstruction layers completed by the deep neural network is positively correlated with the preset frame rate;

[0134] Specifically, the plurality of learning operations are performed on the deep neural network to complete multi-layer reconstruction of the deep neural network, and the deep neural network after the multi-layer reconstruction is output as an intelligent state judgment model, and the number of reconstruction layers completed by the deep neural network is positively correlated with the preset frame rate, including that when the preset frame rate is 60 frames per second, the number of reconstruction layers completed by the deep neural network is 300 layers, when the preset frame rate is 50 frames per second, the number of reconstruction layers completed by the deep neural network is 200 layers, when the preset frame rate is 30 frames per second, the number of reconstruction layers completed by the deep neural network is 150 layers, and so on;

[0135] The intelligent judgment device is connected with the field sensing device, the content analysis device and the multi-layer reconstruction device respectively, and is used for intelligently judging the fatigue level of the setting driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the setting driver and the multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures respectively by using the intelligent state judgment model;

[0136] For example, the test and simulation of the data processing process of intelligently judging the fatigue level of the setting driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the setting driver and the multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures respectively by using the intelligent state judgment model is completed by using the MATLAB toolbox;

[0137] The fatigue level of the setting driver in the current time interval is one of the five fatigue levels of wakefulness, mild fatigue, moderate fatigue, deep fatigue and sleep.

[0138] For example, the test and simulation of the data processing process of intelligently judging the fatigue level of the setting driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the setting driver and the multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures respectively by using the intelligent state judgment model is completed by using the MATLAB toolbox;

[0139] The resolution of the multiple frames of driving environment pictures is the same.

[0140] The imaging feature of the driver's eyes is set as a luminance value distribution range corresponding to the driver's eyes, and the pixel points in the frame of the driving environment picture whose luminance values are within the luminance value distribution range corresponding to the driver's eyes are taken as single eye-constituting pixel points in the frame of the driving environment picture.

[0141] For example, the pixel points in the frame of the driving environment picture whose luminance values are within the luminance value distribution range corresponding to the driver's eyes are taken as single eye-constituting pixel points in the frame of the driving environment picture, including that the luminance values of each pixel point in the frame of the driving environment picture are between 0 and 255.

[0142] The number of layers of reconstruction completed by the deep neural network and the preset shooting frame rate are positively correlated, and the number of layers of reconstruction completed by the deep neural network and the preset shooting frame rate are positively correlated, including that in each learning operation performed on the deep neural network, the known fatigue level of the driver in a certain historical time interval is taken as a single output content of the deep neural network, and the duration length of the certain historical time interval, the average speed of the open-pit mine truck in the certain historical time interval, the instant physiological information of the driver corresponding to the certain historical time interval, and the multiple pieces of customized screening data corresponding to the multiple frames of driving environment pictures in the certain historical time interval are taken as multiple input contents of the deep neural network, and the current learning operation performed on the deep neural network is completed.

[0143] The number of layers of reconstruction completed by the deep neural network and the preset shooting frame rate are positively correlated, including that the content mapping function is used to represent the content mapping relationship between the number of layers of reconstruction completed by the deep neural network and the preset shooting frame rate.

[0144] In addition, the present application can also refer to the following technical contents to highlight the significant technical progress of the present application:

[0145] The setting of the eye imaging feature of the driver is a range of luminance value distribution corresponding to the eyes of the driver, and the pixel points in the frame of the driving environment picture whose luminance values are within the range of luminance value distribution corresponding to the eyes of the driver are taken as single eye constituent pixel points in the frame of the driving environment picture.

[0146] For example, the range of luminance value distribution corresponding to the eyes of the driver is defined by an upper threshold of luminance value and a lower threshold of luminance value corresponding to the eyes of the driver, and the values of the upper threshold of luminance value and the lower threshold of luminance value are both between 0 and 255, and the specific value of the upper threshold of luminance value is greater than the specific value of the lower threshold of luminance value.

[0147] In addition, based on the eye imaging feature of the driver, each eye constituent pixel point in each frame of the driving environment picture is identified, and each part of imaging depth value, each part of horizontal coordinate value and each part of vertical coordinate value corresponding to each eye constituent pixel point in each frame of the driving environment picture are taken as single part of customized filtering data corresponding to the frame of the driving environment picture, and the pixel points in the frame of the driving environment picture whose luminance values are outside the range of luminance value distribution corresponding to the eyes of the driver are taken as single other pixel points in the frame of the driving environment picture.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure, and not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present disclosure, and they should be covered in the scope of the claims and the specification of the present disclosure.

Claims

1. A method of driver fatigue monitoring based on machine vision, characterized in that, The method comprises: obtaining average heart rate data, average high blood pressure value, average low blood pressure value and average blood oxygen saturation of a designated driver driving an open-pit mine truck in a current time interval as instant physiological information of the designated driver; photographing the designated driver at a preset photographing frame rate to obtain a plurality of driving environment pictures in the current time interval, the plurality of driving environment pictures corresponding to a plurality of monitoring time points which are evenly distributed on a time axis; identifying each eye constituent pixel point in each driving environment picture based on eye imaging features of the designated driver, and outputting each imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye constituent pixel point in each driving environment picture as single customized screening data corresponding to the driving environment picture; performing a plurality of learning operations on the deep neural network to complete a plurality of layers of reconstruction of the deep neural network, and outputting the deep neural network after the completion of the plurality of layers of reconstruction as an intelligent state judgment model, the number of layers of reconstruction completed by the deep neural network being positively correlated with the preset photographing frame rate; intelligently judging a fatigue level of the designated driver in the current time interval by the intelligent state judgment model according to a duration of the current time interval, an average speed of the open-pit mine truck in the current time interval, instant physiological information of the designated driver and a plurality of customized screening data corresponding to a plurality of driving environment pictures.

2. The machine vision-based driver fatigue monitoring method of claim 1, wherein: the intelligent judgment of the fatigue level of the designated driver in the current time interval by the intelligent state judgment model according to the duration of the current time interval, the average speed of the open-pit mine truck in the current time interval, the instant physiological information of the designated driver and the plurality of customized screening data corresponding to the plurality of driving environment pictures comprises that the fatigue level of the designated driver in the current time interval is one of five fatigue levels, i.e., awake, mild fatigue, moderate fatigue, deep fatigue and sleep; the plurality of driving environment pictures have the same resolution; the eye imaging features of the designated driver are a brightness value distribution range corresponding to eyes of the designated driver, and each driving environment picture has a single eye constituent pixel point whose brightness value is within the brightness value distribution range corresponding to the eyes of the designated driver. 3.The machine vision based driver fatigue monitoring method of claim 2, wherein: the performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after the multi-layer reconstruction is completed as the intelligent state judgment model, the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate being positively correlated further comprises: in each learning operation performed on the deep neural network, taking the known fatigue level of the set driver in a certain historical time interval as a single output content of the deep neural network, taking the duration length of the certain historical time interval, the average speed of the open-pit mine truck in the certain historical time interval, each item of instant physiological information of the set driver corresponding to the certain historical time interval, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures corresponding to the certain historical time interval as multiple input contents of the deep neural network, and completing the current learning operation performed on the deep neural network; wherein the performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputting the deep neural network after the multi-layer reconstruction is completed as the intelligent state judgment model, the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate being positively correlated comprises: using a content mapping function to represent the content mapping relationship between the number of reconstruction layers completed by the deep neural network and the preset shooting frame rate.

4. The machine vision-based driver fatigue monitoring method of claim 3, wherein, Before the multiple frames of driving environment pictures corresponding to the set driver are obtained by shooting at the preset shooting frame rate, the method further comprises: using the user control interface to realize parameter setting of the preset shooting frame rate according to the input operation of the set driver.

5. The machine vision-based driver fatigue monitoring method of claim 3, wherein, After the intelligent state judgment model is used to intelligently judge the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures, the method further comprises: transmitting the fatigue level of the set driver in the current time interval intelligently judged to the vehicle fleet monitoring server of the nearest open-pit mine through a wireless communication link.

6. The machine vision-based driver fatigue monitoring method of claim 3, wherein, Before the intelligent state judgment model is used to intelligently judge the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver, and multiple pieces of customized screening data corresponding to multiple frames of driving environment pictures, the method further comprises: using the vehicle speed sensor to obtain each piece of vehicle speed corresponding to each time point in the current time interval of the open-pit mine truck, removing the maximum and minimum values of the multiple pieces of vehicle speed to obtain remaining multiple pieces of vehicle speed, and performing mean value calculation on the remaining multiple pieces of vehicle speed to obtain the average speed of the open-pit mine truck in the current time interval.

7. The machine vision-based driver fatigue monitoring method of any one of claims 3-6, wherein: The identifying of each eye-constituted pixel point in each frame of the driving environment picture based on the set driver's eye imaging features comprises: taking a pixel point at the lower left corner of the frame of the driving environment picture as the origin of a two-dimensional coordinate system, taking a pixel column at the leftmost side of the frame of the driving environment picture as the vertical coordinate axis of the two-dimensional coordinate system in the positive direction, and taking a pixel row at the bottom of the frame of the driving environment picture as the horizontal coordinate axis of the two-dimensional coordinate system in the positive direction, and establishing the two-dimensional coordinate system of the frame of the driving environment picture.

8. The machine vision-based driver fatigue monitoring method of any one of claims 3-6, wherein: The obtaining of the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver in the current time interval as the instant physiological information of the set driver comprises: uniformly dividing the current time interval to obtain uniformly distributed heart rate data corresponding to each time point, removing the maximum and minimum values of the heart rate data to obtain remaining heart rate data, and performing mean value calculation on the remaining heart rate data to obtain the average heart rate data of the set driver in the current time interval. The obtaining of the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver in the current time interval as the instant physiological information of the set driver further comprises: uniformly dividing the current time interval to obtain uniformly distributed high blood pressure values corresponding to each time point, removing the maximum and minimum values of the high blood pressure values to obtain remaining high blood pressure values, and performing mean value calculation on the remaining high blood pressure values to obtain the average high blood pressure value of the set driver in the current time interval. The obtaining of the average heart rate data, the average high blood pressure value, the average low blood pressure value and the average blood oxygen saturation of the set driver in the current time interval as the instant physiological information of the set driver further comprises: uniformly dividing the current time interval to obtain uniformly distributed low blood pressure values corresponding to each time point, removing the maximum and minimum values of the low blood pressure values to obtain remaining low blood pressure values, and performing mean value calculation on the remaining low blood pressure values to obtain the average low blood pressure value of the set driver in the current time interval. The process of obtaining the average heart rate, average hypertension, average diastolic blood pressure, and average blood oxygen saturation of the designated driver of the open-pit mining truck within the current time interval as various real-time physiological information outputs of the designated driver further includes: uniformly dividing the current time interval to obtain blood oxygen saturation data corresponding to each uniformly distributed moment; removing the maximum and minimum values ​​from each blood oxygen saturation data to obtain the remaining multiple blood oxygen saturation data; and performing mean calculation on the remaining multiple blood oxygen saturation data to obtain the average blood oxygen saturation of the designated driver of the open-pit mining truck within the current time interval.

9. A machine vision based driver fatigue monitoring system characterized by, The system includes a memory and one or more processors, the memory storing a computer program configured to be executed by the one or more processors to perform the following steps: The average heart rate, average blood pressure, average blood pressure, and average blood oxygen saturation of the designated driver driving the open-pit mining truck are obtained within the current time interval and used as the real-time physiological information output of the designated driver. The driver is captured using a preset frame rate to obtain multiple frames of driving environment images within the current time interval. The multiple monitoring moments corresponding to the multiple frames of driving environment images are evenly distributed on the time axis. Based on the eye imaging features of the driver, each eye-forming pixel in each frame of the driving environment is identified, and the imaging depth value, horizontal coordinate value and vertical coordinate value corresponding to each eye-forming pixel in each frame of the driving environment are output as a single set of customized filtering data for that frame of the driving environment. Multiple learning operations are performed on the deep neural network to complete the multi-layer reconstruction of the deep neural network, and the deep neural network after multi-layer reconstruction is used as the output of the intelligent state judgment model. The number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate. The intelligent state judgment model uses the duration of the current time interval, the average speed of the open-pit mining truck in the current time interval, the real-time physiological information of the driver, and multiple customized filtering data corresponding to multiple frames of driving environment images to intelligently determine the fatigue level of the driver in the current time interval.

10. A machine vision based driver fatigue monitoring system, characterized in that, The system includes: The field sensor is used to acquire the average heart rate, average blood pressure, average blood pressure and average blood oxygen saturation of the designated driver of the open-pit mining truck within the current time interval, so as to output various real-time physiological information of the designated driver. A visual monitor is used to capture images of the set driver at a preset frame rate to obtain multiple frames of driving environment images within the current time interval. The multiple monitoring moments corresponding to the multiple frames of driving environment images are evenly distributed on the time axis. The content analysis device is connected with the visual monitor and is used for identifying each eye constituent pixel in each frame of driving environment picture based on the set driver's eye imaging feature, and outputs each part of imaging depth value, each part of horizontal coordinate value and each part of vertical coordinate value corresponding to each eye constituent pixel in each frame of driving environment picture as single part of customized screening data corresponding to the frame of driving environment picture; The multi-layer reconstruction device is used for performing multiple learning operations on the deep neural network to complete multi-layer reconstruction of the deep neural network, and outputs the deep neural network after the multi-layer reconstruction is completed as an intelligent state judgment model, and the number of reconstruction layers completed by the deep neural network is positively correlated with the preset shooting frame rate; The intelligent judgment device is connected with the field sensing device, the content analysis device and the multi-layer reconstruction device respectively, and is used for intelligently judging the fatigue level of the set driver in the current time interval according to the duration length of the current time interval, the average speed of the open-pit mine truck in the current time interval, each item of instant physiological information of the set driver and multiple parts of customized screening data corresponding to multiple frames of driving environment pictures by using the intelligent state judgment model.

Images