A fatigue driving monitoring method, device and medium based on cloud intelligent learning

Abstract

The application discloses a kind of fatigue driving monitoring method, equipment and medium based on cloud intelligent learning, belong to the technical field of automobile monitoring.The method comprises: obtaining the monitoring video data of driver, and sends to cloud;Monitoring video data is handled based on image preprocessing algorithm, to extract multiple facial features of driver;Multiple facial features are handled based on fatigue identification model, to determine fatigue degree;Fatigue degree is sent to cloud based on early warning;Cloud database is handled based on cloud computing module, and iterative operation training identification model is operated, to obtain the deviation of training identification model, the training identification model of minimum output deviation;The weight factor of minimum training identification model of deviation is obtained by processing minimum training identification model of deviation based on reverse operation algorithm, and fatigue identification model is updated.The application improves the accuracy, popularity and stability of monitoring fatigue driving by the above method, while not affecting the normal driving of driver.

Description

Technical Field

[0001] This application relates to the technical field of vehicle monitoring, and in particular to a fatigue driving monitoring method, device and medium based on cloud-based intelligent learning. Background Technology

[0002] Fatigue driving refers to a decline in driving skills caused by a driver's physiological and psychological dysfunction after prolonged continuous driving. Poor or insufficient sleep quality, coupled with long hours of driving, easily leads to fatigue. Driver fatigue affects various aspects of a driver's attention, senses, perception, thinking, judgment, will, decision-making, and motor skills. Fatigue driving can cause road traffic accidents. Therefore, driving while fatigued is strictly prohibited.

[0003] In existing technologies, the monitoring, research, and application of fatigue driving mainly include the following aspects:

[0004] Firstly, regarding the external manifestations of fatigued driving, monitoring a person's mental state involves observing physiological or behavioral changes such as nervous system function, circulatory function, blood indicators, eye parameters, respiratory function, body temperature fluctuations, head or facial features, and driving behavior. This information is used to form conscious or unconscious reflexes, and combined with monitoring technologies, methods, and evaluation standards, judgments and decisions are made. However, these monitoring devices are typically wearable, which can significantly interfere with the driver and easily affect their normal driving.

[0005] Secondly, regarding vehicle parameters, issues include the car frequently crossing the center line, excessively high or low speeds, incoordination with the surrounding driving environment, and abnormal steering torque. While it offers advantages such as high real-time performance and minimal driver interference, its disadvantages include susceptibility to vehicle type limitations and the difficulty in establishing uniform evaluation standards due to individual differences, resulting in low accuracy in assessing driver fatigue.

[0006] Thirdly, in terms of complex intelligent computing, the cost is too high on the one hand, and the computing power of the vehicle controller is limited, which cannot meet the real-time requirements. In addition, it requires dedicated sensors, so it is difficult to popularize in automotive applications.

[0007] Fourthly, in the field of machine vision, currently common fatigue monitoring methods based on machine vision monitor changes in the driver's head and facial features when the driver is fatigued, such as frequent nodding or prolonged head stillness, pupil constriction, eyelid closure, and slowed blinking rate, and then analyze and judge these changes. The problems with these methods are: scattered information, simple analysis strategies, difficulty in determining judgment criteria, poor accuracy, and susceptibility to changes in experience and environment.

[0008] Therefore, how to improve the accuracy, accessibility, and stability of fatigue driving monitoring without affecting the driver's normal driving has become an urgent problem to be solved. Summary of the Invention

[0009] This application provides a fatigue driving monitoring method, device, and medium based on cloud-based intelligent learning to solve the following technical problem: how to improve the accuracy, accessibility, and stability of fatigue driving monitoring without affecting the driver's normal driving.

[0010] In a first aspect, embodiments of this application provide a fatigue driving monitoring method based on cloud-based intelligent learning. The method includes: acquiring monitoring video data of the driver and sending it to the cloud; wherein the monitoring video data includes videos of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, the cloud database being used to store the monitoring video data; processing the monitoring video data based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein the multiple facial features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump frequency; processing the multiple facial features based on a fatigue recognition model to determine the driver's fatigue level; issuing a warning based on the fatigue level and sending the fatigue level to the cloud database; processing the cloud database based on the cloud computing module and iteratively calculating a preset training recognition model within the cloud computing module to obtain the deviation of the training recognition model, and outputting the training recognition model with the smallest deviation; wherein the training recognition model and the fatigue recognition model are the same type of neural network model; processing the training recognition model with the smallest deviation based on a preset reverse calculation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest deviation; and processing the weight factor of each neuron based on a preset update algorithm to update the fatigue recognition model.

[0011] In one possible implementation, the monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver. Specifically, this includes: processing the monitoring video data based on a preset correction video algorithm, filtering, enhancing, and correcting distortion in the monitoring video data to obtain corrected video data; identifying the corrected video data, segmenting the driver's face based on the driver's facial features to obtain multiple facial features of the driver; and tracking multiple facial features of the driver based on one of the multiple facial features, and extracting the driver's facial features.

[0012] In one possible implementation, multiple facial features are processed based on a fatigue recognition model to determine the driver's fatigue level. Specifically, this includes: inputting multiple facial features as feature parameters into the fatigue recognition model; processing the multiple facial features based on the hidden layer of the fatigue recognition model to determine the first weight of the multiple facial features; and calculating the first weight of the multiple facial features based on a preset nonlinear equation to determine the driver's fatigue level.

[0013] In one possible implementation, a first weight of multiple facial features is calculated based on a preset nonlinear equation to determine the driver's fatigue level. Specifically, this includes: matching multiple facial features to a preset weight matching table to obtain a second weight of different facial features; multiplying the first weight of multiple facial features by the second weight of different facial features to calculate the driver's fatigue value; and comparing the driver's fatigue value with a preset multi-level fatigue threshold to determine the driver's fatigue level.

[0014] In one possible implementation, a cloud computing module processes a cloud database and iteratively calculates a pre-set training recognition model within the cloud computing module to obtain the bias of the training recognition model and outputs the training recognition model with the smallest bias. Specifically, this includes: using the cloud database as samples to train the training recognition model to obtain predicted values ​​for the monitored video; wherein the monitored video data is used as feature values ​​and the fatigue level associated with the monitored video data is used as a label; comparing the predicted values ​​of the training recognition model with the labels to obtain the bias between the predicted values ​​and the actual values; comparing the biases of two adjacent training recognition models one by one to output the training recognition model with the smallest bias; and outputting the training recognition model with the smallest bias when the bias meets a pre-set bias rule.

[0015] In one possible implementation, when the deviation meets a preset deviation rule, the training recognition model with the smallest deviation is output. Specifically, this includes: outputting the training recognition model with the smallest deviation when the update interval of the fatigue recognition model reaches a preset update interval threshold; and / or outputting the training recognition model with the smallest deviation when the difference between the deviation of the training recognition model with the smallest deviation and the deviation of the fatigue recognition model reaches a preset deviation threshold.

[0016] In one possible implementation, the training recognition model with the smallest deviation is processed based on a preset reverse computation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest deviation. Specifically, this includes: matching a preset fatigue warning table according to the fatigue level to determine the warning method; when the fatigue level is mild, reminding the driver through sound and light warnings; when the fatigue level is deep, reporting the driver information to the vehicle monitoring platform, gradually reducing the vehicle speed to a stop and turning on the vehicle's hazard lights.

[0017] In one possible implementation, the weight factors of each neuron are processed based on a preset update algorithm to update the fatigue recognition model. Specifically, this includes: labeling the factor position of the weight factor of each neuron; wherein the factor position is the corresponding position of the weight factor of each neuron in the training recognition model with the smallest deviation; querying the neuron position with the same factor position in the fatigue recognition model, and replacing the corresponding position in the fatigue recognition model with the weight factor of each neuron to update the fatigue recognition model.

[0018] Secondly, embodiments of this application also provide a fatigue driving monitoring device based on cloud-based intelligent learning, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: acquire monitoring video data of the driver and send it to the cloud; wherein the monitoring video data is a video including the driver's face and head, the cloud includes a cloud database and a cloud computing module, the cloud database being used to store the monitoring video data; and process the monitoring video data based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein the multiple facial features include yawning frequency, blinking frequency, and eye... The system analyzes multiple facial features, including eyelid closure time, pupil constriction, gaze deviation from the designated area, and frequency of gaze jumps from the designated area, to determine the driver's fatigue level. It then issues a warning based on the fatigue level and sends the fatigue level data to a cloud database. The system processes the cloud database using a cloud computing module and iteratively calculates a pre-set training recognition model within the cloud computing module to obtain the bias of the training recognition model, outputting the training recognition model with the smallest bias. The training recognition model and the fatigue recognition model are of the same type of neural network model. A pre-set inverse computation algorithm is used to process the training recognition model with the smallest bias to obtain the weight factors of each neuron in the training recognition model with the smallest bias. Finally, a pre-set update algorithm is used to process the weight factors of each neuron to update the fatigue recognition model.

[0019] Thirdly, this application also provides a non-volatile computer storage medium for fatigue driving monitoring based on cloud-based intelligent learning, storing computer-executable instructions. The computer-executable instructions are configured to: acquire monitoring video data of the driver and send it to the cloud; wherein the monitoring video data includes video of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, the cloud database being used to store the monitoring video data; process the monitoring video data based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein the multiple facial features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump. The system employs a frequency thresholding mechanism; it processes multiple facial features based on a fatigue recognition model to determine the driver's fatigue level; it issues a warning based on the fatigue level and sends the fatigue level data to a cloud database; it processes the cloud database using a cloud computing module and iteratively calculates a pre-set training recognition model within the cloud computing module to obtain the bias of the training recognition model, outputting the training recognition model with the smallest bias; wherein, the training recognition model and the fatigue recognition model are of the same type of neural network model; it processes the training recognition model with the smallest bias using a pre-set inverse computation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest bias; and it processes the weight factor of each neuron using a pre-set update algorithm to update the fatigue recognition model.

[0020] This application provides a cloud-based intelligent learning-based method, device, and medium for monitoring driver fatigue. By acquiring monitoring video data of the driver and performing image preprocessing on the video data to obtain multiple facial features that can be processed by a fatigue recognition model, the fatigue recognition model can determine the driver's fatigue level. By determining the driver's fatigue level, it can alert the driver and, to some extent, prevent traffic accidents caused by driver fatigue. Simultaneously, the driver's monitoring video data and corresponding fatigue level are sent to the cloud. A large amount of monitoring video data and corresponding fatigue levels are obtained from the cloud, and the recognition model is iteratively trained using this data. Through extensive training with large amounts of data, a recognition model with the smallest deviation within a fixed time period is obtained. The neuron weights of the recognition model with the smallest deviation within a fixed time period are sent to the fatigue recognition model to update the fatigue recognition model. This improves the accuracy, accessibility, and stability of fatigue driving monitoring without affecting the driver's normal driving performance. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1A flowchart of a fatigue driving monitoring method based on cloud-based intelligent learning is provided for embodiments of this application;

[0023] Figure 2 This is a schematic diagram of the internal structure of a fatigue driving monitoring device based on cloud-based intelligent learning, provided as an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a fatigue driving monitoring method, device, and medium based on cloud-based intelligent learning to solve the following technical problem: how to improve the accuracy, accessibility, and stability of fatigue driving monitoring without affecting the driver's normal driving.

[0026] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This application provides a flowchart of a cloud-based intelligent learning-based fatigue driving monitoring system as an embodiment. Figure 1 As shown in the figure, the fatigue driving monitoring method based on cloud-based intelligent learning provided in this application embodiment specifically includes the following steps:

[0028] Step 1: Obtain the driver's monitoring video data and send it to the cloud.

[0029] The surveillance video data includes footage of the driver's face and head, captured by an in-vehicle video camera. Due to variations in vehicle structure, the choice of in-vehicle video camera also varies. For example, in large vehicles, a flat-mounted camera can be used, placed at the front of the vehicle. In smaller vehicles, a miniature suspended camera or an in-vehicle monitoring system with in-vehicle monitoring capabilities can be used to acquire video data of the driver.

[0030] The device connected to the vehicle-mounted video camera is a fatigue monitoring controller, which predicts the driver's fatigue level based on monitored video data. It should be noted that the connection method between the fatigue detection controller and the vehicle-mounted video camera is not fixed; it can be either wired or wireless.

[0031] The cloud refers to the cloud computing platform that provides support for the embodiments of this application. The cloud includes a cloud database and a computing module. The cloud database is used to store monitoring video data of the driver acquired by various cameras. In subsequent steps, the cloud database is also used to store the fatigue level predicted based on the monitoring video data.

[0032] After acquiring the driver's monitoring video data, the data is uploaded to a cloud database and an in-vehicle fatigue monitoring controller for processing the monitoring video data.

[0033] Step 2: Process the monitoring video data based on the preset image preprocessing algorithm to extract multiple facial features of the driver.

[0034] The image preprocessing algorithm is pre-set in the fatigue monitoring controller. The image preprocessing algorithm is used to process the monitoring video data uploaded by the vehicle camera into multiple facial features that can be processed by the fatigue recognition model.

[0035] The facial features in this embodiment are those related to fatigue.

[0036] Before acquiring the driver's facial features, it is necessary to correct the video monitoring data and track the driver's facial features. Correcting the video monitoring data is necessary because the data is acquired through in-vehicle cameras, and due to environmental factors, the lighting and shadows in the video monitoring data may vary, and there may be shaky movements. Tracking the driver's facial features is necessary because while driving, the driver's facial or head features may shift or turn due to actions such as checking rearview mirrors and reversing, thus requiring tracking of the driver's facial features.

[0037] Correction of monitoring video data: Based on a preset correction video algorithm, the monitoring video data is processed to filter, enhance, and correct distortion in order to obtain corrected video data.

[0038] The video correction algorithm is pre-installed in the fatigue monitoring controller and is used to correct the monitoring video data to obtain corrected video data with uniform specifications. First, the monitoring video data is filtered to remove interference, including white noise and non-facial features. Then, the driver's facial features are enhanced to make them more prominent.

[0039] Distortion correction refers to the correction of image distortion. Because of the optical lenses in the structure of vehicle cameras, the resulting image is distorted; for example, a normally upright tall building may appear warped. Distortion correction algorithms are existing and relatively mature technologies, and will not be elaborated upon here.

[0040] Tracking driver's facial features: Identify and correct video data, segment the driver's face based on the driver's facial features, and obtain multiple facial features of the driver.

[0041] This application determines a driver's fatigue level primarily by assessing and tracking facial and head features. First, the driver's face is segmented based on the five facial features. Then, one of the driver's facial features is selected as the center for tracking; this feature is typically the nose. It should be noted that multiple facial features can be tracked. This involves tracking and extracting facial features such as yawning frequency, blinking frequency, eyelid closure time, pupil constriction, and frequency of gaze deviations and jumps.

[0042] Step 3: Process multiple facial features based on the fatigue recognition model to determine the driver's level of fatigue.

[0043] The fatigue recognition model is a type of neural network model. It includes an input layer, a hidden layer, and an output layer. The fatigue recognition model is used to predict the driver's level of fatigue based on facial features. The specific steps are as follows:

[0044] 1. Input multiple facial features as feature parameters into the fatigue recognition model.

[0045] Multiple facial features, including yawning frequency, blinking frequency, eyelid closure time, pupil constriction, and frequency of gaze deviation and gaze jump, are input into the fatigue recognition model.

[0046] 2. The hidden layer of the fatigue recognition model processes multiple facial features to determine the first weight of the multiple facial features.

[0047] The fatigue recognition model includes at least one hidden layer. Facial features in the hidden layer are passed to the next hidden layer through the neuron weight factors of the hidden layer, until they are transmitted to the output layer.

[0048] In the hidden layer, the weight corresponding to each facial feature can be determined based on multiple facial features. Here, the weight is set as the first weight. It should be noted that the first weight is related to the time when the facial feature appears.

[0049] 3. Calculate the first weight of multiple facial features based on a preset nonlinear equation to determine the driver's fatigue level.

[0050] Based on a pre-defined weighted matching table for multiple facial features, a second weight is obtained for different facial features. Since different facial features correspond to different levels of fatigue, for example, pupil constriction is an uncontrollable factor, so its second weight is relatively high. High blinking frequency may be due to foreign objects entering the driver's eyes, which has many influencing factors, so its second weight is relatively low.

[0051] The driver's fatigue value is calculated by multiplying a first weight of multiple facial features by a second weight of those features. The driver's fatigue value is obtained by weighted multiplication of the first and second weights of multiple facial features.

[0052] There are multiple preset fatigue thresholds, with different fatigue values ​​corresponding to different levels of driver fatigue. By comparing the driver's fatigue value with the preset multi-level fatigue thresholds, the driver's level of fatigue can be determined.

[0053] In a specific case, the multi-level fatigue threshold includes a first-level fatigue threshold, a second-level fatigue threshold, and a third-level fatigue threshold. The first-level fatigue threshold is set to 100, the second-level fatigue threshold is set to 200, and the third-level fatigue threshold is set to 300. The driver's fatigue level can be determined by the fatigue value and the multi-level fatigue threshold.

[0054] Step 4: Issue an alert based on the level of fatigue and send the fatigue level to the cloud database.

[0055] After determining the driver's level of fatigue, different warnings are issued based on the different levels of driver fatigue. In this embodiment, different warnings are issued based on different levels of driver fatigue.

[0056] In specific cases, a fatigue warning table is set up. By comparing the fatigue level with the fatigue warning table, it is possible to know the different warning methods under different fatigue levels. It should be noted that the fatigue level can be set according to the actual situation.

[0057] If the driver's fatigue level is detected as mild, the vehicle's audio system will remind the driver to pay attention to driving safety and provide corresponding suggestions. Specifically, mild fatigue includes Level 1, Level 2, and Level 3 fatigue. When the driver is at Level 1 fatigue, the system will remind the driver that they have reached Level 1 fatigue and play energizing music. When the driver is at Level 2 fatigue, the system will remind the driver that they are at Level 2 fatigue, play energizing music, and suggest that the driver drink energy drinks or coffee to stay alert. When the driver is at Level 3 fatigue, the system will play jarring music and remind the driver to stop at a rest stop within one hour.

[0058] If the driver's fatigue level is detected as deep fatigue, the driver's information is reported to the vehicle management platform, which then monitors the vehicle. Simultaneously, the hazard lights of the vehicle in which the driver is located are activated to warn nearby vehicles to keep their distance. It should be noted that deep fatigue can be categorized into multiple levels, which are set according to the actual situation. If the driver's fatigue level reaches the "drowsy" level, the vehicle will gradually stop operating and the hazard lights will be activated.

[0059] Step 5: Process the cloud database based on the cloud computing module, and iteratively calculate the preset training recognition model in the cloud computing module to obtain the deviation of the training recognition model, and output the training recognition model with the smallest deviation; wherein, the training recognition model and the fatigue recognition model are the same type of neural network model.

[0060] The fatigue recognition model installed in the vehicle is trained using a neural network model, but due to several reasons at the factory, there is still room for further optimization of the fatigue recognition model. These reasons include:

[0061] 1. Compared to the continuous collection of monitoring video data from vehicle owners, the training data sample is not large enough. Acquiring monitoring video data from different drivers will yield a large amount of training data. Furthermore, the in-vehicle fatigue monitoring controller cannot process such a large amount of training data, so it can be processed through the cloud.

[0062] 2. It is impossible to train on monitoring video data of a specific person. For example, a fatigue recognition model trained on monitoring video data of Zhang San is more effective at monitoring Zhang San than at monitoring Li Si.

[0063] Therefore, by acquiring a large amount of monitoring video data from car owners to train a recognition model that is the same type of neural network model as the fatigue recognition model, the fatigue recognition model can be optimized.

[0064] The specific steps are as follows:

[0065] 1. Use the cloud database as a sample to train the recognition model in order to obtain the predicted values ​​of the monitored and recognized videos.

[0066] A large amount of vehicle owner monitoring video data and fatigue levels predicted by an onboard fatigue recognition model are stored in a cloud database, which serves as the training set. The monitoring video data is used as feature values, and the fatigue levels associated with the monitoring video data are used as labels. It should be noted that drivers have the right to choose whether to upload vehicle monitoring data.

[0067] The monitoring video data in the cloud database is used as features to train the monitoring and recognition model, thereby obtaining the predicted value of the monitoring video data.

[0068] 2. Compare the predicted values ​​and labels of the trained recognition model to obtain the deviation between the predicted values ​​and the true values.

[0069] After obtaining the predicted value output by the trained recognition model, the predicted value is compared with the fatigue level (label) corresponding to the monitored video data to obtain the deviation between the predicted value and the true value.

[0070] 3. Compare the deviations of two adjacent training recognition models one by one, and output the training recognition model with the smallest deviation.

[0071] The performance of the training recognition model can be determined by the bias of the training recognition model. The performance of all training recognition models involved in the comparison is obtained by comparing them one by one, and the training recognition model with the best performance (smallest bias) is output.

[0072] 4. When the deviation meets the preset deviation rules, output the training recognition model with the smallest deviation.

[0073] Since the purpose of training the recognition model is to update the fatigue recognition model, iterative training cannot be carried out indefinitely. Therefore, it is necessary to limit the time for updating the fatigue recognition model. The bias rule is the rule that satisfies the need to update the fatigue recognition model.

[0074] The deviation rules are as follows:

[0075] 1) When the fatigue recognition model update interval reaches the preset update interval threshold, output the training recognition model with the smallest deviation;

[0076] 2) When the difference between the deviation of the training recognition model with the smallest deviation and the deviation of the fatigue recognition model reaches a preset deviation threshold, the training recognition model with the smallest deviation is output.

[0077] The training recognition model with the smallest deviation can be output by satisfying one of the above deviation rules.

[0078] Step 6: Process the training recognition model with the smallest deviation based on the preset reverse calculation algorithm to obtain the weight factor of each neuron of the training recognition model with the smallest deviation.

[0079] Because the fatigue recognition model and the training recognition model are the same type of neural network model—meaning their frameworks are identical—the only difference lies in the weight factors of the neurons. During the iterative computation of the neural network model, the framework remains unchanged, but the weight factors of the neurons alter, leading to changes in the model's output. Therefore, changing the weight factors of the neurons in the fatigue recognition model updates the model itself.

[0080] The weight factors of each neuron in the training recognition model are obtained through a preset inverse operation algorithm. In specific examples, the BP algorithm can be used, which will not be elaborated here.

[0081] Step 7: Process the weight factors of each neuron based on the preset update algorithm to update the fatigue recognition model.

[0082] The reverse identification algorithm is an algorithm that, after determining the predicted value, performs layer-by-layer operations from the output layer to the hidden layer and then to the input layer of the neural network model to obtain the weight factor of each neuron in the neural network model.

[0083] After obtaining the training recognition model with the smallest bias, the training recognition model with the smallest bias is processed by the reverse operation algorithm in step 6 to obtain the weight factors of the neurons of the training recognition model with the smallest bias.

[0084] The weight factor of this neuron is sent to the fatigue monitoring controller, and the fatigue factor is modified to correspond to the weight factor of the neuron in the training recognition model with the smallest bias. It is understood that the updated fatigue recognition model is identical to the training recognition model with the smallest bias.

[0085] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a fatigue driving monitoring device based on cloud-based intelligent learning, the structure of which is as follows: Figure 2 As shown.

[0086] Figure 2 This is a schematic diagram of the internal structure of a fatigue driving monitoring device based on cloud-based intelligent learning, provided as an embodiment of this application. Figure 2 As shown, the device includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor; wherein the memory 202 stores instructions executable by the at least one processor, the instructions being executed by the at least one processor 201 to enable the at least one processor 201 to:

[0087] The system acquires monitoring video data of the driver and sends it to the cloud. The monitoring video data includes videos of the driver's face and head. The cloud includes a cloud database and a cloud computing module. The cloud database stores the monitoring video data. The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver. These features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump frequency. Multiple facial features are processed based on a fatigue recognition model to determine the driver's fatigue level. A warning is issued based on the fatigue level and sent to the cloud database. The cloud computing module processes the cloud database and iteratively calculates a preset training recognition model within the cloud computing module to obtain the bias of the training recognition model, outputting the training recognition model with the smallest bias. The training recognition model and the fatigue recognition model are of the same type of neural network model. A preset inverse operation algorithm is used to process the training recognition model with the smallest bias to obtain the weight factor of each neuron in the training recognition model with the smallest bias. A preset update algorithm is used to process the weight factor of each neuron to update the fatigue recognition model.

[0088] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for fatigue driving monitoring based on cloud-based intelligent learning, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0089] The system acquires monitoring video data of the driver and sends it to the cloud. The monitoring video data includes videos of the driver's face and head. The cloud includes a cloud database and a cloud computing module. The cloud database stores the monitoring video data. The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver. These features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump frequency. Multiple facial features are processed based on a fatigue recognition model to determine the driver's fatigue level. A warning is issued based on the fatigue level and sent to the cloud database. The cloud computing module processes the cloud database and iteratively calculates a preset training recognition model within the cloud computing module to obtain the bias of the training recognition model, outputting the training recognition model with the smallest bias. The training recognition model and the fatigue recognition model are of the same type of neural network model. A preset inverse operation algorithm is used to process the training recognition model with the smallest bias to obtain the weight factor of each neuron in the training recognition model with the smallest bias. A preset update algorithm is used to process the weight factor of each neuron to update the fatigue recognition model.

[0090] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0091] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0100] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A fatigue driving monitoring method based on cloud-based intelligent learning, characterized in that, The method includes: The system acquires monitoring video data of the driver and sends it to the cloud; wherein the monitoring video data includes video of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, and the cloud database is used to store the monitoring video data; The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein, the multiple facial features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump from the boundary frequency; The multiple facial features are input as feature parameters into the fatigue recognition model; The hidden layer of the fatigue recognition model processes the multiple facial features to determine a first weight of the multiple facial features, wherein the first weight is related to the time of occurrence of the multiple facial features; Based on the multiple facial features, a preset weighted matching table is used to obtain the second weight of different facial features; The driver's fatigue value is calculated by multiplying the first weight of the multiple facial features by the second weight of the different facial features; The driver's fatigue level is compared with a preset multi-level fatigue threshold to determine the degree of driver fatigue. An early warning is issued based on the fatigue level, and the fatigue level is sent to the cloud database; The cloud computing module processes the cloud database and iteratively calculates the preset training recognition model within the cloud computing module to obtain the deviation of the training recognition model and outputs the training recognition model with the smallest deviation; wherein, the training recognition model and the fatigue recognition model are the same type of neural network model; The training recognition model with the smallest deviation is processed based on a preset reverse calculation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest deviation. The fatigue recognition model is updated by processing the weight factors of each neuron based on a preset update algorithm.

2. The fatigue driving monitoring method based on cloud-based intelligent learning according to claim 1, characterized in that, The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver, specifically including: The monitoring video data is processed based on a preset corrected video algorithm, and the monitoring video data is filtered, enhanced and distorted to obtain corrected video data. The corrected video data is identified, and the driver's face is segmented based on the driver's facial features to obtain multiple facial features of the driver; Using one of the multiple facial features as a reference, the driver's multiple facial features are tracked, and the driver's facial features are extracted.

3. The fatigue driving monitoring method based on cloud-based intelligent learning according to claim 1, characterized in that, The cloud computing module processes the cloud database and iteratively calculates the preset training recognition model within the cloud computing module to obtain the bias of the training recognition model, and outputs the training recognition model with the smallest bias, specifically including: The cloud database is used as a sample to train the training recognition model to obtain the predicted value of the monitoring video data; wherein the monitoring video data is used as the feature value and the fatigue level associated with the monitoring video data is used as the label; Compare the predicted values ​​of the trained recognition model with the labels to obtain the deviation between the predicted values ​​and the true values; Compare the deviations of two adjacent training recognition models one by one, and output the training recognition model with the smallest deviation; When the deviation meets the preset deviation rule, the training recognition model with the smallest deviation is output.

4. The fatigue driving monitoring method based on cloud-based intelligent learning according to claim 3, characterized in that, When the deviation meets a preset deviation rule, the trained recognition model with the smallest deviation is output, specifically including: When the fatigue recognition model update interval reaches a preset update interval threshold, the trained recognition model with the smallest deviation is output; and / or When the difference between the deviation of the training recognition model with the smallest deviation and the deviation of the fatigue recognition model reaches a preset deviation threshold, the training recognition model with the smallest deviation is output.

5. The fatigue driving monitoring method based on cloud-based intelligent learning according to claim 1, characterized in that, Issuing an early warning based on the fatigue level and sending the fatigue level to the cloud database, specifically including: The warning method is determined by matching the fatigue level with a preset fatigue warning table. When the fatigue level is mild, the driver will be alerted via an audible and visual warning. When the fatigue level is considered deep fatigue, the driver's information is reported to the vehicle monitoring platform, the vehicle speed is gradually reduced to a stop, and the vehicle's hazard lights are turned on.

6. The fatigue driving monitoring method based on cloud-based intelligent learning according to claim 1, characterized in that, The fatigue recognition model is updated by processing the weight factors of each neuron based on a preset update algorithm, specifically including: The factor position of the weight factor of each neuron is labeled; wherein the factor position is the corresponding position of the weight factor of each neuron in the training recognition model with the smallest deviation; In the fatigue recognition model, the neuron position that is the same as the factor position is queried, and the weight factor of each neuron is replaced with the corresponding one in the fatigue recognition model to update the fatigue recognition model.

7. A fatigue driving monitoring device based on cloud-based intelligent learning, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The system acquires monitoring video data of the driver and sends it to the cloud; wherein the monitoring video data includes video of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, and the cloud database is used to store the monitoring video data; The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein, the multiple facial features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump from the boundary frequency; The multiple facial features are input as feature parameters into the fatigue recognition model; The hidden layer of the fatigue recognition model processes the multiple facial features to determine a first weight of the multiple facial features, wherein the first weight is related to the time of occurrence of the multiple facial features; Based on the multiple facial features, a preset weighted matching table is used to obtain the second weight of different facial features; The driver's fatigue value is calculated by multiplying the first weight of the multiple facial features by the second weight of the different facial features; The driver's fatigue level is compared with a preset multi-level fatigue threshold to determine the degree of driver fatigue. An early warning is issued based on the fatigue level, and the fatigue level is sent to the cloud database; The cloud computing module processes the cloud database and iteratively calculates the preset training recognition model within the cloud computing module to obtain the deviation of the training recognition model and outputs the training recognition model with the smallest deviation; wherein, the training recognition model and the fatigue recognition model are the same type of neural network model; The training recognition model with the smallest deviation is processed based on a preset reverse calculation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest deviation. The fatigue recognition model is updated by processing the weight factors of each neuron based on a preset update algorithm.

8. A non-volatile computer storage medium for fatigue driving monitoring based on cloud-based intelligent learning, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: The system acquires monitoring video data of the driver and sends it to the cloud; wherein the monitoring video data includes video of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, and the cloud database is used to store the monitoring video data; The system acquires monitoring video data of the driver and sends it to the cloud; wherein the monitoring video data includes video of the driver's face and head, and the cloud includes a cloud database and a cloud computing module, and the cloud database is used to store the monitoring video data; The monitoring video data is processed based on a preset image preprocessing algorithm to extract multiple facial features of the driver; wherein, the multiple facial features include yawning frequency, blinking frequency, eyelid closure time, pupil constriction, gaze deviation from the boundary, and gaze jump from the boundary frequency; The multiple facial features are input as feature parameters into the fatigue recognition model; The hidden layer of the fatigue recognition model processes the multiple facial features to determine a first weight of the multiple facial features, wherein the first weight is related to the time of occurrence of the multiple facial features; Based on the multiple facial features, a preset weighted matching table is used to obtain the second weight of different facial features; The driver's fatigue value is calculated by multiplying the first weight of the multiple facial features by the second weight of the different facial features; The driver's fatigue level is compared with a preset multi-level fatigue threshold to determine the degree of driver fatigue. An early warning is issued based on the fatigue level, and the fatigue level is sent to the cloud database; The cloud computing module processes the cloud database and iteratively calculates the preset training recognition model within the cloud computing module to obtain the deviation of the training recognition model and outputs the training recognition model with the smallest deviation; wherein, the training recognition model and the fatigue recognition model are the same type of neural network model; The training recognition model with the smallest deviation is processed based on a preset reverse calculation algorithm to obtain the weight factor of each neuron in the training recognition model with the smallest deviation. The fatigue recognition model is updated by processing the weight factors of each neuron based on a preset update algorithm.

Images