Early warning method, device, equipment and readable storage medium for vehicle fatigue driving

Abstract

The present invention provides a warning method, device, equipment and readable storage medium for vehicle fatigue driving, which relates to the technical field of automotive safety warning. The method includes determining the facial image information of the driver based on the first video information; acquiring the lane line image information and the steering wheel rotation angle information; determining the risk coefficient of the moving vehicle based on the facial image information of the driver, the lane line image information and the steering wheel rotation angle information; and generating a warning signal when it is determined that the risk coefficient reaches a first preset threshold. The beneficial effect of the present invention is to judge the safety coefficient according to the facial image information of the driver, the lane line image information and the steering wheel rotation angle information, so as to give a speed warning to the driver, effectively avoid the occurrence of speeding behavior caused by driver fatigue or inattention, thereby improving the safety level and driving experience of vehicle driving, avoiding or reducing the occurrence of road traffic accidents, and reducing the accident rate.

Description

Technical Field

[0001] The present invention relates to the technical field of automotive safety warning, and more particularly to a warning method, device, equipment and readable storage medium for vehicle fatigue driving. Background Art

[0002] With the improvement of living standards, the popularity rate of automobiles is getting higher and higher, and the driving experience, comfort, safety and intelligent driving assistance functions of automobiles are receiving more and more attention. With the increase in the number of automobiles, the incidence of vehicle accidents is also gradually increasing, and accidents are often highly correlated with the fatigue driving of drivers.

[0003] With the increase in speed or continuous high-speed driving, drivers will experience varying degrees of driving fatigue. Drowsiness caused by fatigue during driving is deeply felt by most drivers and passengers, especially those who often drive on highways. When driving on highways, the road environment is monotonous, traffic interference is low, the speed is stable, and the noise and vibration frequencies during driving are small, which easily makes drivers feel monotonous and sleepy. The harm of traffic accidents caused by fatigue driving is huge. Therefore, fatigue driving is one of the main culprits of most traffic accidents, and the demand for detecting the state of drivers in the vehicle to prevent fatigue driving is increasing day by day. Summary of the Invention

[0004] The purpose of the present invention is to provide a warning method, device, equipment and readable storage medium for vehicle fatigue driving to improve the above problems. To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0005] In a first aspect, the present application provides a warning method for vehicle fatigue driving, including:

[0006] Based on the first video information, determine the facial image information of the driver; the first video information includes a 15s video recording of the driver's face.

[0007] Obtain the lane line image information and the steering wheel angle information; the lane line image information includes an image in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes an image in which the frequency of steering wheel operation correction decreases or large instantaneous corrections appear alternately.

[0008] Based on the facial image information of the driver, the lane line image information and the steering wheel angle information, determine the risk coefficient of the moving vehicle.

[0009] When it is determined that the risk coefficient reaches a first preset threshold, generate a warning signal.

[0010] Preferably, the determining the facial image information of the driver based on the first video information includes:

[0011] Extract key frame images from the first video information, and perform anomaly detection based on the extracted key frame images. If the anomaly detection result is normal, determine the key frame image as the facial image information of the driver. Among them, the key frame images include images in which the driver's eyes have an increased closing time or an increased number of closures after entering a fatigued state;

[0012] Among them, the process of performing anomaly detection on the key frame images includes: extracting the video content within a preset time period before and after the key frame images and marking the video segments within the video content, and sorting the marked videos in the order of acquisition time before and after to obtain the sorted video segments;

[0013] Traverse the sorted video segments, and determine whether the sorted video segments are the same as the preset video segment sequence of the driver. If they are different, mark the key frame images corresponding to the sorted video segments as abnormal. If they are the same, mark the key frame images corresponding to the sorted video segments as normal.

[0014] Preferably, the extraction of key frame images from the first video information includes:

[0015] Perform preprocessing on the facial images in the first video information using median filtering to obtain a preprocessed first image;

[0016] Use the Otsu thresholding method to perform face detection on the first image to locate the specific facial image of the driver;

[0017] Based on the specific facial image, establish an interested region for the driver's eyes, and use morphological filtering and connected component labeling algorithms to extract and calculate the contour of the eyes to obtain pupil features;

[0018] Use the Kalman filtering method to track the pupil features, detect the blink frequency through the pupils, and then determine whether the driver has entered a fatigued state;

[0019] Determine the images after the driver enters a fatigued state as the key frame images.

[0020] Preferably, the use of the Otsu thresholding method to perform face detection on the first image to locate the specific facial image of the driver includes:

[0021] Select a threshold based on the gray level difference measure of the first image using the Otsu thresholding method;

[0022] Perform binarization processing on the first image based on the threshold, and the formula is as follows:

[0023] pi = n i / N, i = 0, 1, 2, …, L-1

[0024]

[0025] Where N is the number of image pixels, the gray level range is [0, T-1], and the number of pixels corresponding to the gray level i is n i ;

[0026] The calculation formula for the maximum inter-class variance threshold segmentation method is:

[0027] σ 2 B = w 0 (u 0 - u T ) 2 + w 1 (u 1 - u T ) 2 = w 1 w 0 (u 0 - u 1 ) 2

[0028] Where W 0 and W 1 are the probabilities of the occurrence of class C 0 and class C 1 respectively, the mean value is U T , the mean value of C 0 is U 0 , the mean value of C 1 is U 1 ; T takes values in the range of [0, T-1] in turn, so that the maximum value of σ 2 B is the optimal threshold of the maximum inter-class variance threshold segmentation method.

[0029] Preferably, based on the specific facial image, an interested region of the driver's eyes is established, and morphological filtering and connected region labeling algorithms are used to extract and calculate the contour of the eyes to obtain pupil features; the Kalman filtering method is used to track the pupil features, and the blinking frequency is detected through pupil detection, and then it is judged whether the driver is in a fatigued state, including:

[0030] Use the Canny edge detection algorithm to detect the pupil features to obtain the edge of the pupil area;

[0031] Based on the least squares ellipse fitting, operate on the edge of the pupil area to determine the optimal parameter solution vector of the pupil and obtain the state of the driver's line of sight;

[0032] Based on the state within the driver's line of sight, determine the threshold T for judging the open / closed state of the driver's eyes h as follows:

[0033]

[0034] wherein, H b is the height of the upper and lower eyelids when the driver's eyes are closed obtained through historical training and learning, and H 2 is the height of the upper and lower eyelids when the eyes are open obtained through learning, and W b is the width between the inner and outer corner points of the eyes when the eyes are closed obtained through learning, and W 2 is the width between the inner and outer corners of the eyes when the eyes are open obtained through learning, W is the currently detected width between the inner and outer corner points of the eyes, and H is the currently detected height of the upper and lower eyelids;

[0035] According to H and T h , determine whether the driver enters the fatigued state.

[0036] Preferably, the determining the risk coefficient of the traveling vehicle based on the driver's facial image information, the lane line image information, and the steering wheel rotation angle information includes:

[0037] Preprocess the driver's facial image information, the lane line image information, and the steering wheel rotation angle information to determine the preprocessing result;

[0038] Collect the current driving state information of the vehicle, and determine the risk coefficient of the vehicle driving according to the driving state information and the preprocessing result.

[0039] Preferably, when it is determined that the risk coefficient reaches the first preset threshold, generating a warning signal includes:

[0040] When it is determined that the risk coefficient reaches the first preset threshold, obtain the historical risk coefficients of all vehicle models; the historical risk coefficients include the number of warning times, the number of accident occurrences, and the distance between the section where the warning occurs and the current section within the current section range;

[0041] Perform a weighted sum of the historical risk coefficients of all vehicle models to obtain the result of the weighted sum, and determine whether the result is greater than the second preset threshold. If the result is greater than the second preset threshold, generate a warning signal.

[0042] In a second aspect, the present application further provides a warning device for vehicle fatigue driving, including a driver facial image information determination module, an acquisition module, a risk coefficient determination module, and a warning signal generation module, wherein:

[0043] Driver's Facial Image Information Determination Module: Used to determine the driver's facial image information based on the first video information; the first video information includes a 15-second video recording of the driver's face.

[0044] Acquisition Module: Used to acquire lane line image information and steering wheel angle information; the lane line image information includes an image in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes an image in which the frequency of steering wheel operation correction decreases or large instantaneous corrections alternate.

[0045] Hazard Coefficient Determination Module: Used to determine the hazard coefficient of the moving vehicle based on the driver's facial image information, the lane line image information, and the steering wheel angle information.

[0046] Warning Signal Generation Module: Used to generate a warning signal when it is determined that the hazard coefficient reaches the first preset threshold.

[0047] Thirdly, the present application also provides a warning device for vehicle fatigue driving, including:

[0048] A memory, used to store a computer program;

[0049] A processor, used to implement the steps of the warning method for vehicle fatigue driving when executing the computer program.

[0050] Fourthly, the present application also provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above warning method based on vehicle fatigue driving.

[0051] The beneficial effects of the present invention are:

[0052] The present invention uses an image sensor to acquire the driver's facial image information and lane line image information, uses a corner sensor to acquire the steering wheel information, extracts feature vectors, and combines data validity judgment and multi-source information fusion together, deeply excavates the information source data and the complementary verification between each information source, realizes reliable identification of the driver's state, and when the system determines that the driver enters a fatigue state, it will automatically give a warning to the driver from all aspects according to the current fatigue degree of the driver.

[0053] The present invention uses the maximum inter-class variance threshold segmentation method to measure the gray difference of the image and select the threshold, which is simple and has a fast processing speed, improving the efficiency of image processing.

[0054] The present invention utilizes the positional distribution relationship of the facial features of a human face to establish an area of interest for the driver's eyes for calculation. By using the connected region labeling algorithm to label each region within the area of interest of the eyes, eye positioning is obtained using constraint conditions, and the eye contour is extracted for ellipse fitting, providing a good basis for the driver's line-of-sight estimation, mental distraction, and fatigue detection.

[0055] The present invention uses the Canny edge detection algorithm to detect the pupil features, and after the pupil region edge extraction, the pupil region contour in the obtained image is more obvious, and interference is effectively removed, facilitating subsequent pupil center positioning.

[0056] The present invention determines the risk coefficient based on the driver's facial image information, lane line image information, and steering wheel rotation angle information, thereby giving a speed warning to the driver, effectively avoiding speeding behavior caused by driver fatigue or inattention, and then improving the driving safety level and driving experience of the vehicle, avoiding or reducing the occurrence of road traffic accidents, and reducing the accident rate.

[0057] Other features and advantages of the present invention will be described in the subsequent specification, and part of them will become obvious from the specification, or can be understood by implementing the embodiments of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0059] Figure 1 It is a schematic flow chart of the vehicle fatigue driving warning method described in the embodiments of the present invention;

[0060] Figure 2 It is a schematic structural diagram of the vehicle fatigue driving warning device described in the embodiments of the present invention;

[0061] Figure 3 It is a schematic structural diagram of the vehicle fatigue driving warning equipment described in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and illustrated herein generally can be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention provided herein is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0063] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0064] Embodiment 1:

[0065] This embodiment provides a warning method for vehicle fatigue driving.

[0066] See Figure 1 , which shows that this method includes step S100, step S200, step S300 and step S400.

[0067] S100. Based on the first video information, determine the facial image information of the driver; the first video information includes a 15-second video recording of the driver's face.

[0068] It can be understood that in this step S100, S101, S102 and S103 are included, where:

[0069] S101. Extract key frame images from the first video information, and perform anomaly detection based on the extracted key frame images. If the anomaly detection result is normal, determine the key frame image as the facial image information of the driver, where the key frame image includes an image in which the driver's eyes have an increased closing time or an increased number of closures after entering a fatigued state;

[0070] It should be noted that in order to evaluate the detection accuracy of the fatigue detection algorithm, it is evaluated by the method of facial video. The driver's facial video will be segmented into video segments of 15 seconds, selected according to the video recording, and the face will be scored.

[0071] Furthermore, the key-frame images are actually extracted based on the fatigue features of the eye state, the fatigue features of the steering wheel, and the fatigue features of the vehicle driving trajectory. Among them, based on the statistical analysis of the driver's facial video, PERCLOS and the longest closed-eye time MCD can be used as the discrimination criteria for fatigue. As the fatigue level of the driver increases, both PERCLOS and MCD show an increasing trend.

[0072] In this step, an image sensor is used to obtain the driver's facial image information and lane line image information, and a steering wheel sensor is used to obtain the steering wheel information. Feature vectors are extracted, and data validity judgment and multi-source information fusion are combined to deeply explore the complementary verification between the information source data and each information source, so as to realize the reliable identification of the driver's state. When the system determines that the driver is in a fatigue state, it will automatically give an early warning to the driver from all aspects according to the current fatigue level of the driver.

[0073] S102. Among them, the process of performing anomaly detection on the key-frame image includes: extracting the video content within a preset time period before and after the key-frame image and marking the video segments within the video content, and sorting the marked videos in the order of acquisition time before and after to obtain the sorted video segments.

[0074] S103. Traverse the sorted video segments, and determine whether the sorted video segments are the same as the preset video segment sequence of the driver. If they are different, mark the key-frame image corresponding to the sorted video segment as abnormal; if they are the same, mark the key-frame image corresponding to the sorted video segment as normal.

[0075] It should be noted that by sorting each video segment, it is possible to determine the specific time period during which the driver is fatigued while driving, and then determine whether there are other factors causing fatigue during the fatigued driving time period, such as being in a single lane for a long time or feeling sleepy due to the noon sun shining, etc. The facial fatigue levels in different time periods are classified, and different early warning responses can be made according to different levels.

[0076] In this embodiment, the facial fatigue degree of the driver can be divided into four levels: the first level, the second level, the third level, and the fourth level. The first level means that there are no fatigue characteristics on the driver's face. The second level means that there are slight fatigue characteristics on the driver's face. The third level means that the driver is inattentive while driving and the line of sight deviates from the road surface for too long, etc. The fourth level means that the driver has obvious fatigue characteristics. If the comparison between the first video information and the key frame image mark does not match, it is marked as abnormal. If it matches, it is marked as normal, and the first video information corresponding to the video segment marked as normal is sent to the subsequent processing steps to prepare data for establishing a fatigue model later.

[0077] It should be noted that step S101 includes step S1011, step S1012, step S1013, and step S1014, where:

[0078] S1011. Preprocess the facial image in the first video information by using median filtering to obtain the preprocessed first image;

[0079] In this embodiment, considering that the driver will drive at night or during the day, in the case of insufficient light, images cannot be collected. Therefore, infrared imaging technology can be used. The collected face images will inevitably be interfered by noise and must be preprocessed before image processing to eliminate the noise and minimize the impact of noise. The image can be preprocessed by using smoothing filtering. The purpose of filtering has two aspects. One is to smooth the non-edge areas of the image, and the other is to protect the edges of the image.

[0080] S1012. Use the Otsu threshold segmentation method to perform face detection on the first image to locate the specific facial image of the driver;

[0081] It should be noted that step S1012 includes step S10121, step S10122, and step S10123, where:

[0082] S10121. Select a threshold based on the gray difference metric of the first image by using the Otsu threshold segmentation method;

[0083] S10122. Perform binarization processing on the first image based on the threshold, and the formula is as follows: p i =n i / N, i = 0, 1, 2,..., L - 1

[0084]

[0085] where N is the number of image pixels, the gray level range is [0, T - 1], and the number of pixels corresponding to the gray level i is n i ;

[0086] The image is binarized by this method to obtain a probability distribution.

[0087] S10123. The calculation formula of the maximum inter-class variance threshold segmentation method is:

[0088] σ 2 B = w 0 (u 0 - u T ) 2 + w 1 (u 1 - u T ) 2 = w 1 w 0 (u 0 - u 1 ) 2

[0089] Among them, W 0 and W 1 are the probabilities of the occurrence of class C 0 and class C 1 respectively. The mean value is U T , the mean value of C 0 is U 0 , and the mean value of C 1 is U 1 ; T takes values in turn in the range of [0, T - 1], so that the T value that makes σ 2 B the largest is the optimal threshold of the maximum inter-class variance threshold segmentation method.

[0090] It should be noted that when using the maximum inter-class variance threshold segmentation method to select the threshold for measuring the gray difference of the image, it is simple and has a fast processing speed, which improves the efficiency of image processing.

[0091] S1013. Based on the specific facial image, an interested region of the driver's eyes is established. The morphological filtering and connected region labeling algorithms are used to extract and calculate the contour of the eyes to obtain pupil features; the Kalman filtering method is used to track the pupil features, and the blinking frequency is detected through the pupil to further determine whether the driver is in a fatigued state;

[0092] It should be noted that monitoring the state of the driver's eyes is the main way to detect driver fatigue, drowsiness, and distraction. It is necessary to use the positional distribution relationship of the facial features of the human face to establish the region of interest (ROI) of the driver's eyes for calculation. By using the connected component labeling algorithm to label each region within the ROI of the eyes, and using the constraint conditions to obtain eye positioning, the eye contour is extracted for elliptical fitting, providing a good basis for driver gaze estimation, mental distraction, and fatigue detection.

[0093] In this embodiment, the height of the human face is set as H, the height of the region of interest can be set as half of the height of the human face, i.e., H / 2, and the width is W. Morphological filtering uses a structuring element with a certain shape to measure and extract the corresponding shape in the image for the purpose of image analysis and recognition. This includes but is not limited to using erosion and dilation in morphological operations.

[0094] The calculation formula for erosion is as follows:

[0095]

[0096] Where S is the structuring element, B is the original binary image, E is the set of (x, y) that makes up the image after erosion, and S is included in B. Erosion eliminates all boundary points of an object, removes objects smaller than the structuring element from a segmented image, and by selecting structuring elements of different sizes, objects of different sizes in the image can be removed.

[0097] The calculation formula for dilation is as follows:

[0098]

[0099] Where S is the structuring element, B is the original binary image, D is the set of (x, y) that makes up the image after dilation. If the origin of S is displaced to the point (x, y), then its intersection with B is non-empty. Dilation is the process of merging all background points in contact with an object into that object.

[0100] In this step, the connected component labeling algorithm is used to divide the white pixel regions of the binary image into several independent regions according to connectivity. For example, the connected component labeling algorithm analyzes the connection situation between elements in the image, determines which elements are the same object, forms a connected region, and then searches for the elements it connects with the seed element as the center, and labels these elements to form a connected region.

[0101] The Kalman filtering method is used to track the pupil features, and the blink frequency is detected through pupil detection, and then it is judged whether the driver enters the fatigue state;

[0102] It should be noted that assume (x t , y trepresents the pixel position of the center point of the facial area in the image at time t, (u x , v y ) respectively represent the velocities of the center points of the facial and eye regions in the x and y directions in the image at time t. The facial and eye state vectors in the image at time t can be expressed as:

[0103] X t =(x t , y t , u t , v t ) T

[0104] According to the Kalman filtering theory, the facial state vector X t+1 in the image at time t + 1 is linearly related to the current state X t . The system equation is as follows:

[0105] X t+1 =A t X t +W t

[0106] In the formula, A t is the state transition matrix, W t represents the observation matrix, and the zero-mean Gaussian distribution W t ∝N(0, Q). The covariance matrix of W t is denoted as Q.

[0107] It should be noted that step S1013 further includes step S10131, step S10132, step S10133, and step S10134, where:

[0108] S10131. Detect the pupil feature using the Canny edge detection algorithm to obtain the edge of the pupil region;

[0109] It should be noted that the pupil is first subjected to Gaussian smoothing processing, and then the edge of the image is determined by the maximum value of the first-order differential. The calculation formula is as follows:

[0110]

[0111] In the formula, σ is the width of Gaussian filtering. The larger the value of σ, the better the smoothing effect.

[0112] Calculate the partial derivative vectors (G x (x, y), G y(x, y)), the gradient magnitude and gradient direction; divide the edge directions into 0 degrees, 45 degrees, 90 degrees, and 135 degrees, find all adjacent pixels of this pixel along these four edge directions, and filter out non-edge points. Use the double-threshold algorithm to determine and connect the edge points of the image. If there is a pixel with a gray value greater than the high threshold among all adjacent pixels of this pixel, then this pixel is considered to be an edge point of the image, otherwise it is not.

[0113] S10132. Based on the least squares ellipse fitting, operate on the edge of the pupil region to determine the optimal parameter solution vector of the pupil, and obtain the state of the driver's line of sight.

[0114] In this step, the least squares ellipse fitting is used to extract a sub-region centered on the pupil center and with a size of 40*40 in the image, regarded as the initial region, and perform binarization processing. An iterative method is adopted to filter out the non-Purkinje spot region. The shape of the Purkinje spot region is approximately circular. Therefore, the aspect ratio L / W of the minimum circumscribed rectangle of this region is approximately 1. Process the edge of the pupil region to obtain the optimal solution vector. The minor axis of the ellipse is the required upper and lower eyelid heights. Therefore, in this step, the Canny edge detection algorithm and the least squares ellipse fitting are used to detect the pupil characteristics, and the pupil region contour in the image obtained after the edge extraction of the pupil region is more obvious, and the interference is effectively removed, which is convenient for subsequent pupil center positioning.

[0115] S10133. Based on the state of the driver's line of sight, judge the threshold T for the open / closed state of the driver's eyes h is:

[0116]

[0117] where, H b is the height of the upper and lower eyelids when the driver closes his eyes obtained through historical training and learning, H 2 is the height of the upper and lower eyelids when the driver opens his eyes obtained through learning, W b is the width between the inner and outer corner points when the driver closes his eyes obtained through learning, W 2 is the width between the inner and outer corners when the driver opens his eyes obtained through learning, W is the width between the currently detected inner and outer corner points, and H is the currently detected height of the upper and lower eyelids;

[0118] If H is less than or equal to Th, it is considered that the eyes are closed at this time; otherwise, it is considered that the eyes are open. Based on this, it is possible to determine whether the driver's eyes are closed in each frame of the video or image, and then obtain the duration of the driver's eyes remaining closed during a period of driving. In historical experience, when the driver is in a waking state, the duration of the eyes remaining closed generally does not exceed 0.5 seconds. If it is detected that the duration of the eyes remaining closed exceeds the threshold, the driver may be fatigued.

[0119] S10134. Determine whether the driver is in a fatigued state according to H and T h , and judge whether the driver enters a fatigued state.

[0120] S1015. Determine the image after the driver enters the fatigued state as the key frame image.

[0121] Furthermore, lock the image after the driver enters the fatigued state as the key frame image for more convenient extraction and processing of the key frame image later.

[0122] S200. Obtain lane line image information and steering wheel angle information; the lane line image information includes images in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes images in which the frequency of steering wheel operation correction decreases or large instantaneous corrections appear alternately.

[0123] It should be noted that other sensors installed on the vehicle include but are not limited to measuring the distance between the vehicle and obstacles, vehicle speed, acceleration, pitch angle, and heading angle, etc. Among them, the lane line image information includes images in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes images in which the frequency of steering wheel operation correction decreases or large instantaneous corrections appear alternately. "Large instantaneous correction" means that the single correction amplitude of the steering wheel is greater than 5 degrees within 3 seconds.

[0124] It should be noted that when the driver is in a waking state, the driver will control the vehicle by frequently making small corrections to the steering wheel. However, when entering a fatigued state, there are characteristics of a decrease in the frequency of steering wheel operation correction or the alternate appearance of large instantaneous corrections.

[0125] In this embodiment, during the two time windows from 1 to 2 o'clock at noon or from 11 o'clock at night to 1 o'clock in the early morning when the driver is more likely to be sleepy and fatigued, the proportion of the time when the angular velocity of the steering wheel is less than the set threshold reflects the phenomenon of reduced steering wheel correction frequency, and the angular standard deviation reflects the magnitude of the angle fluctuation within the time window. The zero-speed percentage is used to distinguish between the "awake" and "fatigued" states. When the driver enters the fatigued state, the control ability of the vehicle decreases, and then the lane driving trajectory will show an "S" shape. The standard deviation of the lateral position of the vehicle can reflect the degree of lateral fluctuation of the vehicle driving trajectory within a period of time, and thus can distinguish between the "awake" and "fatigued" states.

[0126] S300. Determine the risk coefficient of the moving vehicle based on the facial image information of the driver, the lane line image information, and the steering wheel rotation angle information.

[0127] It should be noted that step S300 includes S301 and S302, where:

[0128] Perform preprocessing on the facial image information of the driver, the lane line image information, and the steering wheel rotation angle information to determine the preprocessing result.

[0129] Furthermore, during the preprocessing of the facial image information of the driver, the lane line image information, and the steering wheel rotation angle information, it includes preprocessing such as image enhancement, image sharpening, image smoothing, and denoising on the determined key frame images, the images with an "S" - shaped lane driving trajectory, and the images with alternately reduced steering wheel operation correction frequency or instant large - scale correction, eliminating irrelevant information in the images, filtering out interference and noise, restoring useful real information, enhancing the detectability and authenticity of relevant information, simplifying the data to the greatest extent, thereby improving the reliability of feature extraction, not only making the images clearer visually, but also making the images more conducive to processing and recognition.

[0130] Collect the current driving state information of the vehicle, and determine the risk coefficient of the vehicle driving based on the driving state information and the preprocessing result.

[0131] It should be noted that the current driving state information of the vehicle includes information related to the vehicle's driving in the current state. For example, the current geographical environment information of the vehicle, weather and meteorological information, and the continuous driving duration of the driver, etc.; among them, the geographical environment information includes the altitude of the current vehicle, the road type of the vehicle, and the center line curvature radius of the road, etc., whether the continuous driving duration of the driver exceeds four hours and takes at least a 20-minute break, whether to change drivers to drive in turns, etc.; the weather and meteorological information includes the precipitation, wind speed of the weather in the current vehicle driving state, and whether there are extreme disaster weather conditions, etc. Score the driving state information according to a preset standard to determine the risk coefficient level, and based on the preprocessing result, comprehensively judge the risk coefficient of the vehicle driving. For example, the risk coefficient of the altitude in the geographical environment information being 0 - 500 meters is 1, and the risk coefficient of 500 - 3000 meters is 2, score the risk coefficient of the driving vehicle and comprehensively evaluate the result.

[0132] S400. When it is determined that the risk coefficient reaches the first preset threshold, generate a warning signal.

[0133] It should be noted that step S400 includes S401 and S402, where:

[0134] S401. When it is determined that the risk coefficient reaches the first preset threshold, obtain the historical risk coefficients of all vehicle models; the historical risk coefficients include the number of warning times, the number of accident occurrences, and the distance between the section where the warning occurs and the current section within the current section range.

[0135] It should be noted that the historical risk coefficients are any one or more, that is, the number of fatigue warning times, the number of accident occurrences, and the distance between the section where the fatigue warning occurs and the current section within the current section range. The cycle can be one year or one month, without limitation. All vehicle models include large trucks, shuttle buses, oil tank trucks, etc. The historical accidents, fatigue warnings, and the number of fatigue warning occurrences within this cycle. The more accidents, the greater the historical risk coefficient.

[0136] S402. Perform a weighted sum of the historical risk coefficients of all vehicle models to obtain the result of the weighted sum, and determine whether the result is greater than the second preset threshold. If the result is greater than the second preset threshold, generate a warning signal.

[0137] It should be noted that according to the weighted ratio summation, multiplication is performed based on the importance coefficients (i.e., weights) of each data and then summed up. The ratio of the weighted sum to the sum of all weights is equal to the weighted arithmetic mean. If the result is greater than the second preset threshold, when the danger coefficient reaches the established preset threshold, corresponding voice prompts or vehicle controls will be provided according to the driving coefficient and driving conditions, and the corresponding voice reminder will also change as the danger coefficient increases. The greater the danger coefficient, the shorter the frequency interval time of the voice reminder until continuous playback. In this embodiment, the voice prompts are as follows: "For driving safety, do not cross the line", "You may need to rest", "For your safety, please drive attentively", "Danger, danger", and the control schemes are as follows: The vehicle decelerates at a speed of 5 Km / h per second until the speed is reduced to 20 Km / h, and the vehicle decelerates at a speed of 15 Km / h per second until the speed is reduced to 20 Km / h, etc.

[0138] Embodiment 2:

[0139] This embodiment provides a warning device for vehicle fatigue driving. Refer to Figure 2 The device includes a driver's facial image information determination module 701, an acquisition module 702, a danger coefficient determination module 703, and a warning signal generation module 704, where:

[0140] The driver's facial image information determination module 701: is used to determine the driver's facial image information based on the first video information; the first video information includes a 15s video recording of the driver's face.

[0141] The acquisition module 702: is used to acquire lane line image information and steering wheel angle information; the lane line image information includes an image in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes an image in which the frequency of steering wheel operation correction decreases or large instantaneous corrections alternate.

[0142] The danger coefficient determination module 703: is used to determine the danger coefficient of the moving vehicle based on the driver's facial image information, the lane line image information, and the steering wheel angle information.

[0143] The warning signal generation module 704: is used to generate a warning signal when it is determined that the danger coefficient reaches the first preset threshold.

[0144] Embodiment 3:

[0145] Corresponding to the above method embodiment, this embodiment also provides a warning device for vehicle fatigue driving. A warning device for vehicle fatigue driving described below can be mutually referred to with the warning method for vehicle fatigue driving described above.

[0146] Figure 3It is a block diagram of a warning device 800 for vehicle fatigue driving shown according to an exemplary embodiment. As Figure 3 shown, the warning device 800 for vehicle fatigue driving includes: a processor 801 and a memory 802. The warning device 800 for vehicle fatigue driving further includes one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0147] Among them, the processor 801 is used to control the overall operation of the vehicle's fatigue driving warning device 800 to complete all or part of the steps in the above-mentioned vehicle fatigue driving warning method. The memory 802 is used to store various types of data to support the operation of the vehicle's fatigue driving warning device 800. These data may include, for example, instructions for any application or method operating on the vehicle's fatigue driving warning device 800, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The multimedia component 803 may include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal can be further stored in the memory 802 or sent through the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be a keyboard, a mouse, or buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the vehicle's fatigue driving warning device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them. Accordingly, the communication component 805 may include: a Wi-Fi module, a Bluetooth module, or an NFC module.

[0148] In an exemplary embodiment, the early warning device 800 for vehicle fatigue driving can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the above-mentioned early warning method for vehicle fatigue driving.

[0149] In another exemplary embodiment, there is also provided a computer-readable storage medium including program instructions, and when the program instructions are executed by a processor, the steps of the above-mentioned early warning method for vehicle fatigue driving are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions can be executed by the processor 801 of the early warning device 800 for vehicle fatigue driving to complete the above-mentioned early warning method for vehicle fatigue driving.

[0150] Embodiment 4:

[0151] Corresponding to the above method embodiment, in this embodiment, there is also provided a readable storage medium, and a readable storage medium described below can be correspondingly referred to with the above-mentioned early warning method for vehicle fatigue driving.

[0152] A computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the early warning method for vehicle fatigue driving in the above method embodiment are implemented.

[0153] Specifically, the readable storage medium can be various readable storage media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc that can store program codes.

[0154] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0155] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. Warning method for vehicle fatigue driving, characterized in that, comprising: Determining the facial image information of the driver based on the first video information; The first video information includes a 15s video recording of the driver's face; Obtaining lane line image information and steering wheel angle information; the lane line image information includes an image in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes an image in which the frequency of steering wheel operation correction decreases or large instantaneous corrections alternate; Determining the risk coefficient of the moving vehicle based on the driver's facial image information, the lane line image information and the steering wheel angle information; When it is determined that the risk coefficient reaches a first preset threshold, generating a warning signal; When it is determined that the risk coefficient reaches the first preset threshold and generating a warning signal, includes: When it is determined that the risk coefficient reaches the first preset threshold, obtaining the historical risk coefficients of all vehicle models; the historical risk coefficients include the number of warning times, the number of accidents, and the distance between the section where the warning occurred and the current section within the current section range; Performing weighted summation on the historical risk coefficients of all vehicle models to obtain the result of the weighted summation, and determining whether the result is greater than a second preset threshold. If the result is greater than the second preset threshold, generating a warning signal.

2. The warning method for vehicle fatigue driving according to claim 1, characterized in that , the determining the facial image information of the driver based on the first video information includes: Extracting key frame images from the first video information, and performing anomaly detection based on the extracted key frame images. If the anomaly detection result is normal, determining the key frame image as the facial image information of the driver, wherein the key frame image includes an image in which the closing time of the driver's eyes increases or the number of closings increases after entering the fatigue state; Among them, the process of performing anomaly detection on the key frame image includes: extracting the video content within a preset time period before and after the key frame image and marking the video segments within the video content, and sorting the marked videos in the order of acquisition time before and after to obtain the sorted video segments; Traversing the sorted video segments, determining whether the sorted video segments are the same as the preset video segment sequence of the driver. If they are different, marking the key frame image corresponding to the sorted video segment as abnormal, and if they are the same, marking the key frame image corresponding to the sorted video segment as normal.

3. The warning method for vehicle fatigue driving according to claim 2, characterized in that , in the extracting key frame images from the first video information, includes: Performing preprocessing on the facial image in the first video information by using median filtering to obtain a preprocessed first image; Using the maximum inter-class variance threshold segmentation method to perform face detection on the first image to locate the specific facial image of the driver; Based on the specific facial image, an interested region of the driver's eyes is established. Using morphological filtering and connected region labeling algorithms, the contour of the eyes is extracted and calculated to obtain pupil features. The Kalman filtering method is used to track the pupil features, and the blinking frequency is detected through the pupils, and then it is judged whether the driver enters the fatigue state; The image after the driver enters the fatigue state is determined as the key frame image.

4. The vehicle fatigue driving warning method according to claim 3, characterized in that , the use of the maximum inter-class variance threshold segmentation method to perform face detection on the first image and locate the specific facial image of the driver includes: Based on the maximum inter-class variance threshold segmentation method, the gray difference measure of the first image is used to select a threshold; Based on the threshold, the first image is binarized, and the formula is as follows: where N is the number of image pixels, and the gray level range , and the number of pixels corresponding to the gray level i is ; The calculation formula of the maximum inter-class variance threshold segmentation method is: Among them, and are the probabilities of occurrence of classes and respectively, with the mean being , the mean of is the mean of ; T takes values in sequence within the range of , such that the maximum T value is the optimal threshold of the maximum between-class variance thresholding method.

5. The vehicle fatigue driving warning method according to claim 1, characterized in that , the determination of the risk coefficient of the driving vehicle based on the driver's facial image information, the lane line image information and the steering wheel angle information includes: Preprocess the driver's facial image information, the lane line image information and the steering wheel angle information to determine the preprocessing result; Collect the current driving state information of the vehicle, and determine the risk coefficient of the vehicle driving according to the driving state information and the preprocessing result.

6. A vehicle fatigue driving warning device, characterized in that it includes: Driver's facial image information determination module: used to determine the driver's facial image information based on the first video information; The first video information includes a 15s video recording of the driver's face; Obtaining module: used to obtain lane line image information and steering wheel angle information; the lane line image information includes an image in which the lane driving trajectory presents an "S" shape, and the steering wheel angle information includes an image in which the steering wheel operation correction frequency decreases or large instantaneous corrections alternate; Risk coefficient determination module: used to determine the risk coefficient of the driving vehicle based on the driver's facial image information, the lane line image information and the steering wheel angle information; Warning signal generation module: used to generate a warning signal when it is judged that the risk coefficient reaches a first preset threshold; The judgment of generating a warning signal when the risk coefficient reaches the first preset threshold includes: When it is judged that the risk coefficient reaches the first preset threshold, obtain the historical risk coefficients of all vehicle models; the historical risk coefficients include the number of warning times, the number of accidents, and the distance between the section where the warning occurs and the current section within the current section range; Perform weighted summation on the historical risk coefficients of all vehicle models to obtain the result of the weighted summation, and judge whether the result is greater than a second preset threshold. If the result is greater than the second preset threshold, generate a warning signal.

7. A vehicle fatigue driving warning device, characterized in that it includes: A memory for storing computer programs; A processor, configured to implement the steps of the warning method for vehicle fatigue driving according to any one of claims 1 to 5 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the warning method for vehicle fatigue driving according to any one of claims 1 to 5 are implemented.

Images