Driving state analysis method based on machine vision and related device

Abstract

The invention discloses a driving state analysis method based on machine vision and a related device, and relates to the technical field of image processing, and the method comprises the steps: carrying out the definition optimization of each frame of driver image; performing feature point extraction on each frame of driver image after definition optimization based on a calibration network to obtain corresponding feature point information; based on the feature point information, an eye closing adaptive threshold value is analyzed, and fatigue driving state detection of the driver is carried out in combination with micro-expression analysis; carrying out hand-held object detection on the driver based on each frame of driver image to obtain hand-held object information; performing sight liveness analysis based on the feature point information so as to perform distracted driving state detection in combination with the handheld object information; and judging whether early warning information needs to be sent based on the fatigue driving state detection result and the distracted driving state detection result. According to the method, false detection or missing detection of the abnormal driving state can be effectively avoided, and the reliability of driving state analysis is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a driving state analysis method based on machine vision and related device. BACKGROUND

[0002] Driver driving state analysis is a field based on advanced technology and data collection methods, aiming to evaluate and understand the behavior and decision-making of drivers on the road. Through sensor technology, big data analysis technology and machine learning technology, the system can identify the driving abnormal behavior of the driver according to the collected video or image, so that the driver can more consciously comply with traffic rules and maintain concentration. Driving state analysis mainly includes fatigue driving state detection and distraction driving state detection. At present, fatigue driving state detection is mainly performed by comparing the degree of eye closure with a fixed threshold, but this method is easily affected by individual differences of drivers, such as smaller eyes of the driver, which can cause false detection or missed detection. At the same time, the current distraction driving state detection is usually performed by identifying whether the user is holding an object, but this method cannot comprehensively detect the distraction driving state of the driver, such as the distraction state of the driver's line of sight attention shift without holding an object, which cannot be detected, resulting in failure to effectively send timely and accurate warning information to the driver. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art, and the present application provides a driving state analysis method based on machine vision and related device, which can effectively avoid the occurrence of driving abnormal state false detection or missed detection and improve the reliability of driving state analysis.

[0004] In order to solve the above technical problems, the present application provides a driving state analysis method based on machine vision, applied to a vehicle-mounted camera and a vehicle management and control terminal system; the method comprises:

[0005] The vehicle management and control terminal system optimizes the clarity of each frame of driver image collected by the vehicle-mounted camera to obtain each frame of driver image after clarity optimization;

[0006] Based on the calibration network, feature points are extracted from each frame of driver image after clarity optimization to obtain corresponding feature point information;

[0007] An eye closure adaptive threshold is analyzed based on the feature point information, and a fatigue driving state detection of the driver is performed based on the eye closure adaptive threshold combined with micro-expression analysis to obtain a fatigue driving state detection result;

[0008] Based on each frame of driver image after clarity optimization, a hand-held object detection of the driver is performed to obtain hand-held object information;

[0009] The line-of-sight activity is analyzed based on the feature point information to obtain line-of-sight activity information, and the line-of-sight activity information is combined with the handheld object information to detect a distracted driving state to obtain a distracted driving state detection result.

[0010] Whether a warning information needs to be sent is determined based on the fatigue driving state detection result and the distracted driving state detection result.

[0011] Optionally, the clarity of each frame of the driver image collected by the vehicle-mounted camera is optimized to obtain each frame of the driver image after clarity optimization, including:

[0012] The bicubic interpolation is performed on each frame of the driver image to obtain each frame of the enlarged driver image;

[0013] The first gradient matrix of each frame of the enlarged driver image is analyzed based on the Sobel algorithm, and the non-maximum suppression is performed on the first gradient matrix to obtain a target gradient matrix;

[0014] The threshold analysis is performed on each frame of the enlarged driver image based on the fireworks algorithm to obtain a high-low double threshold;

[0015] The edge pixel points of each frame of the enlarged driver image are determined based on the high-low double threshold and the target gradient matrix;

[0016] The second gradient matrix is determined based on the normal operator of a plurality of directions, and the clarity of each frame of the driver image is optimized based on the edge pixel points and the second gradient matrix to obtain each frame of the driver image after clarity optimization.

[0017] Optionally, the feature points are extracted from each frame of the driver image after clarity optimization based on the calibration network to obtain corresponding feature point information, including:

[0018] The feature points are positioned based on the calibration network on each frame of the driver image after clarity optimization to obtain a feature point positioning result;

[0019] The feature points are extracted from each frame of the driver image after clarity optimization based on the feature point positioning result and a preset rectangular frame to obtain corresponding feature point information.

[0020] Optionally, the eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold and micro-expression analysis to obtain a fatigue driving state detection result, including:

[0021] The left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame are analyzed based on the feature point information;

[0022] The pupil detection is performed based on the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame to obtain pupil region information;

[0023] calculate a pupil height-width ratio based on the pupil region information, and analyze an eye closure adaptive threshold based on the pupil height-width ratio;

[0024] generate a face texture feature based on the feature point information, and perform fatigue micro-expression feature matching in a micro-expression library based on the face texture feature to obtain fatigue micro-expression feature information;

[0025] detect a fatigue driving state of the driver based on the pupil height-width ratio, the eye closure adaptive threshold, and the fatigue micro-expression feature information, and obtain a fatigue driving state detection result.

[0026] Optionally, the driver's handheld object detection based on the driver image with optimized clarity of each frame is performed to obtain handheld object information, including:

[0027] hand region detection is performed on the driver image with optimized clarity of each frame to obtain a corresponding hand region image;

[0028] The driver's handheld object detection is performed based on the hand region image using an object recognition model to obtain handheld object information.

[0029] Optionally, the gaze activity analysis based on the feature point information is performed to obtain gaze activity information, and the distraction driving state detection is performed based on the gaze activity information in combination with the handheld object information to obtain a distraction driving state detection result, including:

[0030] The driver's gaze concentration area and gaze concentration time are determined based on the feature point information;

[0031] The gaze activity analysis is performed based on the gaze concentration area and the gaze concentration time to obtain gaze activity information;

[0032] The head posture data is determined based on the head contour reference point;

[0033] The first distance between the handheld object and the mouth and the second distance between the handheld object and the eye are calculated based on the handheld object information and the feature point information;

[0034] The distraction driving state detection is performed based on the gaze activity information, the head posture data, the first distance, and the second distance to obtain a distraction driving state detection result.

[0035] Optionally, the determination of whether to issue a warning information based on the fatigue driving state detection result and the distraction driving state detection result, including:

[0036] The warning level and the warning voice information are determined based on the fatigue driving state detection result and the distraction driving state detection result;

[0037] The driving abnormality warning is performed to the driver based on the warning level and the warning voice information.

[0038] In addition, the application further provides a driving state analysis device based on machine vision, which is applied to a vehicle-mounted camera and a vehicle management and control terminal system, and the device comprises:

[0039] An image optimization module is configured to perform definition optimization on each frame of driver image collected by the vehicle-mounted camera by the vehicle management and control terminal system, so as to obtain each frame of driver image after definition optimization.

[0040] A feature point extraction module is configured to perform feature point extraction on each frame of driver image after definition optimization based on a calibration network, so as to obtain corresponding feature point information.

[0041] A fatigue driving detection module is configured to analyze an eye closure adaptive threshold based on the feature point information, and perform fatigue driving state detection of the driver based on the eye closure adaptive threshold and micro-expression analysis, so as to obtain a fatigue driving state detection result.

[0042] A handheld object detection module is configured to perform handheld object detection of the driver based on each frame of driver image after definition optimization, so as to obtain handheld object information.

[0043] A distraction driving detection module is configured to perform gaze activity analysis based on the feature point information, so as to obtain gaze activity information, and perform distraction driving state detection based on the gaze activity information and the handheld object information, so as to obtain a distraction driving state detection result.

[0044] A warning judgment module is configured to judge whether a warning information needs to be sent based on the fatigue driving state detection result and the distraction driving state detection result.

[0045] In addition, the application further provides an electronic device, which comprises a processor and a memory, the memory is configured to store instructions, and the processor is configured to call the instructions in the memory, so that the electronic device executes the above-mentioned driving state analysis method based on machine vision.

[0046] In addition, the application further provides a computer readable storage medium, which stores computer instructions, when the computer instructions run on an electronic device, so that the electronic device executes the above-mentioned driving state analysis method based on machine vision.

[0047] In the embodiment of the present application, the feature point extraction is performed on the driver image of each frame based on the calibration network, which can effectively improve the accuracy of feature point extraction. The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold and micro-expression analysis, which can improve the reliability of fatigue driving state detection and avoid the influence of the driver and complex light on the fatigue driving state detection. The line-of-sight activity is analyzed based on the feature point information to obtain the line-of-sight activity information, and the distraction driving state is detected based on the line-of-sight activity information and handheld object information, which can comprehensively and accurately detect whether the driver has a distraction driving state and avoid the occurrence of missed detection or false detection, so as to timely and accurately send a warning information to the driver. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a flowchart of the driving state analysis method based on machine vision in the embodiment of the present application;

[0050] Figure 2 is a flowchart of the driving state analysis method based on machine vision in another embodiment of the present application;

[0051] Figure 3 is a structural composition diagram of the driving state analysis device based on machine vision in the embodiment of the present application;

[0052] Figure 4 is a structural composition diagram of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] Embodiment one

[0055] Please refer to Figure 1 , Figure 1is a flowchart of a driving state analysis method based on machine vision in an embodiment of the present application, the method is applied to a vehicle camera and a vehicle management terminal system, the method comprises the following steps:

[0056] S11: The vehicle management terminal system optimizes the definition of each frame of driver image collected by the vehicle camera, and obtains each frame of driver image after definition optimization.

[0057] In the specific implementation process of the present application, the definition of each frame of driver image collected by the vehicle camera is optimized, and each frame of driver image after definition optimization is obtained, which comprises: performing bicubic interpolation on each frame of driver image to obtain each frame of enlarged driver image; analyzing the first gradient matrix of each frame of enlarged driver image based on Sobel algorithm, and performing non-maximum suppression on the first gradient matrix to obtain the target gradient matrix; performing threshold analysis on each frame of enlarged driver image based on the fireworks algorithm to obtain the high and low double thresholds; determining the edge pixel points of each frame of enlarged driver image based on the high and low double thresholds and the target gradient matrix; determining the second gradient matrix based on the normal operators of a plurality of directions, and optimizing the definition of each frame of driver image based on the edge pixel points and the second gradient matrix to obtain each frame of driver image after definition optimization.

[0058] Specifically, the vehicle camera is used to shoot a plurality of frames of images of the driver in the vehicle, and the vehicle camera is in communication connection with the vehicle management system, and transmits each frame of driver image shot to the vehicle management terminal system, and the vehicle management terminal system is used to detect and analyze the driving state according to each frame of driver image, to determine whether the driver has the distracted driving state and the fatigue driving state, so as to determine whether the corresponding warning information needs to be sent to the driver.

[0059] The driver images are bicubic interpolated, each pixel point of each frame of the driver images is bicubic interpolated, bicubic interpolation is the most commonly used interpolation method in two-dimensional space in numerical analysis, the core of which is to calculate the target pixel value through the weighted average value of the 16 adjacent pixels of the 4*4 grid around the to-be-interpolated point, the image is enlarged through the target pixel value, and each frame of the enlarged driver image is obtained. The first gradient matrix of each frame of the enlarged driver image is analyzed based on the Sobel algorithm, the Sobel algorithm is an algorithm for detecting edge information of an image, which can calculate the gradient of the image in the corresponding direction, the gradient values in the horizontal direction, the vertical direction, 135° and 45° of the enlarged driver image are calculated through the Sobel operator, the gradient values in the four directions are obtained, and the gradient matrix is constructed based on the gradient values in the four directions; according to the gradient values in the four directions, four directional gradient matrices are formed, the maximum value in the four directional gradient matrices is taken as the gradient maximum value matrix of the image, that is, the first gradient matrix. The non-maximum suppression is performed on the first gradient matrix, all gradient values except the local maximum value are suppressed to zero, the pseudo-edge points of the image are eliminated, the maximum gradient matrix is obtained, and the target gradient matrix is obtained. The threshold value analysis is performed on each frame of the enlarged driver image based on the fireworks algorithm, the maximum inter-class variance is calculated according to the number of pixel points with a gray value less than a gray value threshold, the average gray value of the pixel points with a gray value less than the gray value threshold, the number of pixel points with a gray value greater than the gray value threshold and the average gray value of the pixel points with a gray value greater than the gray value threshold, the initial fireworks population is generated in the feasible solution through the fireworks algorithm optimization of the maximum inter-class variance, the maximum inter-class variance function is set as the fitness function, and the initial fireworks population is subjected to spark generation, fitness calculation, individual updating and selection to the preset iteration number through the fitness function, and the best segmentation threshold value is output. The best segmentation threshold value is taken as the high threshold value, and the half of the high threshold value is taken as the low threshold value, that is, the high-low double threshold value is obtained. The edge pixel points of each frame of the enlarged driver image are determined based on the high-low double threshold value and the target gradient matrix, that is, the target gradient matrix is divided into two threshold values according to the high threshold value and the low threshold value, and the corresponding edge pixel points are obtained. The second gradient matrix is determined based on normal operators in a plurality of directions, the second gradient matrix is constructed through the normal operator vertically upward, the normal operator vertically downward, the normal operator horizontally left, the normal operator horizontally right, the normal operator right upward, the normal operator right downward, the normal operator left upward and the normal operator left downward, and the clarity of each frame of the driver image is optimized based on the edge pixel points and the second gradient matrix. The edge pixel points and the pixel points located on both sides of the edge pixel points are subjected to convolution operation with the second gradient matrix one by one, new pixel points of the pixel points in the maximum gradient direction are obtained, the gradient difference value between the pixel points and the new pixel points is calculated, if the gradient difference value is greater than a preset threshold value, the pixel point is modified, if the gradient difference value is less than or equal to the preset threshold value, the pixel point does not need to be modified, until all the pixel points are modified, and each frame of the driver image with optimized clarity is obtained.

[0060] S12: performing feature point extraction on the driver image of each frame after the definition is optimized based on the calibration network to obtain corresponding feature point information;

[0061] In the implementation of the present application, the feature point extraction on the driver image of each frame after the definition is optimized based on the calibration network to obtain corresponding feature point information includes: performing feature point positioning on the driver image of each frame after the definition is optimized based on the calibration network to obtain a feature point positioning result; and performing feature point extraction on the driver image of each frame after the definition is optimized based on the feature point positioning result and a preset rectangular frame to obtain corresponding feature point information.

[0062] Specifically, the feature point positioning on the driver image of each frame after the definition is optimized based on the calibration network, since the driver image can be in a head posture of a non-frontal face, the driver image of each frame after the definition is optimized is analyzed for face offset by the calibration network, and the face image in the driver image of each frame after the definition is optimized is corrected according to the analyzed face offset angle to avoid inaccurate feature point positioning. The calibration network is a deep neural network, and the feature point positioning is performed on the driver image of each frame after the face offset correction by a face detection algorithm to locate the eye feature region and the mouth feature region of each frame of the driver image, i.e., to obtain the feature point positioning result. The feature point extraction is performed on the driver image of each frame after the definition is optimized based on the feature point positioning result and the preset rectangular frame, the located eye feature region and the mouth feature region are framed according to the preset rectangular frame to avoid unnecessary feature extraction, the eye feature point information such as the eye corner positions of the left and right eyes, the distance between the eyes, the pupil center, the upper and lower eye contour, etc. is extracted from the eye feature region, and the mouth feature information such as the left and right lower and upper mouth contours, etc. is extracted from the mouth feature region, and the eye feature point information and the mouth feature point information are taken as the corresponding feature point information.

[0063] S13: analyzing an eye closure adaptive threshold based on the feature point information, and detecting the fatigue driving state of the driver based on the eye closure adaptive threshold and micro-expression analysis to obtain a fatigue driving state detection result;

[0064] In the implementation of the present application, the eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold combined with micro-expression analysis to obtain the fatigue driving state detection result, including: analyzing the left eye bounding rectangle and the right eye bounding rectangle based on the feature point information; performing pupil detection based on the left eye bounding rectangle and the right eye bounding rectangle to obtain pupil region information; calculating the pupil height-width ratio based on the pupil region information, and analyzing the eye closure adaptive threshold based on the pupil height-width ratio; generating a face texture feature based on the feature point information, and performing fatigue micro-expression feature matching in a micro-expression library based on the face texture feature to obtain fatigue micro-expression feature information; and performing fatigue driving state detection of the driver based on the pupil height-width ratio, the eye closure adaptive threshold, and the fatigue micro-expression feature information to obtain the fatigue driving state detection result.

[0065] Specifically, the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame are analyzed based on the feature point information, that is, the minimum circumscribed rectangle of the left and right eyes is calculated according to the pupil centers and the binocular distance in the feature point information, the calculated minimum circumscribed rectangle is generated to generate a corresponding circumscribed rectangular frame, and the calculation of the circumscribed rectangle can use the smallest_rectangle2 operator, which is a function for calculating the minimum circumscribed rectangle of a region. The pupil detection is performed based on the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame, the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame are combined with the pupil centers in the feature point information and input into the pupil detection model for pupil detection to obtain the pupil region coordinates, that is, the pupil region information is obtained. The pupil height-width ratio is calculated based on the pupil region information, the height and width of the pupil are calculated according to the pupil region information, and the ratio between the height and width of the pupil is calculated, that is, the pupil height-width ratio, and the eye closure adaptive threshold is analyzed based on the pupil height-width ratio, the mean value of the pupil height-width ratios of the driver images of the selected preset number of frames is calculated to obtain the eye closure adaptive threshold. The eye closure adaptive threshold is more suitable for the actual eye closure of the corresponding driver, avoiding the occurrence of the determined threshold not meeting the individual situation of the driver. The face texture feature is generated based on the feature point information, the mouth circumscribed rectangular frame is generated based on the feature point information, the mouth region of the driver image is framed according to the mouth circumscribed rectangular frame, the texture features of the mouth region and the left and right eye regions are extracted, the gray level co-occurrence matrix analysis is performed on the images of the mouth region and the left and right eye regions to obtain the corresponding gray level co-occurrence matrix, the texture energy, texture inertia, texture correlation and texture entropy are determined according to the gray level co-occurrence matrix, the mouth texture feature and the left and right eye texture feature are generated from the texture energy, texture inertia, texture correlation and texture entropy, the mouth texture feature and the left and right eye texture feature are fused to obtain the face texture feature. And the fatigue micro-expression feature matching is performed in the micro-expression library based on the face texture feature, that is, the distance between the face texture feature and the fatigue micro-expression feature in the micro-expression library is calculated, the fatigue micro-expression feature whose distance reaches the preset distance threshold is taken as the target fatigue micro-expression feature, and the fatigue micro-expression feature such as mouth corner drooping and eye face drooping is obtained, that is, the fatigue micro-expression feature information is obtained. The fatigue driving state of the driver is detected based on the pupil height-width ratio, the eye closure adaptive threshold and the fatigue micro-expression feature information, the frequency of the pupil height-width ratio being less than the eye closure adaptive threshold in each frame of the driver image at the preset time is counted, and the frequency of the pupil height-width ratio being less than the eye closure adaptive threshold and the fatigue micro-expression feature are input into the fatigue driving state detection model for fatigue driving state detection. The fatigue driving state detection model is a converged model obtained by inputting sample data sets into a deep neural network for training, and the fatigue driving state detection result is obtained.

[0066] S14: perform hand-held object detection on the driver images after the clarity of each frame is optimized to obtain hand-held object information;

[0067] In the implementation of the present application, the hand-held object detection on the driver images after the clarity of each frame is optimized to obtain hand-held object information includes: performing hand region detection on the driver images after the clarity of each frame is optimized to obtain corresponding hand region images; and performing hand-held object detection on the driver based on the hand region images using an object recognition model to obtain hand-held object information.

[0068] Specifically, the hand region detection on the driver images after the clarity of each frame is optimized can input the driver images after the clarity of each frame is optimized into a hand detection model to obtain corresponding hand region images. The hand-held object detection on the driver based on the hand region images using an object recognition model means that the hand-held object type in the hand region images is identified according to the object recognition model, the hand-held object type includes a mobile phone, a cigarette, etc., the coordinate information of the hand-held object is marked, and the hand-held object information is generated according to the hand-held object type and the coordinate information.

[0069] S15: perform line-of-sight activity analysis based on the feature point information to obtain line-of-sight activity information, and perform distraction driving state detection based on the line-of-sight activity information combined with the hand-held object information to obtain a distraction driving state detection result;

[0070] In the implementation of the present application, the line-of-sight activity analysis based on the feature point information to obtain line-of-sight activity information, and the distraction driving state detection based on the line-of-sight activity information combined with the hand-held object information to obtain a distraction driving state detection result includes: determining a line-of-sight concentration area and a line-of-sight concentration time of the driver based on the feature point information; performing line-of-sight activity analysis based on the line-of-sight concentration area and the line-of-sight concentration time to obtain line-of-sight activity information; determining head posture data based on the feature point information combined with a head contour reference point; calculating a first distance between the hand-held object and the mouth and a second distance between the hand-held object and the eyes based on the hand-held object information and the feature point information; and performing distraction driving state detection based on the line-of-sight activity information, the head posture data, the first distance, and the second distance to obtain a distraction driving state detection result.

[0071] Specifically, the line-of-sight concentration area and the line-of-sight concentration time of the driver are determined based on the feature point information, the eye gaze direction of the driver in each frame of image is analyzed according to the pupil distance and the pupil position in the feature point information, the line-of-sight concentration area of the driver is determined according to the eye gaze direction, and the line-of-sight concentration time of the driver in the line-of-sight concentration area is determined. The line-of-sight activity analysis is performed based on the line-of-sight concentration area and the line-of-sight concentration time, the transformation frequency of the line-of-sight concentration area of the driver is analyzed, and the line-of-sight activity of the driver is analyzed according to the transformation frequency of the line-of-sight concentration area and the line-of-sight concentration time of each line-of-sight concentration area, that is, the line-of-sight activity information is obtained. The head posture data is determined based on the head contour reference point, the head contour reference point such as the central reference point of the forehead is determined in each frame of driver image, the head deflection angle, the head pitch angle and the head roll angle of the driver are analyzed according to the head contour reference point, and the head posture data is generated from the head deflection angle, the head pitch angle and the head roll angle. The first distance between the handheld object and the mouth and the second distance between the handheld object and the eye are calculated based on the handheld object information and the feature point information, the first distance is calculated according to the handheld object coordinate information in the handheld object information and the lip corner position coordinates in the feature point information, and the second distance is calculated according to the handheld object coordinate information and the eye position coordinates in the feature point information. The distraction driving state detection is performed based on the line-of-sight activity information, the head posture data, the first distance and the second distance, when the head posture data is deflected and the line-of-sight activity is greater than a preset threshold, it is indicated that the driver is in a line-of-sight distraction state, such as the driver does not concentrate attention on the front road area, frequently looks around at people or things irrelevant to driving, if the first distance is less than a first preset threshold, it is indicated that the user has a distraction behavior such as smoking or eating, if the second distance is less than a second preset threshold, it is indicated that the user has a distraction behavior such as looking at a mobile phone, and the distraction driving state detection result is obtained. The distraction driving state detection result is more comprehensive because the distraction behavior of the user holding an object and the distraction behavior of the user not holding an object are both detected.

[0072] S16: determining whether a warning information needs to be sent based on the fatigue driving state detection result and the distraction driving state detection result.

[0073] In the specific implementation process of the present application, the determination of whether a warning information needs to be sent based on the fatigue driving state detection result and the distraction driving state detection result includes: determining a warning level and a warning voice information based on the fatigue driving state detection result and the distraction driving state detection result; and performing a driving anomaly warning to the driver based on the warning level and the warning voice information.

[0074] Specifically, the warning level and the warning voice information are determined based on the fatigue driving state detection result and the distraction driving state detection result, that is, when the fatigue driving state and / or the distraction driving state of the driver are detected, the warning level and the warning voice information are determined according to the abnormal driving state of the driver, for example, when the driver has the fatigue driving state and the distraction driving state at the same time, the warning level is the highest, and the warning voice is to remind the driver to concentrate and to find a place to rest as soon as possible. The driving abnormality warning is given to the driver based on the warning level and the warning voice information, and the warning level and the warning voice information can be transmitted to the vehicle terminal of the driver to give the driving abnormality warning.

[0075] In the embodiment of the present application, the feature point information is extracted from the driver image with optimized definition based on the calibration network, which can effectively improve the accuracy of feature point extraction. The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold combined with micro-expression analysis, which can improve the reliability of fatigue driving state detection and avoid the influence of the driver and complex illumination on fatigue driving state detection. The gaze activity is analyzed based on the feature point information to obtain the gaze activity information, and the distraction driving state is detected based on the gaze activity information combined with the handheld object information, which can comprehensively and accurately detect whether the driver has the distraction driving state, and avoid the occurrence of missed detection or false detection, so as to timely and accurately give the warning information to the driver.

[0076] Embodiment two

[0077] Please refer to Figure 2 , Figure 2 is a flowchart of a driving state analysis method based on machine vision in another embodiment of the present application, which is applied to a vehicle camera and a vehicle management and control terminal system. The method comprises the following steps:

[0078] S201: The vehicle management and control terminal system optimizes the definition of each frame of driver image collected by the vehicle camera to obtain each frame of driver image with optimized definition;

[0079] S202: Feature points are extracted from each frame of driver image with optimized definition based on the calibration network to obtain corresponding feature point information;

[0080] S203: The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold combined with micro-expression analysis to obtain the fatigue driving state detection result;

[0081] S204: The handheld object of the driver is detected based on each frame of driver image with optimized definition to obtain the handheld object information;

[0082] S205: determining a driver's line-of-sight concentration area and line-of-sight concentration time based on the feature point information, performing line-of-sight activity analysis based on the line-of-sight concentration area and line-of-sight concentration time, and obtaining line-of-sight activity information;

[0083] S206: determining head posture data based on the head contour reference points;

[0084] S207: calculating a first distance between the handheld object and the mouth and a second distance between the handheld object and the eyes based on the handheld object information and the feature point information;

[0085] S208: performing distraction driving state detection based on the line-of-sight activity information, the head posture data, the first distance, and the second distance, and obtaining a distraction driving state detection result;

[0086] S209: determining whether to issue a warning information based on the fatigue driving state detection result and the distraction driving state detection result.

[0087] In the embodiment of the present application, the feature points are extracted from the driver images with optimized definition based on the calibration network, which can effectively improve the accuracy of feature point extraction. The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold combined with micro-expression analysis, which can improve the reliability of fatigue driving state detection and avoid the influence of the driver and complex lighting on fatigue driving state detection. The line-of-sight activity information is obtained by performing line-of-sight activity analysis based on the feature point information, and the distraction driving state of the driver is detected based on the line-of-sight activity information combined with the handheld object information, which can comprehensively and accurately detect whether the driver has a distraction driving state and avoid the occurrence of missed detection or false detection, so as to timely and accurately issue a warning information to the driver.

[0088] Embodiment three

[0089] Please refer to Figure 3 , Figure 3 is a structural composition diagram of the driving state analysis device based on machine vision in the embodiment of the present application. The device is applied to a vehicle camera and a vehicle management and control terminal system, and the device comprises:

[0090] The image optimization module 31 is used for the vehicle management and control terminal system to optimize the definition of each frame of driver image collected by the vehicle camera, and obtain each frame of driver image with optimized definition;

[0091] The feature point extraction module 32 is used for extracting feature points from each frame of driver image with optimized definition based on the calibration network, and obtaining corresponding feature point information;

[0092] The fatigue driving detection module 33 is configured to analyze an eye closure adaptive threshold based on the feature point information, and detect a fatigue driving state of the driver based on the eye closure adaptive threshold and micro-expression analysis, to obtain a fatigue driving state detection result.

[0093] The handheld object detection module 34 is configured to detect a handheld object of the driver based on the driver image of each frame after the clarity optimization, to obtain handheld object information.

[0094] The distraction driving detection module 35 is configured to analyze a line-of-sight activity based on the feature point information, to obtain line-of-sight activity information, and detect a distraction driving state of the driver based on the line-of-sight activity information and the handheld object information, to obtain a distraction driving state detection result.

[0095] The early warning judgment module 36 is configured to judge whether early warning information needs to be sent based on the fatigue driving state detection result and the distraction driving state detection result.

[0096] In the implementation of the present application, the implementation of the device can refer to the implementation of the method, which will not be repeated here.

[0097] In the embodiment of the present application, the feature points are extracted from the driver image of each frame after the clarity optimization based on the calibration network, which can effectively improve the accuracy of feature point extraction. The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state of the driver is detected based on the eye closure adaptive threshold and micro-expression analysis, which can improve the reliability of fatigue driving state detection and avoid the influence of the driver and complex illumination on fatigue driving state detection. The line-of-sight activity is analyzed based on the feature point information, to obtain line-of-sight activity information, and the distraction driving state of the driver is detected based on the line-of-sight activity information and the handheld object information, which can comprehensively and accurately detect whether the driver has a distraction driving state, to avoid the occurrence of missed detection or false detection, so as to timely and accurately send early warning information to the driver.

[0098] The computer readable storage medium provided by the embodiment of the present application stores a computer program, and the program is executed by a processor to realize the machine vision-based driving state analysis method of any one of the above embodiments. The computer readable storage medium includes but is not limited to any type of disk (including a floppy disk, a hard disk, an optical disk, a CD-ROM, and a magneto-optical disk), a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, a magnetic card or an optical card. That is, the storage device includes any medium that stores or transmits information in a form capable of being read by an equipment (for example, a computer, a mobile phone), and can be a read-only memory, a magnetic disk or an optical disk, etc.

[0099] Embodiment Four

[0100] Please refer to Figure 4 , Figure 4 is a structural component diagram of an electronic device in the embodiment of the present application.

[0101] The embodiment of the present application further provides an electronic device, as shown in Figure 4 , the electronic device includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and executable on the processor 43. Those skilled in the art can understand that Figure 3The electronic device shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine some components. The memory 41 can be used to store computer programs 42 and various functional modules, and the processor 43 runs the computer programs 42 stored in the memory 41 to perform various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include both the internal memory and the external memory. The internal memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. The external memory can include a hard disk, a floppy disk, a ZIP disk, a USB disk, a magnetic tape, etc. The processor 43 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip processor, or the processor 43 can also be any conventional processor, etc. The processor and the memory disclosed in the present application include but are not limited to these types of processors and memories. The processor and the memory disclosed in the present application are only examples and are not limited.

[0102] As an embodiment, the electronic device includes one or more processors 43, a memory 41, and one or more computer programs 42, wherein the one or more computer programs 42 are stored in the memory 41 and configured to be executed by the one or more processors 43, and the one or more computer programs 42 are configured to perform the machine vision-based driving state analysis method in any one of the above embodiments. For specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0103] In the embodiment of the present application, the feature point extraction is performed on the driver image of each frame based on the calibration network, which can effectively improve the accuracy of feature point extraction. The eye closure adaptive threshold is analyzed based on the feature point information, and the fatigue driving state detection of the driver is performed based on the eye closure adaptive threshold and micro-expression analysis, which can improve the reliability of fatigue driving state detection and avoid the influence of the driver and complex light on the fatigue driving state detection. The line of sight activity is analyzed based on the feature point information to obtain the line of sight activity information, and the distraction driving state detection is performed based on the line of sight activity information and handheld object information, which can comprehensively and accurately detect whether the driver has a distraction driving state, avoid the occurrence of missed detection or false detection, and timely and accurately issue a warning information to the driver.

[0104] In addition, the above describes in detail a driving state analysis method and related device based on machine vision provided by the embodiment of the present application, and the principle and implementation mode of the present application are described by using specific examples in this paper. The above embodiment is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A driving state analysis method based on machine vision, characterized in that: Applied to vehicle-mounted cameras and vehicle control terminal systems; the method includes: The vehicle control terminal system optimizes the clarity of each frame of the driver image captured by the vehicle-mounted camera to obtain the driver image with optimized clarity for each frame; Based on the calibration network, feature points of the driver image with optimized clarity in each frame are extracted to obtain corresponding feature point information; Analyzing the eye closure adaptive threshold based on feature point information, and detecting the driver's fatigue driving state based on the eye closure adaptive threshold combined with micro-expression analysis to obtain a fatigue driving state detection result; Based on the driver image with optimized clarity of each frame, the driver's handheld object detection is performed to obtain the handheld object information; Performing gaze activity analysis based on feature point information to obtain gaze activity information, and performing distracted driving status detection based on the gaze activity information combined with handheld object information to obtain a distracted driving status detection result; Determine whether to issue a warning message based on the fatigue driving status detection results and distracted driving status detection results.

2. The driving state analysis method based on machine vision according to claim 1, characterized in that: The step of optimizing the clarity of each frame of the driver image captured by the vehicle-mounted camera to obtain each frame of the driver image with optimized clarity includes: Perform bicubic interpolation on each frame of the driver image to obtain an enlarged driver image of each frame; Analyze the first gradient matrix of the driver image of each frame based on the Sobel algorithm, and perform non-maximum suppression on the first gradient matrix to obtain the target gradient matrix; Threshold analysis is performed on each frame of the driver's image based on the fireworks algorithm to obtain high and low dual thresholds; Determine edge pixel points of the driver image in each frame based on high and low double thresholds and target gradient matrix; A second gradient matrix is ​​determined based on normal operators in several directions, and clarity of each frame of the driver image is optimized based on edge pixels and the second gradient matrix to obtain a driver image with optimized clarity in each frame.

3. The driving state analysis method based on machine vision according to claim 1, characterized in that: The step of extracting feature points from the driver image after each frame of clarity optimization based on the calibration network to obtain corresponding feature point information includes: Based on the calibration network, feature points of the driver image with optimized clarity in each frame are located to obtain feature point location results; Based on the feature point positioning results and the preset rectangular frame, feature points of the driver image with optimized clarity in each frame are extracted to obtain the corresponding feature point information.

4. The driving state analysis method based on machine vision according to claim 1, characterized in that: The method of analyzing the eye closure adaptive threshold based on the feature point information and detecting the driver's fatigue driving state based on the eye closure adaptive threshold combined with micro-expression analysis to obtain a fatigue driving state detection result includes: Analyze the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame based on the feature point information; Perform pupil detection based on the left eye circumscribed rectangular frame and the right eye circumscribed rectangular frame to obtain pupil area information; Calculate pupil aspect ratio based on pupil area information, and analyze eye closure adaptive threshold based on pupil aspect ratio; Generate facial texture features based on feature point information, and perform fatigue micro-expression feature matching in the micro-expression library based on the facial texture features to obtain fatigue micro-expression feature information; The driver's fatigue driving state is detected based on pupil aspect ratio, eye closure adaptive threshold and fatigue micro-expression feature information to obtain fatigue driving state detection results.

5. The driving state analysis method based on machine vision according to claim 1, characterized in that: The detecting of the driver's handheld object based on the driver image with optimized clarity of each frame to obtain the handheld object information includes: Perform hand region detection on the driver image after each frame of clarity optimization to obtain the corresponding hand region image; Based on the hand area image, the object recognition model is used to detect the driver's handheld objects and obtain the handheld object information.

6. The driving state analysis method based on machine vision according to claim 1, characterized in that: The method of performing gaze activity analysis based on feature point information to obtain gaze activity information, and performing distracted driving state detection based on the gaze activity information combined with handheld object information to obtain a distracted driving state detection result includes: Determine the driver's sight focus area and sight focus time based on feature point information; Perform gaze activity analysis based on gaze concentration area and gaze concentration time to obtain gaze activity information; Determining head posture data based on head contour reference points; Calculating a first distance between the handheld object and the mouth and a second distance between the handheld object and the eyes based on the handheld object information and the feature point information; A distracted driving state detection is performed based on the gaze activity information, the head posture data, the first distance and the second distance to obtain a distracted driving state detection result.

7. The driving state analysis method based on machine vision according to claim 1, characterized in that: The determining whether to issue a warning message based on the fatigue driving state detection results and the distracted driving state detection results includes: Determine the warning level and warning voice information based on the fatigue driving state detection results and the distracted driving state detection results; Provides the driver with an abnormal driving warning based on the warning level and warning voice information.

8. A driving state analysis device based on machine vision, characterized in that: Applied to vehicle-mounted cameras and vehicle control terminal systems, the device includes: Image optimization module: used by the vehicle control terminal system to optimize the clarity of each frame of driver images collected by the vehicle-mounted camera, and obtain the driver image with optimized clarity for each frame; Feature point extraction module: used to extract feature points from the driver image after each frame of clarity optimization based on the calibration network to obtain the corresponding feature point information; Fatigue driving detection module: used to analyze the eye closure adaptive threshold based on feature point information, and detect the driver's fatigue driving status based on the eye closure adaptive threshold combined with micro-expression analysis to obtain fatigue driving status detection results; Handheld object detection module: used to detect the driver's handheld object based on the driver image with optimized clarity of each frame and obtain handheld object information; Distracted driving detection module: used to analyze gaze activity based on feature point information to obtain gaze activity information, and then detect distracted driving status based on the gaze activity information combined with handheld object information to obtain distracted driving status detection results; Warning judgment module: used to determine whether a warning message needs to be issued based on the fatigue driving status detection results and distracted driving status detection results.

9. An electronic device comprising a processor and a memory, characterized in that: The memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the driving state analysis method based on machine vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the driving state analysis method based on machine vision according to any one of claims 1 to 7.

Images