A driving state detection method, device, electronic device and readable storage medium
By acquiring and processing the driver's head images in real time, combining head posture and eye feature points, the distraction detection problem in the prior art that cannot recognize the impact of large-angle driver's head and wearing sunglasses is solved, and the accuracy of detection is improved Sex and safety.
Patent Information
- Application Number
- CN202411252748.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-09
AI Technical Summary
The existing distraction detection technology cannot identify facial feature points when the driver's face is too large, and the line of sight is inaccurate when wearing sunglasses or the eye surface reflects, which affects the accuracy of distraction detection.
By acquiring the driver's head image in real time, using the head attitude estimation model to determine the head attitude, and determining the eye characteristic points when the face area is detected, to determine whether the eye image can be used for line of sight direction estimation, combining the head attitude and The gaze area determines the driver's distraction status.
Improve the accuracy of driver distraction status detection, reduce safety risks during driving, and ensure accurate distraction detection under different driver postures and wear conditions.
Smart Images

Figure CN118762353B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of vehicle driving technology, and in particular to a driving state detection method, device, electronic device and readable storage medium. Background Art
[0002] Distraction detection (also known as driver attention detection) is crucial in vehicle safe driving technology. By performing real-time distraction detection while the driver is driving the vehicle, it can significantly improve vehicle driving safety, reduce traffic accidents caused by driver distraction (inattention), and ensure driving safety.
[0003] At present, distraction detection usually uses the solvPnP algorithm (an algorithm for estimating camera posture) to detect whether the driver is distracted. However, this method relies on the driver's facial feature point information. When the driver's face yaw angle in the horizontal direction is too large (for example, the driver looks back), or the driver's face pitch angle in the vertical direction is too large (for example, looking down), so that the camera cannot capture the driver's face image, the driver's facial feature point information cannot be recognized. At this time, the driver's distraction cannot be detected, which undoubtedly increases the safety risk during driving.
[0004] In addition, distraction detection can also use the driver's line of sight to determine whether a distracted state has occurred. However, in some cases, such as when the driver is wearing sunglasses or the surface of the eyes is reflective, line of sight direction recognition becomes unreliable, which can easily lead to errors in line of sight direction estimation, thus affecting the accuracy of distraction detection. Summary of the invention
[0005] In view of this, the purpose of the present application is to provide a driving state detection method, device, electronic device and readable storage medium to improve the accuracy of detecting the driver's distraction state.
[0006] In a first aspect, an embodiment of the present application provides a driving state detection method, comprising:
[0007] When the driver is driving the vehicle, the image to be detected including the driver's head at the current moment is obtained in real time;
[0008] Detecting the driver's head image from the image to be detected, inputting the head image into a trained head posture estimation model, and outputting the driver's head posture through the head posture estimation model; the head posture includes head rotation angle and head position information;
[0009] When the image to be detected contains the driver's facial area, detecting the driver's facial area from the image to be detected;
[0010] Determine the facial feature point position information from the facial region, and determine the eye feature point position information from the facial feature point position information;
[0011] Determining the number of blocked eye feature points according to the eye feature point position information; and determining an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region;
[0012] Determining whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points;
[0013] When the eye image can be used for line of sight direction estimation, inputting the eye image into a trained line of sight direction estimation model, and determining the line of sight direction of the driver by using the line of sight direction estimation model;
[0014] Determining the driver's gaze area according to the driver's sight direction, the head position information, the head rotation angle, and the position information of each area in the vehicle;
[0015] Whether the driving state of the driver is a distracted driving state is determined according to the head rotation angle and the gaze area.
[0016] In combination with the first aspect, the embodiment of the present application provides a first possible implementation of the first aspect, wherein the method further includes:
[0017] When the image to be detected does not include the driver's facial area, it is determined whether the driver's driving state is a distracted driving state according to the rotation angle of the driver's head.
[0018] In combination with the first aspect, an embodiment of the present application provides a second possible implementation manner of the first aspect, wherein the blocked eye feature points are eye feature points of a single eye; and judging whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points includes:
[0019] For each eye, calculating a first ratio of the number of the blocked eye feature points corresponding to the eye to the total number of preset single eye feature points;
[0020] When the first ratio is greater than a first preset threshold, it indicates that the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0021] When the first ratio is not greater than the first preset threshold, inputting the eye image corresponding to the eye into a trained glasses wearing discrimination model, so as to determine whether the eye is wearing glasses through the glasses wearing discrimination model;
[0022] If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation;
[0023] If the eye wears glasses, determine the type of glasses;
[0024] When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0025] When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction;
[0026] When the light spot on the lens corresponding to the eye in the glasses meets the conditions that affect the estimation of the line of sight direction, the eye image representing the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the conditions that affect the estimation of the line of sight direction, the eye image representing the eye can be used for estimating the line of sight direction.
[0027] In combination with the second possible implementation of the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein if the eye wears glasses, determining the type of the glasses includes:
[0028] determining an inner eyelid region and an outer eyelid region from the eye image of the eye;
[0029] Calculate the mean grayscale value of pixels in the inner eyelid area according to the grayscale value of each pixel point in the inner eyelid area; and calculate the mean grayscale value of pixels in the outer eyelid area according to the grayscale value of each pixel point in the outer eyelid area;
[0030] Calculating a second ratio between the mean grayscale value of pixels in the inner eyelid area and the mean grayscale value of pixels in the outer eyelid area;
[0031] When the second ratio is greater than the second preset threshold, the type of the glasses is determined to be sunshade glasses; when the second ratio is not greater than the second preset threshold, the type of the glasses is determined to be transparent glasses.
[0032] In combination with the second possible implementation manner of the first aspect, the embodiment of the present application provides a fourth possible implementation manner of the first aspect, wherein when the type of the glasses is transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses meets the condition that affects the estimation of the line of sight direction includes:
[0033] When the glasses are transparent glasses, calculating, according to the grayscale value of each pixel in the eye image of the eye, a third ratio of the number of pixels having a grayscale value greater than a third preset threshold to the total number of pixels in the eye image;
[0034] If the third ratio is greater than a fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses meets the condition of affecting the estimation of the line of sight direction;
[0035] If the third ratio is not greater than the fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses does not meet the condition affecting the estimation of the line of sight direction.
[0036] In combination with the second possible implementation manner of the first aspect, the embodiment of the present application provides a fifth possible implementation manner of the first aspect, wherein the sight line direction is the sight line direction of a single eye; and determining the driver's gaze area according to the sight line direction of the driver, the head position information, the head rotation angle, and the position information of each area in the vehicle includes:
[0037] For each moment between the current moment and the target historical moment, obtain a first historical eye image of each eye of the driver at a first historical moment of the moment and a second historical eye image at a second historical moment of the moment; wherein the first historical moment is a moment before the moment and a moment after the second historical moment;
[0038] For each eye, determine a first sight line direction of the eye at the first historical moment according to the first historical eye image of the eye, and determine a second sight line direction of the eye at the second historical moment according to the second historical eye image of the eye;
[0039] Predicting a predicted sight line direction of the eye at the moment according to the first sight line direction and the second sight line direction;
[0040] Calculating an angle between the predicted sight line direction and the sight line direction of the eye at the moment, and when the angle is not greater than a third preset angle, retaining the eye image corresponding to the eye at the moment; and when the angle is greater than the third preset angle, deleting the eye image corresponding to the eye at the moment;
[0041] According to the sight lines corresponding to the eye images retained by each eye of the driver at each of the moments, the mean of the sight lines corresponding to the retained eye images is calculated to obtain the sight lines of both eyes of the driver;
[0042] The driver's gaze area is determined based on the sight direction of the driver's eyes, the head position information, the head rotation angle and the position information of each area in the vehicle.
[0043] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the head rotation angle includes a head yaw angle in a horizontal direction and a head pitch angle in a vertical direction; and judging whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area includes:
[0044] When any one or more of the following distraction conditions are continuously satisfied within a preset time period, the driver's driving state is determined to be the distracted driving state; the distraction conditions include: the head yaw angle is greater than a first preset angle, the head pitch angle is greater than a second preset angle, and the gaze area is other areas outside the front windshield area of the vehicle.
[0045] In a second aspect, an embodiment of the present application further provides a driving state detection device, comprising:
[0046] The first acquisition module is used to acquire the image to be detected containing the driver's head at the current moment in real time while the driver is driving the vehicle;
[0047] A first detection module is used to detect the driver's head image from the image to be detected, input the head image into a trained head posture estimation model, and output the driver's head posture through the head posture estimation model; the head posture includes head rotation angle and head position information;
[0048] A second detection module, configured to detect the driver's face area from the image to be detected when the image to be detected contains the driver's face area;
[0049] A first determination module is used to determine the position information of facial feature points from the facial region, so as to determine the position information of eye feature points from the facial feature point position information;
[0050] a second determination module, configured to determine the number of blocked eye feature points according to the eye feature point position information; and to determine an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region;
[0051] A first judgment module, used for judging whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points;
[0052] An input module, used for inputting the eye image into a trained line of sight direction estimation model when the eye image can be used for line of sight direction estimation, and determining the line of sight direction of the driver through the line of sight direction estimation model;
[0053] A third determination module is used to determine the driver's gaze area according to the driver's sight direction, the head position information, the head rotation angle and the position information of each area in the vehicle;
[0054] The second judgment module is used to judge whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area.
[0055] In combination with the second aspect, the embodiment of the present application provides a first possible implementation of the second aspect, wherein the device further includes:
[0056] The third judgment module is used to judge whether the driving state of the driver is a distracted driving state according to the rotation angle of the driver's head when the image to be detected does not contain the driver's facial area.
[0057] In combination with the second aspect, an embodiment of the present application provides a second possible implementation of the second aspect, wherein the blocked eye feature points are blocked eye feature points of a single eye; and the first judgment module is used to judge whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points, specifically for:
[0058] For each eye, calculating a first ratio of the number of the blocked eye feature points corresponding to the eye to the total number of preset single eye feature points;
[0059] When the first ratio is greater than a first preset threshold, it indicates that the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0060] When the first ratio is not greater than the first preset threshold, inputting the eye image corresponding to the eye into a trained glasses wearing discrimination model, so as to determine whether the eye is wearing glasses through the glasses wearing discrimination model;
[0061] If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation;
[0062] If the eye wears glasses, determine the type of glasses;
[0063] When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0064] When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction;
[0065] When the light spot on the lens corresponding to the eye in the glasses meets the conditions that affect the estimation of the line of sight direction, the eye image representing the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the conditions that affect the estimation of the line of sight direction, the eye image representing the eye can be used for estimating the line of sight direction.
[0066] In combination with the second possible implementation of the second aspect, the embodiment of the present application provides a third possible implementation of the second aspect, wherein the first judgment module, when used to determine the type of glasses if the eye is wearing glasses, is specifically used to:
[0067] determining an inner eyelid region and an outer eyelid region from the eye image of the eye;
[0068] Calculate the mean grayscale value of pixels in the inner eyelid area according to the grayscale value of each pixel point in the inner eyelid area; and calculate the mean grayscale value of pixels in the outer eyelid area according to the grayscale value of each pixel point in the outer eyelid area;
[0069] Calculating a second ratio between the mean grayscale value of pixels in the inner eyelid area and the mean grayscale value of pixels in the outer eyelid area;
[0070] When the second ratio is greater than the second preset threshold, the type of the glasses is determined to be sunshade glasses; when the second ratio is not greater than the second preset threshold, the type of the glasses is determined to be transparent glasses.
[0071] In combination with the second possible implementation manner of the second aspect, the embodiment of the present application provides a fourth possible implementation manner of the second aspect, wherein the first judgment module, when used to judge whether the light spot on the lens corresponding to the eye in the glasses meets the condition affecting the line of sight direction estimation when the glasses are transparent glasses, is specifically used to:
[0072] When the glasses are transparent glasses, calculating, according to the grayscale value of each pixel in the eye image of the eye, a third ratio of the number of pixels having a grayscale value greater than a third preset threshold to the total number of pixels in the eye image;
[0073] If the third ratio is greater than a fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses meets the condition of affecting the estimation of the line of sight direction;
[0074] If the third ratio is not greater than the fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses does not meet the condition affecting the estimation of the line of sight direction.
[0075] In combination with the second possible implementation of the second aspect, the embodiment of the present application provides a fifth possible implementation of the second aspect, wherein the sight line direction is the sight line direction of a single eye; when the third determination module is used to determine the driver's gaze area according to the sight line direction of the driver, the head position information, the head rotation angle, and the position information of each area in the vehicle, the third determination module is specifically used to:
[0076] For each moment between the current moment and the target historical moment, obtain a first historical eye image of each eye of the driver at a first historical moment of the moment and a second historical eye image at a second historical moment of the moment; wherein the first historical moment is a moment before the moment and a moment after the second historical moment;
[0077] For each eye, determine a first sight line direction of the eye at the first historical moment according to the first historical eye image of the eye, and determine a second sight line direction of the eye at the second historical moment according to the second historical eye image of the eye;
[0078] Predicting a predicted sight line direction of the eye at the moment according to the first sight line direction and the second sight line direction;
[0079] Calculating an angle between the predicted sight line direction and the sight line direction of the eye at the moment, and when the angle is not greater than a third preset angle, retaining the eye image corresponding to the eye at the moment; and when the angle is greater than the third preset angle, deleting the eye image corresponding to the eye at the moment;
[0080] According to the sight lines corresponding to the eye images retained by each eye of the driver at each of the moments, the mean of the sight lines corresponding to the retained eye images is calculated to obtain the sight lines of both eyes of the driver;
[0081] The driver's gaze area is determined based on the sight direction of the driver's eyes, the head position information, the head rotation angle and the position information of each area in the vehicle.
[0082] In combination with the second aspect, the embodiment of the present application provides a sixth possible implementation of the second aspect, wherein the head rotation angle includes a head yaw angle in a horizontal direction and a head pitch angle in a vertical direction; and the second judgment module is used to judge whether the driver's driving state is a distracted driving state according to the head rotation angle and the gaze area, specifically for:
[0083] When any one or more of the following distraction conditions are continuously satisfied within a preset time period, the driver's driving state is determined to be the distracted driving state; the distraction conditions include: the head yaw angle is greater than a first preset angle, the head pitch angle is greater than a second preset angle, and the gaze area is other areas outside the front windshield area of the vehicle.
[0084] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps in any possible implementation of the first aspect above are performed.
[0085] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in any possible implementation of the first aspect described above are executed.
[0086] The embodiments of the present application provide a driving state detection method, device, electronic device and readable storage medium, wherein the driver's head posture is determined by inputting the driver's head image into the head posture estimation model, and then the head posture is used to judge whether the driver's driving state is a distracted driving state. Compared with the method that relies on facial feature point information, this method in the present application can identify the driver's head posture regardless of the rotation angle of the driver's face or the pitch angle of the face, and can also detect the driver's distraction, which is conducive to reducing the safety risks of the driver during driving.
[0087] At the same time, when the image to be detected contains the driver's facial area, it is determined whether the eye image can be used for the estimation of the line of sight direction. When it can be used for the estimation of the line of sight direction, the line of sight direction is determined using the eye image, and then the driver's gaze area is determined according to the line of sight direction. Then, the driver's driving state is determined to be a distracted driving state by combining the gaze area and the head rotation angle in the head posture. In this embodiment, before using the eye image to determine the line of sight direction, it is determined whether the eye image is available. Only when the eye image is available, the eye image is used to determine the line of sight direction. This is conducive to ensuring that the determined line of sight direction is more accurate, thereby facilitating the improvement of the accuracy of the detection of the driver's distracted state.
[0088] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0090] Figure 1 A flow chart of a driving state detection method provided in an embodiment of the present application is shown;
[0091] Figure 2 A schematic diagram showing the position information of eye feature points of an eye provided in an embodiment of the present application;
[0092] Figure 3 A flow chart showing another driving state detection method provided in an embodiment of the present application is shown;
[0093] Figure 4 A schematic diagram of the structure of a driving state detection device provided in an embodiment of the present application is shown;
[0094] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0095] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0096] Considering that in the prior art, when detecting driver distraction, the solvPnP algorithm (an algorithm for estimating camera posture) is used to detect whether the driver is distracted, which depends on the driver's facial feature point information. When the camera cannot capture the driver's facial image, the driver's facial feature point information cannot be identified. At this time, the driver's distraction cannot be detected, which undoubtedly increases the safety risk during driving. In the method of using the driver's line of sight to determine whether a distracted state has occurred, if the driver wears sunglasses or the surface of the glasses is reflective, the line of sight direction recognition becomes unreliable, which can easily lead to errors in line of sight direction estimation, thereby affecting the accuracy of distraction detection. Based on this, the embodiments of the present application provide a driving state detection method, device, electronic device and readable storage medium, which are described below through embodiments.
[0097] To facilitate understanding of this embodiment, a driving state detection method disclosed in an embodiment of the present application is first introduced in detail. Figure 1 A flow chart of a driving state detection method provided in an embodiment of the present application is shown. Figure 1 As shown, the following steps S101-S109 are included:
[0098] S101: While the driver is driving the vehicle, an image to be detected including the driver's head at the current moment is acquired in real time.
[0099] In this embodiment, the vehicle driven by the driver includes a car, a new energy vehicle, etc. A camera is arranged in the cockpit of the vehicle, and the camera is used to collect a video stream including the driver's head in real time while the driver is driving the vehicle.
[0100] The video stream is obtained from the camera, and the video stream is decomposed into a single frame of the image to be detected containing the driver's head. Generally, the image to be detected at the current moment is the latest image to be detected decomposed from the video stream.
[0101] S102: Detecting the driver's head image from the image to be detected, inputting the head image into a trained head posture estimation model, and outputting the driver's head posture through the head posture estimation model; the head posture includes the head rotation angle and the head position information.
[0102] In this embodiment, the image to be detected includes not only the driver's head area but also the background area. In order to remove background interference, the head detection model is used to detect whether the image to be detected includes the driver's head area. When the driver's head area is included, the head detection model outputs a head detection frame, and then the driver's head image is cropped from the image to be detected along the head detection frame.
[0103] The head image is input into a pre-trained head posture estimation model, and the head posture of the driver is estimated by the head posture estimation model, wherein the head posture estimation model is trained using sample images containing the head.
[0104] The head posture includes the head rotation angle and the head position information. Among them, the head rotation angle includes the head yaw angle in the horizontal direction and the head pitch angle in the vertical direction. The head rotation angle refers to the angle of turning the head left and right. For example, when the driver looks back, the driver's head rotation angle can be 180 degrees. The head pitch angle refers to the angle of pitching the head up and down, for example, the driver looks up or looks down. The head position information is the head position coordinates in a three-dimensional coordinate system constructed with the camera as the center.
[0105] S103: When the image to be detected contains the driver's face area, the driver's face area is detected from the image to be detected.
[0106] In this embodiment, since the image to be detected includes the driver's head, the image to be detected may include the driver's facial area or may not include the driver's facial area.
[0107] When the image to be detected contains the driver's face area, steps S104 to S109 are continued.
[0108] Specifically, the face detection model is used to detect whether the image to be detected contains the driver's face area. When the driver's face area is contained, the face detection model is used to output a face area frame, and then the area framed by the face detection frame is used as the face area.
[0109] In a possible implementation, when the image to be detected does not include the driver's facial area, it is determined whether the driver's driving state is a distracted driving state based on the driver's head rotation angle.
[0110] In this embodiment, when the image to be detected does not include the driver's facial area, the subsequent steps S104-S109 are stopped, and the driver's head rotation angle is directly used to determine whether the driver's driving state is a distracted driving state.
[0111] When judging whether the driver's driving state is a distracted driving state according to the rotation angle of the driver's head, specifically: when the head yaw angle is greater than a first preset angle, or the head pitch angle is greater than a second preset angle, the driver's driving state is judged to be a distracted driving state. When the head yaw angle is not greater than the first preset angle, and the head pitch angle is not greater than the second preset angle, the driver's driving state is judged to be a non-distracted driving state.
[0112] S104: Determine the position information of facial feature points from the face area, and determine the position information of eye feature points from the position information of facial feature points.
[0113] The facial feature point location information includes: eye feature point location information, mouth feature point location information, nose feature point location information, eyebrow feature point location information, and facial contour feature point location information. Figure 2 As shown, the eye feature point position information may be eye contour feature point position information.
[0114] S105: determining the number of blocked eye feature points according to the eye feature point position information; and determining an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region.
[0115] In this embodiment, the total number of eye feature points is fixed, and the difference between the total number of eye feature points and the number of identifiable (ie, unobstructed) eye feature points can be used as the number of obstructed eye feature points.
[0116] S106: Determine whether the eye image can be used for line of sight direction estimation according to the number of blocked eye feature points.
[0117] In a possible implementation, the eye image in step S105 is an eye image of a single eye; the blocked eye feature points are also blocked eye feature points of a single eye. When executing step S106, the following steps S1061-S1068 may be specifically performed:
[0118] S1061: For each eye, calculate a first ratio of the number of blocked eye feature points corresponding to the eye to a preset total number of single eye feature points.
[0119] In this embodiment, when the eyes are not blocked, the number of blocked eye feature points of a single eye should be 0, and the total number of preset single eye feature points should be fixed, such as Figure 2 As shown, the total number of preset feature points of a single eye is 6.
[0120] In reality, the number of eye feature points of a single eye identified may be less than the total number of preset single eye feature points due to wearing glasses or head rotation, that is, there are occluded eye feature points.
[0121] In this embodiment, by calculating a first ratio of the number of occluded eye feature points corresponding to each eye to the total number of preset single eye feature points, it is determined through the first ratio whether the eye image corresponding to each eye can be used for line of sight direction estimation.
[0122] S1062: When the first ratio is greater than a first preset threshold, the eye image corresponding to the eye cannot be used for line of sight direction estimation.
[0123] In this embodiment, one eye of the driver is taken as an example for explanation. Figure 3 As shown, when the first ratio of the eye is greater than the first preset threshold, it indicates that the occluded portion of the eye is relatively large, and the eye image corresponding to the eye cannot be used for line of sight direction estimation. Exemplarily, the first preset threshold is 1 / 2.
[0124] S1063: When the first ratio is not greater than a first preset threshold, the eye image corresponding to the eye is input into a trained glasses wearing discrimination model to determine whether the eye is wearing glasses through the glasses wearing discrimination model.
[0125] like Figure 3 As shown, when the first ratio of the eye is not greater than (ie, less than or equal to) the first preset threshold, it indicates that the blocked portion of the eye is small, and at this time it is necessary to further determine whether each eye wears glasses.
[0126] S1064: If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation.
[0127] S1065: If the eye is wearing glasses, determine the type of the glasses.
[0128] In this embodiment, the types of glasses include shading glasses and transparent glasses, wherein shading glasses are such as sunglasses, and transparent glasses are such as myopia glasses, reading glasses, plain glasses, etc. Generally, the light transmittance of shading glasses is lower than that of transparent glasses.
[0129] In a possible implementation, when executing step S1065, the following steps may be specifically performed: determine the inner eyelid area and the outer eyelid area from the eye image of the eye. Calculate the mean grayscale value of the pixels in the inner eyelid area according to the grayscale value of each pixel in the inner eyelid area; and calculate the mean grayscale value of the pixels in the outer eyelid area according to the grayscale value of each pixel in the outer eyelid area. Calculate a second ratio between the mean grayscale value of the pixels in the inner eyelid area and the mean grayscale value of the pixels in the outer eyelid area. When the second ratio is greater than a second preset threshold, determine that the glasses are light-shielding glasses; when the second ratio is not greater than the second preset threshold, determine that the glasses are transparent glasses.
[0130] In this embodiment, if Figure 2 As shown, the inner eyelid area in the eye image refers to the area within the eye contour formed by eye feature points, and the outer eyelid area refers to other areas in the eye image except the inner eyelid area.
[0131] S1066: When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation.
[0132] S1067: When the glasses are transparent glasses, determine whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction.
[0133] In a possible implementation, when executing step S1067, it can be specifically performed according to the following steps: when the type of the glasses is transparent glasses, calculate the third ratio of the number of pixels whose grayscale values are greater than a third preset threshold to the total number of pixels in the eye image according to the grayscale value of each pixel in the eye image of the eye; if the third ratio of the eye is greater than a fourth preset threshold, then it indicates that the light spot on the lens corresponding to the eye in the glasses meets the condition that affects the estimation of the line of sight direction; if the third ratio of the eye is not greater than the fourth preset threshold, then it indicates that the light spot on the lens corresponding to the eye in the glasses does not meet the condition that affects the estimation of the line of sight direction.
[0134] S1068: When the light spot on the lens corresponding to the eye in the glasses meets the condition that affects the estimation of the line of sight direction, the eye image corresponding to the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the condition that affects the estimation of the line of sight direction, the eye image corresponding to the eye can be used for estimating the line of sight direction.
[0135] S107: When the eye image can be used for line of sight direction estimation, the eye image is input into a trained line of sight direction estimation model, and the line of sight direction of the driver is determined by the line of sight direction estimation model.
[0136] In this embodiment, when the eye image cannot be used to estimate the line of sight direction, the subsequent steps S107-S109 are stopped and the process of directly judging whether the driver's driving state is a distracted driving state according to the driver's head rotation angle is executed.
[0137] like Figure 1 As shown, in this embodiment, the execution order of step S102 and steps S103-S107 can be to execute step S102 first and then execute steps S103-S107; or to execute steps S103-S107 first and then execute step S102; or to execute step S102 and steps S103-S107 simultaneously, which is not limited in this application.
[0138] S108: Determine the driver's gaze area based on the driver's sight direction, head position information, head rotation angle, and position information of various areas in the vehicle.
[0139] In this embodiment, in order to improve the accuracy of determining the driver's gaze area, it is necessary to determine the stability of the driver's sight direction. When the driver's sight direction has a significant jump, it means that the sight direction is unstable, and the accuracy of the gaze area determined using the unstable sight direction will also be affected. Therefore, when determining the driver's annotation area, the stability of the sight direction can be determined first.
[0140] In a possible implementation, the sight direction of the driver is the sight direction of a single eye; when executing step S108, the following steps may be specifically performed:
[0141] S1081: For each moment between the current moment and the target historical moment, obtain the first historical eye image of each eye of the driver at the first historical moment of the moment and the second historical eye image at the second historical moment of the moment; wherein the first historical moment is the previous moment of the moment and is the next moment of the second historical moment.
[0142] S1082: For each eye, determine a first sight line direction of the eye at a first historical moment according to a first historical eye image of the eye, and determine a second sight line direction of the eye at a second historical moment according to a second historical eye image of the eye.
[0143] S1083: Predicting the predicted sight line direction of the eye at the moment according to the first sight line direction and the second sight line direction.
[0144] S1084: Calculate the angle between the predicted sight direction and the sight direction of the eye at that moment. When the angle is not greater than a third preset angle, retain the eye image corresponding to the eye at that moment. When the angle is greater than the third preset angle, delete the eye image corresponding to the eye at that moment.
[0145] S1085: According to the sight lines corresponding to the eye images retained by each eye of the driver at each moment, the mean of the sight lines corresponding to the retained eye images is calculated to obtain the sight lines of both eyes of the driver.
[0146] S1086: Determine the driver's gaze area based on the sight direction of the driver's eyes, head position information, head rotation angle and position information of various areas in the vehicle.
[0147] In step S1081, the target historical moment refers to the historical moment before the current moment, and the duration of the time period between the current moment and the target historical moment is the specified duration. Exemplarily, the current moment is t0, the target historical moment is tn, and when the specified duration is 6, n is 6, that is, the moments between the current moment t0 and the target historical moment t6 are t0, t1, t2, t3, t4, t5, t6, and the order of the moments from first to last is t6, t5, t4, t3, t2, t1, t0.
[0148] Taking time t4 as an example, a first historical eye image of each eye of the driver at a first historical moment t5 of time t4 and a second historical eye image at a second historical moment t6 of time t4 are obtained respectively.
[0149] In step S1082, taking one of the eyes as an example, the first historical eye image of the eye is input into the sight direction estimation model, and the sight direction estimation model outputs the first sight direction of the eye at the first historical moment t5. The second historical eye image of the eye is input into the sight direction estimation model, and the sight direction estimation model outputs the second sight direction of the eye at the second historical moment t6.
[0150] In step S1083, the predicted sight line direction of the eye at time t4 is predicted based on the first sight line direction and the second sight line direction of the eye.
[0151] In step S1084, the angle between the predicted sight line direction of the eye at time t4 determined in step S1083 and the sight line direction of the eye at time t4 determined in step S107 is calculated. When the angle is not greater than the third preset angle, the eye image corresponding to the eye at time t4 is retained; when the angle is greater than the third preset angle, the eye image corresponding to the eye at time t4 is deleted.
[0152] In step S1085, exemplarily, for the driver's left eye, at each moment t6, t5, t4, t3, t2, t1, and t0, if the eye images corresponding to moment t3 and moment t2 are deleted, the eye images corresponding to moment t6, moment t5, moment t4, moment t1, and moment t0 are retained; for the driver's right eye, at each moment t6, t5, t4, t3, t2, t1, and t0, if the eye images corresponding to moment t3 and moment t4 are deleted, the eye images corresponding to moment t6, moment t5, moment t2, moment t1, and moment t0 are retained, the sight line directions of the left eye at moments t6, moment t5, moment t4, moment t1, and moment t0 determined in step S105 and the average of the sight line directions of the right eye at moments t6, moment t5, moment t2, moment t1, and moment t0 determined in step S105 are calculated to obtain the sight line directions of both eyes of the driver.
[0153] In step S1086, the head position information and the position information of each area in the vehicle are both position coordinates in a three-dimensional rectangular coordinate system constructed with the camera as the center. Therefore, the driver's gaze area can be determined based on the line of sight of the driver's eyes, the head position information, the head rotation angle and the position information of each area in the vehicle.
[0154] S109: Determine whether the driver's driving state is a distracted driving state based on the head rotation angle and the gaze area.
[0155] In a possible implementation, the head rotation angle includes a head rotation angle in the horizontal direction and a head pitch angle in the vertical direction. When any one or more of the following distraction conditions are continuously met within a preset time period, the driver's driving state is determined to be a distracted driving state; the distraction conditions include: the head yaw angle is greater than a first preset angle, the head pitch angle is greater than a second preset angle, and the gaze area is other areas other than the front windshield area of the vehicle.
[0156] Based on the same technical concept, the present application also provides a driving state detection device, Figure 4 A schematic diagram of the structure of a driving state detection device provided in an embodiment of the present application is shown. Figure 4 As shown, the device comprises:
[0157] The first acquisition module 401 is used to acquire the image to be detected including the driver's head at the current moment in real time while the driver is driving the vehicle;
[0158] The first detection module 402 is used to detect the driver's head image from the image to be detected, input the head image into a trained head posture estimation model, and output the driver's head posture through the head posture estimation model; the head posture includes head rotation angle and head position information;
[0159] A second detection module 403, configured to detect the driver's face area from the image to be detected when the image to be detected contains the driver's face area;
[0160] A first determination module 404 is used to determine the position information of facial feature points from the facial region, so as to determine the position information of eye feature points from the facial feature point position information;
[0161] A second determination module 405 is used to determine the number of blocked eye feature points according to the eye feature point position information; and determine an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region;
[0162] A first determination module 406, configured to determine whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points;
[0163] An input module 407, configured to input the eye image into a trained line of sight direction estimation model when the eye image can be used for line of sight direction estimation, and determine the line of sight direction of the driver through the line of sight direction estimation model;
[0164] A third determination module 408 is used to determine the driver's gaze area according to the driver's sight direction, the head position information, the head rotation angle and the position information of each area in the vehicle;
[0165] The second judgment module 409 is used to judge whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area.
[0166] Optionally, the device further comprises:
[0167] The third judgment module is used to judge whether the driving state of the driver is a distracted driving state according to the rotation angle of the driver's head when the image to be detected does not contain the driver's facial area.
[0168] Optionally, the blocked eye feature points are blocked eye feature points of a single eye; when the first judgment module 406 is used to judge whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points, it is specifically used to:
[0169] For each eye, calculating a first ratio of the number of the blocked eye feature points corresponding to the eye to the total number of preset single eye feature points;
[0170] When the first ratio is greater than a first preset threshold, it indicates that the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0171] When the first ratio is not greater than the first preset threshold, inputting the eye image corresponding to the eye into a trained glasses wearing discrimination model, so as to determine whether the eye is wearing glasses through the glasses wearing discrimination model;
[0172] If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation;
[0173] If the eye wears glasses, determine the type of glasses;
[0174] When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation;
[0175] When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction;
[0176] When the light spot on the lens corresponding to the eye in the glasses meets the conditions that affect the estimation of the line of sight direction, the eye image representing the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the conditions that affect the estimation of the line of sight direction, the eye image representing the eye can be used for estimating the line of sight direction.
[0177] Optionally, when the first determining module 406 is used to determine the type of glasses if the eye is wearing glasses, it is specifically used to:
[0178] determining an inner eyelid region and an outer eyelid region from the eye image of the eye;
[0179] Calculate the mean grayscale value of pixels in the inner eyelid area according to the grayscale value of each pixel point in the inner eyelid area; and calculate the mean grayscale value of pixels in the outer eyelid area according to the grayscale value of each pixel point in the outer eyelid area;
[0180] Calculating a second ratio between the mean grayscale value of pixels in the inner eyelid area and the mean grayscale value of pixels in the outer eyelid area;
[0181] When the second ratio is greater than the second preset threshold, the type of the glasses is determined to be sunshade glasses; when the second ratio is not greater than the second preset threshold, the type of the glasses is determined to be transparent glasses.
[0182] Optionally, when the glasses are transparent glasses, the first judging module 406 is used to judge whether the light spot on the lens corresponding to the eye in the glasses meets the condition affecting the line of sight direction estimation, specifically for:
[0183] When the glasses are transparent glasses, calculating, according to the grayscale value of each pixel in the eye image of the eye, a third ratio of the number of pixels having a grayscale value greater than a third preset threshold to the total number of pixels in the eye image;
[0184] If the third ratio is greater than a fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses meets the condition of affecting the estimation of the line of sight direction;
[0185] If the third ratio is not greater than the fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses does not meet the condition affecting the estimation of the line of sight direction.
[0186] Optionally, the sight line direction is the sight line direction of a single eye; when the third determination module 408 is used to determine the driver's gaze area according to the sight line direction of the driver, the head position information, the head rotation angle and the position information of each area in the vehicle, it is specifically used to:
[0187] For each moment between the current moment and the target historical moment, obtain a first historical eye image of each eye of the driver at a first historical moment of the moment and a second historical eye image at a second historical moment of the moment; wherein the first historical moment is a moment before the moment and a moment after the second historical moment;
[0188] For each eye, determine a first sight line direction of the eye at the first historical moment according to the first historical eye image of the eye, and determine a second sight line direction of the eye at the second historical moment according to the second historical eye image of the eye;
[0189] Predicting a predicted sight line direction of the eye at the moment according to the first sight line direction and the second sight line direction;
[0190] Calculating an angle between the predicted sight line direction and the sight line direction of the eye at the moment, and when the angle is not greater than a third preset angle, retaining the eye image corresponding to the eye at the moment; and when the angle is greater than the third preset angle, deleting the eye image corresponding to the eye at the moment;
[0191] According to the sight lines corresponding to the eye images retained by each eye of the driver at each of the moments, the mean of the sight lines corresponding to the retained eye images is calculated to obtain the sight lines of both eyes of the driver;
[0192] The driver's gaze area is determined based on the sight direction of the driver's eyes, the head position information, the head rotation angle and the position information of each area in the vehicle.
[0193] Optionally, the head rotation angle includes a head yaw angle in a horizontal direction and a head pitch angle in a vertical direction; when the second judgment module 409 is used to judge whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area, it is specifically used to:
[0194] When any one or more of the following distraction conditions are continuously satisfied within a preset time period, the driver's driving state is determined to be the distracted driving state; the distraction conditions include: the head yaw angle is greater than a first preset angle, the head pitch angle is greater than a second preset angle, and the gaze area is other areas outside the front windshield area of the vehicle.
[0195] Figure 5 A structural diagram of an electronic device provided for an embodiment of the present application includes: a processor 501, a memory 502 and a bus 503, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device runs the above-mentioned information processing method, the processor 501 communicates with the memory 502 through the bus 503, and the processor 501 executes the machine-readable instructions to perform the method steps described in Example 1.
[0196] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method steps described in the first embodiment are executed.
[0197] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, electronic devices, and computer-readable storage media can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0198] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.
[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0201] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0202] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A driving state detection method, characterized in that: include: When the driver is driving the vehicle, the image to be detected including the driver's head at the current moment is obtained in real time; Detecting the driver's head image from the image to be detected, inputting the head image into a trained head posture estimation model, and outputting the driver's head posture through the head posture estimation model; the head posture includes head rotation angle and head position information; When the image to be detected contains the driver's facial area, detecting the driver's facial area from the image to be detected; Determine the facial feature point position information from the facial region, and determine the eye feature point position information from the facial feature point position information; Determining the number of blocked eye feature points according to the eye feature point position information; and determining an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region; Determining whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points; When the eye image can be used for line of sight direction estimation, inputting the eye image into a trained line of sight direction estimation model, and determining the line of sight direction of the driver by using the line of sight direction estimation model; Determining the driver's gaze area according to the driver's sight direction, the head position information, the head rotation angle, and the position information of each area in the vehicle; determining whether the driver's driving state is a distracted driving state according to the head rotation angle and the gaze area; The blocked eye feature points are blocked eye feature points of a single eye; The determining, according to the number of the blocked eye feature points, whether the eye image can be used for line of sight direction estimation includes: For each eye, calculating a first ratio of the number of the blocked eye feature points corresponding to the eye to the total number of preset single eye feature points; When the first ratio is greater than a first preset threshold, it indicates that the eye image corresponding to the eye cannot be used for line of sight direction estimation; When the first ratio is not greater than the first preset threshold, inputting the eye image corresponding to the eye into a trained glasses wearing discrimination model, so as to determine whether the eye is wearing glasses through the glasses wearing discrimination model; If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation; If the eye wears glasses, determine the type of glasses; When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation; When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction; When the light spot on the lens corresponding to the eye in the glasses meets the conditions that affect the estimation of the line of sight direction, the eye image representing the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the conditions that affect the estimation of the line of sight direction, the eye image representing the eye can be used for estimating the line of sight direction.
2. The method according to claim 1, characterized in that: The method further comprises: When the image to be detected does not include the driver's facial area, it is determined whether the driver's driving state is a distracted driving state according to the rotation angle of the driver's head.
3. The method according to claim 1, characterized in that: If the eye is wearing glasses, determining the type of the glasses includes: determining an inner eyelid region and an outer eyelid region from the eye image of the eye; Calculate the mean grayscale value of pixels in the inner eyelid area according to the grayscale value of each pixel point in the inner eyelid area; and calculate the mean grayscale value of pixels in the outer eyelid area according to the grayscale value of each pixel point in the outer eyelid area; Calculating a second ratio between the mean grayscale value of pixels in the inner eyelid area and the mean grayscale value of pixels in the outer eyelid area; When the second ratio is greater than the second preset threshold, the type of the glasses is determined to be sunshade glasses; when the second ratio is not greater than the second preset threshold, the type of the glasses is determined to be transparent glasses.
4. The method according to claim 1, characterized in that: When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction includes: When the glasses are transparent glasses, calculating, according to the grayscale value of each pixel in the eye image of the eye, a third ratio of the number of pixels having a grayscale value greater than a third preset threshold to the total number of pixels in the eye image; If the third ratio is greater than a fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses meets the condition of affecting the estimation of the line of sight direction; If the third ratio is not greater than the fourth preset threshold, it indicates that the light spot on the lens corresponding to the eye in the glasses does not meet the condition affecting the estimation of the line of sight direction.
5. The method according to claim 1, characterized in that: The sight line direction is the sight line direction of a single eye; and determining the driver's gaze area according to the sight line direction of the driver, the head position information, the head rotation angle, and the position information of each area in the vehicle includes: For each moment between the current moment and the target historical moment, obtain a first historical eye image of each eye of the driver at a first historical moment of the moment and a second historical eye image at a second historical moment of the moment; wherein the first historical moment is a moment before the moment and a moment after the second historical moment; For each eye, determine a first sight line direction of the eye at the first historical moment according to the first historical eye image of the eye, and determine a second sight line direction of the eye at the second historical moment according to the second historical eye image of the eye; Predicting a predicted sight line direction of the eye at the moment according to the first sight line direction and the second sight line direction; Calculating an angle between the predicted sight line direction and the sight line direction of the eye at the moment, and when the angle is not greater than a third preset angle, retaining the eye image corresponding to the eye at the moment; and when the angle is greater than the third preset angle, deleting the eye image corresponding to the eye at the moment; According to the sight lines corresponding to the eye images retained by each eye of the driver at each of the moments, the mean of the sight lines corresponding to the retained eye images is calculated to obtain the sight lines of both eyes of the driver; The driver's gaze area is determined based on the sight direction of the driver's eyes, the head position information, the head rotation angle and the position information of each area in the vehicle.
6. The method according to claim 1, characterized in that: The head rotation angle includes a head yaw angle in a horizontal direction and a head pitch angle in a vertical direction; judging whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area includes: When any one or more of the following distraction conditions are continuously satisfied within a preset time period, the driver's driving state is determined to be the distracted driving state; the distraction conditions include: the head yaw angle is greater than a first preset angle, the head pitch angle is greater than a second preset angle, and the gaze area is other areas outside the front windshield area of the vehicle.
7. A driving state detection device, characterized in that: include: The first acquisition module is used to acquire the image to be detected containing the driver's head at the current moment in real time while the driver is driving the vehicle; A first detection module is used to detect the driver's head image from the image to be detected, input the head image into a trained head posture estimation model, and output the driver's head posture through the head posture estimation model; the head posture includes head rotation angle and head position information; A second detection module, configured to detect the driver's face area from the image to be detected when the image to be detected contains the driver's face area; A first determination module is used to determine the position information of facial feature points from the facial region, so as to determine the position information of eye feature points from the facial feature point position information; A second determining module, configured to determine the number of blocked eye feature points according to the eye feature point position information; and determining an eye region from the face region according to the eye feature point position information, so as to determine an eye image according to the eye region; A first judgment module, used for judging whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points; An input module, used for inputting the eye image into a trained line of sight direction estimation model when the eye image can be used for line of sight direction estimation, and determining the line of sight direction of the driver through the line of sight direction estimation model; A third determination module is used to determine the driver's gaze area according to the driver's sight direction, the head position information, the head rotation angle and the position information of each area in the vehicle; A second judgment module is used to judge whether the driving state of the driver is a distracted driving state according to the head rotation angle and the gaze area; The blocked eye feature points are blocked eye feature points of a single eye; when the first judgment module is used to judge whether the eye image can be used for line of sight direction estimation according to the number of the blocked eye feature points, it is specifically used to: For each eye, calculating a first ratio of the number of the blocked eye feature points corresponding to the eye to the total number of preset single eye feature points; When the first ratio is greater than a first preset threshold, it indicates that the eye image corresponding to the eye cannot be used for line of sight direction estimation; When the first ratio is not greater than the first preset threshold, inputting the eye image corresponding to the eye into a trained glasses wearing discrimination model, so as to determine whether the eye is wearing glasses through the glasses wearing discrimination model; If the eye is not wearing glasses, it indicates that the eye can be used for line of sight direction estimation; If the eye wears glasses, determine the type of glasses; When the glasses are shading glasses, the eye image corresponding to the eye cannot be used for line of sight direction estimation; When the glasses are transparent glasses, determining whether the light spot on the lens corresponding to the eye in the glasses satisfies the condition affecting the estimation of the sight direction; When the light spot on the lens corresponding to the eye in the glasses meets the conditions that affect the estimation of the line of sight direction, the eye image representing the eye cannot be used for estimating the line of sight direction; when the light spot on the lens corresponding to the eye in the glasses does not meet the conditions that affect the estimation of the line of sight direction, the eye image representing the eye can be used for estimating the line of sight direction.
8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method as described in any one of claims 1 to 6 are performed.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are executed.
Citation Information
Patent Citations
Driving state detection method and device, driver monitoring system and vehicle
CN111079475A
Sight line area estimation method and system based on head posture and space attention
CN113361441A
Driving state sensing method and device
CN113569785A
Driver state detection method, electronic equipment and storage medium
CN114419606A
Method for monitoring concentration of driver of motor vehicle, involves determining reference direction and time target based on position of indicator, information about road course and / or information of lane maintenance assistants
DE102009005730A1
Cited By
Driving state detection method and apparatus, electronic device, and readable storage medium
EP4734077A1
