Low-light driver distraction detection method based on target and keypoint detection

Abstract

This invention discloses a low-light driver distraction detection method based on target and keypoint detection. The method extracts driver video sequences into images and performs illumination enhancement on these images to increase their brightness. After obtaining the high-brightness driver images, a YOLOv7-based joint target and keypoint detection network is used to identify the driver's face and its keypoints, the telephone and its keypoints, and the water cup and its keypoints. After obtaining the keypoint and target information, the driver's facial pose is calculated based on the face and its keypoint information to determine whether the driver is experiencing cognitive, visual, or mixed distraction. This invention achieves low-light driver distraction detection with a simple and flexible implementation method, effectively solving the problem of insufficient driver images in low-light conditions and improving the accuracy of distraction detection.

Description

Technical Field

[0001] This invention relates to the field of computer network technology, and in particular to a low-light driver distraction detection method based on target and key point detection. Background Technology

[0002] Driver distraction is a major cause of traffic accidents. Research by the Highway Administration has found that approximately 65% ​​of critical collisions and 80% of vehicle collisions are caused by driver distraction. Therefore, effective distraction detection methods can identify potential traffic hazards early, thereby reducing traffic risks. Illumination enhancement can increase the brightness of low-light images; object detection can identify key objects in images, such as faces, telephones, and water cups. The results of object detection can be used to determine whether the driver is distracted.

[0003] However, current driver distraction detection methods hardly consider the low-light problem of the acquired images. In reality, drivers may need to perform driving tasks in different weather conditions or at different times, such as cloudy days or nighttime, resulting in high signal-to-noise ratios and low contrast in the acquired images, which makes distraction detection difficult and reduces the performance of machine vision recognition.

[0004] Current driver distraction detection methods only detect visual distraction, failing to consider the different subcategories of distraction behaviors, such as visual distraction (eyes leaving the driving task) and cognitive distraction (cognitive distraction). The goal is to identify different types of driver distraction.

[0005] Current object detection methods can only obtain bounding boxes of objects and cannot identify key points of the objects. However, in tasks such as face recognition, in addition to the bounding boxes of the face, the key points of the face also provide a lot of important information. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a low-light driver distraction detection method based on target and key point detection.

[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a low-light driver distraction detection method based on target and key point detection, comprising the following steps:

[0008] (1) Use vehicle-mounted camera equipment to collect driver driving videos;

[0009] (2) Capture the driving video collected in step (1) to obtain driving images; construct a low-light enhancement network and train it to obtain a trained low-light enhancement network; perform illumination enhancement processing on the driving images according to the trained low-light enhancement network to obtain an enhanced image after illumination enhancement.

[0010] (3) Determine the key points of three types of targets: face, telephone, and water cup; construct and train a joint detection network based on YOLOv7 targets and key points to obtain a trained joint detection network; process the enhanced image obtained in step (2) according to the trained joint detection network to obtain the target boxes and key points corresponding to the face, telephone, and water cup in the enhanced image.

[0011] (4) Judge the output results corresponding to the target box and key points obtained in step (3) to obtain the driver's visual distraction count and cognitive distraction count.

[0012] (5) Compare the distraction count obtained in step (4) with the set distraction count threshold. When the visual distraction count is greater than the set visual distraction count threshold, the driver's visual distraction flag is set to one. When the cognitive distraction count is greater than the set cognitive distraction count threshold, the driver's cognitive distraction flag is set to one.

[0013] (6) Based on the visual distraction sign and cognitive distraction sign obtained in step (5), a judgment is made to obtain the distraction type corresponding to the driver; the distraction type includes visual distraction, cognitive distraction and mixed distraction.

[0014] Optionally, step (2) includes the following sub-steps:

[0015] (2.1) Extract the driving video acquired in step (1) at a frequency of 2 frames per second to obtain driving images;

[0016] (2.2) Construct a low-light enhancement network, which includes 7 CNN convolutional layers that are symmetrical front and back. Each convolutional layer includes 32 convolutional kernels with a size of 3×3. Each convolutional layer is connected to a ReLU activation function.

[0017] (2.3) Based on the input image and its pixel coordinates, construct a brightness enhancement curve function and iterate the brightness enhancement curve function multiple times to obtain the iterated brightness enhancement curve function;

[0018] (2.4) Determine the loss function of the low-light enhancement network based on the iterated brightness enhancement curve function, wherein the loss function of the low-light enhancement network includes the spatial consistency loss function L. spa Exposure control loss function L exp Color constancy loss function L col and illumination smoothness loss function

[0019] (2.5) The low-light enhancement network is trained using the DARK FACE dataset according to the loss function of the low-light enhancement network to obtain a trained low-light enhancement network.

[0020] (2.6) Input the driving image obtained in step (2.1) into the trained low-light enhancement network to obtain the enhanced image after light enhancement.

[0021] Optionally, the iteratively obtained brightness enhancement curve function is expressed as:

[0022] LE n (x)=LE n-1 (x)+A n LE n-1 (x)(1-LE n-1 (x))

[0023] Among them, LE n (x) represents the enhanced brightness after the nth iteration, where x represents the pixel coordinates of the input image, A n This represents the pixel parameters.

[0024] Optionally, the spatial consistency loss function L spa The expression is:

[0025]

[0026] Where K represents the number of pixels, i and j represent the traversal of pixels, and Y and I represent the brightness of the enhanced image and the brightness of the input image, respectively;

[0027] The exposure control loss function L exp The expression is:

[0028]

[0029] Where M represents the number of non-overlapping regions after the image is decomposed into 16×16, E represents the intermediate brightness value, and Yk represents the brightness value of the k-th pixel.

[0030] The color constancy loss function L col The expression is:

[0031]

[0032] Where J represents the average brightness of the color channel, and p and q represent the pairwise combinations of the three channels {R, G, B}.

[0033] The illumination smoothness loss function The expression is:

[0034]

[0035] Where N represents the number of iterations, ξ represents the RGB channels, and An represents the pixel parameters. These are the horizontal and vertical gradient operators, respectively;

[0036] The expression for the loss function of the low-light enhancement network is:

[0037]

[0038] Where W1, W2, W3, and W4 are the spatial consistency loss functions L spa Exposure control loss function L exp Color constancy loss function L col Illumination smoothness loss function The weight.

[0039] Optionally, step (3) includes the following sub-steps:

[0040] (3.1) Select the center point of the left eye, the center point of the right eye, the nose, the left corner of the mouth and the right corner of the mouth as the key points of the face, and select the coordinates of the upper left corner, the upper right corner, the lower left corner, the lower right corner and the center point of the target box as the key points of the telephone and the water cup, so as to determine the key points of the three types of targets: face, telephone and water cup.

[0041] (3.2) Construct a joint detection network based on YOLOv7 targets and key points. The joint detection network includes an input network, a backbone network, and a detection head network. The input network is used to input the image and align the input image into a 640×640 RGB three-channel image. The backbone network is used to extract image features and includes BConv, E-ELAN, and MPConv layers. The detection head network is used to identify targets and key points and includes SPPCPC, BConv, MPConv, Catconv, and RepVGG layers.

[0042] (3.3) Determine the loss function of the joint detection network, wherein the loss function of the joint detection network includes the regression loss function L. Bounding_box Category loss function L classification Confidence loss function L confidence and keypoint loss function L landmark ;

[0043] (3.4) The joint detection network is trained using the WiderFace dataset based on the target, key points and the loss function of the joint detection network to obtain a trained joint detection network;

[0044] (3.5) Input the enhanced image obtained in step (2) into the trained joint detection network to obtain the target bounding boxes of the face, telephone, and water cup in the enhanced image and the output results corresponding to the key points.

[0045] Optionally, the regression loss function L Bounding_box The expression is:

[0046]

[0047] in, Indicates the prediction box. Represents the true bounding box. express and The minimum bounding rectangle, where 'a' represents the 'a'-th grid, 'd' represents the 'd'-th anchor point, Distance_G represents the diagonal distance of the minimum bounding rectangle, and Distance_AB represents the predicted bounding box. and real frame The Euclidean distance between the two center points, where v measures the consistency of the aspect ratio.

[0048] The category loss function L classification The expression is:

[0049]

[0050] in, This indicates whether the d-th anchor point in the a-th grid is responsible for identifying the target. If it is responsible, then... otherwise s represents the number of grid cells; Let d represent the probability that the predicted bounding box at the d-th anchor point of the a-th grid belongs to category c. The probability that the true bounding box at the d-th anchor point of the a-th grid belongs to class c;

[0051] The confidence loss function L confidence The expression is:

[0052]

[0053] in, This represents the probability that the target object is located within the bounding box of the d-th anchor point in the a-th grid. Let λ represent the probability that the target object is located within the true bounding box of the d-th anchor point in a grid. noobj This represents the weighting coefficient for objects that are not present. This indicates that the d-th anchor point in the a-th grid is not responsible for identifying the target object;

[0054] The keypoint loss function L landmarkThe expression is:

[0055]

[0056] Where Si represents the predicted keypoint vector. Represents the true keypoint vector;

[0057] The expression for the loss function of the joint detection network is:

[0058] Loss = L Bounding_box +L classification +L confiddence +0.5L landmark .

[0059] Optionally, step (4) includes the following sub-steps:

[0060] (4.1) Determine whether there is a face and its key points in the output results corresponding to the target box and key points obtained in step (3). If there is no face and its key points, increment the driver's visual distraction count by one; otherwise, calculate the face pose to determine the Euler angle of the face pose, and determine whether the Euler angle is greater than the set Euler angle threshold. If the Euler angle is greater than the set Euler angle threshold, increment the driver's visual distraction count by one; otherwise, reset the driver's visual distraction count.

[0061] (4.2) Determine whether there is a telephone or a water cup in the output results corresponding to the target box and key points obtained in step (3). If there is a telephone or a water cup, increment the driver's cognitive distraction count by one; otherwise, reset the driver's cognitive distraction count.

[0062] Optionally, the step (4.1) of calculating the face pose to determine the Euler angles of the face pose and determining whether the Euler angles are greater than the set Euler angle threshold specifically includes:

[0063] (4.1.1) Calculate the face pose based on the target bounding box and the output results corresponding to the key points obtained in step (3) to determine the Euler angles of the face pose; the expression for the Euler angles of the face pose is:

[0064]

[0065]

[0066]

[0067] Among them, (x fe y fe ) represents the coordinates of the left eye of the face, (x re y re (x) represents the coordinates of the right eye.nose y nose (x) represents the coordinates of the nose. lm y lm (x) represents the coordinates of the left corner of the mouth. rm y rm Let (x, y) be the coordinates of the right corner of the mouth, (w, h) be the width and height of the face within the bounding box, and (x, y) be the coordinates of the right corner of the mouth. face y face () represents the coordinates of the top-left corner of the target bounding box;

[0068] (4.1.2) Determine whether the Euler angle of the face pose determined in step (4.1.1) is greater than the set Euler angle threshold. If it is greater than the set Euler angle threshold, increment the driver's visual distraction count by one; otherwise, reset the driver's visual distraction count.

[0069] A second aspect of the present invention provides a low-light driver distraction detection device based on target and key point detection, including one or more processors for implementing the above-described low-light driver distraction detection method based on target and key point detection.

[0070] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described low-light driver distraction detection method based on target and key point detection.

[0071] The beneficial effects of this invention are as follows: By constructing a low-light enhancement network to perform illumination enhancement operations on images, the brightness of low-light images is improved, which helps to improve recognition performance and facilitates distraction detection; This invention uses a joint detection network based on YOLOv7 for targets and key points to identify the driver's face and its key points, the telephone and its key points, and the water cup and its key points in the image; Then, based on the driver's face and its key point information, the facial pose is calculated, and this is used to determine whether the driver has cognitive distraction, visual distraction, or a combination of distraction, which helps to more accurately detect the type of driver distraction; The implementation method of this invention is simple and flexible, effectively solving the problem of insufficient driver images in low light and improving the accuracy of distraction detection. Attached Figure Description

[0072] Figure 1 This is a flowchart of the driver distraction detection method in low light conditions based on target and key point detection according to the present invention.

[0073] Figure 2 This is a schematic diagram of a low-light driver distraction detection device based on target and key point detection according to the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] See Figure 1 The low-light driver distraction detection method based on target and key point detection of the present invention includes the following steps:

[0076] (1) Use vehicle-mounted camera equipment to collect driver driving videos.

[0077] In this embodiment, an in-vehicle camera device is used to collect driver driving videos, including relevant driver information that needs to be captured during the journey. Specifically, a camera sensor can be selected to collect driver driving videos and obtain relevant image sequences for subsequent driver distraction detection.

[0078] (2) Capture the driving video collected in step (1) to obtain driving images; construct a low-light enhancement network and train it to obtain a trained low-light enhancement network; perform illumination enhancement processing on the driving images according to the trained low-light enhancement network to obtain an enhanced image after illumination enhancement.

[0079] It should be noted that in actual vehicle driving scenarios, there will be various interferences, such as driving at night or in poor lighting conditions such as cloudy days. In such cases, the signal-to-noise ratio of the driving video and the corresponding driving image is high and the contrast is low. Therefore, it is necessary to perform low-light enhancement operation on the driving image before distraction detection.

[0080] (2.1) Extract the driving video captured in step (1) at a frequency of 2 frames per second to obtain driving images.

[0081] In this embodiment, the driving video is captured at a frequency of 2 frames per second; it should be understood that it can also be captured at a frequency of 1 frame per second, or at a frequency of 1 frame every 2 seconds, depending on the actual situation.

[0082] (2.2) A low-light enhancement network (DCE-Net) is constructed. This DCE-Net consists of seven symmetrical CNN (Convolutional Neural Networks) convolutional layers. Each convolutional layer contains 32 convolutional kernels of size 3×3, and each convolutional layer is connected to a ReLU activation function. Each convolutional layer can learn color and lighting features. This low-light enhancement network can improve image details. When a driving image is input into the network, its features can be extracted to obtain a brightness-enhanced image corresponding to the driving image.

[0083] In this embodiment, the number of iterations n = 8 can be designed, and three channels {R, G, B} are included, for a total of 24 convolution channels.

[0084] (2.3) Based on the input image and its pixel coordinates, construct a brightness enhancement curve function and iterate the brightness enhancement curve function multiple times to obtain the iterated brightness enhancement curve function.

[0085] Specifically, assuming the pixel coordinate is x and the input image is I(x), the enhanced brightness is as follows:

[0086] LE(I(x);α)=I(x)+αI(x)(1-I(x))

[0087] The curve satisfies three conditions: pixel values ​​are normalized to [0,1], the curve is monotonic, and the curve is differentiable. Here, α∈[-1,1] are the parameters that need to be trained. Based on this, to adjust the image within a wider dynamic range and address the insufficient enhancement of the first-order curve, the curve is iterated n times. The image brightness after the iterations is as follows:

[0088] LE n (x)=LE n-1 (x)+α n LE n-1 (x)(1-LE n-1 (x))

[0089] In the above formula, α n It is a global pixel parameter, but due to the uneven brightness in low-light images, the global pixel parameter is prone to over-enhancing or under-enhancing local pixels. Therefore, the global pixel parameter α is... n Adjust the pixelation parameter A to match the image size. n This means that each pixel corresponds to a curve, avoiding over-enhancement or under-enhancement of the image. The enhanced brightness formula is as follows:

[0090] LE n (x)=LE n-1(x)+A n LE n-1 (x)(1-LE n-1 (x))

[0091] (2.4) Determine the loss function of the low-light enhancement network based on the iterated brightness enhancement curve function, wherein the loss function of the low-light enhancement network includes the spatial consistency loss function L. spa Exposure control loss function L exp Color constancy loss function L co1 and illumination smoothness loss function

[0092] In this embodiment, the spatial consistency loss function L spa This is used to control the pixel positions to remain unchanged before and after image enhancement. Y and I represent the brightness of the enhanced image and the brightness of the input image, respectively. i and j represent the pixel traversal, and K represents the number of pixels. The loss function formula is as follows:

[0093]

[0094] In this embodiment, the exposure control loss function L exp To control uneven brightness caused by excessively bright or dark images, the brightness of each pixel needs to be closer to a certain intermediate brightness value E (E=6). M represents the number of non-overlapping regions after the image is decomposed into 16×16. The loss function formula is as follows:

[0095]

[0096] In this embodiment, the color constant loss function L col This is used to control the uneven brightness of a specific color channel compared to other color channels in an image. J represents the average brightness of the color channels, and p and q represent pairwise combinations of the {R, G, B} channels. The color constant loss function L... co1 The expression is:

[0097]

[0098] In this embodiment, the illumination smoothing loss function Used to control uneven brightness between adjacent image pixels. These are the horizontal and vertical gradient operators, respectively, where ξ represents the RGB channels and A... n Represents pixel parameters. Illumination smoothing loss function. The expression is:

[0099]

[0100] Loss function L of low-light enhancement network totalComposed of the above four parts, W1 to W4 represent the weights of the four loss functions, respectively. The expression for the loss function of the low-light enhancement network is:

[0101]

[0102] (2.5) The low-light enhancement network is trained using the DARK FACE dataset based on the loss function of the low-light enhancement network to obtain a trained low-light enhancement network.

[0103] It should be understood that the DARK FACE dataset provides 6,000 real-world low-light images captured at night, including faces with labeled bounding boxes in school buildings, streets, bridges, overpasses, parks, etc. This public dataset can be used to train low-light enhancement networks.

[0104] During training, each batch contains 8 training samples, using a single 1080Ti GPU, with a learning rate of 0.0001. The weights in the loss function of the low-light enhancement network are initialized as follows: W1 = 1, W2 = 1, W3 = 0.5, W4 = 20. Low-light images from the WiderFace dataset are input into the low-light enhancement network batch by batch, and the loss function for each batch is calculated. Training stops when the loss function of the low-light enhancement network no longer changes. It should be understood that the weights in the loss function of the low-light enhancement network can be adjusted according to the learning rate and updated in reverse, and training can continue until the loss function of the low-light enhancement network no longer changes.

[0105] (2.6) Input the driving image obtained in step (2.1) into the trained low-light enhancement network to obtain the enhanced image after light enhancement.

[0106] (3) Determine the key points of three types of targets: face, telephone and water cup; construct and train a joint detection network based on YOLOv7 targets and key points to obtain a trained joint detection network; process the enhanced image obtained in step (2) according to the trained joint detection network to obtain the target boxes of face, telephone and water cup and the corresponding output results of key points in the enhanced image.

[0107] (3.1) Select key points for three types of recognition targets: face, telephone, and water cup. To ensure the consistency of data format, the recognition targets are selected into three categories: face, telephone, and water cup. The key points are selected as follows: In the determination of key points for face, five key points are selected for the face description, namely the center point of the left eye, the center point of the right eye, the nose, the left corner of the mouth, and the right corner of the mouth; In the determination of key points for telephone and water cup, the coordinates of the upper left corner, the upper right corner, the lower left corner, the lower right corner, and the center point are selected.

[0108] (3.2) Construct a joint detection network based on YOLOv7 targets and key points. The joint detection network includes an input network, a backbone network, and a head network. The input network is used to input the image and align the input image into a 640×640 RGB three-channel image. The backbone network is used to extract the features of the image. The head network is used to identify targets and key points.

[0109] In this embodiment, the backbone network consists of three network layers: four BConv layers, four E-ELAN layers, and three MPConv layers. The input image outputs three feature maps in the backbone network and then inputs them into the head network.

[0110] In this embodiment, the detection head network consists of five layers: SPPCPC, BConv, MPConv, Catconv, and RepVGG. The feature map input to the head network is upsampled twice, and then passes through the RepVGG layer to output unprocessed prediction results of sizes 80×80×16, 40×40×16, and 20×20×16.

[0111] (3.3) Determine the loss function of the joint detection network, which includes the regression loss function L. Bounding_box Category loss function L classification Confidence loss function L confiddence and keypoint loss function L landmark .

[0112] In this embodiment, the regression loss function L Bounding_box Used to control the position of the regression box, assuming the predicted box is... The real frame is and The minimum bounding rectangle is The subscript 'a' in the lower right corner represents the 'a'-th grid, and the subscript 'd' in the upper right corner represents the 'd'-th anchor point. The regression loss function is L. Bounding_box The expression is as follows:

[0113]

[0114] Where Distance_G represents the diagonal distance of the minimum bounding rectangle, and Distance_AB represents the predicted bounding box. and real frame The Euclidean distance between the two center points This function uses v to measure the consistency of the aspect ratio, and defines it as:

[0115]

[0116] Among them, w B and h B These represent the width and height of the actual bounding box, respectively. A and h A These represent the width and height of the prediction box, respectively.

[0117] In this embodiment, the category loss function L classification The mathematical expression for class identification used to control targets is:

[0118]

[0119] in, This indicates whether the d-th anchor point in the a-th grid is responsible for identifying the target, and if so, whether it is responsible for identification. Otherwise, the value is set to zero, where s represents the number of grid cells. Let d represent the probability that the predicted bounding box at the d-th anchor point of the a-th grid belongs to category c. This represents the probability that the true bounding box at the d-th anchor point of the a-th grid belongs to category c.

[0120] In this embodiment, the confidence loss function L confidence The cross-entropy loss function is also used to measure the confidence level of the predicted bounding box, and its mathematical expression is as follows:

[0121]

[0122] in, This represents the probability that the target object is located within the bounding box of the d-th anchor point in the a-th grid. Let λ represent the probability that the target object is located within the true bounding box of the d-th anchor point in a grid. noobj This represents the weighting coefficient for objects that are not present. This indicates that the d-th anchor point of the a-th grid is not responsible for identifying the target object.

[0123] In this embodiment, the key point loss function L landmark This is used to measure the location of key points. First, the Wing function is constructed, and its mathematical formula is as follows:

[0124]

[0125] C = ww·ln(1 + |x| / ∈)

[0126] Here, w is used to limit the nonlinear part of the function, and ∈ is used to limit the curvature of the nonlinear part of the function. The constant C = ww·ln(1+|x| / ∈) is used to smoothly connect the linear and nonlinear parts of the function. From this, the keypoint loss function L can be obtained. landmark The expression is:

[0127]

[0128] Where Si represents the predicted keypoint vector. Represents the true keypoint vector.

[0129] In summary, the expression for the loss function of the joint detection network is:

[0130] Loss = L Bounding_box +L classification +L confidence +0.5L landmark

[0131] (3.4) The joint detection network is trained using the WiderFace dataset based on the target, key points and the loss function of the joint detection network to obtain the trained joint detection network.

[0132] The training is based on YOLOv7 targets and keypoints, and the joint detection network is trained using the WiderFace dataset according to the loss function of the joint detection network.

[0133] It should be understood that the WiderFace dataset contains 32,203 images with face annotations, totaling 393,703 face data points. This public dataset can be used to train a joint target and keypoint detection network.

[0134] During training, each batch contains 64 training images, using a single 1080Ti GPU, with a learning rate of 0.001. Training images from the WiderFace dataset are input batch by batch into the joint object and keypoint detection network. The loss function for each batch is calculated, and training stops when the loss function of the joint object and keypoint detection network no longer changes. It should be understood that the weights in the loss function of the joint object and keypoint detection network can be adjusted based on the learning rate and updated in reverse, continuing training until the loss function of the YOLOv7-based joint object and keypoint detection network no longer changes.

[0135] (3.5) Input the enhanced image obtained in step (2) into the trained joint detection network to obtain the target boxes and key points corresponding to the face, telephone, and water cup in the enhanced image.

[0136] (4) Judge the output results corresponding to the target box and key points obtained in step (3) to obtain the driver's visual distraction count and cognitive distraction count.

[0137] (4.1) Determine whether there is a face and its key points in the output results corresponding to the target box and key points obtained in step (3). If there is no face and its key points, increment the driver's visual distraction count by one; otherwise, calculate the face pose to determine the Euler angle of the face pose, and determine whether the Euler angle is greater than the set Euler angle threshold. If the Euler angle is greater than the set Euler angle threshold, increment the driver's visual distraction count by one; otherwise, reset the driver's visual distraction count.

[0138] In this embodiment, the face pose is calculated to determine the Euler angles of the face pose, and it is determined whether the Euler angles are greater than a set Euler angle threshold. Specifically, the steps include the following:

[0139] (4.1.1) Calculate the face pose based on the target box and the output results corresponding to the key points obtained in step (3) to determine the Euler angles of the face pose.

[0140] The output obtained in step (3) includes five key points of the driver's face, namely the coordinates of the left eye (x, y, y). le y le ), right eye coordinates (x) re y re ), nose coordinates (x) nose y nose ), coordinates of the left corner of the mouth (x lm y lm ), coordinates of the right corner of the mouth (x rm y rm The bounding box for the face is defined by the width and height of the face (w, h) and the top-left corner coordinates (x, h). face y face ).

[0141] Euler angles (pitch, yaw, roll) are used to characterize the driver's facial posture, thereby further determining whether the driver's gaze is focused on the driving task. It is assumed that the face is symmetrical about the nose in the x-direction, and the distance from the nose to the top of the head in the y-direction is a fraction of the face height h. The formula for calculating Euler angles is as follows:

[0142]

[0143]

[0144]

[0145] Pitch is defined by the arctangent of the ratio of the difference between the horizontal and vertical coordinates of the two eyes; yaw is defined by the arcsine of the ratio of the difference between the horizontal coordinates of the nose and the center of the face to the face width; and roll is defined by the arcsine of the ratio of the difference between the vertical coordinates of the nose and the center of the face to the face height. An Euler angle threshold is used to determine whether the driver's head deflection is excessive, calculated using the following formula:

[0146]

[0147] Among them, P th R th Y t h represents the set Euler angle threshold.

[0148] (4.1.2) Determine whether the Euler angle of the face pose determined in step (4.1.1) is greater than the set Euler angle threshold. If it is greater than the set Euler angle threshold, increment the driver's visual distraction count by one; otherwise, reset the driver's visual distraction count.

[0149] For example, the Euler angle threshold can be set to: P th =R th =Y th =30°. When the Euler angle of the face pose determined in step (4.1.1) is greater than 30°, the driver's visual distraction count is incremented by one; otherwise, the visual distraction count is reset.

[0150] (4.2) Determine whether there is a telephone or a water cup in the output results corresponding to the target box and key points obtained in step (3). If there is a telephone or a water cup, increment the driver's cognitive distraction count by one; otherwise, reset the driver's cognitive distraction count.

[0151] (5) Compare the distraction count obtained in step (4) with the set distraction count threshold. When the visual distraction count is greater than the set visual distraction count threshold, set the driver's visual distraction flag to one. When the cognitive distraction count is greater than the set cognitive distraction count threshold, set the driver's cognitive distraction flag to one.

[0152] It is easy to understand that when the visual distraction count is less than or equal to the set visual distraction count threshold, the driving video and driving images are re-acquired for distraction detection; when the cognitive distraction count is less than or equal to the set cognitive distraction count threshold, the driving video and driving images are re-acquired for distraction detection.

[0153] (6) Based on the visual distraction signs and cognitive distraction signs obtained in step (5), determine the driver's corresponding distraction type. Among them, the distraction type includes visual distraction, cognitive distraction and mixed distraction.

[0154] Specifically, when both visual distraction signs and cognitive distraction signs are the same, the driver is determined to be mixed distracted; when only one of them is the same, the driver is determined to be the corresponding type of distraction. That is, when the visual distraction sign is the same and the cognitive distraction sign is not the same, the driver is determined to be visually distracted; when the cognitive distraction sign is the same and the visual distraction sign is not the same, the driver is determined to be cognitively distracted.

[0155] Corresponding to the aforementioned embodiments of the low-light driver distraction detection method based on target and key point detection, the present invention also provides embodiments of a low-light driver distraction detection device based on target and key point detection.

[0156] See Figure 2 The present invention provides a low-light driver distraction detection device based on target and key point detection, comprising one or more processors for implementing the low-light driver distraction detection method based on target and key point detection in the above embodiments.

[0157] The embodiments of the low-light driver distraction detection device based on target and key point detection of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 2 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the low-light driver distraction detection device based on target and key point detection according to the present invention. (Except for...) Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0158] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0159] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0160] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the low-light driver distraction detection method based on target and key point detection described in the above embodiments.

[0161] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0162] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-light driver distraction detection method based on target and keypoint detection, characterized in that, The method comprises the following steps: (1) collecting a driving video of a driver using a vehicle-mounted camera device; (2) intercepting the driving video collected in the step (1) to obtain a driving image; constructing a low-light enhancement network and training the low-light enhancement network to obtain a trained low-light enhancement network; performing light enhancement processing on the driving image according to the trained low-light enhancement network to obtain an enhanced image after light enhancement; (3) determining key points of three types of targets, i.e., a face, a phone and a cup; constructing a joint detection network based on yolov7 targets and key points and training the joint detection network to obtain a trained joint detection network; processing the enhanced image after light enhancement obtained in the step (2) according to the trained joint detection network to obtain output results corresponding to target boxes and key points of the face, the phone and the cup in the enhanced image; the joint detection network comprises an input network, a backbone network and a detection head network, wherein the input network is used to realize input of an image, and aligns the input image into an RGB three-channel image with a size of 640x640; the backbone network is used to extract features of the image, and the backbone network comprises a BConv layer, an E-ELAN layer and an MPConv layer; the detection head network is used to identify targets and key points, and the detection head network comprises an SPPCPC layer, a BConv layer, an MPConv layer, a Catconv layer and a RepVGG layer; (4) judging the output results corresponding to the target boxes and the key points obtained in the step (3) to obtain a visual distraction count and a cognitive distraction count of the driver; (5) comparing the distraction counts obtained in the step (4) with set distraction count thresholds, setting a visual distraction flag of the driver to one when the visual distraction count is greater than a set visual distraction count threshold, and setting a cognitive distraction flag of the driver to one when the cognitive distraction count is greater than a set cognitive distraction count threshold; (6) judging the visual distraction flag and the cognitive distraction flag obtained in the step (5) to obtain a distraction type corresponding to the driver; the distraction type comprises a visual distraction, a cognitive distraction and a mixed distraction.

2. The target and keypoint detection based low-light driver distraction detection method of claim 1, wherein, The step (2) comprises the following sub-steps: (2.1) intercepting the driving video collected in the step (1) at a frequency of 2 frames per second to obtain a driving image; (2.2) constructing a low-light enhancement network, wherein the low-light enhancement network comprises 7 CNN convolution layers symmetrically arranged in front and back, each of the convolution layers comprises 32 convolution kernels, the size of the convolution kernel is 3x3, and each of the convolution layers is connected with a ReLU activation function; (2.3) constructing a brightness enhancement curve function according to an input image and pixel coordinates thereof, and iterating the brightness enhancement curve function multiple times to obtain an iterated brightness enhancement curve function; (2.4) determining a loss function of the low-light enhancement network according to the iterated brightness enhancement curve function, wherein the loss function of the low-light enhancement network comprises a spatial consistency loss function L spa , an exposure control loss function L exp , a color constancy loss function L col , and an illumination smoothness loss function (2.5) training the low-light enhancement network using a DARK FACE data set according to a loss function of the low-light enhancement network to obtain a trained low-light enhancement network; (2.6) inputting the driving image obtained in the step (2.1) into the trained low-light enhancement network to obtain an enhanced image after light enhancement.

3. The target and keypoint detection based low-light driver distraction detection method of claim 2, wherein, The iterative brightness enhancement curve function is represented as: LE n (x) = LE n-1 (x) + A n LE n-1 (x)(1 - LE n-1 (x)) where LE n (x) represents the enhanced brightness after the nth iteration, x represents the pixel coordinates of the input image, A n represents the pixel point pixel parameter.

4. The target and keypoint detection based low-light driver distraction detection method of claim 2, wherein, The spatial consistency loss function L spa The expression is: wherein K represents the number of pixels, i and j represent the traversal of pixels, Y and I represent the brightness of the enhanced image and the input image respectively; The exposure control loss function L exp The expression is: where M represents the number of non-overlapping regions after the image is decomposed by 16x16, E indicates the intermediate luminance value, Y k represents the luminance value of the image of the kth pixel; The color constancy loss function L col The expression is: wherein J represents the brightness average value of the color channel, and p and q represent the two-by-two combination of the three channels of {R, G, B}; The illumination smoothness loss function The expression is: wherein N represents the number of iterations, ξ represents the RGB channel, A n representing the pixel parameters of the pixel point, are the horizontal and vertical gradient operators, respectively; The expression of the loss function of the low-light enhancement network is: Wherein, W1, W2, W3, W4 are weights of the spatial consistency loss function L spa , the exposure control loss function L exp , the color constancy loss function L col , the illumination smoothness loss function .

5. The target and keypoint detection based low-light driver distraction detection method of claim 1, wherein, The step (3) comprises the following sub-steps: (3.1) selecting the left eye center point, the right eye center point, the nose, the left corner of the mouth and the right corner of the mouth of the face as the key points of the face, and selecting the top-left corner coordinate, the top-right corner coordinate, the bottom-left corner coordinate, the bottom-right corner coordinate and the center point coordinate of the target frame as the key points of the phone and the cup to determine the key points of the three types of targets of the face, the phone and the cup; (3.2) constructing a joint detection network based on yolov7 targets and key points; (3.3) determining a loss function of the joint detection network, the loss function of the joint detection network comprising a regression loss function L Bounding_box , a category loss function L classification , a confidence loss function L confidence , and a keypoint loss function L landmark ; (3.4) training the joint detection network using the WiderFace dataset according to the targets and key points and the loss function of the joint detection network to obtain the trained joint detection network; (3.5) inputting the enhanced image obtained in the step (2) into the trained joint detection network to obtain the output results corresponding to the target frame and the key points of the face, the phone and the cup in the enhanced image.

6. The target and keypoint detection based low-light driver distraction detection method of claim 5, wherein, The regression loss function L Bounding_box The expression is: wherein, represents a predicted bounding box, represents a real bounding box, represents and a minimum enclosing rectangle, a represents the ath grid, d represents the dth anchor point, Distance_G represents the diagonal distance of the minimum enclosing rectangle, Distance_AB represents the Euclidean distance between the center of the predicted bounding box and the real bounding box two center points, v is a measure of the consistency of the aspect ratio, The category loss function L classification The expression is: wherein, whether the dth anchor point representing the ath grid is responsible for identifying this target, and if responsible for identifying, otherwise s represents the number of grids; a probability that the bounding box of the dth anchor point representing the ath grid belongs to the class c, a probability that the real bounding box of the dth anchor point representing the ath grid belongs to the class c; The confidence loss function L confidence The expression is: wherein, a probability that a target object is within a predicted box representing a dth anchor point of an ath grid, a probability that a target object is within a real box representing a dth anchor point of an ath grid, λ noobj a weight coefficient indicating no object, a dth anchor point of an ath grid is not responsible for identifying the target object; The key point loss function L landmark The expression is: where S i represents a predicted keypoint vector, represents a true keypoint vector; The expression of the loss function of the joint detection network is: Loss = L Bounding_box + L classification + L confidence + 0.5L landmark .

7. The target and keypoint detection based low-light driver distraction detection method of claim 1, wherein, The step (4) comprises the following sub-steps: (4.1) determining whether there is a face and its key points in the output results corresponding to the target frame and the key points obtained in the step (3), if not, increasing the visual distraction count of the driver by one; Otherwise, calculating the face pose to determine the Euler angle of the face pose, and determining whether the Euler angle is greater than the set Euler angle threshold, if the Euler angle is greater than the set Euler angle threshold, increasing the visual distraction count of the driver by one; Otherwise, resetting the visual distraction count of the driver; (4.2) determining whether there is a phone or a cup in the output results corresponding to the target frame and the key points obtained in the step (3), if so, increasing the cognitive distraction count of the driver by one; Otherwise, resetting the cognitive distraction count of the driver.

8. The target and keypoint detection based low-light driver distraction detection method of claim 7, wherein, The step (4.1) of calculating the face pose to determine the Euler angle of the face pose, and determining whether the Euler angle is greater than the set Euler angle threshold specifically comprises: (4.1.1) calculating the face pose according to the output results corresponding to the target frame and the key points obtained in the step (3) to determine the Euler angle of the face pose; the expression of the Euler angle of the face pose is: wherein (x le ,y le ) is the left eye coordinate, (x re ,y re ) is the right eye coordinate, (x nose ,y nose ) is the nose coordinate, (x lm ,y lm ) is the left corner of the mouth coordinate, (x rm ,y rm ) is the right corner of the mouth coordinate, (w, h) is the width and height of the face in the target frame, and (x face ,y face ) is the top-left corner coordinate of the target frame. (4.1.2) determining whether the Euler angle of the face pose determined in the step (4.1.1) is greater than the set Euler angle threshold, if so, increasing the visual distraction count of the driver by one; otherwise, resetting the visual distraction count of the driver.

9. A low-light driver distraction detection device based on target and keypoint detection, characterized in that, The application discloses a target and key point detection based low-light driver distraction detection method.

10. A computer-readable storage medium, characterized in that, A computer readable storage medium having stored thereon a program which, when executed by a processor, implements the target and key point detection based low-light driver distraction detection method.

Images