A driver attention monitoring method and system based on artificial intelligence

Abstract

The present invention discloses a driver attention monitoring method and system based on artificial intelligence, the method comprising: extracting directional histogram features and classifying the driver's frontal image by a support vector machine to obtain a driver's facial region image; detecting the eye region and mouth region in the driver's facial region image, and outputting the center position coordinates of the left eye, the right eye and the mouth; calculating the horizontal rotation angle, the pitch angle and the roll angle of the driver's head according to the center position coordinates; constructing an attention region classification model based on the horizontal rotation angle, the pitch angle and the roll angle of the driver's head to obtain the driver's attention region; using an artificial intelligence-driven long short-term memory network to process the horizontal rotation angle, the pitch angle and the roll angle of the driver's head and the attention region, monitoring the distraction of attention, and outputting the monitoring results. The present invention more accurately identifies the behavior of distraction of attention by analyzing the continuous changes in the driver's head posture.

Description

Technical Field

[0001] The present invention relates to the technical field of driving safety, and in particular to a driver attention monitoring method and system based on artificial intelligence. Background Art

[0002] With the rapid development of intelligent driving technology and the continuous increase in the number of cars, driver status monitoring technology is of great significance in improving driving safety and reducing the incidence of traffic accidents. Especially in long-term driving and complex road environments, it can monitor the driver's attention in real time, detect potential dangerous behaviors in time and issue warnings to ensure driving safety.

[0003] At present, the commonly used driver attention monitoring methods mainly rely on the camera to continuously capture the driver's facial expressions or eye states, and use rule-based or traditional machine learning methods to analyze the captured data. However, most monitoring methods are based on static images and can only detect the driver's attention state at a certain moment, without involving the driver's attention changes in a time series. They are prone to misjudgment or omission, and cannot accurately identify persistent attention distraction problems.

[0004] In addition, many existing monitoring methods only focus on the driver's face or eye area, but the external manifestations of the driver's inattention are diverse, and more information needs to be introduced to improve the accuracy of the monitoring results. For example, the driver's head posture change information, including but not limited to horizontal rotation, pitch, roll, etc., can provide important basis for judging whether the driver is concentrating.

[0005] Therefore, establishing a monitoring method that can more accurately identify the driver's attention status is one of the current research focuses in the field of intelligent driving. Summary of the invention

[0006] In view of this, the present invention provides a driver attention monitoring method and system based on artificial intelligence, aiming to more accurately identify distracting behaviors by analyzing the continuous changes in the driver's head posture.

[0007] In order to achieve the above object, the present invention discloses a driver attention monitoring method based on artificial intelligence, comprising the following steps:

[0008] S1: extracting directional histogram features and performing support vector machine classification on the collected front image of the driver to obtain the coordinates of the circumscribed rectangle of the driver's facial area, and obtaining the driver's facial area image according to the circumscribed rectangle coordinates;

[0009] S2: extract the facial contour in the driver's facial area image by edge detection, locate and detect the eye area and mouth area in the driver's facial area image by using the template matching algorithm, and output the center position coordinates of the left eye, right eye and mouth;

[0010] S3: Calculate the horizontal rotation angle, pitch angle and roll angle of the driver's head using geometric relationships according to the center position coordinates of the left eye, right eye and mouth;

[0011] S4: construct an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area; including:

[0012] Construct an attention area classification model to obtain the driver's attention area. The attention area classification model maps the horizontal rotation angle, pitch angle and roll angle of the driver's head to different attention areas respectively. , the mapping formula is as follows: ;

[0013] In the formula, and are the front area thresholds for horizontal rotation angle and pitch angle respectively; is the lateral gaze threshold; is the lower fixation threshold; is the upper fixation threshold; They represent the attention areas as front, left, right, bottom, top and other areas respectively;

[0014] S5: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver's head and the driver's attention area, monitors distraction, and outputs monitoring results.

[0015] Preferably, the step S1 comprises the following steps:

[0016] S11: Extract the direction histogram features of the collected driver's front image:

[0017] The collected frontal image of the driver Calculate the horizontal gradient of each pixel and vertical gradient ;

[0018] Based on the horizontal gradient and vertical gradient Calculate the gradient magnitude and gradient angle , the calculation formula is as follows:

[0019] ;

[0020] ;

[0021] In the formula, is the pixel position; and The front image of the driver At pixel position The gradient value of the horizontal gradient and the gradient value of the vertical gradient; is the gradient amplitude At pixel position The gradient amplitude at ; is the gradient angle At pixel position The angle value at the gradient angle represents the angle of the gradient direction relative to the horizontal direction, and the range is ;

[0022] Driver front view Divide into multiple non-overlapping image blocks, the size of the image block is , in each image block, the gradient angle of each pixel is Assigned to the corresponding direction interval, is the pixel position in the image block, and the direction interval is Angle range is divided into equally spaced direction intervals; when After the allocation is completed, Corresponding The sum is performed to obtain the direction histogram of each image block. The calculation formula is as follows:

[0023] ;

[0024] In the formula, for dimensional vector, indicating the The orientation histogram of the image block The value of the dimension; , is the total number of image blocks; ; For the The set of pixel positions contained in an image block; For the Direction intervals; is the gradient angle At pixel position The angle value at judge Is the value of If it is in the direction interval, it is 1, otherwise it is 0; is the gradient amplitude At pixel position The gradient amplitude at ;

[0025] S12: Facial region classification using support vectors:

[0026] Input the directional histogram features of each image block to a pre-trained support vector machine to classify the corresponding image block into a facial region. If the image block is classified as having a facial region, the upper left corner pixel position and the lower right corner pixel position of the image block are recorded.

[0027] After completing the facial region classification for all image blocks, among all image blocks classified as having facial regions, the pixel position where the upper left corner horizontal coordinate and the vertical coordinate simultaneously reach the minimum is selected as the upper left corner coordinate of the circumscribed rectangle of the driver's facial region, and the pixel position where the lower right corner horizontal coordinate and the vertical coordinate simultaneously reach the maximum is selected as the lower right corner coordinate of the circumscribed rectangle of the driver's facial region;

[0028] Crop the driver's front image based on the upper left corner coordinates and lower right corner coordinates of the driver's facial area , get the driver's facial area image.

[0029] Preferably, step S2 comprises the following steps:

[0030] S21: Perform Canny edge detection:

[0031] Apply the Canny edge detection algorithm to extract the edge image of the driver's facial area image obtained in step S1 , which is the facial contour;

[0032] S22: Use the eye shape template to perform eye region detection, and output the center position coordinates of the left eye and the right eye in the edge image;

[0033] S23: Use the mouth shape template to detect the mouth area and output the center position coordinates of the mouth in the edge image.

[0034] Further preferably, the step S22 comprises the following steps:

[0035] S221: Eye shape template matching:

[0036] The eye shape template is used for matching. The eye shape template is an image obtained by averaging the pre-collected left and right eye images of the driver after being processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Eye Matching Score , the calculation formula is as follows:

[0037] ;

[0038] In the formula, for The elements in the collection, Template for eye shape The set of pixel positions contained in ; For edge images At pixel position The pixel value at ; Template for eye shape At pixel position The pixel value at ;

[0039] S222: Determine the eye area:

[0040] By maximizing the edge image The eye matching score within the upper left quarter of the image block determines the edge image The center position coordinates of the left eye ; By maximizing the edge image The eye matching score within the upper right quarter of the image block determines the edge image The center position coordinates of the right eye .

[0041] Further preferably, the step S23 comprises the following steps:

[0042] S231: Mouth shape template matching:

[0043] The mouth shape template is used for matching. The mouth shape template is an image of the driver's mouth collected in advance and processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Mouth matching score , the calculation formula is as follows:

[0044] ;

[0045] In the formula, for The elements in the collection, Mouth shape template The set of pixel positions contained in ; For edge images At pixel position The pixel value at ; Mouth shape template At pixel position The pixel value at ;

[0046] S232: Determine the mouth area:

[0047] By maximizing the edge image The mouth matching score within the lower half of the image block determines the edge image The center coordinates of the mouth .

[0048] Preferably, step S3 comprises the following steps:

[0049] S31: Calculate the horizontal rotation angle:

[0050] Calculate the horizontal rotation angle of the driver's head based on the horizontal positions of the left and right eyes , the calculation formula is as follows:

[0051] ;

[0052] In the formula, is the focal length of the camera that captures the frontal image of the driver;

[0053] S32: Calculate the pitch angle:

[0054] Calculate the pitch angle of the driver's head based on the vertical positions of the eyes and mouth , the calculation formula is as follows:

[0055] ;

[0056] S33: Calculate the roll angle:

[0057] Calculate the tilt angle of the driver's head based on the vertical and horizontal positions of the eyes , the calculation formula is as follows:

[0058] .

[0059] Preferably, step S5 comprises the following steps:

[0060] S51: Constructing the input sequence of the long short-term memory network:

[0061] The horizontal rotation angle, pitch angle and roll angle of the driver's head obtained in step S3 and the attention area obtained in step S4 are used as the input data of the long short-term memory network. The input feature vector of each time step of the long short-term memory network is , the eigenvector formula is as follows:

[0062] ;

[0063] In the formula, is the time step, and the driver’s front image is captured once in each time step. and Respectively The horizontal rotation angle, pitch angle, roll angle and attention area corresponding to the front image of the driver taken at each time step;

[0064] The long short-term memory network is based on a The feature vector sequence contained in the time window is taken as input, and the feature vector sequence formula is as follows:

[0065] ;

[0066] In the formula, For the The input feature vector sequence of time steps;

[0067] S52: Building a long short-term memory network:

[0068] Will After inputting into the long short-term memory network, the output is the hidden state , and transform the hidden state through a fully connected layer and activation function Mapped to binary classification output;

[0069] S53: Training Long Short-Term Memory Networks:

[0070] The binary cross entropy loss function is used to train the long short-term memory network. The training formula is as follows:

[0071] ;

[0072] In the formula, is the number of training samples; ; For the The training samples are The true label of each time step is 1 or 0, where 1 represents distraction and 0 represents concentration, obtained by manual annotation; is the output of the long short-term memory network The training samples are The probability of driver distraction in each time step;

[0073] Use the stochastic gradient descent method to train the parameters of the long short-term memory network to minimize the loss function and obtain a trained long short-term memory network;

[0074] S54: Apply the trained LSTM network:

[0075] Use the trained long short-term memory network to judge the real-time data and output each time step If the driver is judged as distracted at multiple consecutive time steps, it is determined that the driver is currently in a distracted state.

[0076] The present invention also discloses a driver attention monitoring system based on artificial intelligence, comprising:

[0077] Facial region extraction module: extracts the direction histogram features and classifies the driver's frontal image using a support vector machine to obtain the coordinates of the circumscribed rectangle of the driver's facial region, and obtains the driver's facial region image based on the circumscribed rectangle coordinates;

[0078] Eye and mouth positioning monitoring module: extracts the facial contour in the driver's facial area image through edge detection, uses the template matching algorithm to locate and detect the eye area and mouth area in the driver's facial area image, and outputs the center position coordinates of the left eye, right eye and mouth;

[0079] Head posture estimation module: Based on the center position coordinates of the left eye, right eye and mouth, the horizontal rotation angle, pitch angle and roll angle of the driver's head are calculated using geometric relationships;

[0080] Attention area extraction module: constructs an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area;

[0081] Monitoring and judgment module: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver's head and the driver's attention area, monitors distraction, and outputs monitoring results;

[0082] The above-mentioned artificial intelligence-based driver attention monitoring method is implemented through this system.

[0083] Beneficial effects:

[0084] The present invention combines directional histogram feature extraction, support vector machine classification, template matching and other advanced image processing technologies to accurately extract the driver's facial features and precisely identify the eye and mouth areas. On this basis, the geometric relationship is further used to calculate the driver's head posture changes, so that the system can not only identify whether the driver is looking forward, but also judge the driver's attention distraction state by analyzing the rotation angle, pitch angle and roll angle of the head. This multi-dimensional analysis greatly improves the accuracy of driver attention monitoring and reduces misjudgment.

[0085] The present invention utilizes an artificial intelligence-driven long short-term memory network to perform in-depth analysis based on time series information. It not only judges the driver's status at a single moment, but also continuously analyzes the changes in the driver's head posture and attention area over a period of time, thereby capturing the changes in the driver's attention in the time series and accurately identifying continuous distraction behaviors, thereby effectively reducing the false alarms caused by single-frame image analysis and improving the system's monitoring effect during long-term driving.

[0086] The present invention effectively avoids false alarms caused by the driver's natural head movement or simple posture adjustment through multi-dimensional analysis of the driver's head posture. In addition, the use of time series analysis based on long short-term memory can identify short-term non-distracting behaviors, such as checking the rearview mirror or looking sideways at traffic conditions, to avoid the system being overly sensitive. The method of the present invention can still maintain a stable monitoring effect under complex lighting conditions and in the presence of certain obstructions, making the driver monitoring system more practical and reliable in actual applications and adaptable to complex driving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a flow chart of a driver attention monitoring method based on artificial intelligence in Example 1 of the present invention. DETAILED DESCRIPTION

[0088] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0089] Example 1

[0090] like Figure 1 As shown, a driver attention monitoring method based on artificial intelligence includes the following steps:

[0091] S1: Extract the direction histogram features and classify the driver's front image using a support vector machine to obtain the coordinates of the circumscribed rectangle of the driver's facial area, and obtain the driver's facial area image based on the circumscribed rectangle coordinates:

[0092] S11: Extract the direction histogram features of the collected driver's front image:

[0093] The collected frontal image of the driver Calculate the horizontal gradient of each pixel and vertical gradient , the calculation formula is as follows:

[0094] ;

[0095] ;

[0096] In the formula, is the pixel position, and are the horizontal and vertical coordinates of the pixel position respectively; and The front image of the driver At pixel position The gradient value of the horizontal gradient and the gradient value of the vertical gradient; , , and The front image of the driver At pixel position , , and The pixel value at ;

[0097] Based on the horizontal gradient and vertical gradient Calculate the gradient magnitude and gradient angle , the calculation formula is as follows:

[0098] ;

[0099] ;

[0100] In the formula, is the gradient amplitude At pixel position The gradient amplitude at ; is the gradient angle At pixel position The angle value at the gradient angle represents the angle of the gradient direction relative to the horizontal direction, and the range is , ignore the direction;

[0101] Driver front view Divide into multiple non-overlapping image blocks, the size of the image block is , in each image block, the gradient angle of each pixel is Assigned to the corresponding direction interval, is the pixel position in the image block, and the direction interval is Angle range is divided into equally spaced direction intervals; when After the allocation is completed, Corresponding The sum is performed to obtain the direction histogram of each image block. The calculation formula is as follows:

[0102] ;

[0103] In the formula, for dimensional vector, indicating the The orientation histogram of the image block The value of the dimension; , is the total number of image blocks; ; For the The set of pixel positions contained in an image block; For the Direction intervals; is the gradient angle At pixel position The angle value at judge Is the value of If it is in the direction interval, it is 1, otherwise it is 0; is the gradient amplitude At pixel position The gradient amplitude at ;

[0104] S12: Facial region classification using support vectors:

[0105] Input the directional histogram features of each image block to a pre-trained support vector machine to classify the corresponding image block into a facial region. If the image block is classified as having a facial region, the upper left corner pixel position and the lower right corner pixel position of the image block are recorded.

[0106] After completing the facial region classification for all image blocks, among all image blocks classified as having facial regions, the pixel position where the upper left corner horizontal coordinate and the vertical coordinate simultaneously reach the minimum is selected as the upper left corner coordinate of the circumscribed rectangle of the driver's facial region, and the pixel position where the lower right corner horizontal coordinate and the vertical coordinate simultaneously reach the maximum is selected as the lower right corner coordinate of the circumscribed rectangle of the driver's facial region;

[0107] Crop the driver's front image based on the upper left corner coordinates and lower right corner coordinates of the driver's facial area , get the driver's facial area image.

[0108] The directional histogram feature can capture the local gradient direction and edge information in the image, and is particularly suitable for detecting targets with obvious edge structures such as faces. Combined with the classification capability of the support vector machine, it can further improve the accuracy and robustness of facial area detection. Therefore, step S1 can effectively and accurately identify the facial area from the frontal image of the driver by extracting the directional histogram features and combining them with the support vector machine classification.

[0109] S2: Extract the facial contour in the driver's facial area image through edge detection, use the template matching algorithm to locate and detect the eye area and mouth area in the driver's facial area image, and output the center position coordinates of the left eye, right eye and mouth:

[0110] S21: Perform Canny edge detection:

[0111] Apply the Canny edge detection algorithm to extract the edge image of the driver's facial area image obtained in step S1 , which is the facial contour;

[0112] S22: Use the eye shape template to perform eye region detection, and output the center position coordinates of the left eye and the right eye in the edge image;

[0113] S221: Eye shape template matching:

[0114] The eye shape template is used for matching. The eye shape template is an image obtained by averaging the pre-collected left and right eye images of the driver after being processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Eye Matching Score , the calculation formula is as follows:

[0115] ;

[0116] In the formula, for The elements in the collection, Template for eye shape The set of pixel positions contained in ; and Eye shape templates The horizontal and vertical coordinates of the pixel position; For edge images At pixel position The pixel value at ; Template for eye shape At pixel position The pixel value at ;

[0117] S222: Determine the eye area:

[0118] By maximizing the edge image The eye matching score within the upper left quarter of the image block determines the edge image The center position coordinates of the left eye ; By maximizing the edge image The eye matching score within the upper right quarter of the image block determines the edge image The center position coordinates of the right eye .

[0119] S23: Use the mouth shape template to detect the mouth area and output the center position coordinates of the mouth in the edge image:

[0120] S231: Mouth shape template matching:

[0121] The mouth shape template is used for matching. The mouth shape template is an image of the driver's mouth collected in advance and processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Mouth matching score , the calculation formula is as follows:

[0122] ;

[0123] In the formula, for The elements in the collection, Mouth shape template The set of pixel locations contained in and Mouth shape templates The horizontal and vertical coordinates of the pixel position; For edge images At pixel position The pixel value at ; Mouth shape template At pixel position The pixel value at ;

[0124] S232: Determine the mouth area:

[0125] By maximizing the edge image The mouth matching score within the lower half of the image block determines the edge image The center coordinates of the mouth .

[0126] Canny edge detection is very sensitive to edge changes in the image and can effectively extract edge contour information of the driver's face, while template matching can accurately locate the center of the eyes and mouth with the help of prior information. This process ensures the accurate positioning of key facial features and reduces the possibility of false detection; therefore, step S2 can accurately identify the key areas in the driver's facial image - the eyes and mouth - by combining Canny edge detection with template matching.

[0127] S3: Based on the center position coordinates of the left eye, right eye and mouth, the horizontal rotation angle, pitch angle and roll angle of the driver's head are calculated using geometric relationships:

[0128] S31: Calculate the horizontal rotation angle:

[0129] Calculate the horizontal rotation angle of the driver's head based on the horizontal positions of the left and right eyes , the calculation formula is as follows:

[0130] ;

[0131] In the formula, is the focal length of the camera that captures the frontal image of the driver;

[0132] S32: Calculate the pitch angle:

[0133] Calculate the pitch angle of the driver's head based on the vertical positions of the eyes and mouth , the calculation formula is as follows:

[0134] ;

[0135] S33: Calculate the roll angle:

[0136] Calculate the tilt angle of the driver's head based on the vertical and horizontal positions of the eyes , the calculation formula is as follows:

[0137] .

[0138] Step S3 calculates the horizontal rotation angle, pitch angle and roll angle of the head based on the geometric relationship between the left and right eyes and the mouth position, avoiding complex three-dimensional reconstruction or deep learning models. It can judge the driver's head posture in real time with low computational complexity and is suitable for driver monitoring systems with high real-time requirements.

[0139] S4: Construct an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area:

[0140] Construct an attention area classification model to obtain the driver's attention area. The attention area classification model maps the horizontal rotation angle, pitch angle and roll angle of the driver's head to different attention areas respectively. , the mapping formula is as follows: ;

[0141] In the formula, and They are the front area thresholds of the horizontal rotation angle and the pitch angle, and in this embodiment, both ; is the lateral gaze threshold, which is ; is the lower gaze threshold, which is ; is the upper gaze threshold, which is ; They represent the attention areas in front, on the left, on the right, below, above and other areas respectively.

[0142] Step S4 infers the driver's attention area by combining the horizontal rotation, pitch and roll posture changes of the driver's head. Compared with the analysis of a single angle, the method of the present invention can effectively capture complex head posture changes based on multiple dimensions, avoid missing or misjudging the driver's attention state, and because the method mainly relies on the geometric calculation of the head angle, it can quickly calculate and output the driver's attention area, which is very suitable for application in real-time driving monitoring systems.

[0143] S5: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver’s head and the driver’s attention area, monitors distraction, and outputs monitoring results:

[0144] S51: Constructing the input sequence of the long short-term memory network:

[0145] The horizontal rotation angle, pitch angle and roll angle of the driver's head obtained in step S3 and the attention area obtained in step S4 are used as the input data of the long short-term memory network. The input feature vector of each time step of the long short-term memory network is , the eigenvector formula is as follows:

[0146] ;

[0147] In the formula, is the time step, and the driver’s front image is captured once in each time step. and Respectively The horizontal rotation angle, pitch angle, roll angle and attention area corresponding to the front image of the driver taken at each time step;

[0148] The long short-term memory network is based on a The feature vector sequence contained in the time window is taken as input, and the feature vector sequence formula is as follows:

[0149] ;

[0150] In the formula, For the The input feature vector sequence of time steps;

[0151] S52: Building a long short-term memory network:

[0152] Will After inputting into the long short-term memory network, the output is the hidden state , and transform the hidden state through a fully connected layer and activation function The mapping is a binary classification output, and the mapping formula is as follows:

[0153] ;

[0154] ;

[0155] In the formula, It is a long short-term memory network. In this embodiment, a long short-term memory network with 3 layers is used, and the dimension of the hidden state is 128; and are the weights and bias values ​​of the fully connected layer respectively; is the activation function, and the sigmoid function is used in this embodiment; For the The probability of driver distraction in each time step;

[0156] S53: Training Long Short-Term Memory Networks:

[0157] The binary cross entropy loss function is used to train the long short-term memory network. The training formula is as follows:

[0158] ;

[0159] In the formula, is the number of training samples; ; For the The training samples are The true label of each time step is 1 or 0, where 1 represents distraction and 0 represents concentration, obtained by manual annotation; is the output of the long short-term memory network The training samples are The probability of driver distraction in each time step;

[0160] Use the stochastic gradient descent method to train the parameters of the long short-term memory network to minimize the loss function and obtain a trained long short-term memory network;

[0161] S54: Apply the trained LSTM network:

[0162] Use the trained long short-term memory network to judge the real-time data and output each time step If the attention is determined to be distracted in multiple consecutive time steps, it is determined that the driver is currently in a distracted state. In this embodiment, the attention is determined in 5 consecutive time steps.

[0163] The long short-term memory network can process the data of the driver's head posture and attention area in real time, and generate a real-time judgment result of the driver's attention state; the advantage of the long short-term memory network is that it can remember the past input data and use this information to make current judgments, which is particularly suitable for processing time series data, so that the system can not only identify the attention state under a single frame image, but also identify the attention changes in a continuous time period, especially for short-term distraction and continuous attention distraction, and the recognition is more accurate. Therefore, step S5 uses the long short-term memory network to process the time series information of the driver's head posture and attention area, and through the monitoring of continuous time steps, effectively captures the driver's state changes during driving, and realizes dynamic driver attention state monitoring. This continuous time monitoring method can effectively reduce single-frame misjudgment and improve the robustness and accuracy of monitoring.

[0164] Example 2

[0165] An artificial intelligence-based driver attention monitoring system includes the following five modules:

[0166] Facial region extraction module: extracts the direction histogram features and classifies the driver's frontal image using a support vector machine to obtain the coordinates of the circumscribed rectangle of the driver's facial region, and obtains the driver's facial region image based on the circumscribed rectangle coordinates;

[0167] Eye and mouth positioning monitoring module: extracts the facial contour in the driver's facial area image through edge detection, uses the template matching algorithm to locate and detect the eye area and mouth area in the driver's facial area image, and outputs the center position coordinates of the left eye, right eye and mouth;

[0168] Head posture estimation module: Based on the center position coordinates of the left eye, right eye and mouth, the horizontal rotation angle, pitch angle and roll angle of the driver's head are calculated using geometric relationships;

[0169] Attention area extraction module: constructs an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area;

[0170] Monitoring and judgment module: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver's head and the driver's attention area, monitors distraction, and outputs monitoring results;

[0171] The driver attention monitoring method based on artificial intelligence in Example 1 is implemented through this system.

[0172] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0174] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A driver attention monitoring method based on artificial intelligence, characterized in that: The following steps are involved: S1: extracting directional histogram features and performing support vector machine classification on the collected front image of the driver to obtain the coordinates of the circumscribed rectangle of the driver's facial area, and obtaining the driver's facial area image according to the circumscribed rectangle coordinates; S2: extract the facial contour in the driver's facial area image by edge detection, locate and detect the eye area and mouth area in the driver's facial area image by using the template matching algorithm, and output the center position coordinates of the left eye, right eye and mouth; S3: Calculate the horizontal rotation angle, pitch angle and roll angle of the driver's head using geometric relationships according to the center position coordinates of the left eye, right eye and mouth; S4: construct an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area; including: Construct an attention area classification model to obtain the driver's attention area. The attention area classification model maps the horizontal rotation angle, pitch angle and roll angle of the driver's head to different attention areas respectively. , the mapping formula is as follows: ; In the formula, and are the front area thresholds for horizontal rotation angle and pitch angle respectively; is the lateral gaze threshold; is the lower fixation threshold; is the upper fixation threshold; They represent the attention areas as front, left, right, bottom, top and other areas respectively; S5: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver's head and the driver's attention area, monitors distraction, and outputs monitoring results.

2. The driver attention monitoring method based on artificial intelligence according to claim 1 is characterized in that: The step S1 comprises the following steps: S11: Extract the direction histogram features of the collected driver's front image: The collected frontal image of the driver Calculate the horizontal gradient of each pixel and vertical gradient ; Based on the horizontal gradient and vertical gradient Calculate the gradient magnitude and gradient angle , the calculation formula is as follows: ； ； In the formula, is the pixel position; and The front image of the driver At pixel position The gradient value of the horizontal gradient and the gradient value of the vertical gradient; is the gradient amplitude At pixel position The gradient amplitude at ; is the gradient angle At pixel position The angle value at the gradient angle represents the angle of the gradient direction relative to the horizontal direction, and the range is ; Driver front view Divide into multiple non-overlapping image blocks, the size of the image block is , in each image block, the gradient angle of each pixel is Assigned to the corresponding direction interval, is the pixel position in the image block, and the direction interval is Angle range is divided into equally spaced direction intervals; when After the allocation is completed, Corresponding The sum is performed to obtain the direction histogram of each image block. The calculation formula is as follows: ； In the formula, for dimensional vector, indicating the The orientation histogram of the image block The value of the dimension; , is the total number of image blocks; ; For the The set of pixel positions contained in an image block; For the Direction intervals; is the gradient angle At pixel position The angle value at judge Is the value of If it is in the direction interval, it is 1, otherwise it is 0; is the gradient amplitude At pixel position The gradient amplitude at ; S12: Facial region classification using support vectors: Input the directional histogram features of each image block to a pre-trained support vector machine to classify the corresponding image block into a facial region. If the image block is classified as having a facial region, the upper left corner pixel position and the lower right corner pixel position of the image block are recorded. After completing the facial region classification for all image blocks, among all image blocks classified as having facial regions, the pixel position where the upper left corner horizontal coordinate and the vertical coordinate simultaneously reach the minimum is selected as the upper left corner coordinate of the circumscribed rectangle of the driver's facial region, and the pixel position where the lower right corner horizontal coordinate and the vertical coordinate simultaneously reach the maximum is selected as the lower right corner coordinate of the circumscribed rectangle of the driver's facial region; Crop the driver's front image based on the upper left corner coordinates and lower right corner coordinates of the driver's facial area , get the driver's facial area image.

3. The driver attention monitoring method based on artificial intelligence according to claim 1, characterized in that: The step S2 comprises the following steps: S21: Perform Canny edge detection: Apply the Canny edge detection algorithm to extract the edge image of the driver's facial area image obtained in step S1 , which is the facial contour; S22: Use the eye shape template to perform eye region detection, and output the center position coordinates of the left eye and the right eye in the edge image; S23: Use the mouth shape template to detect the mouth area and output the center position coordinates of the mouth in the edge image.

4. The driver attention monitoring method based on artificial intelligence according to claim 3 is characterized in that: The step S22 comprises the following steps: S221: Eye shape template matching: The eye shape template is used for matching. The eye shape template is an image obtained by averaging the pre-collected left and right eye images of the driver after being processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Eye Matching Score , the calculation formula is as follows: ； In the formula, for The elements in the collection, Template for eye shape The set of pixel positions contained in ; For edge images At pixel position The pixel value at ; Template for eye shape At pixel position The pixel value at ; S222: Determine the eye area: By maximizing the edge image The eye matching score within the upper left quarter of the image block determines the edge image The center position coordinates of the left eye ; By maximizing the edge image The eye matching score within the upper right quarter of the image block determines the edge image The center position coordinates of the right eye .

5. The driver attention monitoring method based on artificial intelligence according to claim 4 is characterized in that: The step S23 includes the following steps: S231: Mouth shape template matching: The mouth shape template is used for matching. The mouth shape template is an image of the driver's mouth collected in advance and processed by the Canny edge detection algorithm. The edge image is calculated by the sliding window method. Each pixel position Mouth matching score , the calculation formula is as follows: ； In the formula, for The elements in the collection, Mouth shape template The set of pixel positions contained in ; For edge images At pixel position The pixel value at ; Mouth shape template At pixel position The pixel value at ; S232: Determine the mouth area: By maximizing the edge image The mouth matching score within the lower half of the image block determines the edge image The center coordinates of the mouth .

6. The driver attention monitoring method based on artificial intelligence according to claim 5 is characterized in that: The step S3 comprises the following steps: S31: Calculate the horizontal rotation angle: Calculate the horizontal rotation angle of the driver's head based on the horizontal positions of the left and right eyes , the calculation formula is as follows: ； In the formula, is the focal length of the camera that captures the frontal image of the driver; S32: Calculate the pitch angle: Calculate the pitch angle of the driver's head based on the vertical positions of the eyes and mouth , the calculation formula is as follows: ； S33: Calculate the roll angle: Calculate the tilt angle of the driver's head based on the vertical and horizontal positions of the eyes , the calculation formula is as follows: 。 7. The driver attention monitoring method based on artificial intelligence according to claim 6 is characterized in that: The step S5 comprises the following steps: S51: Constructing the input sequence of the long short-term memory network: The horizontal rotation angle, pitch angle and roll angle of the driver's head obtained in step S3 and the attention area obtained in step S4 are used as the input data of the long short-term memory network. The input feature vector of each time step of the long short-term memory network is , the eigenvector formula is as follows: ； In the formula, is the time step, and the driver’s front image is captured once in each time step. and Respectively The horizontal rotation angle, pitch angle, roll angle and attention area corresponding to the front image of the driver taken at each time step; The long short-term memory network is based on a The feature vector sequence contained in the time window is taken as input, and the feature vector sequence formula is as follows: ； In the formula, For the The input feature vector sequence of time steps; S52: Building a long short-term memory network: Will After inputting into the long short-term memory network, the output is the hidden state , and transform the hidden state through a fully connected layer and activation function Mapped to binary classification output; S53: Training Long Short-Term Memory Networks: The binary cross entropy loss function is used to train the long short-term memory network. The training formula is as follows: ； In the formula, is the number of training samples; ; For the The training samples are The true label of each time step is 1 or 0, where 1 represents distraction and 0 represents concentration, obtained by manual annotation; is the output of the long short-term memory network The training samples are The probability of driver distraction in each time step; Use the stochastic gradient descent method to train the parameters of the long short-term memory network to minimize the loss function and obtain a trained long short-term memory network; S54: Apply the trained LSTM network: Use the trained long short-term memory network to judge the real-time data and output each time step If the driver is judged as distracted at multiple consecutive time steps, it is determined that the driver is currently in a distracted state.

8. A driver attention monitoring system based on artificial intelligence, characterized in that: include: Facial region extraction module: extracts the direction histogram features and classifies the driver's frontal image using a support vector machine to obtain the coordinates of the circumscribed rectangle of the driver's facial region, and obtains the driver's facial region image based on the circumscribed rectangle coordinates; Eye and mouth positioning monitoring module: extracts the facial contour in the driver's facial area image through edge detection, uses the template matching algorithm to locate and detect the eye area and mouth area in the driver's facial area image, and outputs the center position coordinates of the left eye, right eye and mouth; Head posture estimation module: Based on the center position coordinates of the left eye, right eye and mouth, the horizontal rotation angle, pitch angle and roll angle of the driver's head are calculated using geometric relationships; Attention area extraction module: constructs an attention area classification model based on the horizontal rotation angle, pitch angle and roll angle of the driver's head to obtain the driver's attention area; Monitoring and judgment module: Uses an AI-driven long short-term memory network to process the horizontal rotation angle, pitch angle, and roll angle of the driver's head and the driver's attention area, monitors distraction, and outputs monitoring results; To implement the artificial intelligence-based driver attention monitoring method as described in any one of claims 1-7.

Images