A driving state monitoring method, system and vehicle

Abstract

The embodiment of the present disclosure provides a driving state monitoring method, system and vehicle, the driving state monitoring method comprises: acquiring first face image data of a driver when driving; extracting relevant face features from the first face image data, wherein the face features comprise features of at least one part of a head and facial features; inputting the face features into a preset driving state monitoring model to monitor a driving state of the driver and obtaining a monitoring result; wherein the driving state monitoring model is trained by using second face image data of the driver in a normal driving state, and the training comprises training a driving state reference feature matrix constructed according to face features extracted from the second face image data.

Description

Technical Field

[0001] This disclosure relates to the field of automotive driving condition monitoring technology, specifically to a driving condition monitoring method, system, and vehicle. Background Technology

[0002] Because there are significant differences in the posture, appearance, and facial features of different individual drivers, existing driver monitoring systems (DMS) use general algorithms for monitoring. This can easily lead to false alarms and misdetections for some individual drivers with special conditions, which greatly affects the driver experience and the driver's trust in the entire vehicle monitoring system. Summary of the Invention

[0003] The purpose of this disclosure is to provide a driving state monitoring method, system, and vehicle to solve the technical problems of existing driver state monitoring systems being prone to false alarms and misdetections, resulting in a poor driver experience.

[0004] To solve the above-mentioned technical problems, the embodiments of this disclosure adopt the following technical solutions:

[0005] A driving status monitoring method, comprising:

[0006] Acquire the first facial image data of the driver while driving;

[0007] Relevant facial features are extracted from the first facial image data, wherein the facial features include features of at least one part of the head and facial features;

[0008] The facial features are input into a preset driving state monitoring model to monitor the driver's driving state and obtain monitoring results;

[0009] The driving state monitoring model is trained using second facial image data of the driver in a normal driving state. The training includes training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data.

[0010] The driving state monitoring method, system, and vehicle provided in this disclosure acquire first facial image data of the driver during daily driving, extract relevant facial features from the first facial image data, and input the facial features into a preset driving state monitoring model trained using second facial image data of the driver in a normal driving state to monitor the driver's driving state and obtain monitoring results. The driving state monitoring model is obtained by training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data. This allows for the construction of baseline state information for drivers based on specific individual driver characteristics, resulting in a more accurate, reasonable, and reliable driving state monitoring model. This improves the accuracy of driving state monitoring and facilitates subsequent precise implementation of corresponding measures (e.g., maintaining a safe distance from vehicles in front and behind) for abnormal driving states, thereby enhancing driver safety.

[0011] In some embodiments, training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data includes:

[0012] The facial key points in the second facial image data are calibrated to obtain the baseline facial key points;

[0013] Extract facial features at the key points of the reference face;

[0014] The facial features are classified to obtain the driver's posture classification result;

[0015] The driving state baseline feature matrix is ​​constructed based on the posture classification results and the feature values ​​corresponding to the facial features;

[0016] The driving state baseline feature matrix is ​​trained to obtain the driving state monitoring model.

[0017] During the training of the driving state monitoring model, a more comprehensive and accurate driving state baseline feature matrix can be constructed based on multi-source features. Baseline facial features can be collected as comprehensively and accurately as possible as the standard for judging driving state monitoring. Then, by training the driving state baseline feature matrix, a more accurate and reliable driving state monitoring model can be obtained.

[0018] In some embodiments, the method further includes:

[0019] Acquire driving environment data, wherein the driving environment data includes at least one of the following: distance data between the driver and in-vehicle equipment, vehicle operating condition data, and environmental data surrounding the vehicle;

[0020] The driving state baseline feature matrix is ​​calculated based on the driving environment data to obtain the probability distribution curve of the facial feature state corresponding to the posture classification result;

[0021] The monitoring standard corresponding to the monitoring result in the driving state monitoring model is determined based on the probability distribution curve. In this embodiment, the driving state baseline feature matrix can be calculated based on the driving environment data to obtain the probability distribution curve of the facial feature state corresponding to the posture classification result. Then, the monitoring standard used to judge the driving state is determined based on the statistical results of the probability distribution curve, resulting in a more accurate driving state monitoring model.

[0022] In some embodiments, the method further includes optimizing the driving state monitoring model based on abnormal driving state data in the monitoring results, thereby obtaining a more accurate driving state monitoring model.

[0023] In some embodiments, optimizing the driving state monitoring model based on abnormal driving state data in the monitoring results includes:

[0024] Obtain vehicle control information for the time period corresponding to the abnormal driving state data;

[0025] Determine whether the vehicle control information meets preset conditions;

[0026] If the conditions are met, the abnormal driving state data is compared with the corresponding driving state baseline feature matrix, and the offset is calculated.

[0027] If the offset is less than a preset offset threshold, the abnormal driving state data is determined to be problematic data;

[0028] The driving state monitoring model is trained using the problem data.

[0029] Based on the vehicle control information of the time period corresponding to the abnormal driving state data, it can be further determined whether the abnormal driving state data is problematic data. The problematic data can be used to train the driving state monitoring model, which can improve the accuracy of training data selection during model training and effectively optimize the model.

[0030] In some embodiments, the method further includes:

[0031] Before the driving state monitoring model is run, the driver's current driving state data is acquired;

[0032] The driver's current driving status data is input into the driving status monitoring model for self-testing;

[0033] If the self-test is successful, the driving state monitoring model is controlled to run and monitor the driver's driving state; if the self-test fails, the driving state baseline feature matrix is ​​adjusted according to the frequency of abnormal driving states.

[0034] The driving state monitoring model is trained and updated based on the adjusted driving state baseline feature matrix.

[0035] Before monitoring the driver's condition, the driving condition monitoring model is self-checked based on the driver's current driving condition. The monitoring parameters (such as monitoring thresholds) of the driving condition monitoring model are adaptively adjusted to ensure the accuracy of monitoring during subsequent vehicle driving operations and improve the user experience.

[0036] In some embodiments, after extracting facial features from the first facial image data and / or the second facial image data, the method further includes:

[0037] Face recognition is performed based on the facial features to obtain the face recognition result;

[0038] The driver's identity information corresponding to the face recognition result is associated with the first facial image data and / or the second facial image data.

[0039] When extracting facial features from the second facial image data, face recognition can be performed based on the facial features. Then, the driver's identity information can be obtained based on the face recognition results. The driver's identity information can then be associated with the second facial image data so that different driving state benchmark feature matrices can be constructed based on different driver identity information. This allows for the construction of benchmark state information for managing identity information based on the specific individual characteristics of drivers. Personalized driving state monitoring models can be trained for different drivers, improving the accuracy of driving state monitoring and enhancing the user experience.

[0040] When extracting facial features from the first facial image data, face recognition is performed based on the facial features. The driver's identity information corresponding to the face recognition result is associated with the first facial image data. This facilitates personalized and accurate monitoring and identification of the driver's identity when using the driving state monitoring model to monitor the driver's state in the future, thus avoiding misidentification.

[0041] In some embodiments, the method further includes:

[0042] The second facial image data of the driver is acquired according to the preset image acquisition cycle, and the second facial image data is updated.

[0043] The driving state reference feature matrix is ​​updated based on the updated second facial image data.

[0044] By periodically updating the second facial image data, second facial image data of the driver in normal driving state can be dynamically collected and used as training data to extract relevant facial features, construct a driving state benchmark feature matrix, and dynamically train the driving state monitoring model. The model parameters are continuously optimized to further improve the accuracy and reliability of the driving state monitoring model.

[0045] This disclosure also provides a driving status monitoring system, including:

[0046] The acquisition module is configured to acquire the first facial image data of the driver while driving;

[0047] The extraction module is configured to extract relevant facial features from the first facial image data, wherein the facial features include features of at least one part of the head and facial features;

[0048] The monitoring module is configured to input the first facial image data into a preset driving state monitoring model to monitor the driver's driving state and obtain monitoring results;

[0049] The driving state monitoring model is trained on second facial image data of the driver in a normal driving state. The training includes training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data.

[0050] This disclosure also provides a vehicle including a control device, the control device including a memory and a processor, the memory storing a computer program, and the processor implementing the above-described method when executing the computer program in the memory.

[0051] This disclosure also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the above-described driving state monitoring method. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of a driving state monitoring method according to an embodiment of the present disclosure;

[0054] Figure 2This is an architecture diagram of the driving status monitoring system according to an embodiment of the present disclosure;

[0055] Figure 3 This is another flowchart of the driving status monitoring method according to an embodiment of the present disclosure;

[0056] Figure 4 This is a key point calibration diagram of the mouth features in the driving state monitoring method of this disclosure embodiment;

[0057] Figure 5 This is a key point calibration map of eye features in the driving state monitoring method of this disclosure embodiment;

[0058] Figure 6 This is a probability distribution curve of a certain facial feature state in the driving state monitoring method of this disclosure embodiment;

[0059] Figure 7 This is a flowchart illustrating the self-test process of the driving state monitoring model in the driving state monitoring method of this disclosure.

[0060] Figure 8 This is a schematic diagram of the driving status monitoring system according to an embodiment of the present disclosure. Detailed Implementation

[0061] Various embodiments and features of this disclosure are described herein with reference to the accompanying drawings.

[0062] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this disclosure will be apparent to those skilled in the art.

[0063] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present disclosure and, together with the general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

[0064] These and other features of this disclosure will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0065] It should also be understood that although this disclosure has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this disclosure, which have the features described in the claims and are therefore all within the scope of protection defined herein.

[0066] The above and other aspects, features and advantages of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0067] Specific embodiments of this disclosure are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this disclosure, which may be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure this disclosure. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely to serve as the basis and representative basis for the claims to teach those skilled in the art to use this disclosure in a variety of substantially any suitable detailed structures.

[0068] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in still another embodiment,” all of which may refer to one or more of the same or different embodiments according to this disclosure.

[0069] When the driver status monitoring system is working, it first collects video images of the driver's upper body (mainly the face) while driving, extracts facial features (including head, eyes, mouth, and other status features), and detects and judges abnormal driver states based on the extracted facial features. Because there are differences in posture, physique, and facial features among different drivers, the driver status monitoring system may misidentify drivers, affecting the system's monitoring and recognition effectiveness.

[0070] The differences in the posture, appearance, and facial features of different individual drivers mainly affect the monitoring and recognition effect of the system through the following reasons: 1) The face image may be obscured, resulting in incomplete extraction of feature information; 2) The key points of the face or other feature extraction algorithms are distorted, resulting in reduced accuracy; 3) The facial feature results are not suitable for judging the driver's state, for example, small eyes are more likely to be interpreted as closed eyes, and a large mouth is more likely to be interpreted as yawning or talking.

[0071] In view of this, embodiments of the present disclosure provide a driving status monitoring method, system, and vehicle.

[0072] Figure 1 A flowchart of a driving state monitoring method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown in the embodiments of this disclosure, a driving status monitoring method includes:

[0073] S101: Acquire the first facial image data of the driver while driving.

[0074] In this embodiment, the driver status monitoring method can be applied to in-vehicle devices such as in-vehicle computers, in-vehicle monitoring equipment, or driving recorders. The in-vehicle device includes a camera and a server. The camera can capture images of the driver's face inside the car to obtain first facial image data. The server can process the acquired first facial image data to monitor the driver's driving status.

[0075] The vehicle-mounted equipment, including the aforementioned camera and server, constitutes the driving status monitoring system of this embodiment. For example... Figure 2 As shown, the driving state monitoring system includes a driver monitoring camera unit, a state monitoring management unit, and a service unit. The driver monitoring camera unit can acquire the driver's first facial image data in real time and transmit it to the state monitoring management unit through a transmission channel (wireless or wired transmission channel) so that the state monitoring management unit can monitor the driver's driving state. The acquired first facial image data can be facial image data at any time or during any period of time while driving.

[0076] Driving status includes normal driving status and abnormal driving status. Abnormal driving status can include driving while fatigued, driving while distracted, or driving dangerously.

[0077] S102: Extract relevant facial features from the first facial image data, wherein the facial features include features of at least one part of the head and facial features.

[0078] After receiving the first facial image data sent by the camera, the server can process the first facial image data and extract facial features related to driving status detection and recognition.

[0079] Facial features include static and / or dynamic features of at least one of the head and facial features (eyes, nose, mouth, eyebrows, and ears). Head features may include head posture, such as head movements like tilting, nodding, or turning, as well as features that further describe head parameters, such as head offset frequency, pitch angle, yaw angle, and roll angle. Eye features may include eye shape and other features related to opening and closing eye movements, as well as features that further describe eye parameters, such as eye opening, interocular distance, blinking frequency, and duration of eye closure. Mouth features may include mouth opening and closing movements, as well as parametric features such as mouth opening. Facial features may also include body contours and other physical features.

[0080] Specifically, during feature extraction, feature values ​​of predetermined feature point locations in the first facial image data can be extracted, thereby extracting facial features of the corresponding parts.

[0081] To improve the accuracy and efficiency of facial feature extraction, before performing step S102, the method further includes: preprocessing the first facial image data.

[0082] Specifically, the acquired first facial image data can be cropped and compressed to improve data processing efficiency; alternatively, the acquired first facial image data can be cleaned to obtain a frontal view of the face that can accurately identify facial features, so as to facilitate the determination of facial key points and the extraction of facial features.

[0083] S103: Input the facial features into a preset driving state monitoring model to monitor the driver's driving state and obtain the monitoring results.

[0084] The driving state monitoring model is trained using second facial image data of the driver in a normal driving state. The training includes training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data.

[0085] In this embodiment, a driving state monitoring model can be trained in advance using second facial image data of the driver in a normal driving state. Specifically, second facial image data can be collected during the driver's daily driving process and used as training data. Relevant facial features are extracted from the second facial image data, and then a driving state baseline feature matrix is ​​constructed based on the extracted facial features. The driving state baseline feature matrix is ​​then trained to obtain the driving state monitoring model. Subsequently, the facial features extracted in step S102 are input into the driving state monitoring model to monitor the driver's driving state and determine whether the driver is in an abnormal driving state (e.g., fatigued driving).

[0086] Optionally, acquiring second facial image data of the driver when the driver is in a normal driving state includes: acquiring facial image data of the driver within a preset time period after the vehicle starts running.

[0087] The preset time for vehicle startup and operation is 3 to 5 minutes after the car's ignition cycle begins. This period is when the user has just entered the car, completed the relevant settings, and is focused on driving. The driver is unlikely to be fatigued. Therefore, the facial image data collected during the above-mentioned time period is used as the second facial image data when the driver is in a normal driving state.

[0088] The constructed driving state baseline feature matrix includes multi-source features; therefore, a multi-source feature fusion algorithm layer (such as...) can be set during the construction of the driving state monitoring model. Figure 2 As shown, the extracted baseline facial features are fused to construct a more comprehensive and accurate driving state baseline feature matrix. The baseline facial features are collected as comprehensively and accurately as possible as the standard for driving state monitoring and judgment. Then, by training the driving state baseline feature matrix, a more accurate and reliable driving state monitoring model can be obtained.

[0089] When monitoring driving status based on the acquired first facial data, the facial features extracted from the first facial data can be fused using a multi-source feature fusion algorithm layer. The fused facial features can then be compared with the driving status baseline feature matrix to identify and determine the driver's driving status.

[0090] In other embodiments, the driver's first facial image data can be directly used as input to the driving state monitoring model, such as... Figure 2 As shown, the constructed driving state monitoring model can include a visual feature detection algorithm layer, which can extract features from the collected first facial image data, and then fuse the extracted facial features through a multi-source feature fusion algorithm layer. Finally, the abnormal state judgment module compares the extracted facial features with the driving state benchmark feature matrix to identify and judge the driver's driving state.

[0091] The aforementioned driving state monitoring model can be a fatigue driving state monitoring model used to monitor whether a user is driving while fatigued, or an autonomous driving state monitoring model used to monitor whether a user is engaging in assisted autonomous driving, etc. The driving state monitoring model can be a multi-state monitoring model, capable of simultaneously monitoring multiple driving states of the driver; for example, it can monitor whether the driver is driving while fatigued, and simultaneously monitor whether the driver is driving while distracted. This disclosure does not specifically limit the specific type of driving state monitoring model.

[0092] In one specific embodiment, when using the above-mentioned driving state monitoring model for monitoring, if the feature value of a certain feature point is not extracted from the first facial image data of the driver to be identified, and the feature value of the feature point is a variable in the above-mentioned driving state reference feature matrix, it can be determined that the feature point may be occluded. Then, the occluded part can be predicted according to the above-mentioned driving state reference feature matrix, and the driving state can be judged and predicted to improve the accuracy of driving state monitoring.

[0093] The aforementioned driving status monitoring model can be a machine learning model such as SVM, decision tree, or neural network (CNN).

[0094] Furthermore, such as Figure 2 As shown, the status monitoring management unit may include a status monitoring message processing module, which can send the monitoring results obtained by the abnormal status judgment module to the service unit, so that the status response strategy module in the service unit can take corresponding strategies to improve driving safety.

[0095] The driving state monitoring method provided in this disclosure acquires first facial image data of the driver during daily driving, extracts relevant facial features from the first facial image data, and inputs the facial features into a preset driving state monitoring model trained using second facial image data of the driver in a normal driving state to monitor the driver's driving state and obtain monitoring results. The driving state monitoring model is trained on a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data. This allows for the construction of baseline state information for drivers based on individual driver characteristics, resulting in a more comprehensive, accurate, reasonable, and reliable driving state monitoring model. This improves the accuracy of driving state monitoring and facilitates subsequent precise implementation of corresponding measures (e.g., maintaining a safe distance from vehicles in front and behind) for abnormal driving states, thereby improving driver safety. In some embodiments, such as... Figure 3 As shown, training the driving state baseline feature matrix constructed based on facial features extracted from the second facial image data includes:

[0096] S201: The facial key points in the second facial image data are calibrated to obtain the baseline facial key points;

[0097] S202: Extract facial features at the key points of the reference face;

[0098] S203: Classify the facial features to obtain the driver's posture classification result;

[0099] S204: Construct the driving state baseline feature matrix based on the posture classification result and the feature values ​​corresponding to the facial features;

[0100] S205: Train the driving state reference feature matrix to obtain the driving state monitoring model.

[0101] like Figure 2 As shown, the constructed driving state monitoring model can include a facial feature calibration algorithm layer, which can calibrate facial key points used to represent facial features to obtain baseline facial key points; then, facial features at these baseline facial key points can be extracted to obtain the baseline facial features when the driver is in a normal driving state. The baseline facial features can be stored in a feature configuration database for subsequent construction of the driving state baseline feature matrix. Figure 4 As shown, 20 key points around the mouth can be calibrated to obtain their coordinates, which is then used to extract mouth features; for example... Figure 5 As shown, 13 key points around the eyes can be marked, of which P1 to P8 are 8 key points of the eyelids, and P9 to P13 are 5 key points of the pupil.

[0102] After extracting facial features based on the calibrated benchmark facial key points, the facial features can be classified. For example, the mouth can be divided into three categories: large, medium, and small, and each category can be further divided into different subcategories such as open mouth and closed mouth based on the size of the mouth opening.

[0103] Based on the ratio between the distance between the eyes and the width of the mouth, mouths can be divided into three categories: when viewed from the front with the mouth closed, a mouth with a ratio of less than 65% is classified as a large mouth, a mouth with a ratio of 65-80% is classified as a medium mouth, and a mouth with a ratio of more than 80% is classified as a small mouth.

[0104] Furthermore, such as Figure 4 As shown, the mouth state can be calculated based on the feature values ​​of the extracted key points of the mouth, and the distance between the upper and lower lips can be calculated: distance between upper and lower lips = ||P15-P19||.

[0105] The maximum lip distance is calculated when the head is facing forward, without facial or mouth expressions, and without lip makeup. It can be obtained through statistical analysis of a large amount of second-face image data of frontal faces.

[0106] The maximum mouth width is the width of the mouth when the mouth is completely closed from the front, relaxed and without smiling, with the lips stretched out. Maximum mouth width = ||P1-P7||.

[0107] To obtain a more accurate maximum mouth width when the mouth is closed from the front, a function or a straight line can be used to fit the line connecting the key points between the upper and lower lips to calculate the maximum mouth width.

[0108] In this embodiment, the mouth aspect ratio (MAR) can also be calculated:

[0109]

[0110] or

[0111]

[0112] The baseline mouth opening under normal driving conditions is calculated. For example, a mouth opening state with a MAR of 0.75 is defined as normal mouth opening, while when MAR > 0.75 and the mouth opening frequency reaches a preset threshold, it can be determined as yawning, indicating that the driver is in a state of fatigue driving.

[0113] In this embodiment, eye shapes can be classified into three categories—small, medium, and large—based on eyelid shape and pupil visibility. Specifically: small eyes have a flat, thin eyelid shape, and nearly half of the pupil is not visible; medium eyes have a mostly visible pupil with almost no sclera (white of the eye) above or below the iris; and large eyes have a fully visible pupil with obvious sclera above and below the iris.

[0114] Furthermore, the pupil diameter and eyelid aspect ratio (EAR) can be calculated based on the feature values ​​of the extracted key eye points.

[0115] Where, pupil diameter = ||P9-P11|| or [(||P9-P13||)×2]

[0116]

[0117] Furthermore, the eye opening can be calculated based on the eyelid distance and pupil diameter:

[0118]

[0119] For large eyes, the normal opening is almost 100%. Therefore, eye features with a normal opening greater than 100% are classified as large eyes. For small eyes, the normal opening is about 50%. Therefore, eye features with a normal opening of about 50% are classified as small eyes.

[0120] More preferably, the eye type can be determined by combining the eyelid aspect ratio and the eye opening when the face is facing forward. Generally speaking, the smaller the eyelid aspect ratio, the flatter the eyes; the smaller the eye opening, the lower the pupil exposure ratio. That is, eyes with an eyelid aspect ratio less than a preset aspect ratio threshold and an eye opening less than a preset opening threshold are classified as small eyes.

[0121] After classifying the extracted facial features, the classification categories and corresponding feature values ​​can be stored as baseline facial features in a system such as... Figure 2 The feature configuration database shown is used for subsequent identification of driving conditions.

[0122] Furthermore, the driving state baseline feature matrix is ​​constructed based on the posture classification results and the feature values ​​corresponding to the facial features obtained from the above calculations. Specifically, a feature value matrix can be constructed based on the feature values ​​(e.g., position coordinates) of each feature point in the facial features corresponding to different posture classification results in normal driving state, thereby constructing the driving state baseline feature matrix. For example, in this embodiment, each column vector can be the feature value of each feature point in a certain part, and then the baseline facial features in normal driving state can be determined based on the feature value of at least one feature point in that part and / or the correlation between different feature points.

[0123] For example, the inter-eye distance can be calculated based on the eigenvalues ​​of feature points in the first column vector containing eye features in the baseline feature matrix, and the mouth width can be calculated based on the eigenvalues ​​of feature points in the second column vector containing mouth features. Then, based on the proportional relationship between the inter-eye distance and the mouth width, the mouth is divided into three categories: large, medium, and small. This allows for further determination of the driver's mouth size based on individual driving differences. Finally, based on the determined mouth size and mouth opening, a baseline facial feature representing the driver in a normal driving state is obtained. In other words, the driving state baseline feature matrix constructed above represents the baseline facial features representing the driver in a normal driving state.

[0124] In this embodiment, constructing the driving state baseline feature matrix in step S204 can obtain the most comprehensive facial baseline feature data possible for effective identification of driving state. Once the driving state baseline feature matrix is ​​constructed, a driving state monitoring model can be built and trained based on this matrix to obtain the final driving state monitoring model used to determine the driver's driving state.

[0125] In some embodiments, the method further includes:

[0126] S301: Acquire driving environment data, wherein the driving environment data includes at least one of the following: distance data between the driver and in-vehicle equipment, vehicle operating condition data, and environmental data surrounding the vehicle;

[0127] S302: Calculate the driving state baseline feature matrix based on the driving environment data to obtain the probability distribution curve of the facial feature state corresponding to the posture classification result;

[0128] S303: Determine the monitoring standard corresponding to the monitoring result in the driving state monitoring model based on the statistical parameters of the probability distribution curve.

[0129] The facial features extracted during normal driving are derived from second facial image data corresponding to different driving environments. Driving environment data such as the distance between the driver and the camera, and the external driving environment may affect the accuracy of the extracted facial features, and consequently the accuracy of the facial feature state calculated based on these features. Therefore, in this embodiment, to ensure a more accurate driving state monitoring model, the driving state baseline feature matrix is ​​calculated based on the driving environment data to obtain the probability distribution curve of the facial feature state corresponding to the posture classification result. Then, the judgment criteria (monitoring criteria) for judging the driving state are determined based on the statistical results of the probability distribution curve.

[0130] like Figure 2As shown, when the vehicle is in motion, driving environment data can be transmitted to the service unit via a transmission channel in the form of CAN signals. The service unit then transmits this data to the status monitoring and management unit for monitoring the driving status. The driver control information module in the service unit can acquire vehicle operating condition data such as gear position / pedal opening, vehicle speed / acceleration, steering wheel angle / acceleration, and lateral / yaw acceleration.

[0131] Figure 6 This is a probability distribution curve for a certain facial feature state according to an embodiment of this disclosure, where the horizontal axis represents the driving environment and the vertical axis represents the probability value of the MAR (Mouth Aspect Ratio) of the facial feature state of mouth opening. Figure 6 As shown, in this embodiment, based on the statistical results of the probability distribution curve, the data within the most concentrated range can be selected, and statistical parameters such as the probability centroid or statistical mean of the facial feature state can be used as the monitoring standard corresponding to that facial feature state, thereby obtaining a more accurate driving state judgment monitoring standard. For example, when judging whether a driver is driving while fatigued by calculating the MAR, the calculated MAR is different in different driving environments. Therefore, in this embodiment, the driving state baseline feature matrix in each driving environment is calculated to obtain the probability distribution curve of the MAR. Since the probability distribution curve roughly satisfies a normal distribution, the probability centroid corresponding to the facial feature state (for example, the mouth opening when MAR>0.75 is the baseline mouth opening for judging whether the driver is in a fatigued driving state) can be used as the judgment standard for fatigued driving. By optimizing the judgment standard, a more accurate driving state monitoring model can be obtained.

[0132] In some embodiments, the method further includes:

[0133] S401: Optimize the monitoring parameters of the driving state monitoring model based on the abnormal driving state data in the monitoring results.

[0134] After monitoring the driver's driving status using steps S101 to S103, abnormal driving status data can be obtained when the driver is detected to be in an abnormal driving status, and the driving status monitoring model can be optimized based on the abnormal driving status data.

[0135] In some embodiments, step S401 specifically includes:

[0136] S4011: Obtain vehicle control information for the time period corresponding to the abnormal driving state data;

[0137] S4012: Determine whether the vehicle control information meets preset conditions;

[0138] S4013: If satisfied, compare the abnormal driving state data with the corresponding driving state reference feature matrix and calculate the offset;

[0139] S4014: If the offset is less than a preset offset threshold, the abnormal driving state data is determined to be problematic data;

[0140] S4015: Train the driving state monitoring model using the problem data.

[0141] Specifically, problematic data (such as misidentified data) during driver monitoring can be filtered based on vehicle control information during driving. In this embodiment, when abnormal driving state data is obtained using the driving state monitoring model, vehicle control information for the time period corresponding to the abnormal driving state can be acquired. It can be determined whether the driver has made sufficient control input to the vehicle during that time period. If so, the corresponding facial features are extracted from the abnormal driving state data and compared with the corresponding driving state baseline feature matrix to determine whether there is an offset. If there is no offset, the abnormal driving state data can be determined as potentially misidentified problematic data. This problematic data can be used as facial image data during normal driving. After collecting a preset number of problematic data, the problematic data and the acquired second facial image data can be used together as a training dataset and input into the driving state monitoring model for training to obtain a driving state baseline feature matrix for judging driving state.

[0142] In other embodiments, the problem data can be fed together with the second facial image data into an independent driving state baseline feature model (which is a data filtering model) for training to obtain baseline features for judging driving state.

[0143] As can be seen from the above, this embodiment can improve the monitoring effect of the driving monitoring state model by performing self-learning training on the benchmark driving state data based on the calibration results with a high degree of deviation after calibrating the driver's benchmark driving state.

[0144] In other embodiments, step S401 can adaptively adjust the monitoring parameters of the driving monitoring state model based on abnormal driving state data, and modify the parameter thresholds used to identify the driving state according to the driving state baseline feature matrix obtained in steps S201 to S204 to improve monitoring accuracy. For example, in the baseline classification, the opening of the small eyes is relatively small. Therefore, based on the small eyes classification, a lower eye opening threshold is set to determine that the driver is in a closed-eye state, and then the duration of the closed-eye state is used to determine whether the driver is in a fatigued driving state.

[0145] In some embodiments, the method further includes:

[0146] S501: Before the driving state monitoring model runs, obtain the driver's current driving state data;

[0147] S502: Input the driver's current driving status data into the driving status monitoring model for self-testing;

[0148] S503: If the self-test is successful, control the driving state monitoring model to run and monitor the driver's driving state; if the self-test fails, adjust the driving state reference feature matrix according to the frequency of abnormal driving states.

[0149] S504: Train and update the driving state monitoring model based on the adjusted driving state baseline feature matrix.

[0150] Specifically, within a preset time (e.g., 5 minutes) before the vehicle is started and the driving status monitoring model is working normally, the driving status monitoring model performs a self-check based on the driver's current driving status, and adaptively adjusts the monitoring parameters (e.g., monitoring thresholds) of the driving status monitoring model to ensure the accuracy of monitoring during subsequent vehicle driving operations and improve user experience.

[0151] like Figure 7 As shown, the self-check of the driving state monitoring model is achieved through the interaction between the driver monitoring algorithm layer (the aforementioned state monitoring management unit) and the driver monitoring function service layer (the aforementioned service unit). During the self-check, if the frequency of abnormal driving states exceeds a preset frequency threshold (e.g., 5 times / minute), the self-check time can be appropriately extended, for example, by 1-2 minutes. If, after the extension, the frequency of abnormal driving states still exceeds the preset frequency threshold, the self-check is considered to have failed, and the driving state baseline feature matrix needs to be readjusted. The driving state monitoring model is then trained and optimized based on the adjusted driving state baseline feature matrix. After a successful self-check, the driving state monitoring model fully enters normal working state, monitoring the driver's state in real time during driving. Upon detecting an abnormal driving state, it reports the driver's abnormal driving state to the service layer. The service layer can execute corresponding prompting strategies to promptly remind the driver to focus on driving and ensure driving safety, such as... Figure 2 As shown, prompts can be given through different output methods, such as TTS broadcast, central control prompts, instrument reminders, or speaker alarms.

[0152] In some embodiments, after extracting facial features from the first facial image data and / or the second facial image data in step S102, the method further includes:

[0153] S601: Perform face recognition based on the facial features to obtain the face recognition result;

[0154] S602: Associate the driver identity information corresponding to the face recognition result with the first facial image data and / or the second facial image data.

[0155] In this embodiment, when extracting facial features from the second facial image data, face recognition can be performed based on the facial features, and then the driver's identity information can be obtained based on the face recognition results. The driver's identity information is then associated with the second facial image data so that different driving state benchmark feature matrices can be constructed based on different driver identity information. This allows for the construction of benchmark state information for managing identity information based on the specific individual characteristics of drivers, enabling the training of personalized driving state monitoring models for different drivers, improving the accuracy of driving state monitoring, and enhancing the user experience.

[0156] When extracting facial features from the first facial image data, face recognition is performed based on the facial features. The driver identity information corresponding to the face recognition result is associated with the first facial image data. This facilitates personalized and accurate monitoring and identification based on the driver's identity when using a driving state monitoring model to monitor the driver's state, avoiding misidentification, etc. (For example, the corresponding driving state monitoring model can be selected based on the face recognition result).

[0157] In some embodiments, the method further includes:

[0158] S701: Acquire second facial image data of the driver while driving according to a preset image acquisition cycle, and update the second facial image data;

[0159] S702: Update the driving state reference feature matrix based on the updated second facial image data.

[0160] In this embodiment, second facial image data of the driver in a normal driving state can be periodically collected and updated to adaptively update the driving state baseline feature matrix, further improving the accuracy of driving state monitoring. For example, for novice drivers, the facial features in a normal state may initially exhibit some randomness; for instance, a large mouth opening might indicate surprise rather than yawning. As driving experience increases, the driving state baseline feature matrix gradually stabilizes. Therefore, by periodically updating this baseline feature matrix, a more accurate dynamic driving state monitoring model can be continuously trained based on the driver's driving experience.

[0161] This embodiment of the present disclosure can collect second facial image data of the driver in normal driving state in real time during the driver's daily driving, dynamically extract facial features, construct a driving state benchmark feature matrix, train the driving state monitoring model, and continuously optimize the parameters of the model to further improve the reliability of the driving state monitoring model.

[0162] Figure 8 A schematic diagram of the structure of a driving status monitoring system according to an embodiment of this disclosure is shown. Figure 8 As shown in the embodiments of this disclosure, a driving status monitoring system is also provided, including:

[0163] The acquisition module 10 is configured to acquire the first facial image data of the driver while driving;

[0164] Extraction module 20 is configured to extract relevant facial features from the first facial image data, wherein the facial features include features of at least one part of the head and facial features;

[0165] The monitoring module 30 is configured to input the facial features into a preset driving state monitoring model to monitor the driver's driving state and obtain monitoring results;

[0166] The driving state monitoring model is trained using second facial image data of the driver in a normal driving state. The training includes training a driving state baseline feature matrix constructed based on facial features extracted from the second facial image data.

[0167] In some embodiments, a training module is also included, configured as follows:

[0168] The facial key points in the second facial image data are calibrated to obtain the baseline facial key points;

[0169] Extract facial features at the key points of the reference face;

[0170] The facial features are classified to obtain the driver's posture classification result;

[0171] The driving state baseline feature matrix is ​​constructed based on the posture classification results and the feature values ​​corresponding to the facial features;

[0172] The driving state baseline feature matrix is ​​trained to obtain the driving state monitoring model.

[0173] In some embodiments, the training module is further configured as follows:

[0174] Acquire driving environment data, wherein the driving environment data includes at least one of the following: distance data between the driver and in-vehicle equipment, vehicle operating condition data, and environmental data surrounding the vehicle;

[0175] The driving state baseline feature matrix is ​​calculated based on the driving environment data to obtain the probability distribution curve of the facial feature state corresponding to the posture classification result;

[0176] The monitoring standard corresponding to the monitoring result in the driving state monitoring model is determined based on the statistical parameters of the probability distribution curve.

[0177] In some embodiments, a model optimization module is also included, configured as follows:

[0178] The driving state monitoring model is optimized based on the abnormal driving state data in the monitoring results.

[0179] In some embodiments, the model optimization module is further configured to:

[0180] Obtain vehicle control information for the time period corresponding to the abnormal driving state data;

[0181] Determine whether the vehicle control information meets preset conditions;

[0182] If the conditions are met, the abnormal driving state data is compared with the corresponding driving state baseline feature matrix, and the offset is calculated.

[0183] If the offset is less than a preset offset threshold, the abnormal driving state data is determined to be problematic data;

[0184] The driving state monitoring model is trained using the problem data.

[0185] In some embodiments, a model self-checking module is also included, configured as follows:

[0186] Before the driving state monitoring model is run, the driver's current driving state data is acquired;

[0187] The driver's current driving status data is input into the driving status monitoring model for self-testing;

[0188] If the self-test is successful, the driving state monitoring model is controlled to run and monitor the driver's driving state; if the self-test fails, the driving state baseline feature matrix is ​​adjusted according to the frequency of abnormal driving states.

[0189] The driving state monitoring model is trained and updated based on the adjusted driving state baseline feature matrix.

[0190] In some embodiments, an association module is further included, configured to: extract facial features from the first facial image data and / or the second facial image data,

[0191] Face recognition is performed based on the facial features to obtain the face recognition result;

[0192] The driver's identity information corresponding to the face recognition result is associated with the first facial image data and / or the second facial image data.

[0193] In some embodiments, an update module is also included, configured as follows:

[0194] The second facial image data of the driver is acquired according to the preset image acquisition cycle, and the second facial image data is updated.

[0195] The driving state reference feature matrix is ​​updated based on the updated second facial image data.

[0196] The driving state monitoring system provided in this disclosure corresponds to the driving state monitoring method in the above embodiments. Any option in the embodiments of the driving state monitoring method is also applicable to the embodiments of the driving state monitoring system, and will not be repeated here.

[0197] This disclosure also provides a vehicle including a control device, the control device including a memory and a processor, the memory storing a computer program, and the processor implementing the above-described driving state monitoring method when executing the computer program in the memory.

[0198] The memory may include volatile memory (e.g., random-access memory (RAM), which may include volatile RAM, magnetic RAM, ferroelectric RAM, and any other suitable form) and non-volatile memory (e.g., disk storage, flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), memristor-based non-volatile solid-state memory, etc.).

[0199] The processor can be a processing device that includes at least one general-purpose processing unit, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that runs other instruction sets, or a processor that runs a combination of instruction sets. The processor can also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), etc.

[0200] Electronic devices may also include a communication interface and a communication bus. The communication interface is used to communicate with external devices (such as network devices), and the processor, memory, and communication interface communicate with each other through the communication bus.

[0201] Electronic devices include, but are not limited to, in-vehicle servers, in-vehicle displays, dashcams and other in-vehicle equipment, as well as handheld devices (such as mobile phones, tablets and other devices) and wearable devices (such as smartwatches, smart bracelets, pedometers and other terminal devices).

[0202] This disclosure also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the above-described driving state monitoring method.

[0203] The computer-executable instructions of embodiments of this disclosure can be organized into one or more computer-executable components or modules. Various aspects of this disclosure can be implemented with any number and combination of such components or modules. For example, aspects of this disclosure are not limited to the specific computer-executable instructions or particular components or modules shown in the drawings and described herein. Other embodiments may include different computer-executable instructions or components having more or fewer functions than those shown and described herein.

[0204] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0205] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0206] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0207] The foregoing has provided a detailed description of several embodiments of this disclosure. However, this disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications based on the concept of this disclosure, and all such variations and modifications should fall within the scope of protection claimed by this disclosure.

Claims

1. A driving state monitoring method characterized by, The method comprises: obtaining first facial image data of a driver during driving; extracting relevant facial features from the first facial image data, wherein the facial features include features of at least one part of the head and facial features; inputting the facial features into a preset driving state monitoring model to monitor the driving state of the driver, and obtaining a monitoring result; wherein the driving state monitoring model is trained using second facial image data of the driver in a normal driving state, and the training includes training a driving state reference feature matrix constructed according to facial features extracted from the second facial image data; The method further comprises: optimizing the driving state monitoring model according to abnormal driving state data in the monitoring result; The optimization of the driving state monitoring model according to the abnormal driving state data in the monitoring result comprises: obtaining vehicle control information of a time period corresponding to the abnormal driving state data; determining whether the vehicle control information meets a preset condition, wherein the preset condition includes that the driver has performed a preset control input on the vehicle during the time period; if the condition is met, comparing the abnormal driving state data with the corresponding driving state reference feature matrix to calculate a deviation; if the deviation is less than a preset deviation threshold, determining that the abnormal driving state data is problem data; training the driving state monitoring model using the problem data.

2. The driving state monitoring method according to claim 1, characterized by, The training of the driving state reference feature matrix constructed according to the facial features extracted from the second facial image data comprises: calibrating facial key points in the second facial image data to obtain reference facial key points; extracting facial features at the reference facial key points; classifying the facial features to obtain a posture classification result of the driver; constructing the driving state reference feature matrix according to the posture classification result and feature values corresponding to the facial features; training the driving state reference feature matrix to obtain the driving state monitoring model.

3. The driving state monitoring method according to claim 2, characterized by, The method further comprises: obtaining driving environment data, wherein the driving environment data includes at least one of distance data between the driver and in-vehicle equipment, vehicle operating condition data, and environmental data around the vehicle; calculating the driving state reference feature matrix according to the driving environment data to obtain a probability distribution curve of a facial feature state corresponding to the posture classification result; determining a monitoring standard corresponding to the monitoring result in the driving state monitoring model according to a statistical parameter of the probability distribution curve.

4. The driving state monitoring method according to claim 1, characterized by, The method further comprises: obtaining current driving state data of the driver before the driving state monitoring model is run; inputting the current driving state data of the driver into the driving state monitoring model for self-checking; if the self-checking is successful, controlling the driving state monitoring model to run to monitor the driving state of the driver; if the self-checking fails, adjusting the driving state reference feature matrix according to the frequency of abnormal driving states; training and updating the driving state monitoring model according to the adjusted driving state reference feature matrix.

5. The driving state monitoring method according to claim 1, characterized by, After extracting the facial features from the first facial image data and / or the second facial image data, the method further comprises: performing face recognition according to the facial features to obtain a face recognition result; associating the driver identity information corresponding to the face recognition result with the first facial image data and / or the second facial image data.

6. The driving state monitoring method according to claim 1, characterized by, The method further comprises: acquiring second facial image data of the driver during driving according to a preset image acquisition period, and updating the second facial image data; updating the driving state reference feature matrix based on the updated second facial image data.

7. A driving state monitoring system characterized by comprising: Comprise: an acquisition module configured to acquire first facial image data of a driver during driving; an extraction module configured to extract relevant facial features from the first facial image data, wherein the facial features include features of at least one part of the head and the facial features; a monitoring module configured to input the first facial image data into a preset driving state monitoring model to monitor the driving state of the driver and obtain a monitoring result; wherein the driving state monitoring model is trained using second facial image data of the driver in a normal driving state, and the training includes training a driving state reference feature matrix constructed according to facial features extracted from the second facial image data; The driving state monitoring system further comprises a model optimization module configured to: optimize the driving state monitoring model according to abnormal driving state data in the monitoring result; The model optimization module is further configured to: acquire vehicle control information of a time period corresponding to the abnormal driving state data; determine whether the vehicle control information meets a preset condition, wherein the preset condition includes that the driver has performed a preset control input on the vehicle during the time period; if so, compare the abnormal driving state data with the corresponding driving state reference feature matrix and calculate an offset; if the offset is less than a preset offset threshold, determine that the abnormal driving state data is problem data; train the driving state monitoring model using the problem data.

8. A vehicle characterized by comprising: A control device comprising a memory and a processor, the memory having a computer program stored thereon, and the processor implementing the method according to any one of claims 1 to 6 when executing the computer program stored on the memory.

Images