Driver state detection method based on driving image and vehicle data

Abstract

The invention provides a driver state detection method based on a driving image and vehicle data, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a driving image and vehicle data at each preset moment in a target time period; inputting the driving image into a trained face detection model, and obtaining a driver face region image output by the face detection model; determining face features of the driver based on the face region image; inputting the input data into a trained state detection model, obtaining a driver state detection result output by the state detection model, the input data at least comprising driver face features, vehicle data and at least one group of historical driver state data, each group of historical driver state data comprises driver state detection results obtained at historical moments and driver face features and vehicle data corresponding to the driver state detection results obtained at the historical moments. The effect of improving the accuracy of the driver state detection result can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a driver state detection method based on driving images and vehicle data. Background Art

[0002] Driver status detection is one of the important technologies to ensure traffic safety. Most current driver status monitoring systems rely mainly on the analysis of image data, capturing the driver's facial features through a camera to assess their attention status and fatigue level. However, the analysis of image data is limited by external environmental factors, such as changes in lighting conditions and instability of camera viewing angles, which can affect the quality of image data and reduce the accuracy of driver status detection results. Summary of the invention

[0003] The present invention provides a driver status detection method based on driving images and vehicle data, so as to solve the defect in the prior art that external environmental factors affect the image data quality and cause the accuracy of the driver status result to be reduced, thereby improving the accuracy of the driver status detection result.

[0004] The present invention provides a driver state detection method based on driving images and vehicle data, the method comprising: Acquire driving images and vehicle data at each preset time within the target time period, wherein the driving image is an image obtained by imaging the driver's position, and the vehicle data reflects the driver's control status of the vehicle, and adjacent preset time periods are separated by a preset time length; Inputting the driving image into a trained face detection model to obtain a driver's face area image output by the face detection model; Determining the driver's facial features based on the driver's facial region image, the driver's facial features at least including a head posture angle, a sight line angle, and an eye aspect ratio, the head posture angle and the sight line angle being obtained by inputting the driver's facial region image into a trained posture angle detection model and a sight line angle detection model, respectively, the eye aspect ratio being obtained based on eye key points, and the eye key points being obtained by inputting the driver's facial region image into a trained key point detection model; Generate input data, input the input data into a trained state detection model, and obtain a driver state detection result output by the state detection model, wherein the input data at least includes the driver's facial features, the vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes a driver state detection result obtained at a historical moment and the driver's facial features and the vehicle data corresponding to the driver state detection result obtained at the historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0005] According to a driver state detection method based on driving images and vehicle data provided by the present invention, generating input data includes: Randomly determine the historical moment from a plurality of moments earlier than the first preset moment within the target time period; Searching for the driver status detection result corresponding to the historical moment in the driver status detection result library; Among them, the driver state detection result library includes driver state detection results corresponding to multiple time moments, and the driver state detection results in the driver state detection result library include a first driver state detection result and a second driver state detection result. The first driver state detection result is obtained based on the state detection model, and the second driver state detection result is obtained based on annotation.

[0006] According to a driver state detection method based on driving images and vehicle data provided by the present invention, the vehicle data at least includes a brake opening change rate, a steering wheel angular velocity and a vehicle acceleration; acquiring the vehicle data at each of the preset moments within the target time period, comprising: The vehicle data at each preset time is obtained through the vehicle CAN bus.

[0007] According to a driver state detection method based on driving images and vehicle data provided by the present invention, the training process of the state detection model includes: Training the state detection model based on multiple sets of first training data to obtain the state detection model after preliminary training, wherein the first training data includes first sample input data and a driver state label corresponding to the first sample input data; Determine the driver's facial features and target data types in the vehicle data based on multiple sets of second training data and the state detection model after preliminary training, wherein the second training data includes second sample input data and a driver state label corresponding to the second sample input data, and the driver's facial features and data types in the vehicle data included in multiple sets of the second sample input data are different; Based on the determined driver facial features and the target data type in the vehicle data, multiple groups of third training data are generated, and the state detection model after preliminary training is re-trained based on the third training data, wherein the third training data includes third sample input data and the driver state label corresponding to the third sample input data, and the driver facial features and the data type of the vehicle data included in the third sample input data are the target data type.

[0008] According to a driver state detection method based on driving images and vehicle data provided by the present invention, the input data is input into a trained state detection model to obtain a driver state detection result output by the state detection model, including: Inputting the driver's facial features into the statistical module in the state detection model, obtaining the statistical features of the driver's facial features at each preset time in the target time period output by the statistical module, wherein the statistical features reflect the changes of the driver's facial features in the target time period; The statistical features, the vehicle data and at least one set of the historical driver status data are input into a classification module in the status detection model to obtain a driver status detection result output by the classification module.

[0009] According to a driver state detection method based on driving images and vehicle data provided by the present invention, before inputting the input data into a trained state detection model, the method comprises: Based on the theoretical data range of the input data, data clipping and normalization processing are performed on the input data.

[0010] The present invention also provides a driver state detection device based on driving images and vehicle data, the device comprising: A data acquisition module, used to acquire driving images and vehicle data at each preset time within a target time period, wherein the driving image is an image obtained by imaging the driver's position, and the vehicle data reflects the driver's control status of the vehicle, and the interval between adjacent preset times is a preset time length; A face detection module, used to input the driving image into a trained face detection model to obtain a driver's face area image output by the face detection model; a facial feature extraction module, for determining the facial features of the driver based on the facial region image of the driver, wherein the facial features of the driver include at least a head posture angle, a sight line angle, and an eye aspect ratio, wherein the head posture angle and the sight line angle are obtained by respectively inputting the facial region image of the driver into a trained posture angle detection model and a sight line angle detection model, and the eye aspect ratio is obtained based on eye key points, and the eye key points are obtained by inputting the facial region image of the driver into a trained key point detection model; A model inference module is used to generate input data, input the input data into a trained state detection model, and obtain a driver state detection result output by the state detection model, wherein the input data at least includes the driver's facial features, the vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes the driver state detection result obtained at a historical moment and the driver's facial features and the vehicle data corresponding to the driver state detection result obtained at the historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-described driver status detection methods based on driving images and vehicle data.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described driver status detection methods based on driving images and vehicle data.

[0013] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned driver state detection methods based on driving images and vehicle data.

[0014] The driver state detection method based on driving images and vehicle data provided by the present invention performs driver face detection on driving images to obtain driver facial features of various types including head posture angle, sight line angle and eye aspect ratio, and also obtains vehicle data reflecting the driver's control state of the vehicle, driver state detection results obtained at historical moments and their corresponding driver facial features and vehicle data. These data can provide supplementary information for the driver facial features obtained based on the driving images, make up for the loss of accuracy of the driver facial features caused by the quality reduction of the driving images due to external influences, and achieve the effect of improving the accuracy of the driver state detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is one of the flow charts of the driver status detection method based on driving images and vehicle data provided by the present invention.

[0017] Figure 2 This is the second flow chart of the driver status detection method based on driving images and vehicle data provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of a driver state detection device based on driving images and vehicle data provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figure 1-2 The present invention describes a driver state detection method based on driving images and vehicle data. Figure 1 As shown, the driver state detection method based on driving images and vehicle data provided by the present invention comprises the steps of: S110, obtaining driving images and vehicle data at each preset time within the target time period, wherein the driving image is an image obtained by imaging the driver's position, and the vehicle data reflects the driver's control status of the vehicle segment, and the interval between adjacent preset times is a preset time length; S120, inputting the driving image into a trained face detection model, and obtaining a driver's face region image output by the face detection model; S130, determining the driver's facial features based on the driver's facial region image, the driver's facial features at least including a head posture angle, a sight line angle, and an eye aspect ratio, the head posture angle and the sight line angle are obtained by inputting the driver's facial region image into a trained posture angle detection model and a sight line angle detection model, respectively, the eye aspect ratio is obtained based on eye key points, and the eye key points are obtained by inputting the driver's facial region image into a trained key point detection model; S140. Generate input data, input the input data into a trained state detection model, and obtain the driver state detection result output by the state detection model, wherein the input data at least includes driver facial features, vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes driver state detection results obtained at a historical moment and driver facial features and vehicle data corresponding to the driver state detection results obtained at a historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0022] The method provided by the present invention can obtain driving images by setting an imaging device in the cab. The position of the imaging device should ensure that the driver's face can be clearly captured. The imaging device can be installed above the steering wheel, at the center console or on the roof interior panel. The imaging device can have a wide-angle field of view and the ability to adapt to different lighting conditions, thereby improving the ability to obtain high-quality driving images in various driving environments.

[0023] A time window can be set in advance, and the length of the target time period is equal to the length of the time window. The driver status detection is performed based on the data obtained in each time window. As the time window slides, multiple driver status detection results can be obtained. The driver status detection result corresponding to each time window can be recorded as the driver status detection result corresponding to the last preset moment of the time window.

[0024] For the driving images at each preset moment within the target time period, they are input into the trained face detection model to obtain the driver's face area image output by the face detection model. Based on the driver's face area image, the driver's facial features can be further extracted. The driver's facial features include at least the head posture angle, the line of sight angle and the eye aspect ratio (Eye Aspect Radio, EAR).

[0025] Specifically, if Figure 2As shown in the figure, the face detection model can be used to detect the face in the driving image to obtain the driver's face area image. The driver's face area image is input into the posture angle detection model and the sight angle detection model respectively to obtain the head posture angle and sight angle. The head posture angle may include pitch, yaw and roll. The driver's face area image is input into the key point detection model to locate the key points. Based on the eye key points detected by the key point detection model, the eye aspect ratio can be calculated.

[0026] When the road conditions are bad, the vehicle shakes, or the lighting conditions are very poor, the quality of the driving image may not be guaranteed, and the driver's facial features extracted based on the driving image may be inaccurate. In the method provided by the present invention, vehicle data is further obtained and added to the input data of the state detection model used to output the driver state detection result.

[0027] Specifically, the vehicle data reflects the driver's control status of the vehicle. The vehicle data at least includes the brake opening change rate, steering wheel angular velocity and vehicle acceleration. The vehicle data at each preset time in the target time period is obtained, including: The vehicle data at each preset time is obtained through the vehicle CAN bus.

[0028] CAN (Controller Area Network) bus is a multi-host local network system, widely used in automotive electronic control systems, it has the advantages of strong real-time performance, strong anti-interference ability and high reliability. Specifically, the data transmission speed of CAN bus is as high as 1Mbps, which can meet the real-time transmission requirements of vehicle dynamic data. At the same time, it adopts differential signal transmission, has strong anti-electromagnetic interference ability, and is suitable for complex automotive working environments. CAN bus also has error detection and recovery mechanism, which can ensure the reliability of data transmission.

[0029] like Figure 2 As shown, the specific process of collecting vehicle data in real time through the vehicle CAN bus includes: the vehicle's built-in sensors collect data such as steering wheel angle, brake force and opening, and driving speed in real time, and the data collected by the sensors are transmitted to the on-board controller (such as ECU) through the CAN bus. The on-board controller performs preliminary processing on the collected data to obtain vehicle data and stores it. Through the CAN bus interface, the processed vehicle data is read into the driver status detection system for driver status detection.

[0030] Vehicle data involves preprocessing and calculating the collected steering wheel angle, brake opening and vehicle speed data. Specifically, the angle data collected by the steering wheel sensor is processed to calculate the steering wheel angular velocity. The formula is: ,in, is the steering wheel angular velocity, is the change in steering wheel angle, The opening data collected by the brake sensor is processed to calculate the brake opening change rate. The formula is: ,in, is the brake opening change rate, is the change in brake opening, The real-time speed data collected by the speed sensor is directly obtained, and after necessary smoothing to reduce noise, the acceleration is calculated. The formula is: ,in, is the vehicle acceleration, is the vehicle speed change, For the time interval.

[0031] The vehicle data reflects the driver's control state of the vehicle, and therefore can provide information for driver state detection. Combined with the driver's facial features, the accuracy of the driver state detection results can be improved. However, the vehicle data and the driver's facial features are also related to the driver's fixed characteristics (appearance, driving control habits). In the method provided by the present invention, in order to further eliminate the influence of the driver's fixed characteristics on the driver's state detection results and improve the accuracy of the driver's state detection results, at least one set of historical driver state data is added to the input data as reference information.

[0032] Specifically, each set of historical driver status data includes the driver status detection results obtained at a historical moment and the corresponding driver facial features and vehicle data. The acquisition process of historical driver status data is as follows: Randomly determine a historical moment from among a plurality of moments earlier than a first preset moment in a target time period; Searching for the driver status detection result corresponding to the historical moment in the driver status detection result library; Among them, the driver state detection result library includes driver state detection results corresponding to multiple time periods. The driver state detection results in the driver state detection result library include a first driver state detection result and a second driver state detection result. The first driver state detection result is obtained based on the state detection model, and the second driver state detection result is obtained based on the annotation.

[0033] The second driver state detection result in the driver state detection result library is obtained by manual annotation. At the initial moment when the state detection model is applied to the driver state detection method based on driving images and vehicle data provided by the present invention, only the second driver state detection result exists in the driver state detection result library. With the application of the state detection model, the first driver state detection result is generated and added to the driver state detection result library. The driver state detection results in the driver state detection result library and the driving images and vehicle data in the target time period correspond to the same driver.

[0034] The method provided by the present invention can be applied to a driver management platform. For drivers who join the driver management platform, a driver status detection result library corresponding to the driver can be established when they register to join the platform. They can drive when they are not tired or distracted, and collect and mark data during the driving process. The second driver status detection result corresponding to the driver is obtained and stored in the driver status detection result library corresponding to the driver.

[0035] The second driver state detection result obtained based on annotation and the first driver state detection result generated at a historical moment can both provide prior knowledge. Adding them to the input data can provide more reference information and improve the accuracy of the driver state detection results output by the state detection model.

[0036] Input the input data into the trained state detection model to obtain the driver state detection results output by the state detection model, including: Inputting the driver's facial features into the statistical module in the state detection model, obtaining the statistical features of the driver's facial features at each moment in the target time period output by the statistical module, the statistical features reflecting the changes in the driver's facial features in the target time period; The statistical features, vehicle data and at least one set of historical driver status data are input into a classification module in the status detection model to obtain the driver status detection result output by the classification module.

[0037] As can be seen from the previous description, the input data includes different types of features. In the process of data fusion, it is an important step to process the normalization of different features. The purpose of normalization is to eliminate the dimensional differences of different feature quality inspections so that each feature has equal weight during data fusion. Due to the different ranges and physical meanings of different features, different normalization strategies should be adopted for each feature. That is, before the input data is input into the trained state detection model, it includes: Based on the theoretical data range of the input data, the input data is clipped and normalized.

[0038] First, the eigenvalues ​​are clipped so that they are within the specified minimum and maximum values. This step is to prevent outliers from affecting the normalized results. According to the range of the eigenvalues ​​and the physical characteristics, an appropriate normalization strategy [-1, 1] or [0, 1] is selected. For features in the range of [-a, a], they are normalized to [-1,1]: , for features in the range [0, b], normalize to [0,1]: , where clipped_value represents the clipped eigenvalue, that is, the eigenvalue within the range of minimum and maximum values.

[0039] For example, the pitch, yaw, and roll angles vary in the positive and negative range, so we use a strategy of normalizing them to [-1, 1]. The steering wheel angular velocity also varies in the positive and negative range, so we use a strategy of normalizing them to [-1, 1]. The brake opening change rate is a feature that only has a positive range, so we use a strategy of normalizing them to [0, 1]. For vehicle acceleration, if we only consider deceleration (negative acceleration), we use a strategy of normalizing them to [0, 1]. EAR is a feature with a specific range, usually varying between [0, 0.4], so we use a strategy of normalizing them to [0, 1].

[0040] In a certain time window, a multidimensional feature vector containing all normalized data is constructed. The selection of the time window should be determined according to the actual application scenario and system requirements, usually ranging from a few seconds to a few minutes. Taking a one-second time window as an example, assuming that the camera has 20 frames of pictures per second, then the image-related features (such as attitude angle, sight angle value, EAR value) can be processed in the following ways: 1. Statistical feature extraction: Perform statistical processing on the 20 values ​​to extract representative statistical features, such as mean, standard deviation, minimum value, maximum value, etc. In this way, each feature (such as pitch angle) will be represented by a multi-dimensional feature (such as mean, standard deviation). Assuming that two statistical features, mean and standard deviation, are extracted, two values ​​will be obtained for each image feature.

[0041] 2. Example of feature vector combination: CAN-related features: Changes in steering wheel angle within 1 second (steering wheel angular velocity ), the change of brake opening within 1 second (brake opening change rate ), the change in speed within 1 second (acceleration ).

[0042] Image-related features: pitch angle mean (p_mean) and standard deviation (p_std), yaw angle mean (y_mean) and standard deviation (y_std), roll angle mean (r_mean) and standard deviation (r_std), horizontal line of sight angle mean (h_mean) and standard deviation (h_std), vertical line of sight angle mean (v_mean) and standard deviation (v_std), EAR mean (e_mean) and standard deviation (e_std).

[0043] 3. Combine the statistical features of the processed image and the vehicle data to obtain a feature vector. Each set of historical driver status data can also be processed in a similar way to obtain the corresponding feature vector, which is spliced ​​together with the combined vector of the processed statistical features and vehicle data and input into the classification module in the state detection model to obtain the driver state detection result output by the classification module.

[0044] This processing method ensures that the image features and CAN features are combined in the same time interval, making the data fusion process more effective and reliable. Through the above steps, the reasonable normalization and fusion of different eigenvalues ​​can be achieved, ensuring that each eigenvalue can be processed with equal weight during data processing and analysis. This method not only improves the accuracy of data fusion, but also enhances the robustness of the model in practical use.

[0045] The classification module in the state detection model is built based on existing machine learning classification models, such as random forests, support vector machines, etc., or regression models. The selection of the model should be based on the dimension and type of data features, the complexity of the prediction task, the drivability of the model, and the real-time requirements. Taking the classification module built based on the SVM classifier as an example, before training, a large number of driving data samples are collected, including normal driving data and data on distracted and fatigued driving. These data samples are used to train and verify the model. The data set should include a variety of driving scenarios and different drivers to ensure the generalization ability of the model. The selected model is trained using the prepared data set. During the training process, the model will learn the relationship between the input data and the driver's state. The goal of model training is to minimize the prediction error. The model parameters are usually optimized through cross-validation techniques to avoid overfitting.

[0046] In a possible implementation, the driver's facial features may include other types of data, such as the proportion of eye closure time, average eyelid closure speed, etc., in addition to the head posture angle, sight angle, and eye aspect ratio listed above. Vehicle data may also include other types of data, such as the vehicle's road guide line offset, in addition to the brake opening change rate, steering wheel angular velocity, and vehicle acceleration listed above. In order to determine which data types are more conducive to driver state detection, in one implementation of the method provided by the present invention, the training process of the state detection model includes: Training the state detection model based on multiple sets of first training data to obtain a preliminarily trained state detection model, wherein the first training data includes first sample input data and a driver state label corresponding to the first sample input data; Determine the target data type in the driver's facial features and the vehicle data based on multiple sets of second training data and the state detection model after preliminary training, wherein the second training data includes second sample input data and a driver state label corresponding to the second sample input data, and the driver's facial features and the data types in the vehicle data included in the multiple second sample input data are different; Based on the determined driver facial features and target data types in the vehicle data, multiple sets of third training data are generated, and the state detection model after the preliminary training is re-trained based on the third training data, wherein the third training data includes third sample input data and the driver state label corresponding to the third sample input data, and the driver facial features and vehicle data data types included in the third sample input data are the target data types.

[0047] In this implementation, the state detection model is first trained using multiple sets of first training data. The driver's facial features in the first sample input data and the data types in the vehicle data included in the first training data include all candidate data types, and the candidate data types are pre-set data types that may be related to the driver's state. The state detection model is first preliminarily trained using the first training data so that the state detection model has a preliminary driver state detection capability. Then, based on the state detection model after the preliminary training and multiple sets of second training data, it is determined which data types have a greater impact on the detection results of the model, thereby retaining the data types that have a greater impact on the output results of the model as the target data type. Finally, the third training data is constructed based on the target data type, and the state detection model after the preliminary training is further trained using the third training data, and finally a state detection model with better performance is obtained. When the state detection model trained in the above manner is used to detect the driver's state, the driver's facial features in the input data and the vehicle data only include data of the target data type.

[0048] There are differences in the types of driver facial features and vehicle data included in multiple second sample input data, that is, the second sample input data can be divided into multiple sets according to the types of driver facial features and vehicle data included therein, each set includes at least one second sample input data, and the second sample input data in the same set include the same types of driver facial features and vehicle data, while the second sample input data input into different sets include different types of driver facial features and vehicle data. From the foregoing description, it can be seen that the method provided by the present invention combines vehicle data and image data to provide a more comprehensive and accurate driver status assessment, and there is no need to use invasive physiological sensors, thus avoiding interference with the driver, improving user experience, and only using one imaging device and CAN bus communication in the vehicle to obtain information. Compared with other solutions that require complex hardware configuration, the implementation cost of the method provided by the present invention is significantly reduced.

[0049] The following is a description of the driver state detection device based on driving images and vehicle data provided by the present invention. The driver state detection device based on driving images and vehicle data described below and the driver state detection method based on driving images and vehicle data described above can be referred to each other. Figure 3 As shown, the driver state detection device based on driving images and vehicle data provided by the present invention includes the following modules: The data acquisition module 310 is used to acquire driving images and vehicle data at each preset time within the target time period. The driving image is an image obtained by imaging the driver's position. The vehicle data reflects the driver's control status of the vehicle. The interval between adjacent preset times is a preset time length. A face detection module 320 is used to input the driving image into a trained face detection model and obtain a driver's face area image output by the face detection model; A facial feature extraction module 330, for determining the facial features of the driver based on the facial region image of the driver, wherein the facial features of the driver include at least a head posture angle, a sight line angle, and an eye aspect ratio, wherein the head posture angle and the sight line angle are obtained by inputting the facial region image of the driver into a trained posture angle detection model and a sight line angle detection model, respectively, and the eye aspect ratio is obtained based on eye key points, and the eye key points are obtained by inputting the facial region image of the driver into a trained key point detection model; The model inference module 340 is used to generate input data, input the input data into the trained state detection model, and obtain the driver state detection result output by the state detection model, wherein the input data includes at least the driver's facial features, vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes the driver state detection result obtained at a historical moment and the driver's facial features and vehicle data corresponding to the driver state detection result obtained at a historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0050] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a driver state detection method based on driving images and vehicle data, the method comprising: obtaining driving images and vehicle data at each preset time within a target time period, the driving image being an image obtained by imaging the driver's position, the vehicle data reflecting the driver's control state of the vehicle, and the interval between adjacent preset moments being a preset time length; inputting the driving image into a trained face detection model to obtain a driver face region image output by the face detection model; determining the driver's face features based on the driver face region image, the driver's face features at least including a head posture angle, a sight line angle and an eye aspect ratio, the head posture angle and the sight line angle being the driver's face region image The eye aspect ratio is obtained by respectively inputting into a trained posture angle detection model and a sight line angle detection model, and the eye aspect ratio is obtained based on the eye key points, and the eye key points are obtained by inputting the driver's facial area image into the trained key point detection model; generating input data, and inputting the input data into a trained state detection model to obtain the driver state detection result output by the state detection model, wherein the input data at least includes the driver's facial features, vehicle data, and at least one set of historical driver state data, and each set of historical driver state data includes the driver state detection result obtained at a historical moment and the driver's facial features and vehicle data corresponding to the driver state detection result obtained at a historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0051] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the driver state detection method based on driving images and vehicle data provided by the above methods. The method includes: obtaining driving images and vehicle data at each preset time within a target time period, the driving image is an image obtained by imaging the driver's position, the vehicle data reflects the driver's control state of the vehicle, and the interval between adjacent preset moments is a preset time length; inputting the driving image into a trained face detection model to obtain the driver's face area image output by the face detection model; determining the driver's facial features based on the driver's face area image, and the driver's facial features at least include head posture angle, view angle, The line angle and eye aspect ratio, the head posture angle and the sight line angle are obtained by inputting the driver's face area image into the trained posture angle detection model and the sight line angle detection model respectively, the eye aspect ratio is obtained based on the eye key points, and the eye key points are obtained by inputting the driver's face area image into the trained key point detection model; generate input data, input the input data into the trained state detection model, and obtain the driver state detection result output by the state detection model, the input data at least includes the driver's facial features, vehicle data and at least one set of historical driver state data, each set of historical driver state data includes the driver state detection result obtained at a historical moment and the driver's facial features and vehicle data corresponding to the driver state detection result obtained at a historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0053] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the driver status detection method based on driving images and vehicle data provided by the above-mentioned methods, the method comprising: obtaining driving images and vehicle data at each preset time within a target time period, the driving image being an image obtained by imaging the driver's position, the vehicle data reflecting the driver's control status of the vehicle, and the interval between adjacent preset moments being a preset time length; inputting the driving image into a trained face detection model to obtain a driver's face area image output by the face detection model; determining the driver's facial features based on the driver's face area image, the driver's facial features at least including a head posture angle, a line of sight angle, and an eye aspect ratio, and the head posture angle The attitude angle and sight line angle are obtained by inputting the driver's facial area image into the trained attitude angle detection model and sight line angle detection model respectively, and the eye aspect ratio is obtained based on the eye key points, and the eye key points are obtained by inputting the driver's facial area image into the trained key point detection model; generate input data, input the input data into the trained attitude detection model, and obtain the driver status detection result output by the status detection model, the input data at least includes the driver's facial features, vehicle data and at least one set of historical driver status data, each set of historical driver status data includes the driver status detection result obtained at a historical moment and the driver's facial features and vehicle data corresponding to the driver status detection result obtained at a historical moment, and the historical moment is earlier than the first preset moment in the target time period.

[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver state detection method based on driving images and vehicle data, characterized in that: The method comprises: Acquire driving images and vehicle data at each preset time within the target time period, wherein the driving image is an image obtained by imaging the driver's position, and the vehicle data reflects the driver's control status of the vehicle, and adjacent preset time periods are separated by a preset time length; Inputting the driving image into a trained face detection model to obtain a driver's face area image output by the face detection model; Determining the driver's facial features based on the driver's facial region image, the driver's facial features at least including a head posture angle, a sight line angle, and an eye aspect ratio, the head posture angle and the sight line angle being obtained by inputting the driver's facial region image into a trained posture angle detection model and a sight line angle detection model, respectively, the eye aspect ratio being obtained based on eye key points, and the eye key points being obtained by inputting the driver's facial region image into a trained key point detection model; Generate input data, input the input data into a trained state detection model, and obtain a driver state detection result output by the state detection model, wherein the input data at least includes the driver's facial features, the vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes a driver state detection result obtained at a historical moment and the driver's facial features and the vehicle data corresponding to the driver state detection result obtained at the historical moment, and the historical moment is earlier than the first preset moment in the target time period.

2. The driver status detection method based on driving images and vehicle data according to claim 1, characterized in that: The generating of input data comprises: Randomly determine the historical moment from a plurality of moments earlier than the first preset moment within the target time period; Searching for the driver status detection result corresponding to the historical moment in the driver status detection result library; Among them, the driver state detection result library includes driver state detection results corresponding to multiple time moments, and the driver state detection results in the driver state detection result library include a first driver state detection result and a second driver state detection result. The first driver state detection result is obtained based on the state detection model, and the second driver state detection result is obtained based on annotation.

3. The driver status detection method based on driving images and vehicle data according to claim 1, characterized in that: The vehicle data includes at least the brake opening change rate, the steering wheel angular velocity and the vehicle acceleration; Acquiring the vehicle data at each of the preset moments within the target time period includes: The vehicle data at each preset time is obtained through the vehicle CAN bus.

4. The driver status detection method based on driving images and vehicle data according to claim 1, characterized in that: The training process of the state detection model includes: Training the state detection model based on multiple sets of first training data to obtain the state detection model after preliminary training, wherein the first training data includes first sample input data and a driver state label corresponding to the first sample input data; Determine the driver's facial features and target data types in the vehicle data based on multiple sets of second training data and the state detection model after preliminary training, wherein the second training data includes second sample input data and a driver state label corresponding to the second sample input data, and the driver's facial features and data types in the vehicle data included in multiple sets of the second sample input data are different; Based on the determined driver facial features and the target data type in the vehicle data, multiple groups of third training data are generated, and the state detection model after preliminary training is re-trained based on the third training data, wherein the third training data includes third sample input data and the driver state label corresponding to the third sample input data, and the driver facial features and the data type of the vehicle data included in the third sample input data are the target data type.

5. The driver status detection method based on driving images and vehicle data according to claim 1, characterized in that: The step of inputting the input data into a trained state detection model to obtain a driver state detection result output by the state detection model includes: Inputting the driver's facial features into the statistical module in the state detection model, obtaining the statistical features of the driver's facial features at each preset time in the target time period output by the statistical module, wherein the statistical features reflect the changes of the driver's facial features in the target time period; The statistical features, the vehicle data and at least one set of the historical driver status data are input into a classification module in the status detection model to obtain a driver status detection result output by the classification module.

6. The driver status detection method based on driving images and vehicle data according to claim 1, characterized in that: Before inputting the input data into the trained state detection model, the method includes: Based on the theoretical data range of the input data, data clipping and normalization processing are performed on the input data.

7. A driver status detection device based on driving images and vehicle data, characterized in that: The device comprises: A data acquisition module, used to acquire driving images and vehicle data at each preset time within a target time period, wherein the driving image is an image obtained by imaging the driver's position, and the vehicle data reflects the driver's control status of the vehicle, and the interval between adjacent preset times is a preset time length; A face detection module, used to input the driving image into a trained face detection model to obtain a driver's face area image output by the face detection model; a facial feature extraction module, for determining the facial features of the driver based on the facial region image of the driver, wherein the facial features of the driver include at least a head posture angle, a sight line angle, and an eye aspect ratio, wherein the head posture angle and the sight line angle are obtained by respectively inputting the facial region image of the driver into a trained posture angle detection model and a sight line angle detection model, and the eye aspect ratio is obtained based on eye key points, and the eye key points are obtained by inputting the facial region image of the driver into a trained key point detection model; A model inference module is used to generate input data, input the input data into a trained state detection model, and obtain a driver state detection result output by the state detection model, wherein the input data at least includes the driver's facial features, the vehicle data, and at least one set of historical driver state data, each set of historical driver state data includes the driver state detection result obtained at a historical moment and the driver's facial features and the vehicle data corresponding to the driver state detection result obtained at the historical moment, and the historical moment is earlier than the first preset moment in the target time period.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the driver state detection method based on driving images and vehicle data as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the driver status detection method based on driving images and vehicle data as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the driver status detection method based on driving images and vehicle data as claimed in any one of claims 1 to 6 is implemented.

Images