Driving prompting method, medium, product, equipment, driving prompting system and vehicle

Abstract

The invention relates to a driving prompting method, a medium, a product, equipment, a driving prompting system and a vehicle. The driving prompting method comprises the following steps: prompting an abnormal driving state of a driver according to driver image data acquired in a vehicle driving process; wherein different abnormal driving states correspond to different prompt contents. The abnormal driving state of the driver is prompted in real time, so that the mental state and the concentration degree of the driver can be improved, the response delay of the driver to dangerous conditions is improved, and the safety is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, and in particular to a driving prompting method, medium, product, device, driving prompting system and vehicle. BACKGROUND

[0002] In the process of driving a vehicle, the reaction delay of the driver to dangerous situations can cause traffic accidents to occur, threatening personal and property safety. Among them, the poor mental state or inattentive of the driver is an important factor causing the reaction delay. How to improve the reaction delay of the driver to improve safety is a technical problem to be solved. SUMMARY

[0003] The embodiments of the present application provide a driving prompting method, medium, product, device, driving prompting system and vehicle, which improves the mental state and concentration of the driver by prompting the abnormal driving state of the driver in real time, improves the reaction delay of the driver to dangerous situations, and improves safety to at least partially solve the above technical problems.

[0004] In order to achieve the above purpose, according to the first aspect of the present application, a driving prompting method is provided, comprising: prompting the abnormal driving state of the driver according to the driver image data collected in the process of driving the vehicle; wherein different abnormal driving states correspond to different prompt contents.

[0005] Optionally, the prompting the abnormal driving state of the driver comprises: prompting the abnormal driving state of the driver through voice.

[0006] Optionally, the abnormal driving state includes a fatigue driving state and / or a distracted driving state.

[0007] Optionally, the prompting the abnormal driving state of the driver according to the driver image data collected in the process of driving the vehicle comprises: acquiring face key point data according to the driver image data collected in the process of driving the vehicle; prompting the abnormal driving state of the driver according to the face key point data.

[0008] Optionally, the acquiring face key point data according to the driver image data collected in the process of driving the vehicle comprises: inputting the driver image data collected in the process of driving the vehicle into a first model to make the first model output face image data; inputting the face image data into a second model to make the second model output face key point data.

[0009] Optionally, the first model comprises a multi-layer feature extraction module and a face prediction module; the inputting of the driver image data collected during the driving of the vehicle into the first model so as to make the first model output face image data comprises: inputting the driver image collected during the driving of the vehicle into the multi-layer feature extraction module so as to make the multi-layer feature extraction module output face features in multiple scales; and inputting the face features in the multiple scales into the face prediction module so as to make the face prediction module output face image data.

[0010] Optionally, the multi-layer feature extraction module comprises convolution modules corresponding to the multiple scales respectively; the inputting of the driver image collected during the driving of the vehicle into the multi-layer feature extraction module so as to make the multi-layer feature extraction module output face features in multiple scales comprises: inputting the driver image collected during the driving of the vehicle into a first convolution module so as to make the first convolution module output the face features in a first scale; and inputting the face features in the first scale into a second convolution module so as to make the second convolution module output the face features in a second scale; wherein the first scale is greater than the second scale.

[0011] Optionally, the face prediction module comprises a classification regression module and a non-maximum suppression module; the inputting of the face features in the multiple scales into the face prediction module so as to make the face prediction module output face image data comprises: inputting the face features in the multiple scales into the classification regression module so as to make the classification regression module output face detection data; and inputting the face detection data into the non-maximum suppression module so as to make the non-maximum suppression module output face image data.

[0012] Optionally, the second model comprises a key point extraction module and an information prediction module; the inputting of the face image data into the second model so as to make the second model output face key point data comprises: inputting the face image data into the key point extraction module so as to make the key point extraction module output key point feature data; and inputting the key point feature data into the information prediction module so as to make the information prediction module output face key point data.

[0013] Optionally, the information prediction module comprises a probability prediction module, a coordinate prediction module and a decoding module; the inputting the key point feature data into the information prediction module, so that the information prediction module outputs the facial key point data, comprises: inputting the key point feature data into the probability prediction module, so that the probability prediction module outputs the existence probability of the key point; inputting the key point feature data into the coordinate prediction module, so that the coordinate prediction module outputs the coordinate offset of the key point; inputting the existence probability and the coordinate offset of the key point into the decoding module, so that the decoding module outputs the facial key point data.

[0014] Optionally, the prompting the abnormal driving state of the driver according to the facial key point data comprises: determining eye closure data according to the facial key point data; and prompting the abnormal driving state of the driver according to the eye closure data.

[0015] Optionally, the determining the eye closure data according to the facial key point data comprises: determining the eye closure data according to first key point data of the upper eyelid, second key point data of the lower eyelid and third key point data of the eye corner in the facial key point data.

[0016] Optionally, the prompting the abnormal driving state of the driver according to the eye closure data comprises: determining a first blink frequency in a first time period according to the eye closure data in the first time period and an eye closure threshold; and prompting the abnormal driving state of the driver according to the first blink frequency.

[0017] Optionally, the prompting the abnormal driving state of the driver according to the eye closure data comprises: prompting the abnormal driving state of the driver according to the eye closure data in a second time period and an eye closure threshold.

[0018] Optionally, the method further comprises: determining the eye closure threshold according to the eye closure data in a starting time period in which the driver drives the vehicle.

[0019] Optionally, the determining the eye closure threshold according to the eye closure data in the starting time period in which the driver drives the vehicle comprises: selecting eye closure data satisfying a first condition from the eye closure data in the starting time period in which the driver drives the vehicle; and determining the eye closure threshold according to the selected eye closure data.

[0020] Optionally, the prompting of the abnormal driving state of the driver according to the face key point data comprises: determining mouth closing data according to the face key point data; and prompting the abnormal driving state of the driver according to the mouth closing data.

[0021] Optionally, the determining of the mouth closing data according to the face key point data comprises: determining the mouth closing data according to fourth key point data of an upper lip, fifth key point data of a lower lip and sixth key point data of a mouth corner in the face key point data.

[0022] Optionally, the prompting of the abnormal driving state of the driver according to the mouth closing data comprises: determining a first yawning frequency in a third time period according to the mouth closing data in the third time period and a mouth closing threshold; and prompting the abnormal driving state of the driver according to the first yawning frequency.

[0023] Optionally, if the mouth closing data in a fourth time period are all greater than the mouth closing threshold, it is determined that the driver is yawning; wherein the fourth time period is one time period in the third time period.

[0024] Optionally, the prompting of the abnormal driving state of the driver according to the face key point data comprises: determining face deflection data according to the face key point data; and prompting the abnormal driving state of the driver according to the face deflection data.

[0025] Optionally, the determining of the face deflection data according to the face key point data comprises: performing key point matching on the face key point data according to a three-dimensional face model to obtain a rotation vector between the face key point data and the three-dimensional face model; and determining face deflection data according to the rotation vector.

[0026] Optionally, the face deflection data comprises at least one of the following: head turning angle data, head lifting angle data and head shaking angle data.

[0027] Optionally, the face deflection data comprises head lifting angle data; and the prompting of the abnormal driving state of the driver according to the face deflection data comprises: determining a first drowsiness frequency in a fifth time period according to the head lifting angle data in the fifth time period and a head lifting angle threshold; and prompting the abnormal driving state of the driver according to the first drowsiness frequency.

[0028] Optionally, if the head lifting angle data in a sixth time period are all greater than the head lifting angle threshold, it is determined that the driver is drowsy; wherein the sixth time period is one time period in the fifth time period.

[0029] Optionally, the face deflection data comprises a head turning angle data; and the prompting the abnormal driving state of the driver according to the face deflection data comprises: prompting the abnormal driving state of the driver according to the head turning angle data in the seventh time period and a head turning angle threshold.

[0030] Optionally, the face deflection data comprises a head shaking angle data; and the prompting the abnormal driving state of the driver according to the face deflection data comprises: prompting the abnormal driving state of the driver according to the head shaking angle data in the eighth time period and a head shaking angle threshold.

[0031] Optionally, the prompting the abnormal driving state of the driver according to the driver image data collected in the driving process of the vehicle comprises: performing target object recognition according to the driver image data collected in the driving process of the vehicle; and if the target object exists in the driver image data, prompting the abnormal driving state of the driver.

[0032] Optionally, the performing target object recognition according to the driver image data collected in the driving process of the vehicle comprises: inputting the driver image data collected in the driving process of the vehicle into a third model, so that the third model outputs a result of target object recognition.

[0033] Optionally, the target object comprises at least one of the following: a cup, a mobile phone, and a cigarette.

[0034] According to a second aspect of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the driving prompting method.

[0035] According to a third aspect of the present application, a computer program product is provided, which comprises a computer program. The computer program is executed by a processor to implement the driving prompting method.

[0036] According to a fourth aspect of the present application, an electronic device is provided, which comprises: a memory storing a computer program; and a processor configured to execute the computer program in the memory to implement the driving prompting method.

[0037] According to a fifth aspect of the present application, a driving prompting system is provided, which comprises a camera and an electronic device. The camera is configured to collect driver image data in a driving process of a vehicle. The electronic device is configured to prompt an abnormal driving state of a driver according to the driver image data. Different abnormal driving states correspond to different prompt contents.

[0038] Optionally, the driving prompt system further comprises a sound device; and the electronic device is further configured to: prompting the abnormal driving state of the driver by voice through the sound device.

[0039] According to a sixth aspect of the present application, a vehicle is provided, comprising the electronic device or the driving prompt system.

[0040] The technical scheme provided by the embodiments of the present application judges whether the driver is in an abnormal driving state according to the image data of the driver collected during the driving of the vehicle, and prompts the abnormal driving state when the driver is in the abnormal driving state. By prompting the abnormal driving state of the driver in real time, the mental state and concentration of the driver can be improved, and the reaction delay of the driver to dangerous situations can be improved, thereby improving the safety. Different abnormal driving states correspond to different prompt contents, and the embodiments of the present application distinguish different abnormal driving states and provide different prompt methods for different abnormal driving states, thereby realizing customized prompt functions, making the prompt contents closer to the actual abnormal driving state, and improving the human-computer interaction experience and the effectiveness of the prompt.

[0041] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0043] In order to more completely understand the present application and its beneficial effects, the following will be described in conjunction with the drawings, wherein the same reference numerals in the following description represent the same parts.

[0044] Figure 1 is a flowchart of a driving prompt method provided by the embodiments of the present application;

[0045] Figure 2 is a schematic diagram of a first model provided by the embodiments of the present application;

[0046] Figure 3 is a schematic diagram of a second model provided by the embodiments of the present application;

[0047] Figure 4 is a schematic diagram of face key point data provided by the embodiments of the present application;

[0048] Figure 5This is a schematic diagram of a driving prompting system provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of a driving prompting method provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of another driving prompting method provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0053] According to a first aspect of this application, embodiments of this application provide a driving prompt method.

[0054] Please see Figure 1 , Figure 1 This is a flowchart of a driving prompting method provided in an embodiment of this application. Figure 1 As shown, the driving prompt method may include the following steps:

[0055] Step S100: Based on the driver image data collected during vehicle operation, provide a prompt regarding the driver's abnormal driving state.

[0056] This application embodiment can collect driver image data in real time during vehicle operation. For example, driver image data can be collected continuously, or it can be collected periodically. Taking continuous driver image data collection as an example, video can be recorded during vehicle operation, and each frame of the video is a driver image data. In practical applications, one or more frames can be extracted from the recorded video as driver image data for analysis and processing.

[0057] The driver image data can be either a frontal or side view of the driver; this application does not limit the specific type of image data. Taking a frontal view as an example, a camera can be installed directly in front of the driver's seat to collect the driver's image data. For instance, a camera can be installed above the vehicle's dashboard, directly aimed at the driver, to capture and record the driver's upper body in real time.

[0058] Based on driver image data, it can be determined whether the driver is in an abnormal driving state. Abnormal driving states include, but are not limited to, fatigued driving and distracted driving. This application embodiment provides prompts when the driver is in an abnormal driving state, for example, through voice prompts, image prompts, or seat vibration prompts. In some embodiments, different abnormal driving states may correspond to different prompt content. This application embodiment can establish a mapping relationship between different abnormal driving states and different prompt content, thereby determining the specific prompt content based on this mapping relationship and the determined abnormal driving state. Different abnormal driving states include, but are not limited to, different types and degrees of abnormal driving states; different prompt content includes, but is not limited to, different prompt types, different prompt frequencies, and different prompt text.

[0059] Taking abnormal driving states, including fatigued driving and distracted driving, as an example, different prompts can be provided for each state. Different levels of fatigue and different types of distraction can also correspond to different prompts. For instance, when providing voice prompts for abnormal driving states, different voice messages can be provided for each state, including fatigued driving, distracted driving, and different types of distraction.

[0060] For example, abnormal driving states include fatigued driving and distracted driving. Fatigue driving includes mild fatigue and severe fatigue, while distracted driving includes drinking water, smoking, not looking directly ahead, and moving out of the detection range. Based on driver image data, when it is determined that the driver has moved out of the detection range and the driver's face cannot be detected, a voice message can be played: "You have moved out of the camera's range. Please adjust your position to ensure normal monitoring." When it is determined that the driver is mildly fatigued, a voice message can be played: "You have been detected to have mild fatigue. Please take a rest." When it is determined that the driver is severely fatigued, a voice message can be played: "You have been detected to have severe fatigue. Please take a rest." When it is determined that the driver is drinking water and is in a distracted driving state, a voice message can be played: "You have been detected to be drinking water while driving. Please concentrate on driving." When it is determined that the driver is smoking and is in a distracted driving state, a voice message can be played: "You have been detected to be smoking while driving. Please concentrate on driving." When it is detected that the driver is not looking directly ahead and is in a distracted driving state, a voice message can be played: "Please look ahead and keep your attention focused."

[0061] In summary, the technical solution provided in this application determines whether the driver is in an abnormal driving state based on driver image data collected during vehicle operation; when the driver is in an abnormal driving state, a prompt is given regarding that abnormal driving state. By providing real-time prompts regarding the driver's abnormal driving state, the driver's mental state and concentration can be improved, and the driver's reaction delay to dangerous situations can be reduced, thereby enhancing safety. Different abnormal driving states correspond to different prompt content. This application distinguishes between different abnormal driving states and provides different prompt methods for different abnormal driving states, realizing a customized prompt function. This makes the prompt content more closely resemble the actual abnormal driving state, improving the human-computer interaction experience and the effectiveness of the prompts.

[0062] In some embodiments, step S100 above includes the following steps:

[0063] Step S110: Obtain facial key point data based on the driver image data collected during vehicle operation;

[0064] Step S120: Based on facial key point data, provide a prompt regarding the driver's abnormal driving state.

[0065] This application embodiment extracts facial key point data from driver image data, and then determines the driver's abnormal driving state based on the extracted facial key point data. The facial key point data can be extracted using an artificial intelligence model, which includes, but is not limited to, at least one of the following: MTCNN (Multi-task Cascaded Convolutional Networks), SSD (Single Shot MultiBox Detector), ASM (Active Shape Models), AAM (Active Appearance Models), CenterFace (center face detection and key point localization algorithm), BlazeFace (fast face detection algorithm), etc. The facial key point data includes the coordinate data of facial key points, which are detection points of key facial features. For example, the facial key point data includes the coordinate data of detection points of at least one of the following features: eyes, mouth, nose, ears, eyebrows, etc.

[0066] In some embodiments, step S110 above includes the following steps:

[0067] Step S111: Input the driver image data collected during the vehicle's operation into the first model so that the first model outputs face image data;

[0068] Step S112: Input the face image data into the second model so that the second model outputs face key point data.

[0069] The first model is an artificial intelligence model for facial recognition. By processing driver image data using the first model, facial image data can be obtained. This facial image data refers to the image data of the facial region. The second model is an artificial intelligence model for key point extraction. By processing facial image data using the second model, facial key point data can be obtained.

[0070] In some embodiments, step S111 above includes the following steps:

[0071] Step S1111: Input the driver image collected during vehicle operation into the multi-layer feature extraction module in the first model so that the multi-layer feature extraction module outputs facial features at multiple scales;

[0072] Step S1112: Input facial features at multiple scales into the face prediction module in the first model so that the face prediction module outputs facial image data.

[0073] The first model includes a multi-layer feature extraction module and a face prediction module. The multi-layer feature extraction module is used to extract face features at multiple scales, and the face prediction module is used to predict the face position to output face image data.

[0074] This application does not limit the number of scales that the multi-layer feature extraction module can extract. In some embodiments, the multi-layer feature extraction module includes convolutional modules corresponding to multiple scales, each convolutional module being used to extract facial features at one scale. Taking a multi-layer feature extraction module including a first convolutional module and a second convolutional module as an example, the above step S1111 may include: inputting a driver image collected during vehicle movement into the first convolutional module, so that the first convolutional module outputs facial features at a first scale; inputting the facial features at the first scale into the second convolutional module, so that the second convolutional module outputs facial features at a second scale. The first scale is larger than the second scale. This application embodiment uses the output of the larger-scale convolutional module as the input of the smaller-scale convolutional module, which allows the facial features obtained by the convolutional modules to have different receptive fields.

[0075] In some embodiments, the face prediction module includes a classification regression module and a non-maximum suppression module; step S1112 above may include: inputting face features at multiple scales into the classification regression module so that the classification regression module outputs face detection data; inputting the face detection data into the non-maximum suppression module so that the non-maximum suppression module outputs face image data. The face detection data may include at least one detection result of a face region, and the detection result may include a face detection box and its confidence score. In this embodiment, at least one detection result of a face region is first obtained through the classification regression module, and then the detection result with the highest confidence and optimal position is obtained through the non-maximum suppression module as the output face image data.

[0076] For example, such as Figure 2 As shown, the first model includes a multi-layer feature extraction module 210 and a face prediction module 220. The multi-layer feature extraction module 210 comprises four convolutional modules 211, with 3, 2, 2, and 3 anchor points respectively. Each convolutional module 211 alternates between 3×3 and 1×1 convolutional kernels. Thus, in the multi-layer feature extraction module 210, features are first extracted using 3×3 convolutions, followed by the application of the ReLU activation function; then, 1×1 convolutions are used to reduce the number of channels, and the ReLU activation function is applied again; then, 3×3 convolutions are used again for feature extraction, and so on. Based on this, the feature maps obtained by the multi-layer feature extraction module 210 have different receptive fields, enabling better detection of targets at different scales. The face features at various scales output by the multi-layer feature extraction module 210 are input into the face prediction module 220. In the face prediction module 220, multiple detection boxes and confidence scores (face detection data) of the face are first obtained through the classification and regression module 221; then, the non-maximum suppression module 222 removes redundant detection boxes and retains the detection boxes with the highest confidence scores and the best positions, thereby improving the accuracy and simplicity of the face image data.

[0077] In some embodiments, step S112 above includes the following steps:

[0078] Step S1121: Input the face image data into the key point extraction module in the second model so that the key point extraction module outputs key point feature data;

[0079] Step S1122: Input the key point feature data into the information prediction module in the second model so that the information prediction module outputs facial key point data.

[0080] The second model includes a key point extraction module and an information prediction module. The key point extraction module is used to extract key point feature data, and the information prediction module is used to integrate the key point feature data to output facial key point data.

[0081] In some embodiments, the information prediction module includes a probability prediction module, a coordinate prediction module, and a decoding module; the above step S1122 may include: inputting key point feature data into the probability prediction module so that the probability prediction module outputs the existence probability of the key point; inputting key point feature data into the coordinate prediction module so that the coordinate prediction module outputs the coordinate offset of the key point; inputting the existence probability and coordinate offset of the key point into the decoding module so that the decoding module outputs facial key point data.

[0082] For example, such as Figure 3 As shown, the second model includes a keypoint extraction module 310 and an information prediction module 320. The keypoint extraction module 310 includes a preprocessing module 311 and a feature extraction module 312. The preprocessing module 311 compresses the face image data, for example, to a size of 160*160*3. The feature extraction module 312 performs feature extraction, for example, using a ResNet network. The information prediction module 320 includes a probability prediction module 321, a coordinate prediction module 322, and a decoding module 323. The probability prediction module 321 is based on the heatmap principle, using a 1×1 convolution kernel and a Sigmoid activation function, outputting a shape (H, W, 1). The probability prediction module 321 predicts whether each grid cell is a positive sample, roughly locating the probability of keypoint existence. The coordinate prediction module 322 predicts coordinate offsets, using a 1×1 convolution kernel, outputting a shape (H, W, 2), and predicting the coordinate offsets. To mitigate problems caused by quantization errors, an anchor point-based encoding method can be employed to improve the accuracy of coordinate information. This method has advantages over direct coordinate regression, especially in terms of location accuracy. Compared to coordinate regression, this method also avoids the adverse effects of Global Average Pooling (GAP) on accuracy. The second model is more efficient in data utilization, requiring no additional data sampling for expansion, making it more convenient in practical applications.

[0083] based on Figure 3 In the second model shown, the coordinates of each key point in the facial key point data can be calculated using the following formulas 1 and 2.

[0084] Formula 1: x = cx + Δx*S

[0085] Formula 2: y = cy + Δy*S

[0086] Where cx and cy are the center coordinates of the grid, and S is the step size of the ResNet feature map downsampling.

[0087] based on Figure 3 The second model shown can have its loss function constructed using the following formula 3.

[0088] Formula 3:

[0089] Where δ represents the absolute difference between the model output and the label; A and C are coefficients; θ is the threshold used to switch between the two different calculation methods of the loss function; ε is a smoothing term used to prevent the denominator from being zero; ω is a parameter used to adjust the slope of the loss function curve; α is a parameter used to adjust the weights; δ is the absolute difference between the model's predicted output and the label; β is the label value. When δ < θ, a logarithmic loss function is used; when δ ≥ θ, a linear loss function is used.

[0090] based on Figure 3 The second model shown allows inputting facial image data to obtain facial landmark data. This data includes the coordinates of multiple landmarks, for example, the x and y coordinates of 98 landmarks. The distribution of these facial landmark data within the facial image data can be as follows: Figure 4 As shown.

[0091] In summary, the technical solution provided in this application extracts facial image data from driver image data using an artificial intelligence model, and then extracts facial key point data from the facial image data to further analyze the driver's abnormal driving state based on the facial key point data. Obtaining facial key point data through an artificial intelligence model can improve the efficiency of facial key point data acquisition. Furthermore, the first and second models provided in this application have simple and lightweight structures, making them suitable for deployment on resource-constrained devices such as Raspberry Pi, without significant loss of accuracy, thus reducing deployment and operating costs.

[0092] In some embodiments, step S120 above includes the following steps:

[0093] Step S1211: Determine the eye closure data based on the facial key point data;

[0094] Step S1212: Based on the human eye closure data, provide a prompt regarding the driver's abnormal driving state.

[0095] This application embodiment can obtain human eye key point data from facial key point data, then determine human eye closure data based on human eye key point data, and then analyze the driver's abnormal driving state, such as fatigue driving state, based on facial closure data.

[0096] Among them, eye closure data is used to indicate the degree of eye closure. Eye keypoint data refers to key point data of the eye area, including but not limited to the first keypoint data of the upper eyelid, the second keypoint data of the lower eyelid, and the third keypoint data of the corner of the eye. In some embodiments, step S1211 may include: determining eye closure data based on the first keypoint data of the upper eyelid, the second keypoint data of the lower eyelid, and the third keypoint data of the corner of the eye from the facial keypoint data. The first keypoint data includes keypoint data of the center of the upper eyelid, keypoint data of the left side of the center of the upper eyelid, and keypoint data of the right side of the center of the upper eyelid; the second keypoint data includes keypoint data of the center of the lower eyelid, keypoint data of the left side of the center of the lower eyelid, and keypoint data of the right side of the center of the lower eyelid.

[0097] This application embodiment can determine eye closure data separately for the driver's left and right eyes. In practical applications, it is also possible to determine the overall eye closure data for both the left and right eyes. For example, the overall eye closure data can be obtained by averaging the eye closure data of the left and right eyes.

[0098] based on Figure 4 The facial key point data shown, the eye closure data of the left eye and the eye closure data of the right eye can be calculated by the following formulas 4 and 5 respectively.

[0099] Formula 4:

[0100] Formula 5:

[0101] Among them, EAR R EAR represents the eye closure data for the right eye. L Represents the left eye's eye closure data, ||E 61 -E 67 || represents the distance between keypoint 61 and keypoint 67, and the distances between other keypoints in the formula follow the same pattern.

[0102] In determining human eye closure data, this application embodiment not only focuses on the area around the eyes, but also fully considers changes in the overall shape of the eyes, thereby making the human eye closure data larger, more sensitive to data, and more conducive to capturing subtle changes in the driver's eye state.

[0103] Based on data on human eye closure, further analysis can be conducted on changes in the driver's eye state, such as the number of blinks or the duration of eye closure, to determine whether the driver is in an abnormal driving state.

[0104] In some embodiments, step S1212 may include: determining the first blink count within the first time period based on eye closure data and an eye closure threshold within the first time period; and providing a prompt to the driver regarding an abnormal driving state based on the first blink count. Specifically, if the eye closure data is less than the eye closure threshold, it is determined that the driver blinked. In practical applications, the driver's eye closure data can be analyzed in real time, and the driver's blink count can be statistically processed to determine whether the driver is in an abnormal driving state based on the first blink count within the first time period. Specifically, if the first blink count is less than a first quantity threshold or greater than a second quantity threshold, it is determined that the driver is in an abnormal driving state, such as mild fatigue, and a prompt is provided to the driver regarding the abnormal driving state. This application embodiment does not specifically limit the duration of the first time period, the first quantity threshold, or the second quantity threshold; these can be flexibly set according to requirements in practical applications. For example, the duration of the first time period can be set to half a minute, one minute, or two minutes, etc.; the first quantity threshold can be set to 3, 4, 5, or 7 times, etc.; and the second quantity threshold can be set to 30, 35, 45, 50, or 55 times, etc. A driver blinking too infrequently or too frequently can be considered an abnormal driving condition, such as fatigue.

[0105] In some embodiments, step S1212 may include: if the eye closure data within the second time period is all less than the eye closure threshold, then a warning is issued to the driver regarding an abnormal driving state. If the driver's eye closure data is less than the eye closure threshold for a longer period of time, it can be considered that the driver has been in a closed-eye state for a long time, such as severe driver fatigue, and a warning is required. This application embodiment does not impose a specific limitation on the duration of the second time period; it can be flexibly set according to needs in practical applications. For example, the second time period can be set to 3 seconds, 4 seconds, or 5 seconds, etc.

[0106] To more accurately determine a driver's blinking or eye-closing behavior and improve the accuracy of judging abnormal driving states, in some embodiments, the above method further includes:

[0107] Step S1210: Determine the eye closure threshold based on the eye closure data during the initial time period of the driver's vehicle operation.

[0108] This application does not impose specific limitations on the duration and start time of the starting time period, and can be flexibly set according to actual needs in practical applications. For example, the start time of the starting time period is the moment when the driver starts driving, or it can be a preset time after the driver starts driving, such as the 1st second or the 2nd second; the duration of the starting time period can be half a minute, 1 minute, or 2 minutes, etc.

[0109] In some embodiments, step S1210 may include: selecting eye closure data that meets a first condition from the eye closure data within the initial time period of the driver's vehicle operation; and determining an eye closure threshold based on the selected eye closure data. The first condition includes selecting the minimum value; for example, four, five, or ten eye closure data points with the minimum value may be selected. Of course, in practical applications, all eye closure data within the initial time period may also be selected to determine the eye closure threshold. Determining the eye closure threshold based on the selected eye closure data includes: averaging the selected eye closure data to obtain the eye closure threshold. Of course, in practical applications, weighted averaging or other processing may also be performed on the selected eye closure data to determine the eye closure threshold.

[0110] In summary, the technical solution provided in this application determines the driver's eye closure data based on facial key point data, and then analyzes the driver's abnormal driving state based on the facial closure data. The process of determining the eye closure data not only focuses on the area around the eyes but also fully considers changes in the overall shape of the eyes, resulting in larger eye closure data values, greater sensitivity to data, and better ability to capture subtle changes in the driver's eye state. Furthermore, when analyzing the driver's abnormal driving state, the eye closure threshold is determined based on the eye closure data during the initial time period of the driver's driving, avoiding misjudgments caused by fixed eye closure thresholds. This fully considers individual differences among drivers, adapting not only to different eye sizes but also to different blinking habits.

[0111] In some embodiments, step S120 above includes the following steps:

[0112] Step S1221: Determine mouth closure data based on facial key point data;

[0113] Step S1222: Based on mouth closure data, provide a prompt regarding the driver's abnormal driving state.

[0114] This application embodiment can obtain mouth key point data from facial key point data, then determine mouth closure data based on the mouth key point data, and then analyze the driver's abnormal driving state, such as fatigue driving state, based on the mouth closure data.

[0115] The mouth closure data is used to indicate the degree of mouth closure. Mouth key point data refers to key point data of the mouth area, including but not limited to the fourth key point data of the upper lip, the fifth key point data of the lower lip, and the sixth key point data of the corners of the mouth. In some embodiments, step S1221 may include: determining mouth closure data based on the fourth key point data of the upper lip, the fifth key point data of the lower lip, and the sixth key point data of the corners of the mouth from the facial key point data. The fourth key point data of the upper lip includes key point data on the left side of the center of the upper lip and key point data on the right side of the center of the upper lip; the fifth key point data of the lower lip includes key point data on the left side of the center of the lower lip and key point data on the right side of the center of the lower lip.

[0116] based on Figure 4 The facial key point data shown, including mouth closure data, can be calculated using the following formula 6.

[0117] Formula 6:

[0118] Where MAR represents mouth closure data, ||M 78 -M 86 || represents the distance between keypoint 78 and keypoint 86, and the distances between other keypoints in the formula follow the same pattern.

[0119] Based on mouth closure data, it can be determined whether the driver is yawning, thereby analyzing the driver's abnormal driving state. In some embodiments, the above step S1222 may include: determining the first number of yawns in the third time period based on the mouth closure data and the mouth closure threshold; and providing a prompt to the driver regarding the abnormal driving state based on the first number of yawns. Wherein, if the mouth closure data in the fourth time period is all greater than the mouth closure threshold, it is determined that the driver is yawning; the fourth time period is a time period within the third time period. Of course, in practical applications, it can also be considered that the driver has yawned when the mouth closure data is greater than the mouth closure threshold, without needing the mouth closure data to be greater than the mouth closure threshold for a specific duration. Wherein, if the first number of yawns is greater than the third quantity threshold, it is determined that the driver is in an abnormal driving state, such as a state of mild fatigue, and a prompt is provided to the driver regarding the abnormal driving state. This application embodiment does not specifically limit the duration of the third time period, the duration of the fourth time period, the mouth closure threshold, and the third quantity threshold; in practical applications, they can be flexibly set according to requirements. For example, the duration of the third time period can be half a minute, one minute, or two minutes; the duration of the fourth time period can be two seconds, three seconds, or five seconds; the mouth closure threshold can be 0.5, 0.68, or 0.7; and the third quantity threshold can be two, three, or five times.

[0120] In summary, the technical solution provided in this application determines mouth closure data based on facial key point data, and then analyzes the driver's abnormal driving state based on the mouth closure data. Specifically, by comparing the mouth closure data with a mouth closure threshold, it is possible to analyze whether the driver is yawning. This application's embodiment determines whether the driver is yawning by comparing mouth closure data and a mouth closure threshold over a period of time, which can avoid false positives. Furthermore, this application's embodiment determines whether the driver is in an abnormal driving state based on the number of yawns over a period of time, which can improve the accuracy of abnormal driving state detection.

[0121] In some embodiments, step S120 above includes the following steps:

[0122] Step S1231: Determine face deflection data based on facial key point data;

[0123] Step S1232: Based on the facial deflection data, provide a prompt regarding the driver's abnormal driving state.

[0124] Face deflection data is used to indicate the deflection state of a face. In some embodiments, step S1231 may include: performing keypoint matching on face keypoint data according to a 3D face model to obtain a rotation vector between the face keypoint data and the 3D face model; and determining face deflection data based on the rotation vector. The 3D face model may be a frontal, undeflected face model. By performing keypoint matching on the face keypoint data using the 3D face model, the transformation relationship between the face keypoint data and the 3D face model can be determined. This mapping relationship can be represented by a rotation vector. For example, the solvePnP function can be used to determine the rotation vector between the face keypoint data and the 3D face model. Face deflection data can be determined based on the rotation vector. In some embodiments, face deflection data includes at least one of the following: head turn angle data, head tilt angle data, and head shake angle data.

[0125] Because Euler angles are more readable and have wider applications, rotation vectors can be converted to Euler angles. For example, the rotation vector can first be converted to a quaternion, as shown in Formula 7 below; then the quaternion can be converted to Euler angles, as shown in Formula 8 below.

[0126] Formula 7:

[0127] Formula 8:

[0128] Where (x,y,z) are unit vectors in the XYZ axis directions, and θ is the angle rotated around the axis.

[0129] Since the results of arctan and arcsin are Since it cannot cover all Euler angles, atan2 can be used instead of arctan, so the above formula 8 can be transformed into the following formula 9.

[0130] Formula 9:

[0131] In the formula, θ is the yaw angle, i.e., the head-turning angle data; θ is the pitch angle, i.e., the head-up angle data; ψ is the roll angle, i.e., the head-down angle data.

[0132] In some embodiments, the face deflection data includes head-up angle data; the above steps S1232 may include: determining the first number of drowsinesses within the fifth time period based on the head-up angle data and the head-up angle threshold; and providing a prompt to the driver regarding abnormal driving status based on the first number of drowsinesses. Wherein, if the head-up angle data within the sixth time period is all greater than the head-up angle threshold, it is determined that the driver is drowsy; the sixth time period is a time period within the fifth time period. Of course, in practical applications, it is also possible to consider the driver as having dozed off once when the head-up angle data is greater than the head-up angle threshold, without requiring the head-up angle data to be greater than the threshold for a specific duration. It should be understood that when determining whether the driver is drowsy, the absolute value of the head-up angle data can be compared with the head-up angle threshold. Wherein, if the first number of drowsinesses is greater than the fourth quantity threshold, it is determined that the driver is in an abnormal driving state, such as a state of severe fatigue, and a prompt is provided to the driver regarding the abnormal driving state. This application embodiment does not impose specific limitations on the duration of the fifth time period, the duration of the sixth time period, the head-up angle threshold, and the fourth quantity threshold; these can be flexibly set according to requirements in practical applications. For example, the duration of the fifth time period can be half a minute, one minute, or two minutes; the duration of the sixth time period can be two seconds, three seconds, or five seconds; the head-up angle threshold can be 0.7, 0.8, or 0.85; and the fourth quantity threshold can be two, three, or five times.

[0133] In some embodiments, the face deflection data includes head-turning angle data; step S1232 may include: providing a prompt for the driver's abnormal driving state based on the head-turning angle data and the head-turning angle threshold within the seventh time period. Specifically, if all the head-turning angle data within the seventh time period is greater than the head-turning angle threshold, the driver is determined to be in an abnormal driving state, such as the driver not looking directly ahead, and a prompt is given regarding the driver's abnormal driving state. This application embodiment does not impose specific limitations on the duration of the seventh time period or the head-turning angle threshold; these can be flexibly set according to actual needs. For example, the duration of the seventh time period can be two seconds, three seconds, or five seconds, etc.; the head-turning angle threshold can be 4, 5, or 6, etc. When determining whether the driver is looking directly ahead, the absolute value of the head-turning angle data can be compared with the head-turning angle threshold.

[0134] In some embodiments, the face deflection data includes head-shaking angle data; step S1232 may include: providing a prompt to the driver regarding abnormal driving status based on the head-shaking angle data and a head-shaking angle threshold within an eighth time period. Specifically, if all head-shaking angle data within the eighth time period are greater than the head-shaking angle threshold, the driver is determined to be in an abnormal driving state, such as the driver not looking directly ahead, and a prompt is issued regarding the driver's abnormal driving status. This application embodiment does not impose specific limitations on the duration of the eighth time period or the head-shaking angle threshold; these can be flexibly set according to actual needs. For example, the duration of the eighth time period can be two seconds, three seconds, or five seconds, etc.; the head-shaking angle threshold can be 0.5, 1, or 1.2, etc. When determining whether the driver is looking directly ahead, the absolute value of the head-shaking angle data can be compared with the head-shaking angle threshold.

[0135] In summary, the technical solution provided in this application determines facial deflection data based on facial key point data, and then analyzes the driver's abnormal driving state based on the facial deflection data. The facial deflection data can include head-up angle data, head-turning angle data, and head-shaking angle data. Based on the facial deflection data, it can be determined whether the driver is drowsy or whether the driver is looking straight ahead, making the analysis of the driver's abnormal driving state more comprehensive and complete.

[0136] In some embodiments, step S100 above includes the following steps:

[0137] Step S130: Based on the driver image data collected during vehicle operation, target object identification is performed;

[0138] Step S140: If a target object is present in the driver's image data, a prompt will be given to indicate the driver's abnormal driving state.

[0139] In step S130, the presence of a target object in the driver image data can be detected. The target object is an object that can affect the driver's attention; for example, the target object includes at least one of the following: a water cup, a mobile phone, or a cigarette. For target object detection, embodiments of this application can be implemented based on an artificial intelligence model. In some embodiments, step S130 includes inputting the driver image data collected during vehicle operation into a third model, so that the third model outputs the target object recognition result. The third model is an artificial intelligence model, such as a YOLOv8nano model. The training of the third model can use the COCO dataset to accurately identify and locate the target object. The target object recognition result obtained through the third model includes the target object's location and type information, such as the starting point coordinates and width and height values ​​of the target object's detection box in the driver image data. The advantage of the YOLOv8nano model lies in its efficient target detection capability, especially suitable for real-time application scenarios. Its training is based on the COCO dataset, which can improve the model's robustness in complex scenarios.

[0140] In step S140, the driver is determined to be in an abnormal driving state, such as a distracted driving state, based on the result of the target object recognition. If the target object recognition result indicates that there is a target object in the driver's image, the driver is determined to be in an abnormal driving state, and a prompt is given to the driver regarding the abnormal driving state.

[0141] In summary, the technical solution provided in this application detects target objects in driver image data. If a target object is detected in the driver image data, it is determined that the driver is in an abnormal driving state, such as distracted driving, and the driver is promptly alerted to the abnormal driving state. This application improves the detection efficiency of target objects by using an artificial intelligence model. Furthermore, this application determines whether the driver is in an abnormal driving state based on the detection of target objects, making the detection of abnormal driving states more comprehensive and complete.

[0142] It should be understood that in the embodiments of this application, the results of the above-mentioned eye closure data, mouth closure data, face deflection data and target object recognition can be combined to determine whether the driver is in an abnormal driving state. By judging the driver's abnormal driving state through multiple conditions, the accurate perception of different situations can be achieved.

[0143] According to a second aspect of this application, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described driving prompting method. This non-transitory computer-readable storage medium possesses all the beneficial effects of the above-described driving prompting method, which will not be elaborated further here.

[0144] According to a third aspect of this application, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the above-described driving prompting method and has all the beneficial effects of the above-described driving prompting method, which will not be elaborated further here.

[0145] According to a fourth aspect of this application, embodiments of this application also provide an electronic device, including: a memory and a processor, wherein a computer program is stored in the memory; the processor is configured to execute the computer program in the memory to implement the steps of the above-described driving prompting method. This electronic device possesses all the beneficial effects of the above-described driving prompting method, which will not be elaborated upon further herein.

[0146] Computer-readable storage media can be, for example, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof, without particular limitation herein. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0147] In some embodiments of this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used or combined with an instruction execution system, apparatus, or device.

[0148] The aforementioned computer-readable storage medium may be included in the aforementioned electronic device, or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0149] Based on driver image data collected during vehicle operation, alerts are provided to drivers regarding abnormal driving states; different abnormal driving states correspond to different alert content.

[0150] Computer program code for performing operations of some embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a Local Area Network (LAN) or a Wide Area Network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.

[0152] It should also be noted that in some alternative implementations, the functions marked in the box may occur in a different order than those marked in the attached figures.

[0153] For example, two consecutively represented blocks can actually be executed in substantially parallel order, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, as well as combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0154] The units described in some embodiments of this application can be implemented in software or in hardware. The described units can also be located in a processor.

[0155] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0156] According to a fifth aspect of this application, a driving prompting system is provided that can be used to perform the driving prompting method described above.

[0157] Please see Figure 5 , Figure 5 This is a schematic diagram of a driving prompting system provided in an embodiment of this application. Figure 5 As shown, the driving prompt system includes a camera 100 and an electronic device 200.

[0158] Camera 100 is used to collect image data of the driver while the vehicle is in motion. Electronic device 200 is used to provide alerts to abnormal driving conditions of the driver based on the driver image data.

[0159] In some embodiments, such as Figure 5 As shown, the driving alert system also includes an audio 300, which is used by the electronic device 200 to alert the driver to abnormal driving conditions.

[0160] The driving prompting system has all the beneficial effects of the aforementioned electronic devices and driving prompting methods, which will not be elaborated upon here.

[0161] According to the sixth aspect of this application, such as Figure 8 As shown, this application also provides a vehicle 10, which includes the aforementioned electronic equipment or driving prompting system. This vehicle possesses all the beneficial effects of the aforementioned electronic equipment or driving prompting system, etc., which will not be elaborated upon here.

[0162] The vehicle may be a gasoline-powered vehicle, a plug-in hybrid electric vehicle, or a new energy vehicle, etc., and this application does not make any specific restrictions.

[0163] The technical solutions provided in the embodiments of this application will be described and illustrated below with some specific examples.

[0164] Please see Figure 6 , Figure 6 This is a schematic diagram of a driving prompting method provided in an embodiment of this application. For example... Figure 6 As shown, the driving prompt method may include the following steps S601 to S617.

[0165] Step S601: During vehicle operation, driver image data is collected via camera.

[0166] The camera can be positioned above the dashboard, directly facing the driver. It is connected to the Raspberry Pi via a serial port to capture and record the driver's upper body status in real time.

[0167] Step S602: Perform data preprocessing on the driver image data.

[0168] The data preprocessing includes extracting driver image data from frames captured by the camera, and adjusting the driver image data to a specific size and feature format, such as scaling it to 240×320 pixels and converting it to RGB format.

[0169] Step S603: Input the driver image data into the first model so that the first model outputs face image data.

[0170] Step S604: Input the face image data into the second model so that the second model outputs face key point data.

[0171] Step S605: Determine the eye closure data based on the facial key point data.

[0172] Step S606: Determine the number of blinks in the first time period based on the human eye closure data and the human eye closure threshold within the first time period.

[0173] If the eye closure data is less than the eye closure threshold, the driver blinks once. The number of blinks by the driver within the first time period is counted as the first blink count.

[0174] Step S607: Based on the first blink count, determine whether the driver is in an abnormal driving state. If yes, proceed to step S617 below; otherwise, continue to step S601.

[0175] If the number of blinks is less than the first threshold or greater than the second threshold, the driver is considered to be in a state of fatigued driving.

[0176] Step S608: Based on the eye closure data and eye closure threshold during the second time period, determine whether the driver is in an abnormal driving state. If yes, proceed to step S617 below; otherwise, continue to step S601.

[0177] If the eye closure data in the second time period are all less than the eye closure threshold, then the driver is in a state of fatigued driving.

[0178] Step S609: Determine mouth closure data based on facial key point data.

[0179] Step S610: Determine the number of first yawns in the third time period based on the mouth closure data and mouth closure threshold in the third time period.

[0180] If the number of mouth closures exceeds the mouth closure threshold during the fourth time period, then the driver yawned once. The number of yawns during the third time period is counted as the first yawn count.

[0181] Step S611: Based on the number of the first yawns, determine whether the driver is in an abnormal driving state. If yes, proceed to step S617 below; otherwise, continue to step S601.

[0182] If the number of the first yawns exceeds the third threshold, the driver is considered to be in a state of fatigued driving.

[0183] Step S612: Determine face deflection data based on facial key point data.

[0184] The facial deflection data includes head turning angle data, head tilting angle data, and head shaking angle data.

[0185] Step S613: Based on the head-up angle data and head-up angle threshold within the fifth time period, determine the number of times the first nap occurs within the fifth time period.

[0186] If the head-up angle data in the sixth time period is greater than the head-up angle threshold, it is determined that the driver has dozed off once. The number of times the driver dozed off in the fifth time period is counted as the first dozing-off count.

[0187] Step S614: Based on the first number of drowsiness counts, determine whether the driver is in an abnormal driving state. If yes, proceed to step S617 below; otherwise, continue to step S601.

[0188] If the number of times a driver falls asleep at first exceeds the fourth threshold, the driver is considered to be in a state of fatigued driving.

[0189] Step S615: Based on the head-turning angle data and the head-turning angle threshold within the seventh time period, determine whether the driver is in an abnormal driving state. If so, proceed to step S617 below; otherwise, continue to step S601.

[0190] If the head-turning angle data in the seventh time period is greater than the head-turning angle threshold, then the driver is not looking directly ahead and is in a state of distracted driving.

[0191] Step S616: Based on the head-shaking angle data and head-shaking angle threshold within the eighth time period, determine whether the driver is in an abnormal driving state. If so, proceed to step S617 below; otherwise, continue to step S601.

[0192] If the head-shaking angle data in the eighth time period is greater than the head-shaking angle threshold, then the driver is not looking straight ahead and is in a state of distracted driving.

[0193] Step S617: Provide voice prompts to the driver regarding abnormal driving conditions via audio.

[0194] Please see Figure 7 , Figure 7 This is a schematic diagram of a driving prompting method provided in an embodiment of this application. For example... Figure 7 As shown, the driving prompt method may include the following steps S701 to S705.

[0195] Step S701: During vehicle operation, driver image data is collected via camera.

[0196] The camera can be positioned above the dashboard, directly facing the driver. It is connected to the Raspberry Pi via a serial port to capture and record the driver's upper body status in real time.

[0197] Step S702: Perform data preprocessing on the driver image data.

[0198] The data preprocessing includes extracting driver image data from frames captured by the camera, and adjusting the driver image data to a specific size and feature format, such as scaling it to 240×320 pixels and converting it to RGB format.

[0199] Step S703: Input the driver image data into the third model so that the third model outputs the target object recognition result.

[0200] Step S704: Based on the result of target object recognition, determine whether the driver is in an abnormal driving state. If yes, proceed to step S705; otherwise, continue to step S701.

[0201] If the result of object recognition indicates that the driver's image data includes an object, then the driver is in a state of distracted driving.

[0202] Step S705: Provide voice prompts to the driver regarding abnormal driving conditions via audio.

[0203] like Figure 5As shown, the driving prompt system in this embodiment includes a camera 100, an electronic device 200, and an audio device 300. The driving prompt method described above can be implemented by the electronic device 200 and by the electronic device 200 controlling the camera 100 and the audio device 300. The electronic device is based on a Raspberry Pi and connects to the camera 100 via the Raspberry Pi's serial port. Furthermore, the electronic device 200 can connect to the audio device 300 via a Bluetooth module to provide voice prompts for abnormal driving conditions.

[0204] In summary, the improvements in the embodiments of this application include at least the following:

[0205] It adopts a lightweight face recognition model and key point detection model, which has a simpler structure compared to traditional deep learning networks;

[0206] It achieves adaptive judgment of human eye closure / squinting, solving the problem of false detection for drivers with small eyes;

[0207] The introduction of a multi-condition supervision module overcomes the limitations of a single detection standard and improves the accuracy of judging abnormal driving conditions.

[0208] Detecting distracted behaviors of the driver, such as drinking water and smoking, improves the system's comprehensive monitoring capabilities, enabling the system to detect fatigued driving and distracted driving in multiple aspects, further enhancing the system's practicality.

[0209] Using ONNX Runtime for inference, combined with model reconstruction, makes model conversion more convenient, improves the real-time inference efficiency of the system, and makes the detection results more timely and accurate.

[0210] It has a more economical computing power requirement, while its accuracy is comparable to that of traditional mainstream deep learning algorithms.

[0211] It can be deployed on Raspberry Pi to achieve real-time monitoring without the need for additional neural network accelerators, reducing costs and providing a feasible and efficient solution for embedded systems.

[0212] By setting multiple conditions to determine the driver's abnormal driving state, the system can accurately perceive different situations and provide customized voice prompts based on specific circumstances. This makes the prompts more relevant to actual fatigue or distraction, improving user experience and the effectiveness of the prompts.

[0213] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0214] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0215] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.

[0216] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although the descriptions of each embodiment in this application have different focuses, and parts not described in detail in a certain embodiment can be referred to the relevant descriptions of other embodiments, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A driving prompting method, characterized in that, The method includes: Based on driver image data collected during vehicle operation, alerts are provided to drivers regarding abnormal driving states; different abnormal driving states correspond to different alert content.

2. The driving prompting method according to claim 1, characterized in that, The method of alerting the driver to abnormal driving conditions includes: The system provides voice prompts to alert the driver to any abnormal driving behavior.

3. The driving prompting method according to claim 1, characterized in that, The abnormal driving state includes fatigued driving state and / or distracted driving state.

4. The driving prompting method according to claim 1, characterized in that, The step of alerting the driver to abnormal driving conditions based on driver image data collected during vehicle operation includes: Based on driver image data collected during vehicle operation, facial key point data is obtained; Based on the facial key point data, alerts are provided to drivers regarding abnormal driving conditions.

5. The driving prompting method according to claim 4, characterized in that, The step of obtaining facial key point data based on driver image data collected during vehicle operation includes: The driver image data collected during the vehicle's operation is input into the first model so that the first model outputs facial image data; The facial image data is input into the second model so that the second model outputs facial key point data.

6. The driving prompting method according to claim 5, characterized in that, The first model includes a multi-layer feature extraction module and a face prediction module; The step of inputting driver image data collected during vehicle operation into the first model, so that the first model outputs facial image data, includes: The driver image captured during vehicle operation is input into the multi-layer feature extraction module so that the multi-layer feature extraction module outputs facial features at multiple scales; The facial features at various scales are input into the face prediction module so that the face prediction module outputs facial image data.

7. The driving prompting method according to claim 6, characterized in that, The multi-layer feature extraction module includes convolution modules corresponding to the various scales respectively; The driver image captured during vehicle operation is input into the multi-layer feature extraction module, so that the multi-layer feature extraction module outputs facial features at multiple scales, including: The driver image captured during vehicle operation is input into the first convolution module, so that the first convolution module outputs the facial features at a first scale; The facial features at the first scale are input into the second convolution module, so that the second convolution module outputs the facial features at the second scale; wherein, the first scale is larger than the second scale.

8. The driving prompting method according to claim 6, characterized in that, The face prediction module includes a classification regression module and a non-maximum suppression module; The step of inputting the facial features at multiple scales into the face prediction module, so that the face prediction module outputs facial image data, includes: The facial features at various scales are input into the classification and regression module so that the classification and regression module outputs facial detection data. The face detection data is input into the nonmaximum suppression module so that the nonmaximum suppression module outputs face image data.

9. The driving prompting method according to claim 5, characterized in that, The second model includes a key point extraction module and an information prediction module; The step of inputting the facial image data into the second model so that the second model outputs facial key point data includes: The face image data is input into the key point extraction module so that the key point extraction module outputs key point feature data; The key point feature data is input into the information prediction module so that the information prediction module outputs facial key point data.

10. The driving prompting method according to claim 9, characterized in that, The information prediction module includes a probability prediction module, a coordinate prediction module, and a decoding module; The step of inputting the key point feature data into the information prediction module, so that the information prediction module outputs facial key point data, includes: The key point feature data is input into the probability prediction module so that the probability prediction module outputs the existence probability of the key points. The key point feature data is input into the coordinate prediction module so that the coordinate prediction module outputs the coordinate offset of the key point; The existence probability and coordinate offset of the key points are input into the decoding module so that the decoding module outputs facial key point data.

11. The driving prompting method according to claim 4, characterized in that, The step of alerting the driver to abnormal driving conditions based on the facial key point data includes: Based on the facial key point data, determine the eye closure data; Based on the aforementioned eye closure data, the driver's abnormal driving state is alerted.

12. The driving prompting method according to claim 11, characterized in that, The step of determining eye closure data based on the facial key point data includes: Based on the first key point data of the upper eyelid, the second key point data of the lower eyelid, and the third key point data of the corner of the eye in the facial key point data, the eye closure data is determined.

13. The driving prompting method according to claim 11, characterized in that, The step of alerting the driver to abnormal driving conditions based on the eye closure data includes: Based on the eye closure data and eye closure threshold within the first time period, determine the first blink count within the first time period; Based on the first blink count, a warning is given to the driver regarding abnormal driving behavior.

14. The driving prompting method according to claim 11, characterized in that, The step of alerting the driver to abnormal driving conditions based on the eye closure data includes: Based on the eye closure data and eye closure threshold during the second time period, the driver's abnormal driving state is alerted.

15. The driving prompting method according to any one of claims 11 to 14, characterized in that, The method further includes: The eye closure threshold is determined based on the eye closure data during the initial time period when the driver was driving the vehicle.

16. The driving prompting method according to claim 15, characterized in that, The step of determining the eye closure threshold based on the eye closure data during the initial time period of the driver driving the vehicle includes: Select eye closure data that meets the first condition from the eye closure data during the initial time period when the driver is driving the vehicle; The human eye closure threshold is determined based on the selected human eye closure data.

17. The driving prompting method according to claim 4, characterized in that, The step of alerting the driver to abnormal driving conditions based on the facial key point data includes: Based on the facial key point data, determine the mouth closure data; Based on the mouth closure data, the driver's abnormal driving state is alerted.

18. The driving prompting method according to claim 17, characterized in that, The step of determining mouth closure data based on the facial key point data includes: Based on the fourth key point data of the upper lip, the fifth key point data of the lower lip, and the sixth key point data of the corner of the mouth in the facial key point data, the mouth closure data is determined.

19. The driving prompting method according to claim 17, characterized in that, The step of alerting the driver to abnormal driving conditions based on the mouth closure data includes: Based on the mouth closure data and mouth closure threshold within the third time period, determine the first number of yawns within the third time period; Based on the first number of yawns, a warning is issued to the driver regarding abnormal driving behavior.

20. The driving prompting method according to claim 19, characterized in that, If the mouth closure data in the fourth time period is greater than the mouth closure threshold, then it is determined that the driver is yawning; wherein, the fourth time period is a time period in the third time period.

21. The driving prompting method according to claim 4, characterized in that, The step of alerting the driver to abnormal driving conditions based on the facial key point data includes: Based on the facial key point data, determine the facial deflection data; Based on the facial deflection data, alerts are provided to drivers regarding abnormal driving conditions.

22. The driving prompting method according to claim 21, characterized in that, The step of determining face deflection data based on the facial key point data includes: Based on the 3D face model, key point matching is performed on the facial key point data to obtain the rotation vector between the facial key point data and the 3D face model; Based on the rotation vector, determine the face deflection data.

23. The driving prompting method according to claim 22, characterized in that, The face deflection data includes at least one of the following: head turning angle data, head raising angle data, and head shaking angle data.

24. The driving prompting method according to claim 22, characterized in that, The face deflection data includes head tilt angle data; The step of alerting the driver to abnormal driving conditions based on the facial deflection data includes: Based on the head-up angle data and head-up angle threshold within the fifth time period, determine the first number of naps within the fifth time period; Based on the first number of times the driver dozes off, a warning is issued regarding the driver's abnormal driving state.

25. The driving prompting method according to claim 24, characterized in that, If the head-up angle data within the sixth time period is greater than the head-up angle threshold, then it is determined that the driver is dozing off; wherein, the sixth time period is a time period within the fifth time period.

26. The driving prompting method according to claim 22, characterized in that, The face deflection data includes head turning angle data; The step of alerting the driver to abnormal driving conditions based on the facial deflection data includes: Based on the head-turning angle data and head-turning angle threshold within the seventh time period, the driver's abnormal driving state is alerted.

27. The driving prompting method according to claim 22, characterized in that, The facial deflection data includes head shaking angle data; The step of alerting the driver to abnormal driving conditions based on the facial deflection data includes: Based on the head-shaking angle data and head-shaking angle threshold within the eighth time period, the driver's abnormal driving state is alerted.

28. The driving prompting method according to claim 1, characterized in that, The step of alerting the driver to abnormal driving conditions based on driver image data collected during vehicle operation includes: Target object identification is performed based on driver image data collected during vehicle operation. If the target object is present in the driver image data, a warning will be issued regarding the driver's abnormal driving state.

29. The driving prompting method according to claim 28, characterized in that, The target object identification based on driver image data collected during vehicle operation includes: The driver image data collected during the vehicle's movement is input into the third model, so that the third model outputs the result of target object recognition.

30. The driving prompting method according to claim 28 or 29, characterized in that, The target object includes at least one of the following: a water cup, a mobile phone, or a cigarette.

31. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the driving prompting method according to any one of claims 1 to 30.

32. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the driving prompting method according to any one of claims 1 to 30.

33. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the driving prompting method according to any one of claims 1 to 30.

34. A driving prompting system, characterized in that, The driving prompting system includes a camera (100) and electronic devices (200); wherein, The camera (100) is used to: collect driver image data during vehicle operation; The electronic device (200) is used to: provide prompts for abnormal driving states of the driver based on the driver image data; wherein different abnormal driving states correspond to different prompt content.

35. The driving prompting system according to claim 34, characterized in that, The driving alert system also includes an audio system (300); wherein, The electronic device is also used to: provide voice prompts to the driver regarding abnormal driving conditions via the audio system (300).

36. A vehicle, characterized in that, This includes the electronic device as described in claim 33, or the driving alert system as described in claim 34 or 35.

Images