Distraction driving alarm method, device, vehicle, medium, equipment and program product

Abstract

The application discloses a method and device for distracting driving alarm, a vehicle, a medium, equipment and a program product, and belongs to the field of data processing. The method comprises the following steps: a perception model detects a current image by perception according to a three-dimensional line-of-sight reference direction and a three-dimensional face reference direction, and outputs a line-of-sight direction angle and a face direction angle of a driver; the line-of-sight direction angle and the face direction angle are compared with corresponding reference angle thresholds, respectively; if at least one of the line-of-sight direction angle and the face direction angle is greater than the corresponding reference angle threshold, a distracting confidence of the current image is set; the distracting confidences of images in a predetermined time period are sorted to obtain a distracting time sequence, and a distracting average value of the distracting time sequence is calculated; and if the distracting average value is greater than a distracting threshold, the driver is alarmed for distraction. The application improves the accuracy of judgment and ensures the real-time of alarm.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, device, vehicle, medium, equipment, and program product for providing a distraction driving alarm. Background Technology

[0002] A Driver Monitor System (DMS) is an aftermarket product installed in the driver's cab. It uses one or more cameras facing the driver and employs deep learning technology to monitor the driver and issue timely warnings when the driver engages in behaviors that affect driving.

[0003] When the driver's face is obscured, or when the vehicle is turning left or right, existing driver fatigue warning systems may misjudge the driver's distracted state, resulting in a poor user experience. Therefore, it is important to reduce the misjudgment rate of driver fatigue warning systems regarding driver distraction without affecting the real-time nature of the alarm. Summary of the Invention

[0004] To achieve the above objectives, this application mainly provides a distraction driving alarm method, device, vehicle, medium, equipment, and program product.

[0005] In a first aspect, embodiments of this application provide a method for alerting distracted driving, comprising:

[0006] By using a perception model to perform perception detection on the driver in a predetermined number of historical images in the camera coordinate system corresponding to the camera parameters, the driver's gaze direction and face direction in the historical images are obtained.

[0007] Using the gaze orientation and face orientation of a predetermined number of historical images, the three-dimensional gaze reference orientation and the three-dimensional face reference orientation are respectively calibrated;

[0008] Based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle.

[0009] The gaze angle and face angle are compared with their corresponding baseline angle thresholds. If at least one of the gaze angle and face angle is greater than its corresponding baseline angle threshold, the distraction confidence of the current image is set numerically.

[0010] The distraction confidence scores of images within a predetermined time period are sorted to obtain a distraction time series. The average distraction score of the distraction time series is calculated. If the average distraction score is greater than the distraction threshold, a distraction alarm is triggered for the driver.

[0011] Optionally, using the gaze orientation and face orientation of a predetermined number of historical images, the three-dimensional gaze reference orientation and the three-dimensional face reference orientation are calibrated respectively. This further includes: transforming the pupil center point coordinates and the nose tip point coordinates in the camera coordinate system to the pupil center point coordinates and the nose tip point coordinates in the image coordinate system, respectively, and establishing a gaze orientation coordinate system and a face orientation coordinate system based on the two-dimensional vertical plane of the image, with the pupil center point coordinates and the nose tip point coordinates as their corresponding origins, respectively, and obtaining the gaze orientation to be calibrated in the gaze orientation coordinate system and the face orientation to be calibrated in the face orientation coordinate system; calculating the gaze orientation offset between the gaze orientation and the gaze orientation to be calibrated, and the face orientation offset between the face orientation and the face orientation to be calibrated; weighting the gaze offset and face orientation offset of the predetermined number of images to obtain the corresponding average gaze orientation and average face orientation; and using the average gaze orientation and average face orientation to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation in the camera coordinate system.

[0012] Optionally, based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle, including: mapping the driver's gaze orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its horizontal plane in the coordinate system corresponding to the three-dimensional gaze reference orientation, to obtain the gaze orientation angle, wherein the gaze orientation arc includes a leftward gaze arc or a rightward gaze arc, and the gaze orientation angle includes a leftward gaze angle or a rightward gaze angle; and mapping the driver's face orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its vertical plane in the coordinate system corresponding to the three-dimensional face reference orientation, to obtain the face orientation angle, wherein the face orientation arc includes an upward face arc or a downward face arc, and the face orientation angle includes an upward face angle or a downward face angle.

[0013] Optionally, the gaze orientation angle and face orientation angle are compared with their corresponding reference angle thresholds, including: comparing the gaze leftward angle with the left reference angle threshold, or comparing the gaze rightward angle with the right reference angle threshold; comparing the face upward angle with the upward reference angle threshold, or comparing the face downward angle with the downward reference angle threshold, wherein the reference angle thresholds include the left reference angle threshold, the right reference angle threshold, the upward reference angle threshold, and the downward reference angle threshold.

[0014] Optionally, the driver's face in the image is occluded by using the facial key point detection results output by the perception model. If the number of facial key points is less than the threshold, or if there is a lack of eye key points or nose tip key points, the image frame is filtered out.

[0015] Optionally, the distraction confidence of the images within a predetermined time period is sorted to obtain a distraction time sequence, which further includes: within the predetermined time period, if a left turn signal or a right turn signal of a driving vehicle is received while sorting the current frame image, the sorting process of the current frame image is terminated.

[0016] Secondly, embodiments of this application provide a distracted driving alarm device, which includes:

[0017] The orientation acquisition module is used to perform perception detection on the driver in a predetermined number of historical images in the camera coordinate system corresponding to the camera parameters through a perception model, so as to obtain the driver's gaze orientation and face orientation in the historical images.

[0018] Orientation calibration module, which is used to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation respectively using the gaze orientation and face orientation of a predetermined number of historical images;

[0019] The orientation angle output module is used to output the driver's gaze orientation angle and face orientation angle based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation. The perception model detects the current image through perception and outputs the driver's gaze orientation angle and face orientation angle.

[0020] The angle comparison module is used to compare the gaze angle and the face angle with their corresponding reference angle thresholds. If at least one of the gaze angle and the face angle is greater than its corresponding reference angle threshold, the distraction confidence of the current image is set numerically.

[0021] The alarm judgment module is used to sort the distraction confidence of images within a predetermined time period to obtain a distraction time sequence, and calculate the average distraction value of the distraction time sequence. If the average distraction value is greater than the distraction threshold, a distraction alarm is triggered for the driver.

[0022] Optionally, using the gaze orientation and face orientation of a predetermined number of historical images, the three-dimensional gaze reference orientation and the three-dimensional face reference orientation are calibrated respectively. This further includes: transforming the pupil center point coordinates and the nose tip point coordinates in the camera coordinate system to the pupil center point coordinates and the nose tip point coordinates in the image coordinate system, respectively, and establishing a gaze orientation coordinate system and a face orientation coordinate system based on the two-dimensional vertical plane of the image, with the pupil center point coordinates and the nose tip point coordinates as their corresponding origins, respectively, and obtaining the gaze orientation to be calibrated in the gaze orientation coordinate system and the face orientation to be calibrated in the face orientation coordinate system; calculating the gaze orientation offset between the gaze orientation and the gaze orientation to be calibrated, and the face orientation offset between the face orientation and the face orientation to be calibrated; weighting the gaze offset and face orientation offset of the predetermined number of images to obtain the corresponding average gaze orientation and average face orientation; and using the average gaze orientation and average face orientation to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation in the camera coordinate system.

[0023] Optionally, based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle, including: mapping the driver's gaze orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its horizontal plane in the coordinate system corresponding to the three-dimensional gaze reference orientation, to obtain the gaze orientation angle, wherein the gaze orientation arc includes a leftward gaze arc or a rightward gaze arc, and the gaze orientation angle includes a leftward gaze angle or a rightward gaze angle; and mapping the driver's face orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its vertical plane in the coordinate system corresponding to the three-dimensional face reference orientation, to obtain the face orientation angle, wherein the face orientation arc includes an upward face arc or a downward face arc, and the face orientation angle includes an upward face angle or a downward face angle.

[0024] Optionally, the gaze orientation angle and face orientation angle are compared with their corresponding reference angle thresholds, including: comparing the gaze leftward angle with the left reference angle threshold, or comparing the gaze rightward angle with the right reference angle threshold; comparing the face upward angle with the upward reference angle threshold, or comparing the face downward angle with the downward reference angle threshold, wherein the reference angle thresholds include the left reference angle threshold, the right reference angle threshold, the upward reference angle threshold, and the downward reference angle threshold.

[0025] Optionally, the driver's face in the image is occluded by using the facial key point detection results output by the perception model. If the number of facial key points is less than the threshold, or if there is a lack of eye key points or nose tip key points, the image frame is filtered out.

[0026] Optionally, the distraction confidence of the images within a predetermined time period is sorted to obtain a distraction time sequence, which further includes: within the predetermined time period, if a left turn signal or a right turn signal of a driving vehicle is received while sorting the current frame image, the sorting process of the current frame image is terminated.

[0027] Thirdly, this application provides a vehicle, wherein the vehicle includes the distraction driving alarm device of Scheme 2.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed, cause the computer to perform the distracted driving alarm method in Scheme 1.

[0029] Fifthly, embodiments of this application provide a computer device including at least one processor, the processor and a memory coupled together, the memory storing computer instructions, and the computer instructions being executed by the processor to implement the distracted driving alarm method in Scheme 1.

[0030] Sixthly, embodiments of this application provide a computer program product comprising computer instructions, wherein when the computer instructions are executed, they implement the distracted driving alarm method as described in Scheme 1.

[0031] The technical solution of this application embodiment calibrates the driver's head orientation and gaze orientation, and then outputs the angle of the driver's face and gaze in the image through a perception model. By comparing the angles and durations in the four dimensions of up, down, left, and right, it determines whether the driver is distracted. This not only improves the accuracy of the judgment, but also ensures the real-time nature of the alarm. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description exemplarily illustrate some embodiments of this application.

[0033] Figure 1 This is a schematic diagram illustrating a specific implementation of a distraction driving alarm method according to this application;

[0034] Figure 2 This is a schematic diagram illustrating a specific embodiment of a distraction driving alarm device according to this application;

[0035] Figure 3 This is a schematic diagram illustrating a specific embodiment of a computer device according to this application.

[0036] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0037] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this application can be more easily understood by those skilled in the art, thereby providing a clearer and more definite definition of the scope of protection of this application.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0039] The application scenario of the entire solution in this application is as follows: The solution for judging whether a driver is distracted is applied in scenarios with higher vehicle speeds (e.g., speeds above 50 km / h). In scenarios with lower vehicle speeds (e.g., speeds below 10 km / h), a general fatigue driving monitoring system can promptly remind the driver, and the same effect can be achieved without this solution.

[0040] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0041] Figure 1 This paper illustrates a specific implementation of a distraction driving alarm method according to this application.

[0042] exist Figure 1In the specific implementation shown, a distracted driving alarm method mainly includes:

[0043] Step S101: Using a perception model, a predetermined number of historical images are used to detect the driver's gaze direction and face direction in the historical images under the camera coordinate system corresponding to the camera parameters.

[0044] In this embodiment, within a camera coordinate system with calibrated camera parameters, the perception model detects the position coordinates of various facial key points of the driver in the image, obtaining the position coordinates of the nose tip and the two pupil points. Furthermore, through complex calculations, it obtains the position coordinates of the pupil center point, located at the center of the line connecting the two pupil points. The perception model also detects the facial orientation based on the nose tip coordinates and nostril orientation; and detects the gaze orientation based on the pupil center point based on the pupil coordinates and eye gaze orientation. The direct output of the perception model makes this scheme more efficient and lays the foundation for subsequent calculations.

[0045] In one specific embodiment of this application, the facial key point detection results of the driver output by the perception model are used to determine whether the driver's face in the image is occluded. When the facial key point detection results show that the number of facial key points is less than the number threshold, or that eye key points are missing, or that nose tip key points are missing, the image frame is filtered out.

[0046] In this embodiment, the perception model obtains the position of the driver's pupil center point, the position of the nose tip point, the direction of the gaze, and the direction of the face in the image based on the position coordinates of the detected facial key points. Therefore, the perception model also outputs the facial key point detection results, including which parts of the key points are missing and the number of key points. If the nose tip key point is missing, or the eye key point is missing, or the number of key points is less than the number threshold, it is considered that the driver's face in the image is occluded, indicating that the frame of the image needs to be discarded to prevent interference to subsequent steps due to incomplete facial key points, and also to reduce the false alarm rate of this application.

[0047] exist Figure 1 In the specific implementation shown, a distracted driving alarm method further includes:

[0048] Step S102: Using the gaze orientation and face orientation of a predetermined number of historical images, the three-dimensional gaze reference orientation and the three-dimensional face reference orientation are respectively calibrated.

[0049] In this embodiment, the driver's head's three-dimensional coordinate system is first calibrated using the pupil center point position, gaze direction, nose tip point position, and face orientation from a predetermined number of historical image frames. Then, the gaze direction and face orientation in the camera coordinate system are calibrated using the gaze direction and face orientation in the head's three-dimensional coordinate system, resulting in a three-dimensional gaze reference orientation and a three-dimensional face reference orientation. This can be used to determine the driver's left-right and up-down distractions, where left-right distraction includes the driver's eyes looking left and right, and up-down distraction includes the driver's eyes looking up and down. The three-dimensional gaze reference orientation and the three-dimensional face reference orientation lay the foundation for subsequent calculations, and the corresponding face orientation and gaze direction are tailored for drivers of different body types, improving calibration efficiency. The calibration process is simple and efficient.

[0050] In one specific embodiment of this application, the three-dimensional gaze reference orientation and three-dimensional face reference orientation are calibrated using the gaze orientation and face orientation of a predetermined number of historical images. The process further includes: transforming the pupil center point coordinates and the nose tip point coordinates in the camera coordinate system to the pupil center point coordinates and the nose tip point coordinates in the image coordinate system, respectively. Using the pupil center point coordinates and the nose tip point coordinates as their corresponding origins, a gaze orientation coordinate system and a face orientation coordinate system based on the two-dimensional vertical plane of the image are established, respectively, to obtain the gaze orientation to be calibrated in the gaze orientation coordinate system and the face orientation to be calibrated in the face orientation coordinate system; calculating the gaze orientation offset between the gaze orientation and the gaze orientation to be calibrated, and the face orientation offset between the face orientation and the face orientation to be calibrated; weighting the gaze offset and face orientation offset of the predetermined number of images to obtain the corresponding average gaze orientation and average face orientation; and using the average gaze orientation and average face orientation to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation in the camera coordinate system.

[0051] In this embodiment, the face orientation and gaze orientation in the camera coordinate system are first determined with the radian values ​​of their three coordinate dimensions. These values ​​are then compared with the radian values ​​of the face orientation and gaze orientation in the gaze orientation coordinate system to obtain the offset of the face orientation and the offset of the gaze orientation. By calculating the average of the offsets, the calibrated reference orientation is obtained. This method is simple and efficient, making the calibration process of the driver's face and gaze more convenient and faster.

[0052] exist Figure 1 In the specific implementation shown, a distracted driving alarm method further includes:

[0053] Step S103: Based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle.

[0054] In this embodiment, obtaining the gaze angle and the face angle requires converting the three-dimensional coordinate system to a two-dimensional coordinate system, which makes the distraction judgment simpler and more convenient.

[0055] In one specific embodiment of this application, based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle, including: in the coordinate system corresponding to the three-dimensional gaze reference orientation, mapping the driver's gaze orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its horizontal plane to obtain the gaze orientation angle, wherein the gaze orientation arc includes a gaze to the left or a gaze to the right arc, and the gaze orientation angle includes a gaze to the left or a gaze to the right angle; and in the coordinate system corresponding to the three-dimensional face reference orientation, mapping the driver's face orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its vertical plane to obtain the face orientation angle, wherein the face orientation arc includes a face-up arc or a face-down arc, and the face orientation angle includes a face-up angle or a face-down angle.

[0056] In this embodiment, in the coordinate system corresponding to the three-dimensional gaze reference orientation, since it is necessary to determine whether the driver is distracted to the left or right, the gaze orientation arc is either a leftward or rightward arc. This requires mapping the gaze orientation arc onto a horizontal plane to obtain an angle, which could be a leftward or rightward angle. Similarly, in the coordinate system corresponding to the three-dimensional face reference orientation, since it is necessary to determine whether the driver is distracted vertically, the face orientation arc is either an upward or downward arc. This requires mapping the face orientation arc onto a vertical plane to obtain an angle, which could be an upward or downward angle. By mapping the gaze orientation arc and face orientation arc to their corresponding two-dimensional coordinate systems to obtain their respective angle values, the determination of distraction becomes more accurate.

[0057] In a specific example of this application, it can also output the driver's gaze curvature and facial curvature, since curvature and angle are interchangeable. Because angle determination is more convenient and accurate, this application uses angle output as an example for illustration, but in practice, it does not impose any specific limitations.

[0058] exist Figure 1 In the specific implementation shown, a distracted driving alarm method further includes:

[0059] Step S104: Compare the gaze angle and face angle with their corresponding reference angle thresholds. If at least one of the gaze angle and face angle is greater than its corresponding reference angle threshold, then set the value of the distraction confidence of the current image.

[0060] In this embodiment, as long as the gaze angle and face angle are greater than their corresponding reference angle thresholds, the driver is considered to be driving with distraction in the frame image. The distraction confidence level of the frame image is set numerically to lay the foundation for subsequent determination of driver distraction and alarm judgment.

[0061] It should be noted that distraction confidence is a numerical value used to judge the degree of driver distraction. Generally, it can be represented by 0 and 1. 1 indicates that the angle of the driver's face or gaze exceeds its corresponding baseline angle threshold, which can determine that the driver in this frame image is suspected of being distracted, while 0 indicates that the angle of the driver's face or gaze does not exceed its corresponding baseline angle threshold.

[0062] In one specific embodiment of this application, the gaze orientation angle and the face orientation angle are compared with their corresponding reference angle thresholds, including: comparing the gaze leftward angle in the gaze orientation angle with the left reference angle threshold, or comparing the gaze rightward angle in the gaze orientation angle with the right reference angle threshold; comparing the face upward angle in the face orientation angle with the upward reference angle threshold, or comparing the face downward angle in the face orientation angle with the downward reference angle threshold, wherein the reference angle thresholds include the left reference angle threshold, the right reference angle threshold, the upward reference angle threshold, and the downward reference angle threshold.

[0063] In this embodiment, corresponding reference angle thresholds are set for the four dimensions of up, down, left, and right, making the alarm judgment of distraction more accurate and comprehensive.

[0064] exist Figure 1 In the specific implementation shown, a distracted driving alarm method further includes:

[0065] Step S105: Sort the distraction confidence scores of images within a predetermined time period to obtain a distraction time series, and calculate the average distraction score of the distraction time series. If the average distraction score is greater than the distraction threshold, then issue a distraction alarm to the driver.

[0066] In this embodiment, each current frame is sorted according to the order in which they arrive, resulting in a corresponding distraction time sequence. If the duration of the distraction time sequence, i.e., a predetermined time period, is greater than the time threshold corresponding to its upper, lower, left, and right sides, then the average distraction value of the distraction time sequence is definitely greater than the distraction threshold, and the current driver's state is definitely distracted. The distraction time sequence within the predetermined time period makes the judgment of distraction behavior more accurate.

[0067] It should be noted that the duration of a driver's distraction is generally around 1-2 seconds, so the predetermined time period is set at 1-2 seconds.

[0068] In a specific example of this application, taking a predetermined time period of 2 seconds and acquiring 10 frames of images per second as an example, within the 2-second timeframe, the driver's state of distraction is determined by whether the average distraction confidence score of the 20 frames in the distraction time sequence is greater than its corresponding distraction threshold. This enables the solution in this application to effectively and promptly safeguard the driver's personal safety.

[0069] In one specific embodiment of this application, the distraction confidence of images within a predetermined time period is sorted to obtain a distraction time sequence. The process further includes: during the predetermined time period, if a left turn signal or a right turn signal of a driving vehicle is received while sorting the current frame image, the sorting process of the current frame image is terminated.

[0070] In this embodiment, when the driver turns on the left or right turn signal, the process will last for a period of time. However, this scenario is special. In order to improve the accuracy of the alarm, the driver should not be given a distraction alarm. Therefore, this scenario needs to be filtered out.

[0071] This application acquires the driver's pupil center point position, nose tip position, gaze direction, and face orientation from a predetermined number of historical image frames using a perception model. Coordinate systems are established for the driver's face and gaze, respectively. Then, using calibrated reference orientations for the face and gaze, the driver's face orientation angle and gaze orientation angle are determined. The face orientation angle is used to determine whether the driver is distracted vertically, and the gaze orientation angle is used to determine whether the driver is distracted horizontally, laying the foundation for subsequent calculations. The three-dimensional coordinate system is mapped to its corresponding two-dimensional coordinate system to obtain the corresponding gaze orientation angle and face orientation angle. By comparing the gaze orientation angle and face orientation angle with their respective reference angle thresholds, a distraction confidence level is set for each frame of the current image. The images with set values ​​are then sorted. When the average distraction level in the sorted distraction time sequence exceeds the distraction threshold, a distraction alarm is issued to indicate the driver's current state. This scheme improves the accuracy of the judgment and ensures the real-time nature of the alarm.

[0072] Figure 2 This paper illustrates a specific embodiment of a distraction driving alarm device according to this application.

[0073] exist Figure 2 In the specific embodiment shown, a distracted driving alarm device mainly includes:

[0074] The orientation acquisition module 201 is used to perform perception detection on the driver in a predetermined number of historical images in the camera coordinate system corresponding to the camera parameters through a perception model, so as to obtain the driver's gaze orientation and face orientation in the historical images.

[0075] Orientation calibration module 202 is used to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation respectively using the gaze orientation and face orientation of a predetermined number of historical images;

[0076] Orientation angle output module 203 is used to output the driver's gaze orientation angle and face orientation angle based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, and the perception model detects the current image through perception.

[0077] Angle comparison module 204 is used to compare the gaze angle and face angle with their corresponding reference angle thresholds. If at least one of the gaze angle and face angle is greater than its corresponding reference angle threshold, the distraction confidence of the current image is set numerically.

[0078] The alarm judgment module 205 is used to sort the distraction confidence of images within a predetermined time period to obtain a distraction time sequence and calculate the average distraction value of the distraction time sequence. If the average distraction value is greater than the distraction threshold, a distraction alarm is triggered for the driver.

[0079] In this embodiment, the orientation acquisition module 201, under the camera coordinate system composed of camera parameters, directly acquires the perception information of a predetermined number of frames of images, which facilitates the foundation for subsequent coordinate system calibration; the orientation calibration module 202, through the pupil center point position, gaze orientation, nose tip point position, and face orientation of a predetermined number of historical images, calibrates the three-dimensional gaze reference orientation and three-dimensional face reference orientation, which lays the foundation for subsequent calculations and can be used to determine the driver's left-right and up-down distractions; the orientation angle output module 203 maps the orientation radian in the three-dimensional coordinate system to its corresponding... The orientation angle in a two-dimensional coordinate system makes distraction detection simpler and more convenient. The angle comparison module 204 determines that the driver is distracted in the image frame if either the gaze angle or the face angle exceeds its corresponding baseline threshold. It then sets a numerical value for the distraction confidence score of that frame, laying the foundation for subsequent determination of driver distraction and triggering an alarm. The alarm judgment module 205 determines that if the duration of the distraction time sequence (i.e., a predetermined time period) exceeds the corresponding time thresholds in its upper, lower, left, and right directions, the average distraction value of the distraction time sequence must be greater than the distraction threshold, indicating that the driver is currently distracted. The distraction time sequence within the predetermined time period makes the judgment of distraction behavior more accurate.

[0080] In one specific embodiment of this application, the three-dimensional gaze reference orientation and three-dimensional face reference orientation are calibrated using the gaze orientation and face orientation of a predetermined number of historical images. The process further includes: transforming the pupil center point coordinates and the nose tip point coordinates in the camera coordinate system to the pupil center point coordinates and the nose tip point coordinates in the image coordinate system, respectively. Using the pupil center point coordinates and the nose tip point coordinates as their corresponding origins, a gaze orientation coordinate system and a face orientation coordinate system based on the two-dimensional vertical plane of the image are established, respectively, to obtain the gaze orientation to be calibrated in the gaze orientation coordinate system and the face orientation to be calibrated in the face orientation coordinate system; calculating the gaze orientation offset between the gaze orientation and the gaze orientation to be calibrated, and the face orientation offset between the face orientation and the face orientation to be calibrated; weighting the gaze offset and face orientation offset of the predetermined number of images to obtain the corresponding average gaze orientation and average face orientation; and using the average gaze orientation and average face orientation to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation in the camera coordinate system.

[0081] In this embodiment, the face orientation and gaze orientation in the camera coordinate system are first determined with the radian values ​​of their three coordinate dimensions. These values ​​are then compared with the radian values ​​of the face orientation and gaze orientation in the gaze orientation coordinate system to obtain the offset of the face orientation and the offset of the gaze orientation. By calculating the average of the offsets, the calibrated reference orientation is obtained. This method is simple and efficient, making the calibration process of the driver's face and gaze more convenient and faster.

[0082] In one specific embodiment of this application, based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle, including: in the coordinate system corresponding to the three-dimensional gaze reference orientation, mapping the driver's gaze orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its horizontal plane to obtain the gaze orientation angle, wherein the gaze orientation arc includes a gaze to the left or a gaze to the right arc, and the gaze orientation angle includes a gaze to the left or a gaze to the right angle; and in the coordinate system corresponding to the three-dimensional face reference orientation, mapping the driver's face orientation arc detected by the perception model in the current image to a two-dimensional coordinate system on its vertical plane to obtain the face orientation angle, wherein the face orientation arc includes a face-up arc or a face-down arc, and the face orientation angle includes a face-up angle or a face-down angle.

[0083] In this embodiment, in the coordinate system corresponding to the three-dimensional gaze reference orientation, since it is necessary to determine whether the driver is distracted to the left or right, the gaze orientation arc is either a leftward or rightward arc. This requires mapping the gaze orientation arc onto a horizontal plane to obtain an angle, which could be a leftward or rightward angle. Similarly, in the coordinate system corresponding to the three-dimensional face reference orientation, since it is necessary to determine whether the driver is distracted vertically, the face orientation arc is either an upward or downward arc. This requires mapping the face orientation arc onto a vertical plane to obtain an angle, which could be an upward or downward angle. By mapping the gaze orientation arc and face orientation arc to their corresponding two-dimensional coordinate systems to obtain their respective angle values, the determination of distraction becomes more accurate.

[0084] In one specific embodiment of this application, the gaze orientation angle and the face orientation angle are compared with their corresponding reference angle thresholds, including: comparing the gaze leftward angle in the gaze orientation angle with the left reference angle threshold, or comparing the gaze rightward angle in the gaze orientation angle with the right reference angle threshold; comparing the face upward angle in the face orientation angle with the upward reference angle threshold, or comparing the face downward angle in the face orientation angle with the downward reference angle threshold, wherein the reference angle thresholds include the left reference angle threshold, the right reference angle threshold, the upward reference angle threshold, and the downward reference angle threshold.

[0085] In this embodiment, corresponding reference angle thresholds are set for the four dimensions of up, down, left, and right, making the alarm judgment of distraction more accurate and comprehensive.

[0086] In one specific embodiment of this application, the facial key point detection results of the driver output by the perception model are used to determine whether the driver's face in the image is occluded. When the facial key point detection results show that the number of facial key points is less than the number threshold, or that eye key points are missing, or that nose tip key points are missing, the image frame is filtered out.

[0087] In this embodiment, the perception model obtains the position of the driver's pupil center point, the position of the nose tip point, the direction of the gaze, and the direction of the face in the image based on the position coordinates of the detected facial key points. Therefore, the perception model also outputs the facial key point detection results, including which parts of the key points are missing and the number of key points. If the nose tip key point is missing, or the eye key point is missing, or the number of key points is less than the number threshold, it is considered that the driver's face in the image is occluded, indicating that the frame of the image needs to be discarded to prevent interference to subsequent steps due to incomplete facial key points, and also to reduce the false alarm rate of this application.

[0088] In one specific embodiment of this application, the distraction confidence of images within a predetermined time period is sorted to obtain a distraction time sequence. The process further includes: during the predetermined time period, if a left turn signal or a right turn signal of a driving vehicle is received while sorting the current frame image, the sorting process of the current frame image is terminated.

[0089] In this embodiment, when the driver turns on the left or right turn signal, the process will last for a period of time. However, this scenario is special. In order to improve the accuracy of the alarm, the driver should not be given a distraction alarm. Therefore, this scenario needs to be filtered out.

[0090] The distraction driving alarm device provided in this application can be used to execute the distraction driving alarm method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0091] In one specific embodiment of this application, the functional modules of the distraction driving alarm device proposed in this application can be directly in hardware, in software modules executed by a processor, or in a combination of both.

[0092] Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in this art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.

[0093] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0094] In another specific embodiment of this application, a vehicle is provided, wherein the vehicle includes the distraction driving alarm device of any embodiment. Optionally, the vehicle includes a processor and a memory, coupled together, and the vehicle is used to implement the appendix to this specification. Figure 1 Distraction driving alarm method in any of the embodiments shown.

[0095] In another specific embodiment of this application, a computer-readable storage medium is provided, which stores computer instructions that, when executed, cause a computer to perform the distracted driving alarm method in any embodiment.

[0096] Figure 3A specific embodiment of a computer device according to this application is shown.

[0097] exist Figure 3 In the specific embodiments shown, a computer device includes at least one processor, the processor and a memory coupled together, the memory storing computer instructions, which, when executed by the processor, implement the distracted driving alarm method in any embodiment.

[0098] In another specific embodiment of this application, a computer program product includes computer instructions, wherein when executed, the computer instructions implement the distracted driving alarm method as in any embodiment.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for alarming distracted driving, characterized in that, The distraction driving alarm method is applied to a distraction driving alarm device, including: The perceptual model performs perceptual detection on a predetermined number of historical images in the camera coordinate system corresponding to the camera parameters, obtaining the driver's gaze orientation and face orientation in the historical images. Specifically, the perceptual model detects the position coordinates of various facial key points in the image to obtain the position coordinates of the nose tip and the two pupil points. It also calculates the position coordinates of the pupil center point located at the center of the line connecting the two pupil points. Furthermore, the perceptual model detects the face orientation based on the nose tip and nostril orientation, and the gaze orientation based on the pupil center point, using the position coordinates of the two pupil points and the gaze direction. Using the gaze orientation and face orientation of the predetermined number of historical images, the three-dimensional gaze reference orientation and the three-dimensional face reference orientation are respectively calibrated; Based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle. The gaze orientation angle and the face orientation angle are compared with their corresponding reference angle thresholds. If at least one of the gaze orientation angle and the face orientation angle is greater than its corresponding reference angle threshold, the distraction confidence of the current image is set numerically. The distraction confidence scores of images within a predetermined time period are sorted to obtain a distraction time series, and the average distraction score of the distraction time series is calculated. If the average distraction score is greater than a distraction threshold, a distraction alarm is triggered for the driver.

2. The distracted driving alarm method as described in claim 1, characterized in that, The step of using the gaze orientation and face orientation of the predetermined number of historical images to respectively calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation further includes: By transforming the pupil center point coordinates and the face nose tip point coordinates in the camera coordinate system to the pupil center point coordinates and the face nose tip point coordinates in the image coordinate system, and taking the pupil center point coordinates and the face nose tip point coordinates as their corresponding origins, a gaze orientation coordinate system and a face orientation coordinate system based on the two-dimensional vertical plane of the image are established respectively, and the gaze orientation to be marked in the gaze orientation coordinate system and the face orientation to be marked in the face orientation coordinate system are obtained. The offset between the gaze direction and the gaze direction to be calibrated, and the offset between the face direction and the face direction to be calibrated are calculated. The gaze orientation offset and face orientation offset of the predetermined number of frames of the image are weighted and calculated to obtain the corresponding average gaze orientation and average face orientation, respectively. The three-dimensional gaze reference orientation and the three-dimensional face reference orientation in the camera coordinate system are obtained by calibration using the average gaze orientation and the average face orientation.

3. The distracted driving alarm method as described in claim 2, characterized in that, Based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, the perception model detects the current image and outputs the driver's gaze orientation angle and face orientation angle, including: In the coordinate system corresponding to the three-dimensional gaze reference orientation, the gaze orientation arc of the driver in the current image, perceived and detected by the perception model, is mapped to a two-dimensional coordinate system on its horizontal plane to obtain the gaze orientation angle. The gaze orientation arc includes a leftward gaze arc or a rightward gaze arc, and the gaze orientation angle includes a leftward gaze angle or a rightward gaze angle. In the coordinate system corresponding to the three-dimensional face reference orientation, the facial orientation arc of the driver in the current image perceived and detected by the perception model is mapped to the two-dimensional coordinate system on its vertical plane to obtain the facial orientation angle. The facial orientation arc includes an upward facial arc or a downward facial arc, and the facial orientation angle includes an upward facial angle or a downward facial angle.

4. The distracted driving alarm method as described in claim 3, characterized in that, The step of comparing the gaze angle and the face angle with their corresponding reference angle thresholds includes: Compare the leftward angle of the line of sight in the line of sight orientation angle with the leftward reference angle threshold, or compare the rightward angle of the line of sight orientation angle with the rightward reference angle threshold; The upward angle of the face in the face orientation angle is compared with the upward reference angle threshold, or the downward angle of the face in the face orientation angle is compared with the downward reference angle threshold, wherein the reference angle threshold includes the left reference angle threshold, the right reference angle threshold, the upward reference angle threshold and the downward reference angle threshold.

5. The distracted driving alarm method as described in claim 1, characterized in that, Also includes: Using the facial key point detection results of the driver output by the perception model, it is determined whether the driver's face is occluded in the predetermined number of historical images. When the facial key point detection results show that the number of facial key points is less than the number threshold, or there is a lack of eye key points, or a lack of nose tip key points, the predetermined number of historical images are filtered out.

6. The distracted driving alarm method as described in claim 1, characterized in that, The step of sorting the distraction confidence scores of images within a predetermined time period to obtain a distraction time series further includes: During the predetermined time period, if a left turn signal or a right turn signal from a driving vehicle is received while sorting the current frame image, the sorting process of the current frame image is terminated.

7. A distracted driving alarm device, characterized in that, include: The orientation perception module is used to perform perception detection on the driver in a predetermined number of historical images in the camera coordinate system corresponding to the camera parameters using a perception model, and obtain the driver's gaze orientation and face orientation in the historical images. Specifically, the perception model obtains the position coordinates of the nose tip and the two pupil points by detecting the position coordinates of various facial key points of the driver in the image, and calculates the position coordinates of the pupil center point located at the center of the line connecting the two pupil points. The perception model also detects the face orientation based on the nose tip and nostril orientation; and detects the gaze orientation based on the pupil center point based on the position coordinates of the two pupil points and the gaze direction. Orientation calibration module, which is used to calibrate the three-dimensional gaze reference orientation and the three-dimensional face reference orientation respectively using the gaze orientation and the face orientation of the predetermined number of historical images; An orientation angle output module is used to output the driver's gaze orientation angle and face orientation angle based on the three-dimensional gaze reference orientation and the three-dimensional face reference orientation, and the perception model detects the current image through perception. An angle comparison module is used to compare the gaze orientation angle and the face orientation angle with their corresponding reference angle thresholds. If at least one of the gaze orientation angle and the face orientation angle is greater than its corresponding reference angle threshold, the distraction confidence of the current image is numerically set. The alarm judgment module is used to sort the distraction confidence of images within a predetermined time period to obtain a distraction time sequence, and calculate the average distraction value of the distraction time sequence. If the average distraction value is greater than the distraction threshold, a distraction alarm is triggered for the driver.

8. A vehicle, characterized in that, The vehicle includes the distraction warning device as described in claim 7.

9. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instruction is executed, it causes the computer to perform the distraction driving alarm method as described in any one of claims 1-6.

10. A computer device comprising at least one processor, said processor coupled to a memory storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the distraction driving alarm method as described in any one of claims 1-6.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed, they implement the distraction driving alarm method as described in any one of claims 1-6.

Images