Automatic driving takeover determination method and system therefor

Abstract

To provide an automatic driving takeover determination method and a system therefor.SOLUTION: An automatic driving takeover determination method includes a driver correction step, a detection module update step, an image acquisition step, a face portion detection step, a driver available determination step, and an automatic driving takeover determination step. The driver correction step includes: acquiring a plurality of correction images of a driver by at least one camera before a vehicle enters an automatic driving mode; and generating a plurality of driver correction parameters which are a plurality of mutual positional parameters of the driver in a cab of the vehicle, by the correction module based on the correction images. The detection module update step includes updating a detection module based on the driver correction parameters. The automatic driving takeover determination step includes determining whether an automatic driving takeover condition is satisfied based on an available determination result. This will improve the safety of the automatic driving system.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an automatic driving handover determination method and its system, and particularly to an automatic driving handover determination method and its system based on a driver image.

Background Art

[0002] While the market for autonomous vehicles is expanding daily, methods to improve safety to reduce accidents have become a top priority in the development of autonomous vehicles. In SAE (Society of Automotive Engineers) Level 3 autonomous vehicles, the driver does not need to hold the steering wheel under certain conditions, but the driver needs to have the ability to take over the driving task. Therefore, the focus of the development of Level 3 autonomous vehicles is not on detecting the driver's concentration like in Level 0 - Level 2 autonomous vehicles, but on detecting whether the driver is still conscious and can take over the driving task at any time.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Based on the above, how to develop an automatic driving handover determination method and its system that can appropriately and accurately detect whether the driver of a Level 3 autonomous vehicle has the ability to take over the driving task has become an urgent problem to be solved in the autonomous vehicle market.

Means for Solving the Problems

[0004] Before entering the automatic driving mode of the vehicle, the present invention acquires a plurality of corrected images of the driver by at least one camera, generates a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver in the driver's cab of the vehicle, by a correction module based on the corrected images, and updates a detection module based on the driver correction parameters, so as to detect whether the driver has the ability to take over the driving task, and provides an automatic driving handover determination method and its system for improving the safety of the automatic driving system.

[0005] According to an embodiment of the present invention, there is provided an automatic driving handover determination method for determining whether a driver sitting in the driver's seat of a vehicle satisfies the automatic driving handover condition in the automatic driving mode, including: a driver correction step including acquiring a plurality of corrected images of the driver by at least one camera before entering the automatic driving mode of the vehicle, and generating, by a correction module based on the corrected images, a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver in the driver's cab of the vehicle; a detection module update step including updating the detection module based on the driver correction parameters; an image acquisition step including acquiring a plurality of driver images of the driver by the camera in a first detection time interval in the automatic driving mode; a face detection step including detecting, by the detection module based on the driver images, whether the driver satisfies at least one face feature condition, which is at least one face detection result; a driver availability determination step including determining, by an availability determination module based on the face detection result, whether the driver satisfies an availability condition, which is an availability determination result; and an automatic driving handover determination step including determining whether the automatic driving handover condition is satisfied based on the availability determination result.

[0006] According to the embodiments of the foregoing embodiments, the automatic driving handover determination method may further include a vehicle body signal acquisition step of acquiring, by at least one vehicle body sensor, a plurality of vehicle body signals in the driver's seat including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals in a first detection time interval in the automatic driving mode, a driver presence determination step of determining, by a presence determination module, whether a driver satisfies a presence condition which is a presence determination result based on at least one of a driver image and the vehicle body signals, and a warning step. The automatic driving handover determination step includes determining whether an automatic driving handover condition is satisfied based on an availability determination result and a presence determination result. When it is determined in the automatic driving handover determination step that the automatic driving handover condition is not satisfied, the warning step includes generating at least one of a visual warning, an auditory warning, and a vibration warning to warn the driver.

[0007] According to the embodiments of the foregoing embodiments, the automatic driving handover determination method may further include a biological signal acquisition step of acquiring, by at least one biological sensor, a plurality of biological signals of the driver including at least one of a plurality of heartbeat signals and a plurality of respiration signals in a first detection time interval in the automatic driving mode, a human body posture detection step of detecting, by a detection module, whether the driver satisfies a non-sleep posture feature condition which is a non-sleep posture detection result based on the driver image, and a heartbeat detection step of detecting, by the detection module, whether the driver satisfies a heartbeat feature condition which is a heartbeat detection result based on the biological signals. The driver availability determination step includes determining, by an availability determination module, whether the driver satisfies an availability condition which is an availability determination result based on a face detection result, a non-sleep posture detection result, and a heartbeat detection result.

[0008] According to the examples of the foregoing embodiments, the at least one facial feature condition may be plural. When the driver satisfies at least two of the facial feature conditions, or satisfies the non-sleep posture feature condition and the heart rate feature condition, and the sum of the weights of the detection results is greater than the sum threshold of the weights, the availability determination module determines that the driver satisfies the availability condition. The weight coefficient of each facial detection result is greater than the weight coefficient of the non-sleep posture detection result and the weight coefficient of the heart rate detection result.

[0009] According to the examples of the foregoing embodiments, the driver correction parameter may include the lateral distance and the longitudinal distance between the driver's line-of-sight range related data and at least one target among the rearview mirror, the left rearview mirror, the right rearview mirror, the car radio, the steering wheel, the instrument panel, and the passenger seat glove box of the driver and the vehicle.

[0010] According to the examples of the foregoing embodiments, the automatic driving handover determination method may further include an initial correction step including: the initial driver in the driver's seat performs at least one action of directly looking at the at least one target based on the initial correction operation instruction of the correction module, obtains at least one initial correction image of the at least one action by a camera, and generates a plurality of initial correction parameters including the viewing angle of the initial driver with respect to the target by the correction module. The driver correction step includes generating driver correction parameters including the viewing angle of the driver with respect to the target by the correction module based on the initial correction parameters and the correction image.

[0011] According to the examples of the foregoing embodiments, the detection module may include an eye-opening detection unit, a viewing angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The face detection step may include an eye-opening detection step of detecting, by the eye-opening detection unit of the detection module based on the driver image, whether the driver satisfies the eye-opening feature condition that is the eye-opening detection result, a viewing angle detection step of detecting, by the viewing angle detection unit of the detection module based on the driver image, whether the driver satisfies the viewing angle feature condition that is the viewing angle detection result, and a head shake detection step of detecting, by the head shake detection unit of the detection module based on the driver image, whether the driver satisfies the head shake feature condition that is the head shake detection result. The at least one face detection result is at least three, and the face detection results include an eye-opening detection result, a viewing angle detection result, and a head shake detection result.

[0012] According to the examples of the foregoing embodiments, the eye-opening detection step may include establishing a first vertical distance by the first upper eye point and the first lower eye point of each eye in each driver image based on the Euclidean distance algorithm, establishing a second vertical distance by the second upper eye point and the second lower eye point, and defining the eye state feature value as the maximum value of the two first vertical distances and the two second vertical distances of the two eyes. If the eye state feature value is greater than the eye-opening threshold, it is defined as satisfying the eye-opening state. The eye-opening threshold is determined by the eye-opening detection unit of the updated detection module and is within the threshold setting range. Whether the eye-opening state is satisfied is used to detect whether the driver satisfies the eye-opening feature condition in the first detection time interval.

[0013] According to the examples of the foregoing embodiments, the viewing angle detection step may include identifying the pupils in each driver image and detecting whether the driver satisfies the viewing angle specified range through coordinate transformation. Whether the driver satisfies the viewing angle specified range is used to detect whether the driver satisfies the viewing angle characteristic condition in the first detection time interval. The head shake detection step may include identifying the tip of the nose in each driver image and detecting whether the driver satisfies the head shake specified range through coordinate transformation. Whether the driver satisfies the head shake specified range is used to detect whether the driver satisfies the head shake characteristic condition in the first detection time interval.

[0014] According to the examples of the foregoing embodiments, the first detection time interval may include a plurality of second detection time intervals, each second detection time interval may include a plurality of third detection time intervals. The first detection time interval may be between 1 second and 30 seconds, each second detection time interval may be between 1 second and 5 seconds, and each third detection time interval may be between 0.5 seconds and 1 second. Each third detection time interval includes a plurality of detection time points. The eye-opening detection step detects whether the driver satisfies the eye-opening state at one detection time point. When more than half of the detection time points in one third detection time interval satisfy the eye-opening state, an eye-opening signal corresponding to the third detection time interval is generated. When there is an eye-opening signal corresponding to each third detection time interval in one second detection time interval, it is detected that the driver satisfies the eye-opening characteristic condition in the first detection time interval.

[0015] According to the examples of the foregoing embodiments, when there is a viewing angle signal corresponding to the viewing angle specified range for each third detection time interval in one second detection time interval, it is detected that the driver satisfies the viewing angle characteristic condition in the first detection time interval. When there is a head shake signal corresponding to the head shake specified range for each third detection time interval in one second detection time interval, it is detected that the driver satisfies the head shake characteristic condition in the first detection time interval.

[0016] According to another embodiment of the present invention, there is provided an automatic driving handover determination system provided in a vehicle, including an automatic driving unit for executing an automatic driving mode of the vehicle, at least one camera for acquiring a plurality of driver images of a driver in the driver's seat inside the vehicle, a processing unit including a correction module, a detection module, and an availability determination module, and an in-vehicle communication network for communicatively connecting the automatic driving unit, the camera, and the processing unit. The processing unit acquires a plurality of corrected images of the driver by the camera before entering the automatic driving mode of the vehicle, generates, by the correction module based on the corrected images, a plurality of driver correction parameters which are a plurality of relative position parameters of the driver in the driver's cab of the vehicle, updates the detection module based on the driver correction parameters, acquires a plurality of driver images of the driver by the camera in a first detection time period in the automatic driving mode, detects, by the detection module based on the driver images, whether the driver satisfies at least one facial feature condition which is at least one facial detection result, determines, by the availability determination module based on the facial detection result, whether the driver satisfies an availability condition which is an availability determination result, and is used to determine whether the automatic driving handover condition is satisfied based on the availability determination result.

[0017] According to the embodiments of the foregoing embodiments, the automatic driving handover determination system may further include at least one vehicle body sensor, at least one biological sensor, and a warning unit. The processing unit may further include a presence determination module. In a first detection time interval in the automatic driving mode, the processing unit obtains a plurality of biological signals of the driver including at least one of a plurality of heartbeat signals and a plurality of respiration signals by the biological sensor, detects by the detection module based on the driver image whether the driver satisfies the non-sleep posture feature condition which is a non-sleep posture detection result, detects by the detection module based on the biological signals whether the driver satisfies the heartbeat feature condition which is a heartbeat detection result, determines by the availability determination module based on at least one facial detection result, non-sleep posture detection result, and heartbeat detection result whether the driver satisfies the availability condition which is an availability determination result, obtains in a first detection time interval in the automatic driving mode a plurality of vehicle body signals in the driver's seat including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals by the vehicle body sensor, determines by the presence determination module based on at least one of the driver image and the vehicle body signals whether the driver satisfies the presence condition which is a presence determination result, determines whether the automatic driving handover condition is satisfied based on the availability determination result and the presence determination result, and when it is determined that the automatic driving handover condition is not satisfied, is used for the warning unit to generate at least one of a visual warning, an auditory warning, and a vibration warning to warn the driver.

[0018] According to the embodiments of the foregoing embodiments, the detection module may include an eye-opening detection unit, a viewing angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The processing unit is used to detect, based on the driver image, whether the driver meets the eye-opening feature condition that is the eye-opening detection result by the eye-opening detection unit of the detection module, whether the driver meets the viewing angle feature condition that is the viewing angle detection result by the viewing angle detection unit of the detection module, and whether the driver meets the head shake feature condition that is the head shake detection result by the head shake detection unit of the detection module. The at least one facial detection result is at least three, and the facial detection results include an eye-opening detection result, a viewing angle detection result, and a head shake detection result.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0020] Hereinafter, a plurality of embodiments of the present invention will be described with reference to the drawings. For the sake of clarity, numerous practical details will be described together in the following description. However, it should be understood that these practical details are not for limiting the present invention. That is, in the embodiments of the present invention, these practical details are not necessary. Also, for the purpose of simplifying the drawings, some conventional structures and components are shown simply and schematically in the drawings, and overlapping components may be denoted by the same number or similar numbers.

[0021] Also, terms such as first, second, etc. are merely for explaining different components and do not limit the components themselves. Thus, the first component may be read as the second component. The combinations of components in this specification are not well-known, conventional, or known combinations in this field, and it is not possible to determine whether the combination relationship can be easily achieved by those skilled in the art depending on whether the components themselves are known.

[0022] FIG. 1 shows a flowchart of an automatic driving handover determination method 100 according to a first embodiment of the present invention, FIG. 2 shows a block diagram of an automatic driving handover determination system 200 according to a second embodiment of the present invention, and FIG. 3 shows a schematic diagram of a vehicle 300 in which the automatic driving handover determination system 200 shown in FIG. 2 is provided. With reference to FIGS. 1 to 3, the automatic driving handover determination method 100 of the first embodiment and the automatic driving handover determination system 200 of the second embodiment will be described together. The automatic driving handover determination method 100 is used to determine whether a driver 400 sitting in the driver's seat 304 in the vehicle 300 satisfies the automatic driving handover conditions (i.e., whether the automatic driving conditions are satisfied) in the automatic driving mode. The vehicle 300 may be a Level 3 autonomous vehicle, and the scenarios of automatic driving handover include the driver 400 starting automatic driving and taking over the vehicle 300, and the automatic driving transferring the driving right to the driver 400. The automatic driving handover determination method 100 includes a driver correction step 112, a detection module update step 114, an image acquisition step 130, a face detection step 134, a driver availability determination step 170, and an automatic driving handover determination step 180.

[0023] Driver correction step 112 includes obtaining a plurality of corrected images of driver 400 by at least one camera 230 before entering the automatic driving mode of vehicle 300, and generating, by correction module (i.e., correction program) 271 based on the corrected images, a plurality of driver correction parameters that are a plurality of relative position parameters of driver 400 in the driver's cab 303 of vehicle 300. Detection module update step 114 includes updating detection module 280 based on the driver correction parameters, so as to adapt to variables such as the height and seat distance of different drivers 400 by means of a self-correction mechanism, and automatically adjusting a reasonable forward field of view.

[0024] Image acquisition step 130 includes obtaining, by camera 230, a plurality of driver images (image frames) that may be visible light or infrared light images of driver 400 in the first detection time interval in the automatic driving mode. Facial detection step 134 includes detecting, by detection module 280 based on the driver images, whether driver 400 satisfies at least one facial feature condition that is at least one facial detection result.

[0025] Driver availability determination step 170 includes determining, by availability determination module 277 based on the facial detection result, whether driver 400 satisfies an availability condition that is an availability determination result. Automatic driving handover determination step 180 includes determining whether the automatic driving handover condition is satisfied based on the availability determination result. Thereby, it accurately detects whether driver 400 has the ability to take over the driving task, and contributes to improving the safety of the automatic driving system.

[0026] Please refer to FIGS. 2 and 3. The automatic driving handover determination system 200 of the second embodiment is provided in the vehicle 300 and includes an automatic driving unit 240, at least one camera 230, a processing unit 270, and an in-vehicle communication network 203. The automatic driving unit 240 is used to execute the automatic driving mode of the vehicle 300. The camera 230 is used to acquire a plurality of driver images of the driver 400 sitting in the driver's seat 304 in the vehicle 300. The processing unit 270 includes a correction module 271, a detection module 280, and an availability determination module 277. The in-vehicle communication network 203 is used for communication connection of the automatic driving unit 240, the camera 230, and the processing unit 270.

[0027] Before the vehicle 300 enters the automatic driving mode, the processing unit 270 acquires a plurality of corrected images of the driver 400 by the camera 230, generates a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver 400 in the cab 303 of the vehicle 300, by the correction module 271 based on the corrected images, updates the detection module 280 based on the driver correction parameters, acquires a plurality of driver images of the driver 400 by the camera 230 in the first detection time interval in the automatic driving mode, detects by the detection module 280 whether the driver 400 satisfies at least one facial feature condition, which is at least one facial detection result, based on the driver images, determines by the availability determination module 277 whether the driver 400 satisfies an availability condition, which is an availability determination result, based on the facial detection result, and determines whether the automatic driving handover condition is satisfied based on the availability determination result. Thereby, it can be appropriately and accurately detected whether the driver 400 of the Level 3 autonomous vehicle has the ability to take over the driving task.

[0028] FIG. 4 shows another schematic view of the vehicle 300 provided with the automatic driving handover determination system 200 in FIG. 2. Refer to FIGS. 3 and 4. The driver correction parameter may include a lateral distance (e.g., lateral distances h1, h3, h5, h7) and a vertical distance (e.g., vertical distances m1, m2, m4) between the driver 400 and at least one target among the rearview mirror 313, the left rearview mirror 310, the right rearview mirror 317, the car radio 314, the steering wheel 311, the instrument panel 312, and the passenger glove box 315 of the vehicle 300 in the line-of-sight range related data of the driver 400. Thereby, it adapts to different drivers 400 by the self-correction mechanism and automatically adjusts a reasonable forward field of view. Specifically, the line-of-sight range of the driver 400 takes into account the head swing angle, that is, it is in a state where the presence or absence (rotation) of head swing is integrated into the line-of-sight range, which includes the line-of-sight movement range (owl type) in the head swing state and the line-of-sight movement range (lizard type) in the state without head swing.

[0029] Specifically, refer to FIGS. 1 to 4. The automatic driving handover determination method 100 may further include an initial correction step 110. The initial correction step 110 targets a vehicle 300 of an unknown vehicle type. The initial correction step 110 acquires an image of an initial driver (which may be the same as or different from the driver 400) by the camera 230, finds a region of interest (ROI) or a face position, and instructs the initial driver in the driver's seat 304 to perform at least one action of directly looking at the at least one target according to the initial correction operation instruction of the correction module 271, acquires at least one initial correction image of the at least one action by the camera 230, and generates a plurality of initial correction parameters by the correction module 271. The initial correction parameters include the viewing angle of the initial driver with respect to the target. The driver correction step 112 targets a vehicle 300 of a known vehicle type. The driver correction step 112 includes generating driver correction parameters by the correction module 271 based on the initial correction parameters and the correction images. The driver correction parameters include the viewing angle of the driver 400 with respect to the target. Thereby, in the detection module update step 114, the detection module 280 is updated based on the driver correction parameters of different drivers 400.

[0030] In the initial correction step 110, for example, refer to FIG. 3. If the upper endpoint of the handle 311 is set as the origin, the vertical distance m1 between the center point of the handle 311 and the origin, the vertical distance m2 between the instrument panel 312 and the origin, and the vertical distance m4 between the face of the driver 400 and the origin can be obtained. Here, the vertical distances m1, m2, and m4 may be 0.1 m, 0.1 m, and 0.45 m respectively. Refer to FIG. 4. If the left rear mirror 310 is set as the origin, the horizontal distance h1 between the driver 400 (or the handle 311, the instrument panel 312, the pedal) and the origin, the horizontal distance h3 between the car radio 314 (or the rear mirror 313) and the origin, the horizontal distance h5 between the passenger glove box 315 and the origin, and the horizontal distance h7 between the right rear mirror 317 and the origin can be obtained. Here, the horizontal distances h1, h3, h5, and h7 may be 0.75 m, 1.1 m, 1.45 m, and 2.1 m respectively. In the initial correction step 110, it can be seen that the viewing angle at which the initial driver directly views the rear mirror 313 is 37 degrees to the right and 24 degrees upward, the viewing angle at which the initial driver directly views the left rear mirror 310 is 60 degrees to the left and 6 degrees downward. The viewing angle at which the initial driver directly views the right rear mirror 317 is 71 degrees to the right and 6 degrees downward, and the viewing angle at which the initial driver directly views the car radio 314 is 37 degrees to the right and 18 degrees downward. The viewing angle at which the initial driver directly views the handle 311 is 29 degrees downward, the viewing angle at which the initial driver directly views the instrument panel 312 is 18 degrees downward, and the viewing angle at which the initial driver directly views the passenger glove box 315 is 55 degrees to the right and 33 degrees downward. The above-mentioned vertical distances, horizontal distances, and viewing angles can be derived from each other by trigonometric functions. Even when the driver 400 is different from the initial driver in the driver correction step 112, the driver correction parameters applied to the driver 400 can be generated based on the initial correction parameters and the correction image.

[0031] As a result, important positions, lines of sight, viewing angle ranges, etc. during the driving of different vehicle models and different drivers 400 are not fixed numerical values, and the factors affecting these numerical values are the height, body type, face shape, seat habits of the driver 400, or the structural design and mounting position of the vehicle 300, etc. In the driver correction step 112, distance information of important positions can be obtained from the structural parameters of the vehicle 300, and the viewing angles of each important position can be dynamically estimated based on the position of the driver 400. This contributes to the fact that the automatic driving handover determination system 200 provided in the Level 3 autonomous vehicle performs a wide range of detections covering scenarios where the head posture is not a frontal view. This improves the overall detection rate and is different from the detection method in which Level 0 to Level 2 autonomous vehicles regard it as abnormal if they cannot detect a frontal face.

[0032] Please refer to FIGS. 1 and 2. The detection module 280 may include an eye-opening detection unit 285, a viewing angle detection unit 286, and a head shake detection unit 287, all of which may be detection algorithms based on machine learning (e.g., image deep learning). The face detection step 134 may include an eye-opening detection step 135, a viewing angle detection step 136, and a head shake detection step 137, all of which include image preprocessing. The eye-opening detection step 135 includes detecting by the eye-opening detection unit 285 of the detection module 280 based on the driver image whether the driver 400 satisfies the eye-opening feature condition that is the eye-opening detection result. The viewing angle detection step 136 includes detecting by the viewing angle detection unit 286 of the detection module 280 based on the driver image whether the driver 400 satisfies the viewing angle feature condition that is the viewing angle detection result. For example, in the second detection time interval within the first detection time interval, the designated range formed by targets such as the rearview mirror 313, left rearview mirror 310, right rearview mirror 317, car radio 314, steering wheel 311, instrument panel 312, and passenger seat glove box 315 that the line of sight of the driver 400 hits may be considered to satisfy the viewing angle feature condition. The head shake detection step 137 includes detecting by the head shake detection unit 287 of the detection module 280 based on the driver image whether the driver 400 satisfies the head shake feature condition that is the head shake detection result. For example, in the second detection time interval within the first detection time interval, the designated range formed by targets such as the rearview mirror 313, left rearview mirror 310, right rearview mirror 317, car radio 314, steering wheel 311, instrument panel 312, and passenger seat glove box 315 that the head of the driver 400 shakes may be considered to satisfy the head shake feature condition. The at least one face detection result is at least three, and the face detection results include an eye-opening detection result, a viewing angle detection result, and a head shake detection result. This contributes to accurately detecting whether the driver 400 has the ability to take over the driving task.

[0033] FIG. 5 shows a schematic diagram of the eye state feature values in the eye opening detection step 135 in FIG. 1. Refer to FIGS. 1 and 5. The eye opening detection step 135 establishes a first vertical distance v1 by the first upper eye point p1 and the first lower eye point p2 of each eye in each driver image based on the Euclidean Distance algorithm, and establishes a second vertical distance v2 by the second upper eye point p3 and the second lower eye point p4, and may include defining the eye state feature value as the maximum value of the two first vertical distances v1 and the two second vertical distances v2 of both eyes. If the eye state feature value is greater than the eye opening threshold (threshold value), it is defined as satisfying the eye opening state. The eye opening threshold is determined by the eye opening detection unit 285 of the updated detection module 280 and is within the threshold setting range (for example, between 6 pixel values and 15 pixel values). Whether the eye opening state is satisfied or not is used to detect whether the driver 400 satisfies the eye opening feature condition in the first detection time interval. Thereby, it is determined based on the valid eye opening detection result whether the driver 400 has the ability to take over the driving task.

[0034] FIG. 6 shows a schematic diagram of the eye-opening detection step 135 in FIG. 1, and FIG. 7 shows another schematic diagram of the eye-opening detection step 135 in FIG. 1. Refer to FIGS. 6 and 7. The first detection time interval t1 may include a plurality of second detection time intervals t2, each second detection time interval t2 may include a plurality of third detection time intervals t3, the first detection time interval t1 may be between 1 second and 30 seconds, each second detection time interval t2 may be between 1 second and 5 seconds, and each third detection time interval t3 may be between 0.5 seconds and 1 second. Each third detection time interval t3 includes a plurality of detection time points, and the eye-opening detection step 135 detects whether the driver 400 satisfies the eye-opening state at one detection time point. When more than half of the detection time points in the third detection time interval t3 satisfy the eye-opening state, an eye-opening signal corresponding to the third detection time interval t3 is generated. For example, the first detection time interval t1 is 30 seconds, the second detection time interval t2 is 5 seconds, and the third detection time interval t3 is 1 second. As shown in FIG. 6, in the second detection time interval t2 (0 seconds to 5 seconds in FIG. 6), if an eye-opening signal corresponds to each third detection time interval t3, it is detected that the driver 400 satisfies the eye-opening characteristic condition in the first detection time interval t1 (even if no eye-opening signal corresponds from 5 seconds to 30 seconds in FIG. 6). Conversely, as shown in FIG. 7, in the second detection time interval t2, if an eye-opening signal does not correspond to at least one third detection time interval t3 (0 seconds to 1 second in FIG. 7), and no eye-opening signal corresponds from 5 seconds to 30 seconds in FIG. 7, it is detected that the driver 400 does not satisfy the eye-opening characteristic condition in the first detection time interval t1. Thereby, accurate eye-opening detection of the present invention is realized.

[0035] Please refer to FIG. 1. The viewing angle detection step 136 may include identifying the pupil images of each driver and detecting whether the driver 400 satisfies the viewing angle specified range by coordinate transformation (for example, Rodrigues’ Rotation Formula). Whether the driver satisfies the viewing angle specified range is used to detect whether the driver 400 satisfies the viewing angle characteristic condition in the first detection time interval t1. The head shake detection step 137 may include identifying the tip of the nose in each driver image and detecting whether the driver 400 satisfies the head shake specified range by coordinate transformation (for example, Rodrigues’ Rotation Formula). Whether the driver satisfies the head shake specified range is used to detect whether the driver 400 satisfies the head shake characteristic condition in the first detection time interval t1. Thus, it is determined based on the effective viewing angle detection result and the head shake detection result whether the driver 400 has the ability to take over the driving task.

[0036] If there is a corresponding viewing angle signal that satisfies the viewing angle specified range in each third detection time interval t3 in the second detection time interval t2, it is detected that the driver 400 satisfies the viewing angle characteristic condition in the first detection time interval t1. If there is a corresponding head shake signal that satisfies the head shake specified range in each third detection time interval t3 in the second detection time interval t2, it is detected that the driver 400 satisfies the head shake characteristic condition in the first detection time interval t1. Thus, the accurate viewing angle and head shake detection of the present invention are realized.

[0037] The automatic driving handover determination method 100 may further include a biological signal acquisition step 150, a human body posture detection step 132, and a heart rate detection step 152, and may also include a respiration detection step (not shown). The automatic driving handover determination system 200 may further include at least one biological sensor 250 (for example, a bio-radar). The biological signal acquisition step 150 includes acquiring, by the biological sensor 250, a plurality of physiological signals of the driver 400 including a plurality of heart rate signals and / or a plurality of respiration signals in a first detection time interval t1 in the automatic driving mode. The human body posture detection step 132 includes detecting, by the human body posture detection unit 282 of the detection module 280 based on the driver image, whether the driver 400 satisfies the non-sleep posture feature condition which is a non-sleep posture detection result, by integrating the facial and body images of the driver 400 using a deep learning algorithm and extracting posture feature points. The heart rate detection step 152 includes detecting, by the detection module 280 based on the biological signals, whether the driver 400 satisfies the heart rate feature condition which is a heart rate detection result. The driver availability determination step 170 includes determining, by the availability determination module 277 based on the facial detection result, the non-sleep posture detection result, and the heart rate detection result, whether the driver 400 satisfies the availability condition which is an availability determination result. Thereby, with the assistance of the biological sensor 250 such as a bio-radar, the problem that biological signs cannot be detected only by the camera 230 (for example, the heart rate and respiration images are restricted by the light source, occlusion, and the movement of the driver 400) is solved, the accuracy of driver availability is improved, and the misjudgment caused by the image losing the features of the driver 400 is avoided.

[0038] There may be a plurality of facial feature conditions. When the driver 400 satisfies at least two of the facial feature conditions, and / or satisfies at least one of the non-sleep posture feature condition and the heart rate feature condition, and satisfies the non-sleep posture feature condition and the heart rate feature condition, and the total weight of the detection results is greater than the total weight threshold, the available determination module 277 determines that the driver 400 satisfies the available conditions. Also, the weight coefficient of each facial detection result is greater than the weight coefficient of the non-sleep posture detection result and the weight coefficient of the heart rate detection result. Thereby, in the detection of availability, the detection accuracy of the image and the biological signal is the core of the whole. However, in actual applications, external light source interference, environmental changes, the appearance of the driver 400, etc. affect the detection accuracy. The automatic driving handover determination system 200 according to the present invention designs a feature classifier, and the weight of the feature classifier complements the image and the biological signal by assignment, and avoids the loss of the available detection function when the discrimination is insufficient due to excessive dependence on a single feature.

[0039] For example, refer to Table 1 below. When the driver 400 satisfies two of the facial feature conditions (the open-eye state ≥ Sth seconds, the time when the line of sight hits within the specified range ≥ Sth seconds) and one of the biological feature conditions (the time when the heart rate is higher than 50 beats per minute ≥ Sth seconds), if the total weight of the detection results calculated in this way is greater than the total weight threshold, the available determination module 277 determines that the driver 400 satisfies the available conditions. Here, the aforementioned Sth seconds is the second detection time interval t2, and the Sth seconds may be between 1 second and 5 seconds.

Table 1

[0040] Also, please refer to Table 2 below. When the driver 400 does not meet the facial feature conditions and meets two of the biometric feature conditions (non-sleep state ≥ Sth seconds, time with heart rate higher than 50 beats per minute ≥ Sth seconds), if the sum of the weights of the detection results calculated in this way is less than the sum-of-weights threshold, the availability determination module 277 determines that the driver 400 does not meet the availability conditions. Here, the aforementioned Sth seconds is the second detection time interval t2, and the Sth seconds may be between 1 second and 5 seconds.

Table 2

[0041] The automatic driving handover determination method 100 may further include a vehicle body signal acquisition step 120, a driver presence determination step 160, and a warning step 190. The automatic driving handover determination system 200 may further include at least one vehicle body sensor 220 and a warning unit 290. The processing unit 270 may further include a presence determination module 276. The vehicle body signal acquisition step 120 includes acquiring, by at least one vehicle body sensor 220, a plurality of vehicle body signals including at least some of a plurality of seat belt engagement signals and a plurality of driver seat pressure signals at the driver seat 304 during a first detection time interval t1 in the automatic driving mode. The driver presence determination step 160 includes determining, by the presence determination module 276, whether a driver 400 satisfies a presence condition that is a presence determination result, based on at least one of a driver image, vehicle body signals, and biometric signals. The automatic driving handover determination step 180 includes determining whether an automatic driving handover condition is satisfied based on an availability determination result and a presence determination result. If the automatic driving handover determination step 180 determines that the automatic driving handover condition (i.e., at least one of the presence condition and the availability condition is not satisfied) is not satisfied, the warning step 190 includes generating at least one of a visual warning, an auditory warning, and a vibration warning to warn the driver 400. Thereby, satisfying the presence condition indicates that there is a driver 400 at the driver seat 304, and satisfying the availability condition indicates that the driver 400 has the ability to take over. When the driver 400 does not satisfy at least one of the presence condition and the availability condition, that is, when the driver 400 does not have the ability to take over the operation for a while (for example, is dozing off), or when the vehicle 300 needs to be taken over by the driver 400 (for example, the automatic driving state has been cancelled), the warning unit 290 issues a warning. If there is no reaction after the warning, the automatic driving unit 240 takes a minimum risk action for the safety of the automatic driving system of the vehicle 300.

[0042] Although the present invention has been disclosed as above based on the embodiments, the embodiments do not limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present invention is based on the scope defined in the appended claims.

Explanation of Reference Numerals

[0043] 100 Automatic driving handover determination method 110 Initial correction step 112 Driver correction step 114 Detection module update step 120 Vehicle body signal acquisition step 130 Image acquisition step 132 Human body posture detection step 134 Facial detection step 135 Eye opening detection step 136 Viewing angle detection step 137 Head shake detection step 150 Biosignal acquisition step 152 Heart rate detection step 160 Driver presence determination step 170 Driver availability determination step 180 Automatic driving handover determination step 190 Warning step 200 Automatic driving handover determination system 203 Vehicle-mounted communication network 220 Vehicle body sensor 230 Camera 240 Automatic driving unit 250 Biosensor 270 Processing unit 271 Correction module 276 Presence determination module 277 Availability determination module 280 Detection module 282 Human body posture detection unit 285 Eye opening detection unit 286 Viewing angle detection unit 287 Head shake detection unit 290 Warning unit 300 Vehicle 303 Driver's cab 304 Driver's seat 310 Left rearview mirror 311 Steering wheel 312 Instrument panel 313 Rearview mirror 314 Car radio 315 Passenger glove box 317 Right rearview mirror 400 Driver h1, h3, h5, h7 Lateral distance m1, m2, m4 Vertical distance p1 First upper point p2 First lower point p3 Second upper point p4 Second lower point t1 First detection time interval t2 Second detection time interval t3 Third detection time interval v1 First vertical distance v2 Second vertical distance

Claims

1. An automatic driving handover determination method for determining whether a driver in the driver's seat inside a vehicle satisfies an automatic driving handover condition in an automatic driving mode, comprising: Before entering the automatic driving mode of the vehicle, obtaining a plurality of corrected images of the driver by at least one camera, and generating, by a correction module based on the corrected images, a plurality of driver correction parameters that are a plurality of relative position parameters of the driver in the driver's cab of the vehicle; A detection module update step of updating a detection module based on the driver correction parameters; An image acquisition step of obtaining a plurality of driver images of the driver by the camera in a first detection time interval in the automatic driving mode; A face detection step of detecting, by the detection module based on the driver images, whether the driver satisfies at least one face feature condition that is at least one face detection result; A driver availability determination step of determining, by an availability determination module based on the at least one face detection result, whether the driver satisfies an availability condition that is an availability determination result; An automatic driving handover determination step of determining whether the automatic driving handover condition is satisfied based on the availability determination result; An automatic driving handover determination method comprising the above steps.

2. A vehicle body signal acquisition step of obtaining, by at least one vehicle body sensor, a plurality of vehicle body signals in the driver's seat including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals in a first detection time interval in the automatic driving mode; A driver presence determination step of determining, by a presence determination module based on at least one of the driver images and the vehicle body signals, whether the driver satisfies a presence condition that is a presence determination result; A warning step; further comprising: The automatic driving handover determination step includes determining whether the automatic driving handover condition is satisfied based on the availability determination result and the presence determination result; When the automatic driving handover determination step determines that the automatic driving handover condition is not satisfied, the warning step includes generating at least one of a visual warning, an auditory warning, and a vibration warning to warn the driver. The automatic driving handover determination method according to Claim 1.

3. In the first detection time interval in the automatic driving mode, a biological signal acquisition step of acquiring, by at least one biological sensor, a plurality of biological signals of the driver including at least one of a plurality of heartbeat signals and a plurality of respiration signals; A human body posture detection step of detecting, by the detection module based on the driver image, whether the driver satisfies a non-sleep posture feature condition which is a non-sleep posture detection result; A heartbeat detection step of detecting, by the detection module based on the biological signal, whether the driver satisfies a heartbeat feature condition which is a heartbeat detection result, and further including: The driver availability determination step includes determining, by the availability determination module based on the at least one face detection result, the non-sleep posture detection result, and the heartbeat detection result, whether the driver satisfies the availability condition which is the availability determination result. The automatic driving handover determination method according to claim 1.

4. There are a plurality of the at least one face feature condition. When the driver satisfies at least two of the face feature conditions, or satisfies the non-sleep posture feature condition and the heartbeat feature condition, and the sum of the weights of the detection results is greater than the weight sum threshold, the availability determination module determines that the driver satisfies the availability condition. The weight coefficient of each of the face detection results is greater than the weight coefficient of the non-sleep posture detection result and the weight coefficient of the heartbeat detection result. The automatic driving handover determination method according to claim 3.

5. The driver correction parameter includes horizontal distance and vertical distance between the driver's line-of-sight range related data and at least one target among the rearview mirror, left rearview mirror, right rearview mirror, car radio, steering wheel, instrument panel, and passenger seat glove box of the driver and the vehicle. The automatic driving handover determination method according to claim 1.

6. An initial correction step in which the initial driver in the driver's seat executes at least one operation of directly looking at the at least one target based on an initial correction operation instruction of the correction module, the camera acquires at least one initial correction image of the at least one operation, and the correction module generates a plurality of initial correction parameters including the viewing angle of the initial driver with respect to the at least one target. The driver correction step includes generating, by the correction module, the driver correction parameters based on the initial correction parameters and the correction image, the driver correction parameters including a viewing angle with respect to at least one target of the driver. The automatic driving handover determination method according to claim 5.

7. The detection module includes an eye-opening detection unit, a viewing angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The face detection step includes: An eye-opening detection step of detecting, by the eye-opening detection unit of the detection module, based on the driver image whether the driver satisfies an eye-opening feature condition that is an eye-opening detection result; A viewing angle detection step of detecting, by the viewing angle detection unit of the detection module, based on the driver image whether the driver satisfies a viewing angle feature condition that is a viewing angle detection result; A head shake detection step of detecting, based on the driver image, by the head shake detection unit of the detection module whether the driver satisfies a head shake feature condition that is a head shake detection result; including The at least one face detection result is at least three, and the face detection result includes the eye-opening detection result, the viewing angle detection result, and the head shake detection result. The automatic driving handover determination method according to claim 1.

8. The eye-opening detection step includes establishing a first vertical distance by a first upper eye point and a first lower eye point of both eyes in each driver image based on a Euclidean distance algorithm, and establishing a second vertical distance by a second upper eye point and a second lower eye point, and defining an eye state feature value as the maximum value of the two first vertical distances and the two second vertical distances of both eyes. If the eye state feature value is greater than an eye-opening threshold, it is defined that the eye-opening state is satisfied. The eye-opening threshold is determined by the eye-opening detection unit of the updated detection module and is within a threshold setting range. Whether the eye-opening state is satisfied is used to detect whether the driver satisfies the eye-opening feature condition in the first detection time interval. The automatic driving handover determination method according to claim 7.

9. The viewing angle detection step includes identifying the pupils in each of the driver images and detecting whether the driver satisfies a viewing angle specified range through coordinate transformation. Whether the driver satisfies the viewing angle specified range is used to detect whether the driver satisfies the viewing angle characteristic condition in the first detection time interval. The head shake detection step includes identifying the tip of the nose in each of the driver images and detecting whether the driver satisfies a head shake specified range through coordinate transformation. Whether the driver satisfies the head shake specified range is used to detect whether the driver satisfies the head shake characteristic condition in the first detection time interval. The automatic driving handover determination method according to claim 7.

10. The first detection time interval includes a plurality of second detection time intervals. Each of the second detection time intervals includes a plurality of third detection time intervals. The first detection time interval is between 1 second and 30 seconds. Each of the second detection time intervals is between 1 second and 5 seconds. Each of the third detection time intervals is between 0.5 second and 1 second. Each of the third detection time intervals includes a plurality of detection time points. The eye-opening detection step detects whether the driver satisfies an eye-opening state at one of the detection time points. When more than half of the detection time points in one of the third detection time intervals satisfy the eye-opening state, an eye-opening signal corresponding to one of the third detection time intervals is generated. When the eye-opening signal corresponds to each of the third detection time intervals in one of the second detection time intervals, it is detected that the driver satisfies the eye-opening characteristic condition in the first detection time interval. The automatic driving handover determination method according to claim 7.

11. When the viewing angle signal satisfying the viewing angle specified range corresponds to each of the third detection time intervals in one of the second detection time intervals, it is detected that the driver satisfies the viewing angle characteristic condition in the first detection time interval. When the head shake signal satisfying the head shake specified range corresponds to each of the third detection time intervals in one of the second detection time intervals, it is detected that the driver satisfies the head shake characteristic condition in the first detection time interval. The automatic driving handover determination method according to claim 10.

12. An automatic driving handover determination system provided in a vehicle, An automatic driving unit for executing the automatic driving mode of the vehicle, At least one camera for acquiring a plurality of driver images of a driver in the driver's seat inside the vehicle, A processing unit including a correction module, a detection module, and an availability determination module, An in-vehicle communication network used for communication connection between the automatic driving unit, the camera, and the processing unit, Comprising, The processing unit, Before entering the automatic driving mode of the vehicle, acquire a plurality of corrected images of the driver by the camera, and generate, by the correction module based on the corrected images, a plurality of driver correction parameters that are a plurality of relative position parameters of the driver in the driver's cab of the vehicle, Update the detection module based on the driver correction parameters, In a first detection time interval in the automatic driving mode, acquire a plurality of driver images of the driver by the camera, Detect, by the detection module based on the driver images, whether the driver satisfies at least one facial feature condition that is at least one facial detection result, Determine, by the availability determination module based on the at least one facial detection result, whether the driver satisfies an availability condition that is an availability determination result, An automatic driving handover determination system used to determine whether an automatic driving handover condition is satisfied based on the availability determination result.

13. At least one vehicle body sensor, At least one biological sensor, A warning unit, Further comprising, The processing unit further includes a presence determination module, The processing unit, In the first detection time interval in the automatic driving mode, acquire a plurality of biological signals of the driver including at least one of a plurality of heart rate signals and a plurality of respiration signals by the at least one biological sensor, Detect, by the detection module based on the driver images, whether the driver satisfies a non-sleep posture feature condition that is a non-sleep posture detection result, Detect, by the detection module based on the biological signals, whether the driver satisfies a heart rate feature condition that is a heart rate detection result, determining, by the availability determination module based on the at least one face detection result, the non-sleep posture detection result, and the heart rate detection result, whether the driver satisfies the availability condition that is the availability determination result; acquiring, by the at least one vehicle body sensor, a plurality of vehicle body signals in the driver's seat including at least some of the plurality of seat belt engagement signals and the plurality of driver's seat pressure signals during the first detection time interval in the automatic driving mode; determining, by the presence determination module based on at least one of the driver image and the vehicle body signal, whether the driver satisfies the presence condition that is the presence determination result; determining whether the automatic driving handover condition is satisfied based on the availability determination result and the presence determination result, and when it is determined that the automatic driving handover condition is not satisfied, generating at least one of a visual warning, an auditory warning, and a vibration warning by the warning unit to warn the driver. The automatic driving handover determination system according to claim 12.

14. The detection module includes an eye-opening detection unit, a viewing angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The processing unit is detecting, by the eye-opening detection unit of the detection module based on the driver image, whether the driver satisfies the eye-opening feature condition that is the eye-opening detection result; detecting, by the viewing angle detection unit of the detection module based on the driver image, whether the driver satisfies the viewing angle feature condition that is the viewing angle detection result; used for detecting, by the head shake detection unit of the detection module based on the driver image, whether the driver satisfies the head shake feature condition that is the head shake detection result; The automatic driving handover determination system according to claim 12, wherein the at least one face detection result is at least three, and the face detection result includes the eye-opening detection result, the viewing angle detection result, and the head shake detection result.

Images