Method and system for determining automatic driving takeover

Abstract

To provide an automatic driving takeover determination method and a system therefor.SOLUTION: An automatic driving takeover determination method includes a driver correction step, a detection module update step, an image acquisition step, a face portion detection step, a driver available determination step, and an automatic driving takeover determination step. The driver correction step includes: acquiring a plurality of correction images of a driver by at least one camera before a vehicle enters an automatic driving mode; and generating a plurality of driver correction parameters which are a plurality of mutual positional parameters of the driver in a cab of the vehicle, by the correction module based on the correction images. The detection module update step includes updating a detection module based on the driver correction parameters. The automatic driving takeover determination step includes determining whether an automatic driving takeover condition is satisfied based on an available determination result. This will improve the safety of the automatic driving system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method and system for determining whether to take over automatic driving, and more particularly to a method and system for determining whether to take over automatic driving based on a driver image. [Background technology]

[0002] As the market for autonomous vehicles expands day by day, how to improve safety to reduce accidents has become a priority in autonomous vehicle development. SAE (Society of Automotive Engineers) Level 3 autonomous vehicles do not require a driver to hold the steering wheel under certain conditions, but the driver must have the ability to take over the driving task. Therefore, the focus of Level 3 autonomous vehicle development is not on detecting driver concentration like Level 0 to Level 2 autonomous vehicles, but on detecting whether the driver is still conscious and able to take over the driving task at any time. Summary of the Invention [Problem to be solved by the invention]

[0003] Based on the above, the issue of how to develop a method and system for determining whether a Level 3 autonomous vehicle driver has the ability to take over the driving task appropriately and accurately is an issue that must be resolved urgently in the autonomous vehicle market. [Means for solving the problem]

[0004] The present invention provides an automatic driving takeover determination method and system that detects whether the driver has the ability to take over the driving task and improves the safety of the automatic driving system by acquiring multiple corrected images of the driver using at least one camera before the vehicle enters automatic driving mode, generating multiple driver correction parameters, which are multiple relative position parameters of the driver in the vehicle's cab, using a correction module based on the corrected images, and updating a detection module based on the driver correction parameters.

[0005] According to one embodiment of the present invention, an automatic driving takeover determination method for determining whether a driver in the driver's seat of a vehicle satisfies automatic driving takeover conditions in automatic driving mode includes: a driver correction step including acquiring multiple corrected images of the driver using at least one camera before entering the automatic driving mode of the vehicle, and generating multiple driver correction parameters, which are multiple relative position parameters of the driver in the driver's cab of the vehicle, using a correction module based on the corrected images; a detection module update step including updating the detection module based on the driver correction parameters; an image acquisition step including acquiring multiple driver images of the driver using a camera during a first detection time interval in the automatic driving mode; a face detection step including detecting by the detection module based on the driver image whether the driver satisfies at least one facial feature condition, which is at least one facial detection result; a driver availability determination step including determining by the availability determination module based on the face detection result whether the driver satisfies the availability condition, which is an availability determination result; and an automatic driving takeover determination step including determining whether the automatic driving takeover conditions are met based on the availability determination result.

[0006] According to an example of the aforementioned embodiment, the method for determining whether to take over automatic driving may further include: a vehicle body signal acquisition step including acquiring, by at least one vehicle body sensor, a plurality of vehicle body signals at the driver's seat including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals during a first detection time interval in the automatic driving mode; a driver presence determination step including determining, by a presence determination module, whether or not the driver satisfies a presence condition, which is a presence determination result, based on at least one of a driver image and the vehicle body signals; and a warning step. The automatic driving takeover determination step includes determining whether or not the automatic driving takeover condition is satisfied based on the availability determination result and the presence determination result, and if it is determined in the automatic driving takeover determination step that the automatic driving takeover condition is not satisfied, the warning step includes generating at least one of a visual warning, an audible warning, and a vibration warning to warn the driver.

[0007] According to an example of the above-described embodiment, the autonomous driving takeover determination method may further include: a biological signal acquisition step including acquiring, by at least one biological sensor, a plurality of biological signals of the driver, including at least one of a plurality of heart rate signals and a plurality of respiratory signals, during a first detection time interval in the autonomous driving mode; a human body posture detection step including detecting, by a detection module based on a driver image, whether the driver satisfies a non-sleep posture characteristic condition, which is a non-sleep posture detection result; and a heart rate detection step including detecting, by the detection module based on the biological signals, whether the driver satisfies a heart rate characteristic condition, which is a heart rate detection result. The driver availability determination step includes determining, by the availability determination module, whether the driver satisfies the availability condition, which is an availability determination result, based on the face detection result, the non-sleep posture detection result, and the heart rate detection result.

[0008] According to an example of the aforementioned embodiment, the at least one facial feature condition may be a plurality of conditions, and if the driver satisfies at least two of the facial feature conditions, or satisfies the non-sleeping posture feature condition and the heart rate feature condition, and the sum of the weights of the detection results is greater than a sum of weights threshold, the availability determination module determines that the driver satisfies the availability condition, and the weight coefficient of each face detection result is greater than the weight coefficient of the non-sleeping posture detection result and the weight coefficient of the heart rate detection result.

[0009] According to an example of the aforementioned embodiment, the driver correction parameters may include driver's line of sight related data, lateral distance and longitudinal distance between the driver and at least one target of the vehicle's rear mirror, left rear mirror, right rear mirror, car radio, steering wheel, instrument panel and passenger glove box.

[0010] According to an example of the aforementioned embodiment, the method for determining automatic driving takeover may further include an initial correction step including an initial driver in the driver's seat performing at least one action of looking directly at the at least one target based on an initial correction operation instruction of a correction module, acquiring at least one initial correction image of the at least one action by a camera, and generating, by the correction module, a plurality of initial correction parameters including a visual angle of the initial driver with respect to the target. The driver correction step includes generating, by the correction module, a driver correction parameter including a visual angle of the driver with respect to the target based on the initial correction parameter and the corrected image.

[0011] According to an example of the aforementioned embodiment, the detection module may include an eye-opening detection unit, a visual angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The face detection step may include an eye-opening detection step including detecting, based on a driver image, by the eye-opening detection unit of the detection module, whether the driver satisfies an eye-opening characteristic condition, which is an eye-opening detection result; a visual angle detection step including detecting, based on the driver image, by the visual angle detection unit of the detection module, whether the driver satisfies the visual angle characteristic condition, which is a visual angle detection result; and a head shake detection step including detecting, based on the driver image, by the head shake detection unit of the detection module, whether the driver satisfies a head shake characteristic condition, which is a head shake detection result. The at least one face detection result may include at least three, and the face detection result includes an eye-opening detection result, a visual angle detection result, and a head shake detection result.

[0012] According to an example of the aforementioned embodiment, the eye-open detection step may include: establishing a first vertical distance based on a first eye-upper point and a first current-lower point of each of the eyes in each driver image, and establishing a second vertical distance based on a second eye-upper point and a second current-lower point of each of the eyes in each driver image based on a Euclidean distance algorithm; and defining the eye state feature value as the maximum value of the two first vertical distances and the two second vertical distances of the eyes. If the eye state feature value is greater than an eye-open threshold, the eye-open state is defined to be satisfied. The eye-open threshold is determined by the eye-open detection unit of the updated detection module and is within a threshold setting range. Whether the eye-open state is satisfied is used to detect whether the driver satisfies the eye-open feature condition in the first detection time interval.

[0013] According to an example of the aforementioned embodiment, the visual angle detecting step may include identifying a pupil in each driver image and detecting whether the driver satisfies a visual angle specification range through coordinate transformation, where the satisfaction of the visual angle specification range is used to detect whether the driver satisfies a visual angle characteristic condition in a first detection time interval. The head shake detecting step may include identifying a nose tip in each driver image and detecting whether the driver satisfies a head shake specification range through coordinate transformation, where the satisfaction of the head shake specification range is used to detect whether the driver satisfies a head shake characteristic condition in a first detection time interval.

[0014] According to an example of the aforementioned embodiment, the first detection time interval may include a plurality of second detection time intervals, each of which may include a plurality of third detection time intervals. The first detection time interval may be between 1 and 30 seconds, each of which may be between 1 and 5 seconds, and each of which may be between 0.5 and 1 second. Each of the third detection time intervals may include a plurality of detection time points. The eye-open detection step detects whether the driver's eye is open at one detection time point. If more than half of the detection time points in one third detection time interval satisfy the eye-open state, an eye-open signal corresponding to the third detection time interval is generated. If all of the third detection time intervals in one second detection time interval correspond to an eye-open signal, the driver is detected as satisfying the eye-open characteristic condition in the first detection time interval.

[0015] According to the example of the above-described embodiment, if each of the third detection time intervals in one second detection time interval corresponds to a visual angle signal that satisfies the visual angle specified range, it is determined that the driver satisfies the visual angle characteristic condition in the first detection time interval. If each of the third detection time intervals in one second detection time interval corresponds to a head shake signal that satisfies the head shake specified range, it is determined that the driver satisfies the head shake characteristic condition in the first detection time interval.

[0016] According to another embodiment of the present invention, an automatic driving takeover determination system provided in a vehicle includes an automatic driving unit for executing the automatic driving mode of the vehicle, at least one camera for acquiring multiple driver images of a driver in the driver's seat in the vehicle, a processing unit including a correction module, a detection module, and an availability determination module, and an in-vehicle communication network for communicatively connecting the automatic driving unit, the camera, and the processing unit. The processing unit is used to acquire multiple corrected images of the driver using the camera before entering the automatic driving mode of the vehicle, generate multiple driver correction parameters, which are multiple relative position parameters of the driver in the driver's cab of the vehicle, using the correction module based on the corrected images, update the detection module based on the driver correction parameters, acquire multiple driver images of the driver using the camera during a first detection time interval in the automatic driving mode, detect using the detection module based on the driver image whether the driver satisfies at least one facial feature condition, which is at least one facial detection result, determine using the availability determination module based on the facial detection result whether the driver satisfies an availability condition, which is an availability determination result, and determine whether the automatic driving takeover condition is met based on the availability determination result.

[0017] According to an example of the aforementioned embodiment, the autonomous driving takeover determination system may further include at least one vehicle body sensor, at least one biosensor, and a warning unit. The processing unit may further include a presence determination module, and the processing unit may acquire a plurality of biosignals of the driver, including at least one of a plurality of heart rate signals and a plurality of respiratory signals, by the biosensor during a first detection time interval in the autonomous driving mode, detect by the detection module based on a driver image whether the driver satisfies a non-sleeping posture characteristic condition, which is a non-sleeping posture detection result, by the detection module based on the biosensor, detect by the detection module whether the driver satisfies a heart rate characteristic condition, which is a heart rate detection result, and perform an availability determination based on at least one of the face detection result, the non-sleeping posture detection result, and the heart rate detection result, The system is used to determine whether the driver satisfies the presence condition, which is a presence determination result, using a presence determination module; acquiring a plurality of vehicle body signals at the driver's seat including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals using a vehicle body sensor during a first detection time interval in the autonomous driving mode; determining whether the driver satisfies the presence condition, which is a presence determination result, using a presence determination module based on at least one of the driver image and the vehicle body signals; determining whether the autonomous driving handover condition is met based on the availability determination result and the presence determination result, and if it is determined that the autonomous driving handover condition is not met, the warning unit generates at least one of a visual warning, an audible warning, and a vibration warning to warn the driver.

[0018] According to an example of the aforementioned embodiment, the detection module may include an eye-opening detection unit, a visual angle detection unit, and a head shake detection unit, all of which are detection algorithms based on machine learning. The processing unit is used to detect, based on the driver image, whether the driver satisfies an eye-opening characteristic condition, which is an eye-opening detection result, using the eye-opening detection unit of the detection module; to detect, based on the driver image, whether the driver satisfies the visual angle characteristic condition, which is a visual angle detection result, using the visual angle detection unit of the detection module; and to detect, based on the driver image, whether the driver satisfies a head shake characteristic condition, which is a head shake detection result, using the head shake detection unit of the detection module. The at least one face detection result is at least three, and the face detection result includes an eye-opening detection result, a visual angle detection result, and a head shake detection result. [Brief explanation of the drawings]

[0019] [Figure 1] 3 is a flowchart showing a method for determining whether to take over automatic driving according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a block diagram showing an automated driving takeover determination system according to a second embodiment of the present invention. [Figure 3] 3 is a schematic diagram showing a vehicle equipped with the automatic driving takeover determination system shown in FIG. 2. [Figure 4] 3 is another schematic diagram showing a vehicle equipped with the automatic driving takeover determination system in FIG. 2. [Figure 5] 2 is a schematic diagram showing eye state feature values ​​in an eye open detection step in FIG. 1; FIG. [Figure 6] FIG. 2 is a schematic diagram showing an eye-opening detection step in FIG. [Figure 7] 1. FIG. 4 is another schematic diagram showing the eye-open detection step in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. For clarity of explanation, numerous practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. That is, these practical details are not necessary in the embodiments of the present invention. In addition, to simplify the drawings, some conventional structures and components are simply and diagrammatically shown in the drawings, and overlapping components may be designated by the same or similar numbers.

[0021] Furthermore, terms such as "first" and "second" are merely used to describe different components and do not limit the components themselves, so a "first component" may be read as a "second component." The combinations of components in this specification are not well-known, conventional, or known combinations in this field, and whether the combination relationship can be easily achieved by a person skilled in the art cannot be determined based on whether the components themselves are known.

[0022] FIG. 1 shows a flowchart of an automated driving takeover determination method 100 according to a first embodiment of the present invention. FIG. 2 shows a block diagram of an automated driving takeover determination system 200 according to a second embodiment of the present invention. FIG. 3 shows a schematic diagram of a vehicle 300 equipped with the automated driving takeover determination system 200 shown in FIG. 2. The automated driving takeover determination method 100 according to the first embodiment and the automated driving takeover determination system 200 according to the second embodiment will be described together with reference to FIGS. 1 to 3. The automated driving takeover determination method 100 is used to determine whether a driver 400 in the driver's seat 304 of the vehicle 300 satisfies the automated driving takeover conditions in the automated driving mode (i.e., whether the automated driving conditions are satisfied). The vehicle 300 may be a Level 3 automated vehicle, and the automated driving takeover scenario includes the driver 400 starting automated driving and taking over the vehicle 300, and the automated driving transferring driving rights to the driver 400. The method 100 for determining whether to take over automatic driving includes a driver correction step 112, a detection module update step 114, an image acquisition step 130, a face detection step 134, a driver availability determination step 170, and an automatic driving takeover determination step 180.

[0023] The driver correction step 112 includes acquiring a plurality of corrected images of the driver 400 by at least one camera 230 before the vehicle 300 enters the autonomous driving mode, and generating a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver 400 in the cab 303 of the vehicle 300, by a correction module (i.e., correction program) 271 based on the corrected images. The detection module update step 114 includes updating the detection module 280 based on the driver correction parameters, thereby adapting to variables such as different heights and seat distances of the driver 400 through a self-correction mechanism and automatically adjusting a reasonable forward field of view range.

[0024] The image acquisition step 130 includes acquiring a plurality of driver images (image frames), which may be visible light or infrared light images of the driver 400, by the camera 230 during a first detection time interval in the autonomous driving mode. The face detection step 134 includes detecting, by the detection module 280 based on the driver images, whether the driver 400 satisfies at least one facial feature condition, which is at least one face detection result.

[0025] The driver availability determination step 170 includes determining whether or not the driver 400 satisfies the availability conditions, which are the availability determination results, by the availability determination module 277 based on the face detection result. The automated driving takeover determination step 180 includes determining whether or not the automated driving takeover conditions are satisfied based on the availability determination result. This makes it possible to accurately detect whether the driver 400 has the ability to take over the driving task, contributing to improving the safety of the automated driving system.

[0026] 2 and 3. The automated driving takeover determination system 200 of the second embodiment is provided in a vehicle 300, and includes an automated driving unit 240, at least one camera 230, a processing unit 270, and an in-vehicle communication network 203. The automated driving unit 240 is used to execute the automated driving mode of the vehicle 300. The camera 230 is used to acquire multiple driver images of a driver 400 sitting in a driver's seat 304 in the vehicle 300. The processing unit 270 includes a correction module 271, a detection module 280, and an availability determination module 277. The in-vehicle communication network 203 is used for communication connection between the automated driving unit 240, the camera 230, and the processing unit 270.

[0027] The processing unit 270 is used to acquire a plurality of corrected images of the driver 400 using the camera 230 before the vehicle 300 enters the autonomous driving mode, generate a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver 400 in the cab 303 of the vehicle 300, using the correction module 271 based on the corrected images, update the detection module 280 based on the driver correction parameters, acquire a plurality of driver images of the driver 400 using the camera 230 during a first detection time interval in the autonomous driving mode, detect whether the driver 400 satisfies at least one facial feature condition, which is at least one facial detection result, using the detection module 280 based on the driver images, determine whether the driver 400 satisfies an availability condition, which is an availability determination result, using the availability determination module 277 based on the facial detection result, and determine whether the autonomous driving takeover condition is satisfied based on the availability determination result. This makes it possible to appropriately and accurately detect whether the driver 400 of a Level 3 autonomous vehicle has the ability to take over the driving task.

[0028] 4 shows another schematic diagram of the vehicle 300 equipped with the automatic driving takeover determination system 200 in FIG. 2 (see FIGS. 3 and 4). The driver correction parameters may include data related to the driver's 400's line of sight range, and the lateral distances (e.g., lateral distances h1, h3, h5, h7) and longitudinal distances (e.g., longitudinal distances m1, m2, m4) between the driver 400 and at least one target selected from the rearview mirror 313, the left rearview mirror 310, the right rearview mirror 317, the car radio 314, the steering wheel 311, the instrument panel 312, and the passenger glove compartment 315 of the vehicle 300. This allows the self-correction mechanism to adapt to different drivers 400 and automatically adjust a reasonable forward field of view range. Specifically, the line of sight range of driver 400 takes into consideration the head shaking angle, i.e., the state in which the presence or absence of head shaking (rotation) is integrated into the line of sight range, and it includes the line of sight movement range with head shaking (owl type) and the line of sight movement range with no head shaking (lizard type).

[0029] For details, please refer to FIGS. 1 to 4. The automated driving takeover determination method 100 may further include an initial correction step 110. The initial correction step 110 is targeted at a vehicle 300 of an unknown model. The initial correction step 110 includes obtaining an image of an initial driver (which may be the same as or different from the driver 400) using the camera 230 to find a region of interest (ROI) or a face position, instructing the initial driver in the driver's seat 304 to perform at least one action of looking directly at the at least one target according to an initial correction operation instruction from the correction module 271, obtaining at least one initial correction image of the at least one action using the camera 230, and generating a plurality of initial correction parameters using the correction module 271, the initial correction parameters including the initial driver's viewing angle with respect to the target. The driver correction step 112 is targeted at a vehicle 300 of a known model. The driver correction step 112 includes generating driver correction parameters by the correction module 271 based on the initial correction parameters and the correction image, where the driver correction parameters include the viewing angle of the driver 400 to the target. Thereby, the detection module update step 114 updates the detection module 280 based on the driver correction parameters of the different drivers 400.

[0030] In the initial correction step 110, for example, see FIG. 3. If the top end point of the steering wheel 311 is set as the origin, the vertical distance m1 between the center point of the steering wheel 311 and the origin, the vertical distance m2 between the instrument panel 312 and the origin, and the vertical distance m4 between the face of the driver 400 and the origin can be obtained, where the vertical distances m1, m2, and m4 may be 0.1 m, 0.1 m, and 0.45 m, respectively. See FIG. 4. If the left rearview mirror 310 is set as the origin, the lateral distance h1 between the driver 400 (or the steering wheel 311, the instrument panel 312, or the pedals) and the origin, the lateral distance h3 between the car radio 314 (or the rearview mirror 313) and the origin, the lateral distance h5 between the passenger glove compartment 315 and the origin, and the lateral distance h7 between the right rearview mirror 317 and the origin can be obtained. Here, the lateral distances h1, h3, h5, and h7 may be 0.75 m, 1.1 m, 1.45 m, and 2.1 m, respectively. In the initial correction step 110, it is found that the visual angle when the initial driver looks directly at the rearview mirror 313 is 37 degrees to the right and 24 degrees up, and the visual angle when the initial driver looks directly at the left rearview mirror 310 is 60 degrees to the left and 6 degrees down. The visual angle when the initial driver looks directly at the right rearview mirror 317 is 71 degrees to the right and 6 degrees down, and the visual angle when the initial driver looks directly at the car radio 314 is 37 degrees to the right and 18 degrees down. The visual angle when the initial driver looks directly at the steering wheel 311 is 29 degrees down, the visual angle when the initial driver looks directly at the instrument panel 312 is 18 degrees down, and the visual angle when the initial driver looks directly at the passenger glove compartment 315 is 55 degrees to the right and 33 degrees down. The vertical distance, horizontal distance, and visual angle can be derived from each other using trigonometric functions. Even if the driver 400 is different from the initial driver in the driver correction step 112, the driver correction parameters to be applied to the driver 400 can be generated using the initial correction parameters and the correction image.

[0031] As a result, the important positions, line of sight, and visual angle ranges for different vehicle types and different drivers 400 while driving are not fixed values. Factors that affect these values ​​include the driver's 400 height, body type, face shape, seat habits, or the structural design and installation position of the vehicle 300. In the driver correction step 112, distance information for important positions is obtained from the structural parameters of the vehicle 300, and the visual angle of each important position can be dynamically estimated based on the position of the driver 400. This contributes to the automated driving takeover determination system 200 installed in a Level 3 automated vehicle performing a wide range of detection that covers scenarios in which the head pose is not looking straight ahead. This improves the overall detection rate and differs from the detection method used by Level 0 to Level 2 automated vehicles, which considers it abnormal if they cannot detect a face looking straight ahead.

[0032] 1 and 2. The detection module 280 may include an eye-open detection unit 285, a visual angle detection unit 286, and a head shake detection unit 287, each of which may be a detection algorithm based on machine learning (e.g., image deep learning). The face detection step 134 may include an eye-open detection step 135, a visual angle detection step 136, and a head shake detection step 137, each of which includes image preprocessing. The eye-open detection step 135 includes detecting, by the eye-open detection unit 285 of the detection module 280, based on the driver image, whether the driver 400 satisfies an eye-open feature condition, which is an eye-open detection result. The visual angle detection step 136 includes detecting, by the visual angle detection unit 286 of the detection module 280, based on the driver image, whether the driver 400 satisfies a visual angle feature condition, which is a visual angle detection result. For example, in a second detection time interval in the first detection time interval, a designated range formed by targets such as rearview mirror 313, left rearview mirror 310, right rearview mirror 317, car radio 314, steering wheel 311, instrument panel 312, and passenger glove compartment 315, which are on which driver 400's line of sight, can be deemed to satisfy the visual angle characteristic condition. Head shake detection step 137 includes detecting, by head shake detection unit 287 of detection module 280, based on the driver image, whether driver 400 satisfies a head shake characteristic condition, which is a head shake detection result. For example, in a second detection time interval in the first detection time interval, a designated range formed by targets such as rearview mirror 313, left rearview mirror 310, right rearview mirror 317, car radio 314, steering wheel 311, instrument panel 312, and passenger glove compartment 315, on which driver 400's head shakes, can be deemed to satisfy the head shake characteristic condition. The at least one face detection result includes at least three face detection results, including an eye opening detection result, a visual angle detection result, and a head shaking detection result, which contributes to accurately detecting whether the driver 400 has the ability to take over the driving task.

[0033] FIG. 5 shows a schematic diagram of the eye state feature value in the eye open detection step 135 in FIG. 1. Please refer to FIGS. 1 and 5. The eye open detection step 135 may include, based on the Euclidean distance algorithm, establishing a first vertical distance v1 between a first upper eye point p1 and a first lower eye point p2 of each of the two eyes in each driver image, and establishing a second vertical distance v2 between a second upper eye point p3 and a second lower eye point p4 of each of the two eyes, and defining the eye state feature value as the maximum value of the two first vertical distances v1 and the two second vertical distances v2 of the two eyes. If the eye state feature value is greater than an eye open threshold (threshold), it is defined as satisfying the eye open state. The eye open threshold is determined by the eye open detection unit 285 of the updated detection module 280 and is within a threshold setting range (e.g., between 6 pixel values ​​and 15 pixel values). Whether the eye open state is satisfied is used to detect whether the driver 400 satisfies the eye open feature condition in the first detection time interval. This allows a determination to be made as to whether the driver 400 has the ability to take over the driving task based on a valid eye-open detection result.

[0034] FIG. 6 is a schematic diagram of the eye-open detection step 135 in FIG. 1, and FIG. 7 is another schematic diagram of the eye-open detection step 135 in FIG. 1. Please refer to FIGS. 6 and 7. The first detection time interval t1 may include multiple second detection time intervals t2, and each second detection time interval t2 may include multiple third detection time intervals t3. The first detection time interval t1 may be between 1 and 30 seconds, each second detection time interval t2 may be between 1 and 5 seconds, and each third detection time interval t3 may be between 0.5 and 1 second. Each third detection time interval t3 includes multiple detection time points. The eye-open detection step 135 detects whether the driver 400 has an eye-open state at one detection time point. If more than half of the detection time points in the third detection time interval t3 satisfy the eye-open state, an eye-open signal corresponding to the third detection time interval t3 is generated. For example, the first detection time interval t1 is 30 seconds, the second detection time interval t2 is 5 seconds, and the third detection time interval t3 is 1 second. As shown in FIG. 6, if an eye-open signal corresponds to each of the third detection time intervals t3 in the second detection time interval t2 (0 to 5 seconds in FIG. 6), it is determined that the driver 400 satisfies the eye-open characteristic condition in the first detection time interval t1 (even if no eye-open signal corresponds to any of the intervals 5 to 30 seconds in FIG. 6). Conversely, as shown in FIG. 7, if an eye-open signal does not correspond to at least one of the third detection time intervals t3 (0 to 1 second in FIG. 7) in the second detection time interval t2 and also if no eye-open signal corresponds to any of the intervals 5 to 30 seconds in FIG. 7, it is determined that the driver 400 does not satisfy the eye-open characteristic condition in the first detection time interval t1. This allows for accurate eye-open detection according to the present invention.

[0035] See FIG. 1. The visual angle detection step 136 may include identifying a pupil image of each driver and detecting whether the driver 400 satisfies a visual angle specification range through coordinate transformation (e.g., Rodrigues' Rotation Formula), where the satisfaction of the visual angle specification range is used to detect whether the driver 400 satisfies a visual angle characteristic condition in a first detection time interval t1. The head shake detection step 137 may include identifying a nose tip in each driver image and detecting whether the driver 400 satisfies a head shake specification range through coordinate transformation (e.g., Rodrigues' Rotation Formula), where the satisfaction of the head shake specification range is used to detect whether the driver 400 satisfies a head shake characteristic condition in a first detection time interval t1. Thus, whether the driver 400 has the ability to take over the driving task is determined based on the valid visual angle detection result and head shake detection result.

[0036] If each of the third detection time intervals t3 in the second detection time interval t2 corresponds to a visual angle signal that satisfies the visual angle designation range, it is determined that the driver 400 satisfies the visual angle characteristic condition in the first detection time interval t1. If each of the third detection time intervals t3 in the second detection time interval t2 corresponds to a head shake signal that satisfies the head shake designation range, it is determined that the driver 400 satisfies the head shake characteristic condition in the first detection time interval t1. This allows accurate visual angle and head shake detection of the present invention to be achieved.

[0037] The automated driving takeover determination method 100 may further include a biological signal acquisition step 150, a human body posture detection step 132, and a heart rate detection step 152, and may also include a respiration detection step (not shown). The automated driving takeover determination system 200 may further include at least one biological sensor 250 (e.g., a bioradar). The biological signal acquisition step 150 includes acquiring, by the biological sensor 250, a plurality of physiological signals of the driver 400, including a plurality of heart rate signals and / or a plurality of respiration signals, during a first detection time interval t1 in the automated driving mode. The human body posture detection step 132 includes detecting whether the driver 400 satisfies a non-sleep posture feature condition, which is a non-sleep posture detection result, by using a deep learning algorithm to integrate facial and body images of the driver 400 and extract posture feature points based on the driver image, by the human body posture detection unit 282 of the detection module 280. The heart rate detection step 152 includes detecting whether the driver 400 satisfies a heart rate feature condition, which is a heart rate detection result, by the detection module 280 based on the biological signal. The driver availability determination step 170 includes determining whether the driver 400 satisfies an availability condition, which is an availability determination result, by the availability determination module 277 based on the face detection result, the non-sleep posture detection result, and the heart rate detection result. This solves the problem of not being able to detect vital signs using the camera 230 alone (for example, images for heart rate and breathing are limited by light sources, occlusions, and the movement of the driver 400), improves the accuracy of driver availability, and avoids erroneous determination due to the image losing the features of the driver 400.

[0038] There may be multiple facial feature conditions. If the driver 400 satisfies at least two of the facial feature conditions, or / and at least one of the non-sleeping posture feature condition and the heart rate feature condition, and the sum of the weights of the detection results is greater than a total weight threshold, the availability determination module 277 determines that the driver 400 satisfies the availability condition. The weight coefficient of each facial detection result is greater than the weight coefficient of the non-sleeping posture detection result and the weight coefficient of the heart rate detection result. Thus, in availability detection, the accuracy of the image and biometric signals is the core of the overall detection. However, in practical applications, factors such as external light source interference, environmental changes, and the appearance of the driver 400 can affect the detection accuracy. The automated driving takeover determination system 200 according to the present invention designs a feature classifier, and the weights of the feature classifier are assigned to complement the image and biometric signals to avoid over-reliance on a single feature, which would result in the loss of availability detection function due to insufficient discrimination.

[0039] For example, see Table 1 below. When driver 400 satisfies two of the facial feature conditions (eyes open≧Sth seconds, time during which gaze is within a specified range≧Sth seconds) and one of the biometric feature conditions (time during which heart rate is higher than 50 beats per minute≧Sth seconds), if the sum of the weights of the detection results calculated in this way is greater than the sum of weight threshold, available use determination module 277 determines that driver 400 satisfies the available use condition, where Sth seconds is the second detection time interval t2, and Sth seconds may be between 1 second and 5 seconds. [Table 1]

[0040] Also, see Table 2 below. When the driver 400 does not satisfy the facial feature condition and satisfies two of the biometric feature conditions (non-sleep state≧Sth seconds, time during which the heart rate is higher than 50 beats / minute≧Sth seconds), if the sum of the weights of the detection results calculated in this way is smaller than the sum of weight threshold, the availability determination module 277 determines that the driver 400 does not satisfy the availability condition, where the aforementioned Sth seconds is the second detection time interval t2, and Sth seconds may be between 1 second and 5 seconds. [Table 2]

[0041] The automatic driving takeover determination method 100 may further include a vehicle body signal acquisition step 120, a driver presence determination step 160, and a warning step 190. The automatic driving takeover determination system 200 may further include at least one vehicle body sensor 220 and a warning unit 290. The processing unit 270 may further include a presence determination module 276. The vehicle body signal acquisition step 120 includes acquiring, by the at least one vehicle body sensor 220, a plurality of vehicle body signals including at least some of a plurality of seat belt engagement signals and a plurality of driver seat pressure signals in the driver's seat 304 during a first detection time interval t1 in the automatic driving mode. The driver presence determination step 160 includes determining, by the presence determination module 276, whether the driver 400 satisfies a presence condition, which is a presence determination result, based on at least one of a driver image, a vehicle body signal, and a biological signal. The automatic driving takeover determination step 180 includes determining whether the automatic driving takeover condition is satisfied based on the availability determination result and the presence determination result. If the automatic driving takeover determination step 180 determines that the automatic driving takeover conditions are not met (i.e., at least one of the presence condition and the availability condition is not met), the warning step 190 includes generating at least one of a visual warning, an audible warning, and a vibration warning to warn the driver 400. Thus, if the presence condition is met, it indicates that the driver 400 is in the driver's seat 304, and if the availability condition is met, it indicates that the driver 400 has the ability to take over. If the driver 400 does not meet at least one of the presence condition and the availability condition, that is, if the driver 400 has not been able to take over driving for a while (e.g., is dozing), or if the vehicle 300 is in a state where the driver 400 needs to take over driving (e.g., the automatic driving state is deactivated), the warning unit 290 issues a warning, and if there is no reaction after the warning, the automatic driving unit 240 takes a minimum-risk action for the safety of the automatic driving system of the vehicle 300.

[0042] Although the present invention has been disclosed above based on the embodiments, the embodiments do not limit the present invention, and a person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present invention is based on the scope defined in the appended claims. [Explanation of symbols]

[0043] 100 Automatic Driving Takeover Determination Method 110 Initial Correction Step 112 Driver Correction Steps 114 Detection module update step 120 Vehicle signal acquisition step 130 Image Acquisition Steps 132 Human Body Posture Detection Steps 134 Face detection step 135 Eye Opening Detection Step 136 Visual Angle Detection Steps 137 Head shake detection step 150 biosignal acquisition steps 152 heart rate detection steps 160 Driver presence determination step 170 Driver availability determination step 180 Autonomous Driving Takeover Judgment Step 190 Warning Steps 200 Autonomous Driving Handover Judgment System 203 In-vehicle communication network 220 Body Sensor 230 Camera 240 Autonomous Driving Unit 250 Biometric Sensors 270 processing units 271 Correction Module 276 Presence Determination Module 277 Availability Determination Module 280 Detection Module 282 Human Body Posture Detection Unit 285 Eye Opening Detection Unit 286 Visual Angle Detection Unit 287 Head movement detection unit 290 Warning Unit 300 vehicles 303 Driver's cab 304 Driver's seat 310 Left rear mirror 311 Handle 312 Instrument Panel 313 Rear mirror 314 Car Radio 315 Passenger glove box 317 Right rear mirror 400 Driver h1, h3, h5, h7 horizontal distance m1, m2, m4 Vertical distance p1 First superior point p2 First Present Point p3 Second superior point p4 Second Present Point t1 First detection time interval t2 Second detection time interval t3 Third detection time interval v1 1st vertical distance v2 2nd vertical distance

Claims

1. An automatic driving takeover determination method for determining whether a driver in a driver's seat in a vehicle satisfies an automatic driving takeover condition in an automatic driving mode, a driver correction step of acquiring a plurality of corrected images of the driver by at least one camera before the vehicle enters the autonomous driving mode, and generating a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver in a cab of the vehicle, by a correction module based on the corrected images; a detection module updating step of updating a detection module based on the driver correction parameters; an image acquisition step of acquiring a plurality of driver images of the driver by the camera during a first detection time interval in the autonomous driving mode; a face detection step of detecting, by the detection module based on the driver image, whether the driver satisfies at least one facial feature condition, which is at least one face detection result; a driver availability determination step of determining whether the driver satisfies an availability condition, which is an availability determination result, by an availability determination module based on the at least one face detection result; an automatic driving takeover determination step of determining whether the automatic driving takeover condition is satisfied based on the availability determination result; A method for determining whether to take over automatic driving, including:

2. a vehicle body signal acquisition step of acquiring, by at least one vehicle body sensor, a plurality of vehicle body signals at the driver's seat, including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals, during the first detection time interval in the autonomous driving mode; a driver presence determination step of determining whether the driver satisfies a presence condition, which is a presence determination result, by a presence determination module based on at least one of the driver image and the vehicle body signal; Warning steps and Further comprising: The automatic driving takeover determination step includes determining whether the automatic driving takeover condition is satisfied based on the availability determination result and the presence determination result, 2. The method for determining whether to take over automatic driving as described in claim 1, wherein if the automatic driving takeover determination step determines that the automatic driving takeover conditions are not met, the warning step includes generating at least one of a visual warning, an audible warning, and a vibration warning to warn the driver.

3. a biological signal acquisition step of acquiring, by at least one biological sensor, a plurality of biological signals of the driver, including at least one of a plurality of heart rate signals and a plurality of respiratory signals, during the first detection time interval in the autonomous driving mode; a human body posture detection step of detecting, by the detection module, whether the driver satisfies a non-sleeping posture characteristic condition, which is a non-sleeping posture detection result, based on the driver image; a heart rate detection step of detecting, by the detection module, whether the driver satisfies a heart rate characteristic condition based on the biological signal, the heart rate being a heart rate detection result; The automatic driving takeover determination method according to claim 1, wherein the driver availability determination step includes determining by the availability determination module whether the driver satisfies the availability conditions, which are the availability determination results, based on the at least one face detection result, the non-sleep posture detection result, and the heart rate detection result.

4. the at least one facial feature condition is a plurality of conditions, and when the driver satisfies at least two of the facial feature conditions, or when the driver satisfies the non-sleeping posture feature condition and the heart rate feature condition, and when a sum of weights of the detection results is greater than a sum of weights threshold, the availability determination module determines that the driver satisfies the availability condition; The method for determining whether to take over automatic driving according to claim 3 , wherein the weighting coefficient of each of the face detection results is greater than the weighting coefficient of the non-sleep posture detection result and the weighting coefficient of the heart rate detection result.

5. 2. The method for determining whether to take over automatic driving as described in claim 1, wherein the driver correction parameters include data related to the driver's line of sight range, and the lateral and vertical distances between the driver and at least one target selected from the vehicle's rearview mirror, left rearview mirror, right rearview mirror, car radio, steering wheel, instrument panel, and passenger glove box.

6. an initial correction step of having an initial driver in the driver's seat perform at least one action of looking directly at the at least one target based on an initial correction operation instruction of the correction module, obtaining at least one initial correction image of the at least one action by the camera, and generating a plurality of initial correction parameters including a visual angle of the initial driver with respect to the at least one target by the correction module; The driver correction step includes generating the driver correction parameters by the correction module based on the initial correction parameters and the correction image, and the driver correction parameters include the driver's visual angle with respect to the at least one target. The automatic driving takeover determination method described in claim 5.

7. the detection module includes an eye opening detection unit, a visual angle detection unit, and a head shake detection unit, each of which is a detection algorithm based on machine learning; The face detection step includes: an eye-opening detection step of detecting whether the driver satisfies an eye-opening characteristic condition, which is an eye-opening detection result, based on the driver image by the eye-opening detection unit of the detection module; a visual angle detection step of detecting whether the driver satisfies a visual angle characteristic condition, which is a visual angle detection result, based on the driver image by the visual angle detection unit of the detection module; a head shake detection step of detecting whether the driver satisfies a head shake characteristic condition, which is a head shake detection result, based on the driver image and the head shake detection unit of the detection module; Including, The method for determining whether to take over automatic driving as described in claim 1, wherein the at least one face detection result is at least three, and the face detection result includes the eye open detection result, the visual angle detection result, and the head shake detection result.

8. the eye-open detection step includes: establishing a first vertical distance by a first eye-upper point and a first eye-lower point of each of the eyes in each of the driver images based on a Euclidean distance algorithm; and establishing a second vertical distance by a second eye-upper point and a second eye-lower point of each of the eyes; and defining an eye state feature value as the maximum value of the two first vertical distances and the two second vertical distances of the eyes; The method for determining automatic driving takeover described in claim 7, wherein the eye state characteristic value is defined as satisfying the eye open state if it is greater than an eye open threshold, the eye open threshold is determined by the eye open detection unit of the updated detection module and is within a threshold setting range, and whether the eye open state is satisfied is used to detect whether the driver satisfies the eye open characteristic condition in the first detection time interval.

9. the visual angle detection step includes identifying pupils in each of the driver images and detecting whether the driver satisfies a visual angle designation range through coordinate transformation, and whether the driver satisfies the visual angle designation range is used to detect whether the driver satisfies the visual angle characteristic condition in the first detection time interval; The head shake detection step includes identifying the tip of the nose in each of the driver images and detecting whether the driver satisfies a designated head shake range by coordinate transformation, and whether the designated head shake range is satisfied is used to detect whether the driver satisfies the head shake characteristic condition in the first detection time interval. The method for determining whether to take over automatic driving described in claim 7.

10. the first detection time interval includes a plurality of second detection time intervals, each of the second detection time intervals includes a plurality of third detection time intervals, the first detection time intervals are between 1 second and 30 seconds, each of the second detection time intervals are between 1 second and 5 seconds, and each of the third detection time intervals are between 0.5 seconds and 1 second; Each of the third detection time periods includes a plurality of detection time points, and the eye-open detection step detects whether the driver has an eye-open state at one of the detection time points, and generates an eye-open signal corresponding to one of the third detection time periods if more than half of the detection time points in one of the third detection time periods satisfy the eye-open state; An automatic driving takeover determination method as described in claim 7, wherein if the eye open signal corresponds to each of the third detection time intervals in one second detection time interval, the driver is detected to satisfy the eye open characteristic condition in the first detection time interval.

11. When the visual angle signals satisfying the visual angle designation range correspond to the respective third detection time periods in one of the second detection time periods, it is determined that the driver satisfies the visual angle characteristic condition in the first detection time period; The method for determining whether to take over automatic driving as described in claim 10, wherein if each of the third detection time intervals in one of the second detection time intervals corresponds to a head shake signal that satisfies the head shake specified range, the driver is detected as satisfying the head shake characteristic condition in the first detection time interval.

12. An automatic driving takeover determination system provided in a vehicle, an autonomous driving unit for executing an autonomous driving mode of the vehicle; at least one camera for capturing a plurality of driver images of a driver in a driver's seat within the vehicle; a processing unit including a correction module, a detection module, and an availability determination module; an in-vehicle communication network used for communication connection between the autonomous driving unit, the camera, and the processing unit; Equipped with The processing unit Before the vehicle enters the autonomous driving mode, acquiring a plurality of corrected images of the driver by the camera, and generating a plurality of driver correction parameters, which are a plurality of relative position parameters of the driver in a cab of the vehicle, by the correction module based on the corrected images; updating the detection module based on the driver compensation parameters; acquiring a plurality of driver images of the driver by the camera during a first detection time interval in the autonomous driving mode; Detecting by the detection module based on the driver image whether the driver satisfies at least one facial feature condition, which is at least one facial detection result; determining by the availability determination module whether the driver satisfies an availability condition, which is an availability determination result, based on the at least one face detection result; An automatic driving takeover determination system used to determine whether the automatic driving takeover conditions are met based on the availability determination result.

13. at least one vehicle body sensor; at least one biosensor; a warning unit; Further provided with the processing unit further includes a presence determination module; The processing unit acquiring, by the at least one biological sensor, a plurality of biological signals of the driver, including at least one of a plurality of heart rate signals and a plurality of respiratory signals, during the first detection time interval in the autonomous driving mode; Detecting whether the driver satisfies a non-sleeping posture characteristic condition, which is a non-sleeping posture detection result, based on the driver image by the detection module; Detecting by the detection module whether the driver satisfies a heart rate characteristic condition based on the biological signal, which is a heart rate detection result; determining by the availability determination module whether the driver satisfies the availability condition, which is the availability determination result, based on the at least one face detection result, the non-sleep posture detection result, and the heart rate detection result; During the first detection time interval in the autonomous driving mode, acquiring a plurality of vehicle body signals at the driver's seat, including at least some of a plurality of seat belt engagement signals and a plurality of driver's seat pressure signals, by the at least one vehicle body sensor; determining by the presence determination module whether the driver satisfies a presence condition, which is a presence determination result, based on at least one of the driver image and the vehicle body signal; The automatic driving takeover determination system of claim 12 is used to determine whether the automatic driving takeover conditions are met based on the availability determination result and the existence determination result, and if it is determined that the automatic driving takeover conditions are not met, the warning unit generates at least one of a visual warning, an audible warning, and a vibration warning to alert the driver.

14. the detection module includes an eye opening detection unit, a visual angle detection unit, and a head shake detection unit, each of which is a detection algorithm based on machine learning; The processing unit detecting whether the driver satisfies an eye-opening characteristic condition, which is an eye-opening detection result, based on the driver image by the eye-opening detection unit of the detection module; Detecting whether the driver satisfies a visual angle characteristic condition, which is a visual angle detection result, based on the driver image by the visual angle detection unit of the detection module; and detecting whether the driver satisfies a head shake characteristic condition, which is a head shake detection result, based on the driver image by the head shake detection unit of the detection module; The autonomous driving handover determination system of claim 12, wherein the at least one face detection result is at least three, and the face detection result includes the eye open detection result, the visual angle detection result, and the head shake detection result.

Images