Method and system for detecting driver handover.

Abstract

This invention provides a method and system for detecting a driver handover. [Solution] A method for detecting a driver handover in an automated driving mode, which determines whether a driver in the driver's seat of a vehicle meets the conditions for handover, comprising: an image acquisition step; a face feature detection step; a reliability determination step including determining whether the overall reliability is equal to or greater than a reliability threshold by a reliability determination module according to the face detection results; a driver availability detection step including determining whether the driver meets the availability conditions by an availability determination module if the overall reliability is equal to or greater than a reliability threshold; and a driver handover determination step. This method is applicable to varied and complex environments and scenarios.

Description

Technical Field

[0001] The present disclosure relates to a method and system for detecting driving handover, and particularly to a method and system for detecting driving handover based on a driver image.

Background Art

[0002] While the market for autonomous vehicles is growing, how to enhance safety to reduce the occurrence of accidents is a top consideration for the development of autonomous vehicles. For SAE (Society of Automotive Engineers) Level 3 autonomous vehicles, the driver does not need to hold the steering wheel under specific conditions but must have the ability to take over the driving task. Therefore, the development focus of Level 3 autonomous vehicles is not to detect the driver's concentration as in Level 0 to Level 2 autonomous vehicles, but to detect whether the driver has the awareness to take over the driving task at any time.

[0003] Based on the above, how to develop a method and system for detecting driving handover that can appropriately and accurately detect whether the driver of a Level 3 autonomous vehicle has the ability to take over the driving task has become an urgent issue to be solved in the autonomous vehicle market.

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the present disclosure, a method and system for detecting driving handover are provided. The reliability determination step of the driving handover detection method includes determining by a reliability determination module whether the overall reliability is greater than or equal to a reliability threshold according to the face detection result. The driver availability detection step of the driving handover detection method includes determining by an availability determination module whether the driver meets the availability conditions when the overall reliability is greater than or equal to the reliability threshold, and is applicable to complex environments and scenarios rich in changes in driving handover detection.

Means for Solving the Problems

[0005] According to one embodiment of the present disclosure, a method for detecting a driver handover in an automated driving mode, comprising: an image acquisition step including capturing multiple driver images of the driver with at least one camera; a face feature detection step including detecting whether the driver satisfies at least one face feature threshold using a detection module according to the driver images and obtaining at least one face detection result; a reliability determination step including determining whether the overall reliability is equal to or greater than a reliability threshold using a reliability determination module according to the face detection result; a driver availability detection step including determining whether the driver satisfies availability conditions using an availability determination module if the overall reliability is equal to or greater than a reliability threshold and obtaining an availability determination result; and a driver handover determination step including determining whether the driver handover conditions are met according to the availability determination result.

[0006] In the embodiment of the driving handover detection method described above, the detection module may include a personnel database which includes a plurality of personnel parameter sets corresponding to a plurality of known personnel, each personnel parameter set which includes a plurality of feature parameter values, and the driving handover detection method may further include a face recognition step which includes recognizing whether the driver is one of the known personnel, and a posture feature detection step which includes detecting whether the driver satisfies at least one posture feature threshold according to the driver image using the detection module and obtaining at least one posture detection result, and if the face recognition step recognizes that the driver is one of the known personnel, the face feature detection step and the posture feature detection step are executed, and the confidence determination step includes determining whether the overall confidence level is equal to or greater than a confidence threshold using the confidence determination module according to the face detection result and the posture detection result.

[0007] In the embodiment of the above-described method for detecting a driver handover, the face recognition step may further include a local training step, which is performed when the face recognition step recognizes that the driver is not one of the known personnel, and which includes training a detection module using a local training module, which is a machine learning algorithm, with the driver's image; a personnel parameter set addition step, which includes adding a personnel parameter set corresponding to the driver to the personnel database; and a personnel database update step, which includes updating the personnel database, after which the image acquisition step is performed.

[0008] In the embodiment of the driving handover detection method described above, the following steps may be performed when the overall confidence level is less than the confidence threshold in the confidence determination step: an image cloud upload step, which includes uploading driver images to a cloud server; a cloud training step, which includes training a cloud classifier used to resemble or update the detection module using a cloud training module, which is a machine learning algorithm, with the driver images; a personnel parameter set update step, which includes updating the personnel parameter set corresponding to the driver in the cloud classifier; and a personnel parameter set download step, which includes downloading the updated personnel parameter set corresponding to the driver in the cloud classifier to the detection module, wherein the confidence threshold follows the minimum confidence value of being positive in the confusion matrix.

[0009] In the embodiment of the operation handover detection method described above, the cloud training step may include fixing a portion of the feature parameter values ​​of the personnel parameter set and training the cloud classifier with the cloud training module, and the personnel parameter set update step may include updating another portion of the feature parameter values ​​of the personnel parameter set in the cloud classifier.

[0010] In the embodiment of the driver handover detection method described above, the cloud training step may include marking a driver image and determining a fixed portion of the feature parameter values ​​using the marked driver image.

[0011] In the embodiment of the above-described method for detecting the handover of driving, the image cloud upload step may be performed after the end of the automatic driving mode, or the image acquisition step may be performed after the personnel parameter set download step.

[0012] In the embodiment of the driving handover detection method described above, the detection module may include an eye-opening detection unit, a visual angle detection unit and a head yaw detection unit, and the face feature detection step includes an eye-opening detection step which includes detecting whether the driver satisfies an eye-opening feature threshold using the eye-opening detection unit according to the driver image and obtaining the eye-opening detection result, a visual angle detection step which includes detecting whether the driver satisfies a visual angle feature threshold using the visual angle detection unit according to the driver image and obtaining the visual angle detection result, and a head yaw detection unit which includes detecting whether the driver satisfies a head yaw feature threshold according to the driver image and obtaining the head The process may include a head yaw detection step, which includes obtaining a yaw detection result, and the face detection result is at least three, including an eye-opening detection result, a visual angle detection result, and a head yaw detection result. The confidence determination step includes calculating the eye-opening confidence, visual angle confidence, head yaw confidence, and posture confidence, respectively, using a confidence determination module according to the eye-opening confidence, visual angle confidence, head yaw confidence, and posture confidence, and calculating the overall confidence using the confidence determination module according to the eye-opening confidence, visual angle confidence, head yaw confidence, and posture confidence.

[0013] In the embodiment of the operation handover detection method described above, in the reliability determination step, the eye-opening reliability, visual angle reliability, head yaw reliability, and posture reliability may each have corresponding eye-opening weights, visual angle weights, head yaw weights, and posture weights, respectively, in order to calculate the overall reliability by the reliability determination module, and each of the eye-opening weight and visual angle weight is greater than each of the head yaw weight and posture weight, and the characteristic parameter values ​​of the personnel parameter set corresponding to each known personnel include eye-opening feature threshold, visual angle feature threshold, head yaw feature threshold, posture feature threshold, eye-opening weight, visual angle weight, head yaw weight, and posture weight.

[0014] In the embodiment of the driving handover detection method described above, at least one of the eye-opening feature threshold, visual angle feature threshold, head yaw feature threshold, and posture feature threshold may each have thresholds suitable for a face-unobstructed state and at least one face-obstructed state, where the face-obstructed state is when the driver's face is obscured by an object.

[0015] In the embodiment of the driver handover detection method described above, a reliability adjustment step may be performed in the reliability determination step when the overall reliability is equal to or greater than a reliability threshold, and which includes increasing or decreasing the overall reliability according to the frequency and duration of occurrence of the same or similar images among the driver images.

[0016] In an embodiment of the above-described method for detecting a driver handover, the method may further include: a vehicle body signal acquisition step, which includes acquiring a plurality of vehicle body signals on the driver's seat, including at least some of the signals from a plurality of seat belt lock signals and a plurality of driver's seat pressure signals, using at least one vehicle body sensor; a driver presence detection step, which includes determining whether the driver meets the presence conditions using a presence determination module according to at least one of the driver image and vehicle body signals, and determining whether the driver meets the presence conditions; and a warning step, which is performed when the driver handover determination step determines that the driver handover conditions are not met, and includes warning the driver by generating at least one of a visual warning, an auditory warning, and a vibration warning, wherein the driver handover determination step includes determining whether the driver handover conditions are met according to the availability determination result and the presence determination result.

[0017] According to other embodiments of the present disclosure, a driving handover detection system is provided which includes an autonomous driving unit installed in a vehicle and for executing an autonomous driving mode of the vehicle; at least one camera installed in the vehicle and for capturing multiple driver images of a driver in the driver's seat inside the vehicle; a local processing unit installed in the vehicle and including a detection module, a reliability determination module and an availability determination module; and an in-vehicle communication network installed in the vehicle for communication connectivity between the autonomous driving unit, the camera and the local processing unit. The local processing unit is used to capture a driver image of the driver by the camera, to detect whether the driver satisfies at least one facial feature threshold by the detection module according to the driver image and to obtain at least one face detection result, to determine whether the overall reliability is equal to or greater than a reliability threshold by the reliability determination module, to determine whether the driver satisfies availability conditions and to obtain an availability determination result if the overall reliability is equal to or greater than a reliability threshold, and to determine whether the driver satisfies driving handover conditions according to the availability determination result.

[0018] The embodiment of the driving handover detection system may further include a local wireless communication unit and a cloud server, the local wireless communication unit being installed in the vehicle, the in-vehicle communication network being used for communication connectivity between the autonomous driving unit, the camera, the local processing unit and the local wireless communication unit, the cloud server including a cloud processing unit and a cloud wireless communication unit, the cloud processing unit including a cloud training module and a cloud classifier and being communicated to the cloud wireless communication unit, the local processing unit and the cloud processing unit being wirelessly connected via the local wireless communication unit and the cloud wireless communication unit, and the local processing unit and the cloud wireless communication unit being used to upload driver images to the cloud server when the overall confidence level is less than a confidence threshold, and to train a cloud classifier, which is a machine learning algorithm, using the driver images, by a cloud training module. [Brief explanation of the drawing]

[0019] [Figure 1] This is a flowchart of the method for detecting a driver handover according to the first embodiment of this disclosure. [Figure 2] Figure 1 is a schematic diagram of the facial feature detection process in the driver handover detection method. [Figure 3] Figure 1 is a schematic diagram of the multitask detection architecture used in the driver handover detection method. [Figure 4] This is a block diagram of the operation handover detection system according to the second embodiment of the present disclosure. [Figure 5] Figure 4 is a schematic diagram of the driver handover detection system. [Modes for carrying out the invention]

[0020] Several embodiments of this disclosure will be described below with reference to the drawings. For clarity, many practical details will be mentioned in conjunction with the following description. However, it should be understood that these practical details are not used to limit this disclosure; that is, these practical details are not necessary in the embodiments of this disclosure. Also, for the sake of simplification of the drawings, some conventional structures and elements are briefly and schematically illustrated in the drawings. Repeating elements may also be represented by the same number.

[0021] Furthermore, terms such as "first element," "second element," etc., are merely used to describe different elements and do not imply any restriction on the elements themselves; therefore, "first element" can be interpreted as "second element." Also, the combinations of elements described herein are not common, customary, or well-known combinations in the art, and whether the elements themselves are well-known cannot be used to determine whether the combination relationship can be easily achieved by a person skilled in the art.

[0022] FIG. 1 is a flowchart of a driving takeover detection method 100 according to a first embodiment of the present disclosure, FIG. 2 is a schematic diagram of a face feature detection step 130 in the driving takeover detection method 100 of FIG. 1, FIG. 3 is a schematic diagram of a multi-task detection architecture in the driving takeover detection method 100 of FIG. 1, FIG. 4 is a block diagram of a driving takeover detection system 200 according to a second embodiment of the present disclosure, and FIG. 5 is a schematic diagram of the driving takeover detection system 200 of FIG. 4 (not drawn based on the actual ratio). Please refer to FIGS. 1 to 5. The driving takeover detection system 200 of the second embodiment is used to assist in explaining the driving takeover detection method 100 of the first embodiment, and it can be understood that the driving takeover detection method 100 according to the present disclosure is not limited to implementation in the driving takeover detection system 200, and the driving takeover detection system 200 according to the present disclosure is not limited to the use of the driving takeover detection method 100. The driving takeover detection method 100 of the first embodiment is used to determine whether a driver (for example, driver 400) in the driver's seat 241 in the vehicle 210 satisfies the driving takeover condition in the autonomous driving mode, and includes an image capture step 120, a face feature detection step 130, a confidence determination step 140, a driver availability detection step 160, and a driving takeover determination step 170.

[0023] The image acquisition step 120 includes acquiring driver images 181 of the driver 400 in multiple frames using at least one camera 214. Specifically, the driver images 181 acquired by the camera 214 may be infrared night vision images and have a spontaneous infrared supplemental lighting function so that they can be used in two scenarios: daytime and nighttime when there is insufficient light. The camera 214 may also be a depth camera. The face feature detection step 130 includes detecting, according to the driver images 181, whether the driver 400 satisfies at least one face feature threshold (threshold value), which is at least one face detection result 188, using the detection module 230. The confidence determination step 140 includes determining, according to the face detection results 188, whether the overall confidence level is equal to or greater than the confidence threshold using the confidence determination module 224. The driver availability detection step 160 includes determining (detecting) whether the driver 400 meets the availability conditions using the availability determination module 227 if the overall reliability is equal to or greater than the reliability threshold, and obtaining this as the availability determination result. The operation handover determination step 170 includes determining whether the operation handover conditions are met according to the availability determination result. Thus, the operation handover detection method 100 according to this disclosure updates the reliability as a model update decision criterion in order to be applied to varied and complex environments and scenarios.

[0024] Specifically, the detection module 230 may include a personnel database 239 that includes a plurality of personnel parameter sets corresponding to a plurality of known personnel, respectively, and each personnel parameter set includes a plurality of feature parameter values. The driving handover detection method 100 may further include a face recognition step 122 and a posture feature detection step 136. The face recognition step 122 includes recognizing whether the driver 400 is one of the known personnel. The posture feature detection step 136 includes detecting whether the driver 400 satisfies at least one posture feature threshold by the posture detection unit 232 of the detection module 230 according to the driver image 181, and obtaining at least one posture detection result. Specifically, the posture feature threshold may include a head placement angle threshold, a torso placement angle threshold, and a hand posture threshold, and the posture detection result may include a head placement angle detection result, a torso placement angle detection result, and a hand posture detection result. When it is recognized in the face recognition step 122 that the driver 400 is one of the known personnel, the face feature detection step 130 and the posture feature detection step 136 are executed. The reliability determination step 140 includes determining whether the overall reliability is greater than or equal to the reliability threshold by the reliability determination module 224 according to the face detection result 188 and the posture detection result. Thereby, it can be further determined whether the driver 400 is in a waking state or a distracted state by the face feature detection step 130 and the posture feature detection step 136. For example, in the face feature detection step 130 and the posture feature detection step 136, it is detected whether the driver 400 continues to maintain the same posture for a preset time (which may be between 5 seconds and 10 minutes) or satisfies the nodding situation. If satisfied, it is determined that the driver 400 is not in a waking state; if not satisfied, it is determined that the driver 400 is in a waking state. When it is determined that the driver 400 is in a waking state, important feature points such as the driver 400's left eye, right eye, left ear, right ear, nose, left shoulder, right shoulder, etc. are further detected to determine whether the driver 400 is in a distracted state.

[0025] The driver handover detection method 100 may further include a local training step 124, a personnel parameter set addition step 126, and a personnel database update step 128. If the face recognition step 122 recognizes that the driver 400 is not one of the known personnel, the local training step 124 is performed, which includes training the detection module 230 with a local training module 223, which is a machine learning algorithm, using the driver image 181. The personnel parameter set addition step 126 includes adding a personnel parameter set corresponding to the driver 400 to the personnel database 239. The personnel database update step 128 includes updating the personnel database 239. After the personnel database update step 128, the image acquisition step 120 is performed. As a result, the local training process 124 trains and calculates feature thresholds using a small multitask model, performing personalized parameter learning on the local side, and the thresholds for face (e.g., visual angle, head yaw angle, etc.) and posture differ for different drivers during their driving periods. Possible factors influencing these thresholds include the driver's height, body type, face shape, seating habits, driving behavior, or the vehicle's mechanical design, mounting position, etc.

[0026] To enable the driver handover detection method 100 to accommodate various drivers and to eliminate the need to manually adjust thresholds for each driver, if the face recognition process 122 recognizes that driver 400 is not one of the known personnel, the driver handover detection system 200 instructs driver 400 by voice or image to look at a predetermined location in the cockpit, such as the rearview mirror, left rearview mirror, right rearview mirror, in-vehicle infotainment, steering wheel, dashboard, passenger glove box, etc. (but not limited to these), and captures the visual angle, head yaw angle, etc. detected within a certain time, sets the maximum or minimum value as a threshold based on the detected type, and further adds a personnel parameter set corresponding to driver 400 to the personnel database 239 through the local training process 124 and the personnel parameter set addition process 126. Furthermore, the locally personalized parameter learning in the driver handover detection method 100 is performed for the driver 400 and is intended to improve the driver 400's availability detection. It is trained locally and can be used only with vehicle 210 without being shared with other vehicles. In addition, the locally personalized parameter learning in the driver handover detection method 100 is performed by registering personnel information in the personnel parameter database on the cloud server 270 (not in-vehicle) and combining it with the vehicle parameter database in vehicle 210 (e.g., mounting position of camera 214, seat information) and Over The Air (OTA) technology. This allows the driver 400 to achieve accuracy in driver availability detection across different vehicles.

[0027] The detection module 230 may include an eye-opening detection unit 235, a visual angle (line of sight) detection unit 236, and a head yaw detection unit 237. The face feature detection step 130 may include an eye-opening detection step 131, a visual angle detection step 132, and a head yaw detection step 133. The eye-opening detection step 131 includes detecting whether the driver 400 meets an eye-opening feature threshold using the eye-opening detection unit 235 according to the driver image 181, and obtaining the eye-opening detection result. The visual angle detection step 132 includes detecting whether the driver 400 meets a visual angle feature threshold using the visual angle detection unit 236 according to the driver image 181, and obtaining the visual angle detection result. The head yaw detection step 133 includes detecting whether the driver 400 meets a head yaw feature threshold using the head yaw detection unit 237 according to the driver image 181, and obtaining the head yaw detection result. Furthermore, the head yaw detection step 133 in the face feature detection step 130 is calculated using face feature points, and the head position detection step in the posture feature detection step 136 is calculated based on the relationship between the head feature points and the torso of the human body, for example, based on the relationship between the head feature points and the shoulders. The face detection result 188 consists of at least three results, including eye open detection results, visual angle detection results, and head yaw detection results. The confidence determination step 140 includes calculating the eye open confidence, visual angle confidence, head yaw confidence, and posture confidence, respectively, using the confidence determination module 224 according to the eye open confidence, visual angle confidence, head yaw confidence, and posture confidence, and also calculating the overall confidence using the confidence determination module 224 according to the eye open confidence, visual angle confidence, head yaw confidence, and posture confidence. This allows the camera 214 and the detection module 230 (artificial intelligence multitasking detection model) to obtain multiple physiological signs of the driver 400 (face, eye condition, gaze, head posture, body posture, etc.), enabling the development of a personalized detection model for the driver 400 that can be applied to varied and complex environments and scenarios, improving safety in the handover of driving tasks in the Level 3 autonomous driving system.

[0028] Please refer to Figure 3. This is a multi-task detection architecture used in the driver handover detection method 100, which includes a Feature Extraction Backbone network 182, a head prediction branch 184, and an eye prediction branch 192. The head prediction branch 184 corresponds to the face feature detection process 130 and the posture feature detection process 136, and the eye prediction branch 192 corresponds to the face feature detection process 130. Furthermore, the Feature Extraction Backbone Network 182 in the driver handover detection method 100 performs lightweight feature extraction using MobileNet V2 and connects each prediction branch to the feature extraction block 183 of the Feature Extraction Backbone Network 182 to obtain the features necessary for task prediction. After obtaining the driver image 181 by the image acquisition process 120, head cue features 185 and eye cue features 193 are extracted to perform predictions for various tasks, such as the prediction of head posture and important points of the face. In the process, feature aggregation modules 186 and 196 and cue feature interaction module 187 merge the different stage features of the two prediction branches from the feature extraction backbone network 182, and finally input the prediction modules for their respective downstream tasks and output the face detection results 188.

[0029] Please refer to Figures 1, 2, 4, and 5. In the confidence determination step 140, the eye-opening confidence, visual angle confidence, head yaw confidence, and posture confidence may each have corresponding eye-opening weights, visual angle weights, head yaw weights, and posture weights, respectively, in order for the confidence determination module 224 to calculate the overall confidence, and each of the eye-opening weight and visual angle weight is greater than each of the head yaw weight and posture weight. The feature parameter values ​​of the personnel parameter set corresponding to each known person include eye-opening feature threshold, visual angle feature threshold, head yaw feature threshold, posture feature threshold, eye-opening weight, visual angle weight, head yaw weight, and posture weight. Thus, since eye-opening and visual angle are related to safe driving behavior, both the eye-opening weight and visual angle weight need to be greater than the head yaw weight and posture weight to ensure the accuracy of the calculated confidence. Furthermore, the eye-opening weight, visual angle weight, head yaw weight, and posture weight are each between 0.2 and 0.6, the eye-opening weight may be greater than, equal to, or less than the visual angle weight, and the sum of the eye-opening weight, visual angle weight, head yaw weight, and posture weight is 1.

[0030] For example, the head yaw feature threshold is when the head yaw angle is between 5 degrees and 5 degrees (0 degrees indicates that the driver 400's head is facing directly forward), the visual angle feature threshold is when the visual angle is between 45 degrees and 45 degrees (0 degrees indicates that the driver 400 is looking directly forward, and a visual angle pointing to the left or right may be defined as the positive direction). The eye-opening confidence score for satisfying and not satisfying the eye-opening feature threshold is 1 point and 0 points, respectively (there may be more score grades), the visual angle confidence score for satisfying and not satisfying the visual angle feature threshold is 1 point and 0 points, respectively, the head yaw confidence score for satisfying and not satisfying the head yaw feature threshold is 1 point and 0 points, respectively, and the posture confidence score for satisfying and not satisfying the posture feature threshold is between 0 and 1 point and there are various score grades. The eye-opening weight and visual angle weight are both 0.3, and the head yaw weight and posture weight are both 0.2. The formula "Overall confidence = eye-opening weight × eye-opening confidence + visual angle weight × visual angle confidence + head yaw weight × head yaw confidence + posture weight × posture confidence" may be defined, and the confidence threshold may be between 0.2 and 0.6, or specifically 0.4.

[0031] At least one of the eye-opening feature threshold, visual angle feature threshold, head yaw feature threshold, and posture feature threshold may have thresholds suitable for a face-unobstructed state and at least one face-obstructed state, where the face-obstructed state is when the driver's face 400 is obscured by an object. Thus, the driving handover detection method 100 has an online deep learning mechanism that can solve the problem of losing the driver's features due to the light source or object obstruction of the camera 214, and can detect different drivers, improve accuracy in daytime, nighttime, low light, profile, and with face accessories (e.g., masks, glasses, sunglasses, hats, etc., but not limited to these), and can be applied to varied and complex environments and scenarios, improving safety in the handover of driving tasks in a Level 3 autonomous driving system.

[0032] Refer to Figures 1, 4, and 5. The driver handover detection method 100 may further include an expiration notification step 144, an image cloud upload step 146, a cloud training step 148, a personnel parameter set update step 152, and a personnel parameter set download step 154. If the overall confidence level in the confidence level determination step 140 is less than the confidence level threshold, the expiration notification step 144 and the image cloud upload step 146 are executed. The expiration notification step 144 includes notifying the driver 400 by the warning unit 216 of the vehicle 210 that the driver handover detection system 200 is not functioning properly, using at least one of visual, auditory, and vibrational methods. For example, the visual method may be a text message displayed on the in-vehicle infotainment screen, the auditory method may be a warning sound effect or warning voice, and the vibrational method may be, but is not limited to, vibration of the driver's seat or vibration of the steering wheel. The image cloud upload step 146 includes encrypting the driver image 181 and uploading it to the cloud server 270. The cloud training process 148 includes training a cloud classifier 284, which is used to resemble or update the detection module 230, using driver images 181 with a cloud training module 283, which is a machine learning algorithm. The personnel parameter set update process 152 includes updating the personnel parameter set corresponding to the driver 400 in the cloud classifier 284. The personnel parameter set download process 154 includes downloading the updated personnel parameter set corresponding to the driver 400 in the cloud classifier 284 to the detection module 230 using over-the-air technology. The confidence threshold also follows the minimum confidence value of positive in the confusion matrix. This allows the model to be updated for varied and complex environments and scenarios.Furthermore, the "Multitasking Model Online / Continuous Learning" method 100 for detecting driver handover primarily detects human characteristics, and therefore, regardless of vehicle type or driver, its training can be performed locally or in the cloud. The parameters of the multitasking model are shared through cloud updates and over-the-air technology, allowing all vehicles to adapt to various scenarios, such as changing light or (severely) obscured faces.

[0033] For example, the driver handover detection method 100 disclosed herein can achieve the following: If the driver 400 is in a concentrated driving state required for Level 3 and is facing the camera 214 with their face, the overall confidence level determined in the confidence level determination step 140 is equal to or greater than the confidence threshold. If the driver 400 is not in a concentrated driving state required for Level 3 and is facing the camera 214 with their face, the overall confidence level determined in the confidence level determination step 140 is less than the confidence threshold, and the image cloud upload step 146 and cloud training step 148 are performed in order thereafter, and it is determined in the personnel parameter set update step 152 that there is no need to update the personnel parameter set corresponding to the driver 400 in the cloud classifier 284.

[0034] The cloud training process 148 may include training the cloud classifier 284 by the cloud training module 283 with a fixed portion of the feature parameter values ​​of the personnel parameter set, and the personnel parameter set update process 152 may include updating another portion of the feature parameter values ​​of the personnel parameter set in the cloud classifier 284. Thus, the driving handover detection method 100 according to this disclosure primarily uses a camera 214 and artificial intelligence technology to detect the state of the driver 400, and by adding an online deep learning mechanism, optimizes the driving state detection technology in the cockpit, improves its discrimination accuracy, and develops a personalized detection model for the driver 400 (i.e., the personnel parameter set in the personnel database 239), which can be applied to varied and complex environments and scenarios, improving safety in the handover of driving tasks in Level 3 autonomous vehicle autonomous driving systems.

[0035] The cloud training process 148 may include marking the driver images 181 and determining a fixed portion of the feature parameter values ​​using the marked driver images 181. This is effective in fine-tuning network parameters for small samples. Furthermore, the cloud training process 148 uses continuous learning techniques, such as experience recall, to learn new information and reduce the occurrence of catastrophic forgetting.

[0036] The driver handover detection method 100 may further include a reliability adjustment step 142. If the overall reliability in the reliability determination step 140 is equal to or greater than the reliability threshold, the reliability adjustment step 142 is executed, and the reliability adjustment step 142 includes increasing or decreasing the overall reliability according to the frequency and duration of the occurrence of the same or similar images in the driver images 181. This improves the accuracy of the subsequent driver availability detection step 160 and the driver handover determination step 170.

[0037] Refer to Figures 1, 4, and 5. The driver handover detection method 100 may further include a vehicle signal acquisition step 110, a driver presence detection step 112, and a warning step 172. The vehicle signal acquisition step 110 includes acquiring a plurality of vehicle signals on the driver's seat 241, including signals from a plurality of seat belt lock signals and a plurality of driver's seat pressure signals, by at least one vehicle sensor 215 of the vehicle 210. Furthermore, the vehicle sensor 215 may, but is not limited to, be attached to the driver safety buckle or to the driver's seat. If the vehicle sensor 215 is attached to the driver's seat, it may be attached to the seat cushion, the seat back, or a combination thereof, and is not limited to being attached to the inner layer or surface of the driver's seat. The driver presence detection step 112 includes determining (detecting) whether the driver 400 meets the presence conditions by the presence determination module 228 according to at least one of the driver image 181 and the vehicle signals, and obtaining the presence determination result. The driver handover determination step 170 includes determining whether the driver handover conditions are met according to the availability determination result and the existence determination result. If the driver handover determination step 170 determines that the driver handover conditions are not met, the warning step 172 is executed. In other words, if either the availability determination result or the existence determination result does not meet the driver handover conditions, the warning step 172 is executed. The warning step 172 includes warning the driver 400 by generating at least one of a visual warning, an auditory warning, and a vibration warning using the warning unit 216 of the vehicle 210. If, after executing the warning step 172, the driver 400 determines that they do not have the ability to control the vehicle 210, the automatic driving unit 213 enters the "minimum risk control mechanism," which means slowly braking the vehicle 210 to a stop or pulling it over to the side of the road and stopping it. In the embodiments of this disclosure (not shown), the driver handover detection method includes determining whether the driver maintains a state of concentration on driving when switching from manual driving mode to automatic driving mode. If it is determined that the driver is not maintaining concentration while driving, a warning process will be executed, and the system will be prevented from switching to autonomous driving mode.

[0038] After the autonomous driving mode ends or the engine stops, the image cloud upload process 146 can be executed. After the personnel parameter set download process 154, the image acquisition process 120 can be executed. This allows the system to capture and store a picture if the overall confidence level of the human body posture and facial feature parameters is lower than the confidence threshold, and to re-enter the picture of this frame into the cloud learning system to achieve the effect of online learning and increase the detection accuracy of posture and facial features, for example, by strengthening the detection accuracy and confidence of faces other than frontal faces. Furthermore, in the local training process 124, training and calculating the feature threshold using a small multitask model helps in storing the facial feature threshold and the weights of the personalized multitask model in the current driver number in subsequent processes.

[0039] Please refer to Figures 4 and 5. The driving handover detection system 200 according to the second embodiment of this disclosure includes an autonomous driving unit 213, a camera 214, a local processing unit 220, and an in-vehicle communication network 211. The autonomous driving unit 213 is installed in the vehicle 210 and is used to execute the autonomous driving mode of the vehicle 210. The camera 214 is installed in the vehicle 210 and is used to capture multiple driver images 181 of a driver 400 in the driver's seat 241 inside the vehicle 210. The local processing unit 220 is installed in the vehicle 210 and includes a local training module 223, a reliability determination module 224, an availability determination module 227, a presence determination module 228, and a detection module 230. The in-vehicle communication network 211 is installed in the vehicle 210 and is used for communication connectivity between the autonomous driving unit 213, the camera 214, the vehicle sensor 215, the warning unit 216, and the local processing unit 220. The local processing unit 220 captures multiple driver images 181 of the driver 400 using the camera 214, detects whether the driver 400 meets at least one facial feature threshold using the detection module 230 according to the driver images 181, and uses this as the face detection result 188. The confidence determination module 224 then determines whether the overall confidence level is equal to or greater than the confidence threshold according to the face detection result 188. If the overall confidence level is equal to or greater than the confidence threshold, it determines whether the driver 400 meets the availability condition, uses this as the availability determination result, and uses this to determine whether the driver handover condition is met according to the availability determination result. Thus, the driver handover detection system 200 according to this disclosure executes the image acquisition step 120, face feature detection step 130, confidence determination step 140, driver availability detection step 160, and driver handover determination step 170 of the driver handover detection method 100, and can update the confidence level as a model update discrimination criterion to be applied to varied and complex environments and scenarios.

[0040] The driver handover detection system 200 may further include a local wireless communication unit 217 and a cloud server 270. The local wireless communication unit 217 is provided in the vehicle 210, and the in-vehicle communication network 211 is used for communication between the autonomous driving unit 213, the camera 214, the local processing unit 220, and the local wireless communication unit 217. The cloud server 270 includes a cloud processing unit 280 and a cloud wireless communication unit 287, the cloud processing unit 280 includes a cloud training module 283 and a cloud classifier 284, and is communicated with the cloud wireless communication unit 287, and the local processing unit 220 and the cloud processing unit 280 are wirelessly communicated with each other via the local wireless communication unit 217 and the cloud wireless communication unit 287. The local processing unit 220 and the cloud wireless communication unit 287 are used to upload the driver image 181 to the cloud server 270 when the overall confidence level is less than the confidence threshold, and to train the cloud classifier 284, which is used to make the detection module similar to or to update the detection module, using the driver image 181 with the cloud training module 283, which is a machine learning algorithm. This allows the driver handover detection system 200 according to this disclosure to perform the image cloud upload step 146 and the cloud training step 148 of the driver handover detection method 100, which helps the detection module 230 update its model for varied and complex environments and scenarios.

[0041] Further details of the operation handover detection system 200 of the second embodiment can be found in the description of the operation handover detection method 100 of the first embodiment, and are omitted here.

[0042] Although the present invention has been disclosed in the embodiments described above, these are not intended to limit the invention, and any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be based on the claims that are subsequently appended. [Explanation of Symbols]

[0043] 100: Method for detecting driver handover 110: Vehicle body signal acquisition process 112: Driver presence detection process 120: Image acquisition process 122: Face Recognition Process 124: Local Training Process 126: Personnel parameter set addition process 128: Personnel Database Update Process 130: Facial Feature Detection Process 131: Eye opening detection process 132: Visual angle detection process 133: Head yaw detection process 136: Pose feature detection process 140: Confidence Determination Process 142: Reliability adjustment process 144: Expiration Notification Process 146: Image Cloud Upload Process 148: Cloud Training Process 152: Personnel parameter set update process 154: Personnel parameter set download process 160: Driver availability detection process 170: Operation handover determination process 172: Warning process 181: Driver's image 182: Feature Extraction Backbone Network 183: Feature Extraction Block 184: Head Prediction Branch 185: Head Clues and Characteristics 186, 196: Feature aggregation module 187: Clue Feature Interaction Module 188: Face detection results 192: Ocular Prediction Branch 193: Eye-related clue features 200: Driver handover detection system 210: Vehicle 211: In-vehicle communication network 213: Autonomous driving unit 214: Camera 215: Vehicle sensor 216: Warning Unit 217: Local Wireless Communication Unit 220: Local Processing Unit 223: Local training module 224: Confidence Determination Module 227: Availability Determination Module 228: Existence Determination Module 230: Detection Module 232: Attitude detection unit 235: Eye opening detection unit 236: Viewing angle detection unit 237: Head yaw detection unit 239: Personnel Database 241: Driver's seat 270: Cloud Server 280: Cloud Processing Unit 283: Cloud Training Module 284: Cloud Classifier 287: Cloud Wireless Communication Unit 400: Driver

Claims

1. A method for detecting a driver handover in order to determine whether the driver in the driver's seat inside the vehicle meets the conditions for taking over driving in autonomous driving mode, An image acquisition step of capturing multiple driver images of the driver using at least one camera, A face feature detection step in which a detection module detects whether the driver satisfies at least one face feature threshold according to the plurality of driver images, and obtains at least one face detection result, A confidence determination step in which a confidence determination module determines whether the overall confidence level is equal to or greater than a confidence threshold according to at least one of the face detection results, If the overall reliability is equal to or greater than the reliability threshold, the availability determination module determines whether the driver meets the availability conditions, and the driver availability detection step is to obtain the availability determination result. A drive handover determination step that determines whether the conditions for operation handover are met according to the availability determination result, A method for detecting a driver handover, including the method for detecting a driver handover.

2. The detection module includes a personnel database, each containing multiple personnel parameter sets corresponding to multiple known personnel, each of which contains multiple characteristic parameter values, and the operation handover detection method is A facial recognition step to determine whether the driver is one of the multiple known persons, A posture feature detection step in which a detection module detects whether the driver satisfies at least one posture feature threshold according to the plurality of driver images, and obtains at least one posture detection result, It further includes, If the face recognition step recognizes that the driver is one of the multiple known persons, the face feature detection step and the posture feature detection step are executed. The method for detecting a driver handover according to claim 1, wherein the reliability determination step includes determining whether the overall reliability is equal to or greater than the reliability threshold by the reliability determination module in accordance with the at least one face detection result and the at least one posture detection result.

3. A local training step is performed when the face recognition step recognizes that the driver is not one of the multiple known persons, and includes training the detection module using the multiple driver images with a local training module which is a machine learning algorithm, A personnel parameter set addition step, which adds a personnel parameter set corresponding to the driver to the personnel database, A personnel database update process for updating the aforementioned personnel database, It further includes, The method for detecting a driver handover according to claim 2, wherein the image acquisition step is performed after the personnel database update step.

4. An image cloud upload step is performed when the overall reliability is less than the reliability threshold in the reliability determination step, and includes uploading the multiple driver images to a cloud server. A cloud training step involves using the aforementioned multiple driver images to train a cloud classifier, which is a machine learning algorithm, using a cloud training module to train a cloud classifier that is similar to or used to update the detection module. A personnel parameter set update step, which updates the personnel parameter set corresponding to the driver in the cloud classifier, A personnel parameter set download step, which downloads the updated personnel parameter set corresponding to the driver in the cloud classifier to the detection module, It further includes, The method for detecting a driver handover according to claim 2, wherein the confidence threshold is determined to be the minimum confidence value of positive in the confusion matrix.

5. The cloud training process includes fixing some of the feature parameter values ​​of the personnel parameter set and training the cloud classifier with the cloud training module, The method for detecting a driver handover according to claim 4, wherein the personnel parameter set update step includes updating another portion of the plurality of feature parameter values ​​of the personnel parameter set in the cloud classifier.

6. The driver handover detection method according to claim 5, further comprising the cloud training step of marking the plurality of driver images and determining a fixed portion of the plurality of feature parameter values ​​using the marked plurality of driver images.

7. After the end of the aforementioned autonomous driving mode, the image cloud upload process is executed. The method for detecting a driver handover according to claim 4, wherein the image acquisition step is performed after the personnel parameter set download step.

8. The detection module includes an eye-opening detection unit, a visual angle detection unit, and a head yaw detection unit. The aforementioned facial feature detection step is: An eye-opening detection step in which the eye-opening detection unit detects whether the driver satisfies the eye-opening characteristic threshold according to the plurality of driver images, and takes the eye-opening detection result as follows: A visual angle detection step in which the visual angle detection unit detects whether the driver satisfies the visual angle feature threshold according to the plurality of driver images, and takes the result as a visual angle detection result, A head yaw detection step in which the head yaw detection unit detects whether the driver satisfies the head yaw characteristic threshold according to the plurality of driver images, and takes the result as a head yaw detection result, Includes, The aforementioned at least one face detection result is at least three, and includes the eye open detection result, the visual angle detection result, and the head yaw detection result. The method for detecting a driver handover according to claim 2, wherein the reliability determination step includes calculating the eye-opening reliability, visual angle reliability, head yaw reliability and posture reliability, respectively, by the reliability determination module according to the eye-opening detection result, the visual angle detection result, the head yaw detection result and the at least one posture detection result, and calculating the overall reliability by the reliability determination module according to the eye-opening reliability, the visual angle reliability, the head yaw reliability and posture reliability.

9. In the confidence determination step, the eye-opening confidence, the visual angle confidence, the head yaw confidence, and the posture confidence each have corresponding eye-opening weights, visual angle weights, head yaw weights, and posture weights, respectively, in order to calculate the overall confidence using the confidence determination module, and each of the eye-opening weight and the visual angle weight is greater than each of the head yaw weight and the posture weight. The method for detecting a driver handover according to claim 8, wherein the plurality of feature parameter values ​​of the personnel parameter set corresponding to each known personnel include the eye-opening feature threshold, the visual angle feature threshold, the head yaw feature threshold, the at least one posture feature threshold, the eye-opening weight, the visual angle weight, the head yaw weight, and the posture weight.

10. The method for detecting a driver handover according to claim 9, wherein at least one of the eye-opening feature threshold, the visual angle feature threshold, the head yaw feature threshold, and the at least one posture feature threshold each has a threshold suitable for a face-unobstructed state and at least one face-obstructed state, and the face-obstructed state is a state in which the driver's face is obscured by an object.

11. The method for detecting a driver handover according to claim 1, further comprising a reliability adjustment step, which is performed when the overall reliability in the reliability determination step is equal to or greater than the reliability threshold, and which includes increasing or decreasing the overall reliability according to the occurrence rate and duration of identical or similar images among the plurality of driver images.

12. A vehicle body signal acquisition step of acquiring multiple vehicle body signals on the driver's seat, including at least some of the signals from multiple seat belt lock signals and multiple driver's seat pressure signals, by at least one vehicle body sensor, A driver presence detection step in which a presence determination module determines whether the driver meets the presence conditions according to at least one of the plurality of driver images and the plurality of vehicle body signals, and takes the presence determination result as follows: A warning step is performed when the driver handover determination step determines that the driver handover conditions are not met, and which generates at least one of a visual warning, an auditory warning, and a vibration warning to warn the driver. It further includes, The operation handover determination step includes determining whether the operation handover conditions are met according to the availability determination result and the existence determination result, as described in the operation handover determination step.

13. An automatic driving unit provided in the vehicle and for executing the automatic driving mode of the vehicle, The vehicle is provided with at least one camera for capturing multiple driver images of the driver in the driver's seat inside the vehicle, A local processing unit provided in the vehicle, which includes a detection module, a reliability determination module, and an availability determination module, An in-vehicle communication network provided in the vehicle, through which the automatic driving unit, the at least one camera, and the local processing unit are connected for communication, Includes, The aforementioned local processing unit is The process involves capturing the multiple driver images of the driver using at least one camera, The detection module detects whether the driver satisfies at least one facial feature threshold according to the plurality of driver images, and obtains at least one face detection result. The confidence level determination module determines whether the overall confidence level is equal to or greater than the confidence level threshold based on at least one of the face detection results, If the overall reliability is equal to or greater than the reliability threshold, the system determines whether the operator meets the availability conditions and uses this as the availability determination result. In accordance with the availability determination result, it is determined whether the conditions for taking over operations are met, A driver handover detection system used in [the application].

14. It further includes a local wireless communication unit and a cloud server, The local wireless communication unit is provided in the vehicle, and the in-vehicle communication network is used for communication connectivity between the autonomous driving unit, the at least one camera, the local processing unit, and the local wireless communication unit. The cloud server includes a cloud processing unit and a cloud wireless communication unit, the cloud processing unit includes a cloud training module and a cloud classifier, and is connected to the cloud wireless communication unit, the local processing unit and the cloud processing unit are wirelessly connected to each other via the local wireless communication unit and the cloud wireless communication unit. The local processing unit and the cloud wireless communication unit are, If the overall confidence level is less than the confidence threshold, the multiple driver images are uploaded to the cloud server. Using the aforementioned multiple driver images, the cloud training module, which is a machine learning algorithm, trains the cloud classifier used to be similar to or to update the detection module. The operation handover detection system according to claim 13, used in the above.

Images