Visual and auditory concentration judgment method suitable for driver of vehicle, host and driver monitoring system

Abstract

The invention discloses a visual and auditory concentration judgment method suitable for a driver of a vehicle, a host and a driver monitoring system. The method comprises the following steps: judging the visual concentration degree of the driver according to an image of the head of the driver obtained by a camera device in the vehicle; judging the cognitive concentration degree of the driver according to the sound in the vehicle obtained by a sound receiving device in the vehicle; according to the visual concentration degree and the cognitive concentration degree, whether a prompt is given to the driver or not is judged; and executing one or any combination of the following steps when the driver is judged to be prompted: enabling a display device in the vehicle to give out a visual prompt; and making a loudspeaking device in the carrier give out an auditory prompt.

Description

[0001] priority

[0002] This Chinese invention patent application claims the priority of a Taiwan patent with an application date of August 18, 2023 and application number 112131108. This patent application combines the technical solution of the above-mentioned Taiwan patent. Technical Field

[0003] The present invention relates to the field of driving safety, and in particular to a monitoring and feedback mechanism for a driver's concentration, and specifically to a method for judging the visual and auditory concentration of a driver of a vehicle, a host, and a driver monitoring system. Background Art

[0004] There have been significant advances in recent technology developments for assisted or autonomous vehicle driving. Improvements in the yield of systems on a chip have led to significant advances in vehicle cognition, sensing, and control systems, making it possible to achieve more reliable and safe fully automated vehicles. However, before this ideal can be achieved, the presence of a driver is necessary.

[0005] When ensuring vehicle safety, one of the important factors is the concentration of the vehicle driver, which includes both cognitive and visual aspects. Analysis of accident reports in Europe, the United States and Australia between 2019 and 2021 shows that approximately 17% of fatal vehicle accidents are caused by driver distraction and / or inattention. According to the recommendations of the Centers for Disease Control and Prevention, driver distraction can be classified into three types: visual distraction (line of sight deviation events), control distraction (hand deviation events), and psychological distraction (focus deviation events). Examples of driver lack of concentration include interacting with passengers, using the phone, looking inside or outside the vehicle, eating, drinking, singing or dancing, putting on makeup, smoking, operating devices, and picking up items. The above behaviors lead to slower reaction times, slowing down appropriate planning and control decisions, etc., highlighting the need for a driver monitoring system (DMS). Based on events detected by the sensors, the driver monitoring system is used to generate feedback to the driver, which may include audible, visual and / or tactile warning messages, to inform them when the driver's attention level falls below a certain range.

[0006] Conventional driver monitoring systems only include optical sensors. However, in low light conditions, it is difficult to determine the driver's concentration. Therefore, there is an urgent need for a driver monitoring system, device, and method that can improve the above shortcomings, so as to better monitor the driver and give a warning when his concentration level is low. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for judging the visual and auditory concentration of a vehicle driver, a host and a driver monitoring system, which aims to solve the problem that the traditional driver monitoring system is difficult to judge the driver's concentration in low light conditions.

[0008] The technical solution of the present invention is as follows:

[0009] According to one embodiment of the present invention, a method for determining the visual and auditory concentration of a driver of a vehicle is provided, comprising: determining the driver's visual concentration based on an image of the driver's head obtained by a camera device in the vehicle; determining the driver's cognitive concentration based on sounds inside the vehicle obtained by a sound receiving device in the vehicle; determining whether to issue a prompt to the driver based on the above-mentioned visual concentration and the above-mentioned cognitive concentration; and when the above-mentioned determination requires issuing a prompt to the driver, executing one of the following steps or any combination thereof: causing a display device in the vehicle to issue a visual prompt; and causing a speaker device in the vehicle to issue an auditory prompt.

[0010] Preferably, in order to save computing resources and time, the above-mentioned step of determining the driver's visual concentration level further includes: using a neural network to detect the face on the image and annotate multiple facial features; estimating the facial vector of the driver's head using the positions of the multiple facial features; and when the facial vector is outside a range, determining that the driver's visual concentration level is poor.

[0011] Preferably, in order to save computing resources and time, when the facial vector is within the range, it is determined whether the driver's eyes are open based on the multiple facial features; and when it is determined that the driver's eyes are closed, it is determined that the driver's visual concentration is poor.

[0012] Preferably, in order to calculate where the driver's pupils are looking, when it is determined that the driver's two eyes are open, the positions of the two pupils are marked and the corresponding two pupil vectors are estimated; and based on the face vector and the two pupil vectors, the driver's visual concentration is determined.

[0013] Preferably, in order to represent a facial vector and / or a pupil vector, wherein the facial vector is known based on the relationship between positions of a plurality of features of the head model and positions of the plurality of facial features of the image, the facial vector is represented by three Euler angles.

[0014] Preferably, in order to represent the face vector and / or the pupil vector, the pupil vector is represented by three Euler angles.

[0015] Preferably, in order to reduce high-frequency changes in facial vectors or Euler angles, the step of determining the driver's visual concentration further comprises: filtering the facial vector through one or any combination of the following: a time series filter; a Kalman filter; and a low-pass filter.

[0016] Preferably, in order to obtain clear images in low light conditions, the images are captured using infrared wavelengths.

[0017] Preferably, in order to give the driver the best chance of seeing the visual prompt, the display device comprises a heads-up display in front of the driver.

[0018] Preferably, in order to have the best chance for the driver to hear the auditory prompt, the above-mentioned speaker device is located on the headrest behind the driver's head.

[0019] Preferably, in order to stop prompting the driver after the driver regains concentration, the above-mentioned visual and auditory concentration judgment method further includes: when the above-mentioned judgment does not require prompting the driver, executing one of the following steps or any combination thereof: making the display device not to issue a visual prompt; and making the speaker device not to issue an auditory prompt.

[0020] Preferably, in order to determine the level of cognitive concentration, the sound inside the vehicle is a spectrogram, and the step of determining the driver's cognitive concentration further includes inferring the spectrogram using a neural network to determine whether it includes background sounds and prominent sounds that distract the driver.

[0021] According to one embodiment of the present invention, a host suitable for judging the visual and auditory concentration of a vehicle driver is provided, and the host includes at least one processor for executing instructions in a non-volatile memory (i.e., non-volatile memory) to implement the visual and auditory concentration judgment method as described above.

[0022] According to an embodiment of the present invention, a driver monitoring system is provided, further comprising the above-mentioned host, camera device, sound receiving device, display device and speaker device.

[0023] Beneficial effects: The method, host and driver monitoring system for judging the visual and auditory concentration of a vehicle driver provided by the present invention can improve the disadvantage of being difficult to monitor the driver in low-light conditions, use images and sounds to better monitor the driver, and give warnings when his concentration level is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 FIG. 1 is a schematic diagram of a driver monitoring system 100 according to an embodiment of the present invention.

[0025] Figure 2 FIG. 1 is a block diagram of the driver monitoring system 100 .

[0026] Figure 3 FIG. 3 is a flow chart of a method 300 for determining visual concentration according to an embodiment of the present invention.

[0027] Figure 4 4 is a schematic diagram of the projection of the three-dimensional model of the driver's 199 head and the two-dimensional image 410 according to an embodiment of the present invention.

[0028] Figure 5 FIG. 4 is a schematic diagram of a face vector according to an embodiment of the present invention.

[0029] Figure 6 FIG. 6 is a flow chart of a method 600 for determining auditory concentration according to an embodiment of the present invention.

[0030] Figure 7 FIG. 7 is a flow chart of a method 700 for determining visual and auditory concentration according to an embodiment of the present invention.

[0031] Description of reference numerals:

[0032] 100: driver monitoring system;

[0033] 110: host; 111: processor; 112: sub-processor; 113: peripheral device connection device.

[0034] 120: camera device;

[0035] 130: radio receiver;

[0036] 140: display device;

[0037] 150: speaker device;

[0038] 199: driver;

[0039] 300: Visual concentration judgment method;

[0040] 310-390: steps;

[0041] 410: Image;

[0042] 510: face vector;

[0043] 520: model center;

[0044] 600: Method for judging auditory concentration;

[0045] 610-630: steps;

[0046] 700: Method for judging visual and auditory concentration;

[0047] 710~730: Steps. DETAILED DESCRIPTION

[0048] The present invention provides a method for judging the visual and auditory concentration of a driver of a vehicle, a host and a driver monitoring system. In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, not a comprehensive embodiment. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] The terms "first", "second", "third", etc. (if any) in the specification and scope of the invention and the drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects described in the above descriptions can be interchanged where appropriate. In the description of the present invention, "plurality" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. Such functional entities can be implemented in software form, or in one or more hardware circuits or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0050] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection", and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, or an indirect connection through an intermediate medium, and it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] Please refer to Figure 1 FIG. 1 is a schematic diagram of a driver monitoring system 100 according to an embodiment of the present invention. Figure 2 As shown, it is a block diagram of the driver monitoring system 100. Figure 1The driver monitoring system 100 shown is installed in a car cabin, but a person skilled in the art will appreciate that the driver monitoring system 100 can be applied to all vehicles, such as motorcycles, tricycles, aircraft, marine vehicles and spacecraft, as long as there is at least one driver 199 in the vehicle responsible for controlling the vehicle.

[0053] The driver monitoring system 100 includes a host 110, and at least one camera 120, at least one sound receiving device 130, at least one display device 140, and at least one speaker device 150 connected to the host 110. Figure 1 As shown, the field of view of the camera device 120 at least includes the face or face of the driver 199. The sound receiving range of the sound receiving device 130 at least includes the mouth of the driver 199. The driver 199 can see the visual content displayed by the display device 140 and hear the auditory content emitted by the speaker device 150. In one embodiment, the display device 140 is included in a head-up display in front of the driver. In one embodiment, the speaker device 150 is located on the headrest behind the driver's head.

[0054] In one embodiment, the display device 140 may include a forward head-up display (HUD). In other embodiments, the display device 140 may include a display of virtual reality, augmented reality, or mixed reality. A person skilled in the art will appreciate that as long as it can visually prompt the driver 199, even a simple lighting device can be used as the above-mentioned display device 140. Similarly, as long as it can auditorily prompt the driver 199, even a simple buzzer can be used as the above-mentioned speaker device 150.

[0055] In some embodiments, the display device 140 is not fully controlled by the driver monitoring system 100. For example, the display device 140 can be an independent display device that receives input from the driver monitoring system 100 and other systems, and is responsible for blending display contents from multiple sources together for display. Similarly, in some embodiments, the speaker device 150 is not fully controlled by the driver monitoring system 100. For example, the speaker device 150 can be an independent audio device that receives input from the driver monitoring system 100 and other systems, and is responsible for blending audio contents from multiple sources together for playback.

[0056] In one embodiment, the camera 120 may include an infrared sensor. In another embodiment, the camera 120 may also include one or more infrared lights so that the infrared sensor can clearly capture the driver's 199 face and features in low visible light conditions. In one embodiment, the sound receiving device 130 may include a directional sound receiving function so as to aim at the driver's 199 face, especially the mouth, to screen out external noise. For example, the sound receiving device 130 may include an array composed of multiple sound receiving components, which can focus or scan the source direction of the sound wave by signal processing. One or more of the sound receiving devices 130 can also be used to collect background sound so as to determine whether the sound comes from the background or the driver 199.

[0057] The host 110 mentioned above may include one or more processors 111 to control the entire host 110 and the driver monitoring system 100. The processor 111 may execute an operating system, as well as drivers and applications executed in the operating system environment. These software and / or firmware (i.e., firmware) may be stored in a non-volatile memory, including instructions and data that can be executed by the processor 111, so as to be used to implement the methods and functions provided by the present invention. A person of ordinary skill in the art should have all the common sense of a university undergraduate degree in computer organization, computer structure, operating system, system software, etc., and may understand that various possible changes and derivatives of the hardware and software of the host 110 are applicable to the present invention.

[0058] In order to accelerate the execution of artificial intelligence and / or neural network-like algorithms so as to monitor the driver in real time, the host 110 may include one or more co-processors (i.e., coprocessors) 112 for providing vectorized arithmetic and logical computing capabilities. The above-mentioned neural network-like algorithms may include one or more deep neural networks (DNNs) or a collection of multiple deep neural networks. The above-mentioned neural network-like algorithms may include convolutional neural networks (CNNs) or their derivative neural network-like algorithms. The above-mentioned co-processor 112 may be a general-purpose graphics acceleration processor (GPU) or a processor dedicated to neural network-like algorithms (Neural network Processor Unit, NPU). Those of ordinary skill in the art will appreciate that such co-processors for edge artificial intelligence have been put into practical use, so they will not be described in detail here.

[0059] The host 110 may further include one or more peripheral device connection devices 113 for connecting the at least one camera device 120, the at least one sound receiving device 130, the at least one display device 140 and the at least one speaker device 150. The peripheral device connection device 113 may include one or more interfaces that comply with one or more industrial standards, such as Bluetooth, USB, IEEE1394, UART, iSCSI, SATA, PCI-Express, PCT, etc. It may also include a proprietary interface.

[0060] The driver monitoring system 100 provided by the present invention is used to assess the driver's visual concentration using a visual sensor such as a camera 120, and to assess the recognition concentration using an auditory sensor such as a sound receiving device 130, so as to provide the driver's overall concentration. By integrating information from multiple modes such as vision and hearing, the driver monitoring system 100 provided by the present invention can infer the driver's visual and cognitive concentration.

[0061] Unlike conventional driver monitoring systems that only include optical sensors, the driver monitoring system 100 provided by the present invention can provide more reliable and more valuable information on the driver's concentration by analyzing the driver's cognitive concentration. The driver monitoring system 100 provided by the present invention can use an image including the driver's face to estimate the visual concentration of the driver 199, and the above estimation can use an algorithm to obtain the face and eye direction of the driver 199 based on the image. In order to estimate the cognitive concentration of the driver 199, an algorithm can also be used to estimate auditory events based on sound. The above auditory events can be classified into multiple categories. For example, the algorithm can determine whether the driver 199 is in a noisy environment or a distracting environment. Finally, after determining that the driver 199's concentration is low based on visual and cognitive concentration, the display device 140 and / or the speaker device 150 can be used to prompt or warn the driver 199.

[0062] Please refer to Figure 3 FIG. 3 is a flow chart of a method 300 for determining visual concentration according to an embodiment of the present invention. The method 300 for determining visual concentration can be performed by Figure 1 and Figure 2 The driver monitoring system 100 shown is implemented, in particular, by a program executed by a host 110. If there is no causal relationship between any two steps, the present invention does not limit the execution order of the two steps. The visual concentration determination method 300 can start from step 310.

[0063] Step 310: Receive or read parameters used by the visual concentration determination method 300, such as one or more transformation matrices, deflection angle thresholds of facial vectors, etc. In one embodiment, the camera device 120 may be calibrated before the visual concentration determination method 300 is executed.

[0064] Please refer to Figure 4 As shown, it is a schematic diagram of the projection of the three-dimensional model of the driver 199 head and the two-dimensional image 410 according to an embodiment of the present invention. In this embodiment, a general human head model can be used as the three-dimensional head model of the driver 199. The above-mentioned general human head model includes various facial features and their corresponding positions, such as eyes, nose, mouth, ears, eyebrows, etc. These facial features have a three-dimensional coordinate in the human body model coordinate system. For example Figure 4 C1 to C4 shown.

[0065] When the camera 120 takes a picture of the driver 199's face to form a two-dimensional image 410, the facial features mentioned above will appear in the image 410, corresponding to a two-dimensional coordinate u respectively. i After being calibrated by the camera 120, the above-mentioned conversion matrix can be generated to convert the two-dimensional coordinates in the above-mentioned image 410 into the above-mentioned three-dimensional coordinates. According to the three-dimensional coordinates of each facial feature, it is possible to know where the face of the head model is facing. In other words, the face vector of the head model can be known.

[0066] In another embodiment, the head model of the driver 199 may be corrected to obtain a specific conversion matrix to reduce the estimation error of the facial vector. Alternatively, different head models may be selected based on age, gender, race, etc., and different conversion matrices may be used to reduce the estimation error of the facial vector.

[0067] Please refer to Figure 5 As shown in FIG. 1 , it is a schematic diagram of a face vector according to an embodiment of the present invention. Figure 5 The coordinate system shown can be Figure 4 After obtaining the three-dimensional coordinates of one or more facial features, the pointing vector of the head model can be obtained by averaging, weighting or various algorithms. The present invention refers to it as a facial vector 510.

[0068] The facial vector 510 corresponds to a reference point of the head model in the human body model coordinate system, and the reference point can be regarded as the model center 520. The facial vector 510 can be represented by three angles, namely, the pitch angle, the yaw angle, and the roll angle for three mutually perpendicular axes. These angles are also called Euler angles. For the sake of convenience, when the facial vector 510 of the driver 199 looks straight ahead, these three Euler angles are all 0 degrees. In one embodiment, the facial vector looking straight ahead is parallel to the local horizontal plane of the vehicle and parallel to the forward direction of the vehicle. It can be understood by those of ordinary skill in the art that 0 degrees of the Euler angle can be set to other angles.

[0069] Step 320 : Receive an image from the camera device 120 .

[0070] Step 330: Detect the face on the image and mark the facial features. In this step, the trained neural network can be used to infer the image to determine whether the image contains a face. When the image does not contain a face, the process of the visual concentration determination method 300 can be directly terminated. When the image contains a face, the trained neural network can be used to mark the facial features on the image. Then, the process proceeds to step 340.

[0071] Step 340: Estimate the face vector using the above facial features. In this step, you can use Figure 3 and Figure 4 The aforementioned technique uses the parameters received in step 310 to find the three-dimensional coordinates of the head model corresponding to the above-mentioned facial features. Then, the above-mentioned three-dimensional coordinates are used to estimate the face vector. In one embodiment, the face vector is represented by three Euler angles. Then, the process proceeds to step 350.

[0072] In one embodiment, in order to reduce the high-frequency changes of the facial vectors or Euler angles or to add an adaptive filter for state estimation, one of the following or any combination thereof may be added to this step: a filtering step of a time-series filter, a Kalman filter or a low-pass filter.

[0073] Step 350: Determine whether the facial vector deviates beyond a range. For example, when the facial vector of the driver deviates to a certain angle to the left or right, such as 30 degrees, that is, the absolute value of the yaw angle is greater than 30 degrees, it can be almost considered that the driver is not looking forward, and the process can proceed to step 390. In this way, the computing resources of the subsequent steps can be omitted. In another example, when the facial vector of the driver reaches 15 degrees upward or downward, that is, the absolute value of the pitch angle is greater than 15 degrees, it can be almost considered that the driver is not looking forward, and the process can proceed to step 390. In another example, when the sum of the absolute values ​​of the three Euler angles is greater than a certain critical value, it can be almost considered that the driver is not looking forward, and the process can proceed to step 390. In other words, it can be determined whether the facial vector deviates beyond a range based on one of the three Euler angles or any combination thereof. If it does not exceed the range, the process continues to step 360.

[0074] Step 360: Labeling eyes. In this step, the trained neural network can be used to infer the image so as to label a pair of eyes. Then, the process proceeds to step 370.

[0075] Step 370: Determine whether a pair of eyes in the image are open. If at least one eye is open, the process can proceed to step 380. When no eye is open, the process can proceed to step 390.

[0076] Step 380: Label the pupil and estimate the pupil vector. In this step, you can use Figure 3 and Figure 4 The above-mentioned technique uses the parameters received in step 310 to find the three-dimensional coordinates of the head model corresponding to the pupil position. Then, one or two pupil vectors are estimated using the three-dimensional coordinates. In one embodiment, the pupil vector is represented by three Euler angles. Then, the process proceeds to step 390.

[0077] In one embodiment, in order to reduce the high frequency changes of the facial vectors or Euler angles or add an adaptive filter for state estimation, one of the following or any combination thereof may be added to this step: a time series filter; a Kalman filter; or a low-pass filter filtering step.

[0078] Step 390: Determine visual concentration based on the above-mentioned face vector and / or pupil vector. In one embodiment, if the face vector deviates too much from the normal range, or the deviation time is too long, it can be known that the visual concentration of the driver 199 is lower than the standard. In another embodiment, although the face vector is within the normal range, if the pupil vector cannot be detected for a long time, it can be known that the driver 199 is dozing off and his visual concentration is lower than the standard.

[0079] Please refer to Figure 6 FIG. 6 is a flow chart of a method 600 for determining auditory concentration according to an embodiment of the present invention. The method 600 for determining auditory concentration may be performed by: Figure 1 and Figure 2 The driver monitoring system 100 shown is implemented, in particular, by a program executed by the host 110. If there is no causal relationship between any two steps, the present invention does not limit the execution order of the two steps. The auditory concentration determination method 600 can start from step 610 or step 620.

[0080] Optional step 610: receiving or reading the parameters used by the auditory concentration determination method 600, such as the frequency range and time length of the sound data to be determined (ie, sound data).

[0081] Step 620: Receive sound data from a sound receiving device. In one embodiment, the sound data may include a spectrogram. The spectrogram may be a two-dimensional array. The first dimension of the array represents time, the second dimension represents frequency, and the array element value is the intensity of the sound. Then, the process proceeds to step 630.

[0082] Step 630: Determine whether there is background noise (ambient noise) and / or prominent noise based on the sound data. The sound data can be inferred using a trained quasi-neural network to determine whether the sound data contains background noise and / or sounds that may distract the driver. In one embodiment, the above-mentioned background noise may include engine sounds, air conditioning sounds, brake sounds, etc. The above-mentioned prominent sounds may include human conversation sounds, music playing sounds, or any sounds that mean the driver is distracted. For example, the above-mentioned prominent sounds can be analyzed to determine whether the driver is talking, or listening to music, etc.

[0083] When no background sound is detected, it can be assumed that the vehicle is not moving. Therefore, even if there is a prominent sound, there is no need to worry about the driver's cognitive concentration. When background sound is detected, it means that the vehicle's engine is running, and the driver's cognitive concentration must be further concerned. Therefore, when both background sound and prominent sound are detected, it can be said that the driver's cognitive concentration is low.

[0084] Please refer to Figure 7 FIG. 7 is a flow chart of a method 700 for determining visual and auditory concentration according to an embodiment of the present invention. The method 700 for determining visual and auditory concentration may be performed by: Figure 1 and Figure 2The driver monitoring system 100 shown is implemented, in particular, by a program executed by a host computer 110. The visual and auditory concentration determination method 700 includes a loop. In each executed loop, the instructions can be started by the visual concentration determination method 300 or the auditory concentration determination method 600. The present invention does not limit the execution order of the visual concentration determination method 300 or the auditory concentration determination method 600. After the execution of the two methods 300 and 600, the process can proceed to step 710.

[0085] Step 710: Comprehensively judge whether the driver is distracted based on the results obtained by the visual concentration determination method 300 and the auditory concentration determination method 600. When the result of the comprehensive judgment is yes, the process proceeds to step 720. When the result of the comprehensive judgment is no, the process can proceed to step 730 or repeat the execution cycle to execute methods 300 and 600 again. In one embodiment, this step can also refer to the historical results of methods 300 and 600 executed in one or more previous cycles to make a comprehensive judgment.

[0086] In one example, when the visual concentration level obtained by method 300 and the cognitive concentration level obtained by method 600 are both inattentive, the result of the above comprehensive judgment may be inattentive.

[0087] In another example, when the visual concentration level obtained by method 300 and the cognitive concentration level obtained by method 600 are both concentration, the result of the above comprehensive judgment may be concentration.

[0088] In other examples, the above comprehensive judgment can be performed after table lookup or weighted calculation based on various results and / or historical results of method 300 and method 600. For example, when the pupil vector is continuously not detected in several consecutive cycles, no matter what result is obtained by method 600, the result of the comprehensive judgment of the latest cycle is inattention. In another example, when method 600 detects a very loud human voice volume, no matter what result is obtained by method 300, the result of the comprehensive judgment of the latest cycle is inattention.

[0089] Those skilled in the art will appreciate that the above comprehensive judgment can be based on many combinations of visual concentration and cognitive concentration to determine the result. The present invention is not limited to the above combination types, as long as the results and / or historical results of the above two visual concentration and cognitive concentration are referenced.

[0090] Step 720: Prompt the driver. In one embodiment, the display device 140 and / or the speaker device 150 may provide visual and / or auditory prompts to the driver. For example, when the method 300 has determined that the visual concentration level is low, this step may rely more on hearing to issue a prompt or warning to the driver 199. Conversely, when the method 600 has determined that the cognitive concentration level is low, this step may rely more on hearing to issue a prompt or warning to the driver 199.

[0091] In one embodiment, when there are multiple display devices 140, and the method 300 determines that the driver's 199 head is deflected toward one of the display devices 140, this step can cause the display device 140 to issue a prompt so as to provide a visual warning to the driver 199. For example, when the driver's 199 head turns to the left, the display device 140 on the left side is caused to issue a prompt.

[0092] Step 730: Stop issuing prompts. When the comprehensive judgment in the previous cycle believes that the driver 199 is not concentrating and the prompt is issued in step 720, and the comprehensive judgment in the latest cycle believes that the driver 199 has recovered the concentration level, the above-mentioned visual and / or audible prompts can be stopped.

[0093] According to one embodiment of the present invention, a method for determining the visual and auditory concentration of a driver of a vehicle is provided, comprising: determining the driver's visual concentration based on an image of the driver's head obtained by a camera device in the vehicle; determining the driver's cognitive concentration based on sounds inside the vehicle obtained by a sound receiving device in the vehicle; determining whether to issue a prompt to the driver based on the above-mentioned visual concentration and the above-mentioned cognitive concentration; and when the above-mentioned determination requires issuing a prompt to the driver, executing one of the following steps or any combination thereof: causing a display device in the vehicle to issue a visual prompt; and causing a speaker device in the vehicle to issue an auditory prompt.

[0094] Preferably, in order to save computing resources and time, the above-mentioned step of determining the driver's visual concentration level further includes: using a neural network to detect the face on the image and annotate multiple facial features; estimating the facial vector of the driver's head using the positions of the multiple facial features; and when the facial vector is outside a range, determining that the driver's visual concentration level is poor.

[0095] Preferably, in order to save computing resources and time, when the facial vector is within the range, it is determined whether the driver's eyes are open based on the multiple facial features; and when it is determined that the driver's eyes are closed, it is determined that the driver's visual concentration is poor.

[0096] Preferably, in order to calculate where the driver's pupils are looking, when it is determined that the driver's two eyes are open, the positions of the two pupils are marked and the corresponding two pupil vectors are estimated; and based on the face vector and the two pupil vectors, the driver's visual concentration is determined.

[0097] Preferably, in order to represent a facial vector and / or a pupil vector, wherein the facial vector is known based on the relationship between positions of a plurality of features of the head model and positions of the plurality of facial features of the image, the facial vector is represented by three Euler angles.

[0098] Preferably, in order to represent the face vector and / or the pupil vector, the pupil vector is represented by three Euler angles.

[0099] Preferably, in order to reduce high frequency changes of the facial vector or Euler angle, the step of determining the driver's visual concentration further comprises: filtering the facial vector through one or any combination of the following: a time series filter; a Kalman filter; and a low-pass filter.

[0100] Preferably, in order to obtain clear images under low light conditions (ie, low brightness), the above-mentioned images are captured in the infrared band.

[0101] Preferably, in order to give the driver the best chance of seeing the visual prompt, the display device comprises a heads-up display in front of the driver.

[0102] Preferably, in order to give the driver the best chance of hearing the auditory prompt, the speaker is located on the headrest behind the driver's head.

[0103] Preferably, in order to stop prompting the driver after the driver regains concentration, the above-mentioned visual and auditory concentration judgment method further includes: when the above-mentioned judgment does not require prompting the driver, executing one of the following steps or any combination thereof: making the display device not to issue a visual prompt; and making the speaker device not to issue an auditory prompt.

[0104] Preferably, in order to determine the level of cognitive concentration, the sound inside the vehicle is a spectrogram, and the step of determining the driver's cognitive concentration further includes inferring the spectrogram using a neural network to determine whether it includes background sounds and prominent sounds that distract the driver.

[0105] According to one embodiment of the present invention, a host suitable for judging the visual and auditory concentration of a vehicle driver is provided. The host includes at least one processor for executing instructions in a non-volatile memory to implement the visual and auditory concentration judgment method as described above.

[0106] According to an embodiment of the present invention, a driver monitoring system is provided, further comprising the above-mentioned host, camera device, sound receiving device, display device and speaker device.

[0107] The method, host and driver monitoring system for judging the visual and auditory concentration of a vehicle driver provided by the present invention can improve the disadvantage of being difficult to monitor the driver in low-light conditions, use images and sounds to better monitor the driver, and give warnings when his concentration level is low.

Claims

1. A method for judging the visual and auditory concentration of a vehicle driver, characterized in that: Include: determining the driver's visual concentration level based on an image of the driver's head obtained by a camera device in the vehicle; Determining the driver's cognitive concentration level based on the sounds inside the vehicle acquired by a sound receiving device in the vehicle; Determining whether to issue a prompt to the driver based on the visual concentration level and the cognitive concentration level; and When the above judgment requires that the driver be prompted, perform one or any combination of the following steps: causing a display device in the vehicle to issue a visual prompt; and A speaker device in the vehicle is caused to issue an auditory prompt.

2. The method for judging visual and auditory concentration according to claim 1, wherein: The above-mentioned step of determining the driver's visual concentration level further includes: Detecting faces on the image and labeling multiple facial features using a neural network; estimating a facial vector of the driver's head using the positions of the plurality of facial features; and When the face vector is outside the range, it is determined that the driver's visual concentration is poor.

3. The method for determining visual and auditory concentration as claimed in claim 2, further comprising: When the facial vector is within the range, determining whether both eyes of the driver are open according to the plurality of facial features; and When it is judged that both eyes of the driver are closed, it is judged that the visual concentration of the driver is poor.

4. The method for judging visual and auditory concentration according to claim 3, wherein: Also includes: When it is determined that both eyes of the driver are open, the positions of the two pupils are marked, and the corresponding two pupil vectors are estimated; and The visual concentration level of the driver is determined according to the face vector and the two pupil vectors.

5. The method for judging visual and auditory concentration according to claim 2, wherein: The facial vector is obtained based on the relationship between the positions of the multiple features of the head model and the positions of the multiple facial features of the image. The facial vector is represented by three Euler angles.

6. The method for judging visual and auditory concentration according to claim 4, wherein: The pupil vector mentioned above is represented by three Euler angles.

7. The method for judging visual and auditory concentration according to claim 2, wherein: The above step of determining the driver's visual concentration level further includes: filtering the facial vector through one or any combination of the following: a time series filter; a Kalman filter; and a low-pass filter.

8. The method for judging visual and auditory concentration according to claim 1, wherein: The above images were taken in infrared wavelengths.

9. The method for judging visual and auditory concentration according to claim 1, wherein: The above-mentioned display device includes a head-up display in front of the driver.

10. The method for judging visual and auditory concentration according to claim 1, wherein: The speaker device is located on the headrest behind the driver's head.

11. The method for judging visual and auditory concentration according to claim 1, wherein: Also includes: When the above determination does not require prompting the driver, perform one or any combination of the following steps: causing the display device to not emit a visual prompt; and The speaker device does not emit an auditory prompt.

12. The method for judging visual and auditory concentration according to claim 1, wherein: The above-mentioned sound inside the vehicle is a spectrum diagram, wherein the above-mentioned step of determining the driver's cognitive concentration level further includes using a neural network to infer the spectrum diagram to determine whether it contains background sounds and prominent sounds that distract the driver.

13. A host computer for judging the visual and auditory concentration of a vehicle driver, characterized in that: The host comprises at least one processor for executing instructions in a non-volatile memory, so as to implement the visual and auditory concentration determination method as claimed in any one of claims 1 to 12.

14. A driver monitoring system, characterized in that: It further comprises the host, camera, sound receiving device, display device and speaker device as described in claim 13.

Images