Driver distraction from uncertain gaze estimation

Abstract

A method and apparatus, including computer programs, for providing a driver distraction signal and / or providing a system degradation signal indicative of reliability of the driver distraction signal based on a gaze of a driver of a vehicle. Images of a driver of a vehicle are received, the images containing information indicative of a gaze of the driver. For each image, a gaze area of the driver is determined, the gaze area including a gaze point and a gaze uncertainty value. For each image, the gaze area is associated with an area of several areas around the vehicle, where the areas include at least one predefined attention-focused area and at least one predefined attention-infocused area. A driver distraction level and / or a system degradation level indicative of reliability of the driver distraction level is determined based on the determined gaze area for the plurality of images, the gaze uncertainty value, and the aforementioned area. A driver distraction signal and / or a system degradation signal indicative of the determined driver distraction level and / or system degradation level is output.

Description

Technical Field

[0001] The present disclosure relates to gaze estimation. More specifically, the present disclosure generally relates to a system and method for monitoring a driver of a vehicle based at least in part on gaze estimation. Background Art

[0002] Gaze estimation is the process of predicting the direction of a person's gaze by analyzing the position and movement of their eyes. It involves using computer vision techniques to detect and track eyes, and then typically using machine learning algorithms to analyze eye movements to determine and / or predict where a person is looking. Gaze estimation has applications in a variety of fields, including human-computer interaction or attention computing, virtual / augmented reality, and health or advertising research.

[0003] One application of gaze estimation is in the context of driver monitoring systems (DMS), which allow the driver of a vehicle to be monitored, e.g., to be alerted if the driver is not looking at the road, thereby improving the safety of the driver and those around the vehicle. Such monitoring is typically done by using the driver's gaze estimate in combination with information about the geometry of the car (such as the position of the windshield, dashboard, screens, rearview mirrors, etc. in the car) to determine where the driver is looking. This is typically done as a relatively simple gaze-object mapping, which can be implemented by a person of ordinary skill in the art and using known techniques.

[0004] However, there are often situations in vehicles where gaze determination is degraded. Some examples of gaze determination degradation include when there is strong sunlight above the eyes, when the driver is wearing dark sunglasses, when there is a high head angle (i.e., only a portion of one eye is seen in the camera used for gaze tracking). Typically, these situations are managed by classifying whether the driver's eyes are "visible enough", that is, a binary quality signal of the gaze is obtained, which indicates whether gaze determination is usable. Summary of the invention

[0005] In some aspects, the technology described herein relates to a system configured to provide a driver distraction signal and / or a system degradation signal indicating the reliability of the driver distraction signal based on the gaze of the driver of the vehicle. The system includes a processor and a memory, the memory including instructions executable by the processor. The system is configured to: receive a plurality of images of the driver of the vehicle, wherein the images contain information indicating the driver's gaze; determine the driver's gaze area for each image, wherein the gaze area includes a gaze point and a gaze uncertainty value; for each image, associate the gaze area with an area in a plurality of areas around the vehicle, wherein the plurality of areas include at least one predefined attention concentration area and at least one predefined inattention area; determine the driver distraction level and / or the system degradation level indicating the reliability of the driver distraction level based on the gaze areas determined for the plurality of images, the determined gaze uncertainty values, and the associated areas of the vehicle; and output the driver distraction signal and / or the system degradation signal, wherein the driver distraction signal indicates the determined driver distraction level, and the system degradation signal indicates the determined system degradation level.

[0006] In one embodiment, the fixation region is a probability distribution, wherein the fixation point is the mean of the probability distribution and the fixation uncertainty value is the standard deviation of the probability distribution.

[0007] In one embodiment, the driver distraction level is based on which portions of the gaze area fall within the attention zone and the inattention zone, respectively.

[0008] In one embodiment, a high driver distraction level is obtained when a substantial portion of the gaze area falls within the inattention region, and a low driver distraction level is obtained when a substantial portion of the gaze area falls within the attention region.

[0009] In one embodiment, the system is further configured to set one or more threshold signal values ​​for the distraction signal and the system degradation signal to indicate one or more levels of driver distraction and one or more levels of system degradation, respectively.

[0010] In one embodiment, the system is further configured to: classify the driver distraction signal into one of the following categories: attentive, distracted, degraded, and uncertain; and associate the determined reliability with the determined category.

[0011] In one embodiment, a gaze estimation algorithm based on machine learning is used to determine the gaze area.

[0012] In one embodiment, the system is further configured to control a device for the vehicle based on the output driver distraction signal and / or the system degradation signal.

[0013] In some aspects, the technology described herein relates to a method for providing a driver distraction signal and / or a system degradation signal indicating the reliability of the driver distraction signal based on the gaze of a driver of a vehicle, the method comprising: receiving multiple images of the driver of the vehicle, wherein the images contain information indicating the driver's gaze; determining a gaze area of ​​the driver for each image, wherein the gaze area includes a gaze point and a gaze uncertainty value; for each image, associating the gaze area with an area of ​​multiple areas around the vehicle, wherein the multiple areas include at least one predefined attention focus area and at least one predefined inattention focus area; determining a driver distraction level and / or a system degradation level indicating the reliability of the driver distraction level based on the gaze areas determined for the multiple images, the determined gaze uncertainty values, and the associated areas of the vehicle; and outputting a driver distraction signal and / or a system degradation signal, wherein the driver distraction signal indicates the determined driver distraction level and the system degradation signal indicates the determined system degradation level.

[0014] In some aspects, the technology described herein relates to a computer program including instructions that, when executed on at least one processor, cause the at least one processor to perform the method. The computer program product may be implemented by a non-transitory computer-readable medium encoded with instructions that cause one or more hardware processors in at least one computer hardware device in a system of the type described to perform the method steps discussed.

[0015] In some aspects, the technology described herein relates to an eye tracking system configured to provide a driver distraction signal based on gaze.

[0016] In some aspects, the technology described herein relates to a vehicle including the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the following, the present invention will be described in detail with reference to exemplary embodiments of the present invention and to the accompanying drawings, in which:

[0018] Figure 1 is a flow chart illustrating a method for providing a driver distraction signal based on a gaze of a driver of a vehicle and / or providing a system degradation signal indicating the reliability of the driver distraction signal, according to some embodiments.

[0019] Figure 2 is a block diagram illustrating a system for providing a driver distraction signal based on a gaze of a driver of a vehicle and / or providing a system degradation signal indicating the reliability of the driver distraction signal, according to some embodiments.

[0020] Figure 3is a diagram showing a method according to some embodiments Figure 2 A block diagram of a more detailed view of a portion of the system.

[0021] Figure 4 is a schematic diagram of a vehicle geometry used in the system according to some embodiments.

[0022] Figure 5 yes Figure 4 Schematic diagram of the car geometry, including the gaze area indicating that the driver is looking at the road ahead.

[0023] Figure 6 yes Figure 4 Schematic diagram of the car geometry, including the gaze area indicating that the driver is not looking at the road ahead.

[0024] Figure 7 yes Figure 4 Schematic diagram of the car geometry, including the gaze region representing the uncertainty about whether the driver is looking at the road ahead.

[0025] Figure 8 yes Figure 4 Schematic diagram of the car geometry, including the gaze region with high uncertainty about where the driver is looking.

[0026] Fig. 9 yes Figure 4 Schematic diagram of the car geometry, including two gaze regions with different uncertainties, both indicating that the driver is not looking at the road ahead.

[0027] Fig.10 is a diagram schematically illustrating a category combining attention region probability and gaze uncertainty according to some embodiments.

[0028] Fig.11 is a block diagram illustrating a computer system in which an embodiment of the method may be implemented.

[0029] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0030] Various embodiments of the present invention are directed to techniques for providing a driver distraction signal based on a gaze of a driver of a vehicle and / or providing a system degradation signal indicating the reliability of the driver distraction signal. As described above, in most conventional systems for gaze detection in a vehicle context, a classification is made as to whether the driver's eyes are sufficiently visible (e.g., due to strong sunlight above the eyes, use of sunglasses, high head angle, etc. as described above), i.e., a binary quality signal of the presence of a gaze.

[0031] In contrast, according to various embodiments of the invention described herein, there is an estimation error in the gaze estimation based on how "hard" it is to look at the image. Thus, a more fine-grained idea of ​​how good the estimated gaze is can be obtained, which expands the possible use cases of the collected gaze data.

[0032] For example, this information can be used to reduce visual distraction, as well as perform diagnostics or “self-tests” and alert the car / driver when a gaze detection / driver distraction feature is not working properly.

[0033] The term "gaze tracking" is used herein to refer to any method or system for detecting and / or monitoring the position of eyes and / or the gaze direction of (eyes). Such systems will be known and understood by those skilled in the art. Therefore, this article will not further describe the system itself, but will focus on how the data collected by the system is processed and used to provide information to the driver of the vehicle or other interested parties.

[0034] It should be appreciated that while the term "automobile" is used throughout this specification to represent a vehicle, the techniques described herein can be applied to essentially any vehicle, including a car, a truck, or even a train crew or an airplane, or a simulator of such an environment where it may be valuable to not only understand the driver / engineer / pilot's gaze, but also to have some information or quality measure about the reliability of the information. Various embodiments of the invention will now be described by way of example and with reference to the accompanying drawings. However, it should be noted that these are merely exemplary embodiments and that many other embodiments fall within the scope of the claims.

[0035] Figure 1 is a flow chart illustrating a method 100 for providing a driver distraction signal based on a gaze of a driver of a vehicle and providing a system degradation signal indicating the reliability of the driver distraction signal, according to some embodiments. Figure 2 is a block diagram illustrating a system 200 in which method 100 may be performed according to some embodiments. Figure 1 As shown, method 100 begins with receiving 102 driver's image 202 of the driver of vehicle.The resolution of image 202 needs to be high enough to allow estimation of driver's gaze.This will depend on the placement and the visual field of camera usually, but generally speaking, several megapixels are enough.In certain embodiments, image is near infrared image, and in certain embodiments, image is color image, such as RGB (red green blue) image.In general, the placement of camera in vehicle makes that under most head rotation situations, typical driver's head position and driver's eyes are visible for the camera that captures them.There are several different camera placements and camera specifications to meet this requirement and can be determined by those of ordinary skill in the art.

[0036] The driver image 202 is then processed by the core processing module 204 of the eye tracking system. Eye tracking systems and methods (sometimes referred to as gaze detection systems and methods) include, for example, products produced by and purchased from Tobii Technology AB, and they operate by detecting reflections from the driver's eyes using near-infrared illumination and image sensors. An example of such a gaze detection system is described in U.S. Patent No. 7,572,008. The present invention may also employ other alternative gaze detection systems, regardless of the technology behind the gaze detection system. The eye tracking system may employ its own processor or a processor of another device (i.e., a processor / computer), or even a cloud-based distributed system in some embodiments, to interpret and process the received data. When the eye tracking system is mentioned herein, two possible methods of processing data are all mentioned.

[0037] In one embodiment, the processing performed by the core processing module 204 involves determining 104 a fixation area and fixation uncertainty, where the driver of the vehicle is looking (e.g., windshield, dashboard, etc.) is indicated by the fixation area. The fixation area is determined by the core processing module 204 by defining a fixation vector (e.g., on a windshield or dashboard) at a fixation point originating from the driver's eye (cornea) and terminating in the vehicle interior. There are many different types of available algorithms that allow determination of a fixation area. An example is a head pose estimation algorithm, which can give an indication of where the driver is looking based on determining the driver's head pose. The head pose can be determined based on a three-dimensional reference system, wherein (i) the three-dimensional position indicates the position of the head, and wherein, (ii) the tilt around the front-rear axis, the tilt around the left-right axis, and the rotation around the upper-lower axis can be measured to indicate the orientation of the driver's head. When the driver's head position has been determined, based on the assumption that the driver usually looks straight ahead, the driver's fixation position can also be determined. Although this method may not be as accurate as some precise fixation point estimation algorithms, in some cases, it is suitable for determining a fixation area by itself.

[0038] In other embodiments, the gaze estimation algorithm determines the pupil position of at least one eye of the driver, so as to determine where the driver is looking. This method is known in the art, and only a brief discussion will be carried out here. This can be based on the knowledge about the distance between the driver's pupil relative to one or more facial markers (for example, the driver's nose, mouth, ears or other facial features) to achieve. These can be determined when the driver looks forward, and then any change in these distances can indicate that the pupil deviates from the position change of the forward-looking position. The position of the pupil can then be used to determine which direction the driver is looking at. In certain embodiments, at least three facial markers are used to determine the relative distance to the pupil. Similar to head posture estimation, this method is sometimes not as accurate as an accurate gaze point estimation algorithm, but is very suitable for determining a rougher gaze area, which may be sufficient depending on the current environment.

[0039] In some other embodiments, the gaze estimation algorithm includes a gaze estimation algorithm based on machine learning. The algorithm can be trained based on several ground truth gaze positions generated by the device for rendering visual stimulation. For example, the driver can be required to look at one of the rows of lights that are lit at different positions corresponding to different areas inside or outside the vehicle in the environment. For example, as in the conventional data collection and calibration of the eye tracking system, some stimulus points are normal points on the driver image 202. These ground truth gaze positions can be presented in two-dimensional and / or three-dimensional positions relative to the driver. The machine learning system can observe the driver when looking at different stimuli, and learn when the driver is looking at different areas. For example, the system can take the image when the driver looks at a specific stimulus, and identify certain features (for example, the driver's head posture or pupil position) from the image. The system can use these features in combination with the geometry of the system (for example, the distance between the driver, stimulus and / or the device that captures the driver's image). In this way, the algorithm learns the features of the driver's image, and these features indicate that the driver is looking at a specific position. Then, the trained machine learning algorithm can be used to determine when the driver of the vehicle is looking at different areas associated with the vehicle. As with the algorithms discussed above, this approach may not be as accurate as a precise gaze point estimation algorithm, but is well suited for determining a coarser gaze region, which may be sufficient in some embodiments.

[0040] As described above, in addition to the gaze area, the various embodiments described herein also determine gaze uncertainty. Gaze uncertainty can be conceptually considered as a "circle" or area around the gaze point, where the larger the circle, the greater the gaze uncertainty. As described above, the gaze point and the gaze area represent the location where the driver is looking, and can be measured using many of the above-mentioned techniques. According to the various embodiments described herein, gaze uncertainty can be derived by interpreting the output from the gaze determination algorithm as a probability distribution. These probability distributions should be considered as conditions on the input image. Although it may seem strange at first glance to establish a distribution for a single image because the image has a clear ground truth, it should be noted that the ground truth may not be the only result determined by the image alone. At least in theory, there are the same images with different ground truths. For example, consider the case where a person's eyes cannot be fully seen in the image. Although the person is definitely looking in a specific direction when the image is captured, there is a distribution of possible directions that the person may look in (i.e., the distribution of the gaze point) considering the information captured in the image, for example, the person may be looking up or down or looking at a specific location, however, since the eyes cannot be fully seen in the image, this situation will result in an image that looks the same. This distribution is not available, but can be handled using a properly constructed loss function for training the model. For example, the Kullback-Leibler divergence (KL-divergence) can be used as a measure of the "difference" between two distributions. By minimizing the KL-divergence, the predicted distribution can be fitted to the actual distribution. This choice can be justified by noting that the KL divergence can be interpreted as the average difference in the number of bits required to encode samples of the actual distribution using a code optimized for the predicted distribution rather than a code optimized for the actual distribution. If the KL divergence is zero, the prediction is optimal in the sense that all uncertainty comes from the values ​​being measured rather than from the process of measuring the values. In some embodiments, the predicted distribution can be modeled as a normal distribution, where the mean is the fixation point and the standard deviation is the fixation uncertainty.

[0041] Next, the gaze vector and the gaze uncertainty are associated 106 with the specific geometry of the vehicle by the degradation module 206. To better understand this, see Figure 4 , which shows a schematic diagram of display / view-related features of a vehicle geometry according to some embodiments. In particular, Figure 4A schematic diagram of a car geometry 400 is shown, which includes a windshield 402, an instrument panel 404, an entertainment panel 406, a left side mirror 408, and a right side mirror 410. The windshield 402 is schematically divided from the middle down into an attention-focusing area and an attention-distracting area, which will be described in further detail below. The instrument panel 404 (also commonly referred to as the dashboard) is used to display information related to the operation of the vehicle to the driver, as is generally known. The entertainment panel 406 can be, for example, a music system or a screen connected to the driver's mobile phone. These different features (or portions thereof) of the car geometry 400 can be classified as attention-focusing areas and attention-distracting areas, respectively. It is clear that some of these areas can be inside the vehicle (e.g., the instrument panel 404 and the entertainment panel 406), and some of these areas can be located outside the vehicle (e.g., the left side mirror 408, the right side mirror 410, and the road visible through the driver's side windshield 402). Therefore, the attention-focusing area and the attention-distracting area should not be considered to be only inside the vehicle. An example of an attention zone may be the portion of windshield 402 directly in front of the driver, and it is known that when the driver's gaze falls within this region, her attention is on the road ahead of her. An example of an attention zone may be an entertainment panel 406, or portions of instrument panel 404 that display information unrelated to the operation of the vehicle while driving. When the driver's gaze falls within the inattention zone, it is reasonable to assume that her attention to what is happening on the road ahead is distracted or weakened. Therefore, which areas are designated as attention and inattention, respectively, will depend on the specific configuration of the vehicle.

[0042] As mentioned above, the vehicle also has an image capture device or an eye tracking system (not shown) to enable the driver's gaze to be determined. The image capture device can be used to capture images of the driver, which allow the determination of the gaze area. For example, the image capture device can capture images showing the driver's head position and / or pupil position. In some embodiments, an eye tracking system associated with or including an image capture device can be present in the vehicle. The eye tracking system can be used to determine the driver's gaze point or gaze area, or the change of the gaze point or gaze area. Eye tracking systems and methods (sometimes referred to as gaze detection systems and methods) include products such as those produced by Tobii Technology AB and purchased therefrom, and they operate by using near-infrared illumination and image sensors to detect reflections from the driver's eyes. An example of such a gaze detection system is described in U.S. Patent No. 7,572,008. The present invention can also adopt other alternative gaze detection systems, regardless of the technology behind the gaze detection system. The eye tracking system can use its own processor or a processor of another device (i.e., processor / computer), or even a cloud-based distributed system in some embodiments, to interpret and process the received data. When eye tracking systems are mentioned in this article, two possible ways of processing the data are mentioned.

[0043] Next, a driver distraction level and a system degradation level are determined 108 by the degradation module 206. The system degradation level indicates the reliability of the determined driver distraction level. Figure 3 A more detailed view of the demotion module 206 according to one embodiment is shown. Figure 3 As can be seen in FIG. 4 , the attention probability module 302 receives the gaze vector, the gaze uncertainty, and the vehicle geometry 400, and determines the attention focus area probability based on these factors. In one embodiment, the attention focus area probability can be determined by sampling a plurality of possible gaze points (e.g., 100 points) from a normal distribution of possible gaze points and determining the proportion of them that are within the attention focus area. For example, when less than 30% of the possible gaze points fall within the attention focus area, the driver may be characterized as "distracted", and when more than 50% of the possible gaze points fall within the attention focus area, the driver may be characterized as "focused". However, it should be recognized that this is only one example of determining the attention focus area probability, and there are other ways in which the attention focus area probability can be determined, and many ways in which different thresholds can be set based on the current situation.

[0044] The driver distraction level and the system degradation level are determined based on the determined attention focus area probability and gaze uncertainty by the degradation logic module 304. According to different embodiments, various logic rules and / or machine learning functions can be set to make such a determination. Figures 5 to 9 Some examples of various scenarios where attention region probabilities and gaze uncertainties can be used to determine driver distraction levels and system degradation levels are schematically shown. Figures 5 to 9 In the Figure 4 4 and superimposed thereon the determined gaze region (shown as a circle with a dashed circumference), where the center represents the gaze point and the radius represents the gaze uncertainty.

[0045] Figure 5 yes Figure 4 FIG. 4 is a schematic diagram of a car geometry of FIG. 40 including a gaze area 502 indicating that the driver is looking at the road ahead. The gaze area 502 is clearly within the attention focus area of ​​the windshield 402, i.e., the attention focus area probability is very high. Further, the gaze area 502 has a small radius, which means that the gaze uncertainty is low. In this case, the degradation logic module 304 generally indicates that the driver is paying attention and the system degradation level is low.

[0046] exist Figure 6 In FIG. 4 , the gaze area 602 is located on the entertainment panel 406. Although the radius of the gaze area 602 is slightly larger than Figure 5 The radius of the fixation region 502 in (i.e., indicating the fixation uncertainty and Figure 5 406, which is considered to be an area of ​​inattention while driving. Therefore, the gaze uncertainty is low. The degradation logic module 304 will indicate that the driver is inattentive and slightly high, and the system degradation level is low.

[0047] Figure 7 4 shows a case where the fixation region 702 is located at the edge of the attention focus region (i.e., the portion of the windshield 402 that is located in front of the driver), so that some possible fixations in the fixation point distribution are within the attention focus region and some possible fixations in the fixation point distribution are outside the attention focus region. However, the radius of the fixation region 702 is still relatively small, which is similar to Figure 6 That is, the driver's attention is uncertain, but the accuracy of the determination is high, or in other words, the level of gaze uncertainty is relatively low. Therefore, the degradation logic module 304 will indicate that the driver may be partially distracted (or at least unable to make a clear determination), but the system degradation level is low and the gaze uncertainty is low.

[0048] Next, Figure 8A situation with a large gaze area 802 is shown (i.e., a large gaze area indicates high gaze uncertainty about where the driver is looking). The gaze area 802 includes the vehicle's focused attention area (e.g., windshield 402), and the vehicle's inattentive area (e.g., entertainment panel 406), and / or possible combinations thereof (e.g., instrument panel 404). In this case, the driver is likely concentrating on looking at the road ahead, but this is difficult to assess due to the large uncertainty, so typically the degradation logic module 304 will indicate that a clear determination cannot be made, indicating high gaze uncertainty and a high system degradation level. For example, this may be due to the factors discussed at the beginning, such as strong sunlight above the eyes, the driver wearing dark sunglasses, a high head angle, etc.

[0049] Fig. 9 The last example schematically illustrated in shows two gaze areas 902 and 904, each having a different gaze uncertainty (i.e., a different radius of the gaze area), but both indicating that the driver is not looking at the road ahead. Thus, although the gaze uncertainty of both gaze areas 902 and 904 is relatively high, it is clear that both gaze areas are completely outside the attention focus area (i.e., the windshield), and the degradation logic module 304 will therefore indicate a distracted driver, high gaze uncertainty, and a high system degradation level.

[0050] However, it should be noted that Figures 5 to 9 All of the examples presented in are based on the driver moving the vehicle in a forward direction. If the driver is in reverse, the analysis of the degradation logic module 304 will most likely be at least partially different. For example, the side mirrors 408 and 410 may be designated as the attention area, and the windshield will be the inattention area, which will affect the analysis. Therefore, it should be clear that the designation of the attention area and the inattention area is not static, nor is the analysis of the degradation logic module 304, but rather these will change based on the current environment.

[0051] Finally, the driver distraction signal and the system degradation signal are output 110 to the receiving system 208, thereby ending the method 100. The receiving system 208 can then use these signals to provide various reminders or warning signals to the driver, or to control the operation of the vehicle, or for diagnostic purposes, etc. For example, if the driver is looking at the entertainment panel 406 of the vehicle, and the safety system of the vehicle senses an obstacle in front, the system can generate an audio or visual reminder to direct the driver's attention to the focus area represented by the windshield 402 directly in front. Similarly, if the safety system of the vehicle senses that there is something behind the vehicle, the system can generate an audio or visual reminder to direct the driver's attention to the area associated with the rearview mirrors 408, 410 of the vehicle. This can enhance the safety features of the vehicle. In different embodiments, various thresholds can also be set for the content represented by different signals.

[0052] Fig.10 Schematic diagram 1000 showing various possible outputs from system 200 according to some embodiments. Fig.10 The horizontal axis in shows the attention focus area probability, i.e., the probability that the driver's gaze is in the attention focus area. The vertical axis shows the gaze uncertainty. Fig.10 As can be seen in FIG. 1 , when the attention region probability is low, the system 200 outputs a “distracted” signal 1002 regardless of the gaze uncertainty level. That is, when the driver is clearly not looking at the attention region (e.g., as described in relation to FIG. 1 ), the system 200 outputs a “distracted” signal 1002 regardless of the gaze uncertainty level. Figure 6 and Fig. 9 ), it can be concluded that the driver is distracted, of course, regardless of the level of gaze uncertainty within a certain threshold region. Accordingly, when the probability of the focused attention region is high, the system 200 outputs the "focused attention" signal 1004, regardless of the level of gaze uncertainty. That is, when it is clear that the driver is looking at the focused attention region (e.g., as described with respect to Figure 5 described), it can be concluded that the driver is paying attention, regardless of the level of fixation uncertainty (i.e., Figure 5 The radius of the gaze area 502 in can be larger and the output from the system 200 still remains the same).

[0053] Between these "distracted" signals 1002 and "focused" signals 1004, there are "partially distracted" signals 1006 and "degraded" signals 1008. When the probability of the focused area is between the thresholds of the "distracted" signals 1002 and the "focused" signals 1004, the "partially distracted" signals 1006 and "degraded" signals 1008 are output, that is, the probability of the focused area can be determined, but the driver's attention cannot be clearly determined. When the gaze uncertainty is low (for example, as with respect to Figure 7 As described above, where it is relatively certain where the driver is looking), a "partial distraction" signal 1006 is output, whereas when the gaze uncertainty is high (e.g., as described above, the driver is relatively certain where he is looking), a "partial distraction" signal 1006 is output. Figure 8 As described, where it is unclear where the driver is looking), a "degraded" signal 1008 is output. Of course, the criteria for when to output a specific signal and what thresholds to use between the "distracted" signal 1002, the "focused" signal 1004, the "partially distracted" signal 1006, and the "degraded" signal 1008, etc., will vary depending on the specific circumstances at hand and are within the capabilities of those skilled in the art.

[0054] Finally, it should be noted that while all of the above analyses are presented in the context of a single image, in a real-world scenario, they would be done based on video data (i.e., a stream of images captured at a rate of 30 frames per second) (although other frame rates are possible), and therefore, the driver distraction signal and the system degradation signal would typically be continuous signals available throughout the operation of the vehicle. Furthermore, in order to have a more stable signal, various filtering functions (e.g., moving average, etc.) may be applied to achieve a more robust system. Such modifications are well within the capabilities of one of ordinary skill in the art.

[0055] Embodiments of the invention described herein may be systems, methods and / or computer program products at any possible level of technical detail integration for providing a driver distraction signal and providing a system degradation signal indicating the reliability of the driver distraction signal based on the gaze of the driver of the vehicle according to what has been described above. The computer program product may include (one or more) computer-readable storage media having computer-readable program instructions thereon for causing a processor to perform various aspects of the present invention.

[0056] Fig.11 is a block diagram illustrating a computer system in which an embodiment of the method may be implemented. Fig.11As can be seen in the computer system 1100, the computer system 1100 can be used in whole, in part, or with various modifications to provide the functionality of the disclosed system. The computer system 1100 includes hardware elements that can be electrically coupled via a bus 1122. The hardware elements can include one or more central processing units 1102, one or more input devices 1104 (e.g., a touch screen, a keyboard, etc.), and one or more output devices 1106 (e.g., a display device, etc.). The computer system 1100 may also include one or more storage devices 1108. For example, the storage device(s) 1108 may be a disk drive, an optical storage device, a solid-state storage device (such as a programmable, flash-updatable random access memory ("RAM") and / or a read-only memory ("ROM")), and / or the like.

[0057] The computer system 1100 may further include a computer-readable storage medium reader 1110, a communication system 1112 (eg, a modem, a network card (wireless or wired), an infrared communication device, a Bluetooth TM Device, cellular communication device, etc.), and working memory 1116 (which may include RAM devices and ROM devices as described above). In some embodiments, computer system 1100 may also include a processing acceleration unit 1114, which may include a digital signal processor, a dedicated processor and / or the like.

[0058] The computer-readable storage media reader 1110 may further be connected to computer-readable storage media to collectively represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing computer-readable information (and optionally in conjunction with storage device(s) 1108). The communication system 1112 may permit data to be exchanged with the networks, systems, computers, and / or other components described above.

[0059] The computer system 1100 may also include software elements, which are shown as currently located in the working memory 1116, including an operating system 1118 and / or other code 1120. It should be understood that alternative embodiments of the computer system 1100 may have many variations different from those described above. For example, custom hardware may also be used and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. In addition, connections to other computing devices such as network input / output devices and data acquisition devices may also occur.

[0060] The software of computer system 1100 may include code for implementing any or all of the functionality of the various elements of the architecture as described herein. For example, software stored on and / or executed by a computer system such as system 1100 may provide the functionality of the disclosed system. Methods that may be implemented by software on some of these components have been discussed in more detail above.

[0061] Although this specification includes many implementation details, these should not be interpreted as limitations on the scope of the present invention or the scope of things that may be claimed, but rather as descriptions of features specific to implementations of the present invention. Certain features described in this specification in the context of different embodiments may also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even claimed at first, one or more features of these combinations may be removed from the claimed combinations in some cases, and the claimed combinations may involve sub-combinations or variants of sub-combinations. Therefore, unless otherwise expressly stated, or unless the knowledge of a person of ordinary skill in the art clearly indicates otherwise, any feature of the above-described embodiments may be combined with any other feature of the above-described embodiments.

[0062] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the order shown or in an ordered order, or that all of the operations shown be performed to obtain the desired results. In some cases, multitasking and / or parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments. The described program components and systems may be integrated into a single software product or packaged into multiple software products.

[0063] Thus, embodiments of the invention have been described. Other embodiments are within the scope of the appended claims. Thus, many variations of the above examples are well within the scope of the appended claims and within the capabilities of one of ordinary skill in the art.

Claims

1. A system configured to provide a driver distraction signal and / or a system degradation signal indicating the reliability of the driver distraction signal based on a gaze of a driver of a vehicle, the system comprising: processor; as well as a memory comprising instructions executable by the processor, wherein the system is configured to: receiving a plurality of images of a driver of the vehicle, wherein the images include information indicative of a gaze of the driver; Determining a gaze area of ​​the driver for each image, wherein the gaze area includes a gaze point and a gaze uncertainty value; for each image, associating the fixation region with a region of a plurality of regions around the vehicle, wherein the plurality of regions include at least one predefined attention region and at least one predefined inattention region; determining a driver distraction level and / or a system degradation level indicative of reliability of the driver distraction level based on the determined gaze areas for the plurality of images, the determined gaze uncertainty values, and the associated area of ​​the vehicle; and A driver distraction signal and / or a system degradation signal are output, wherein the driver distraction signal indicates the determined driver distraction level and the system degradation signal indicates the determined system degradation level.

2. The system of claim 1, wherein: The fixation region is a probability distribution, wherein the fixation point is a mean value of the probability distribution, and the fixation uncertainty value is a standard deviation of the probability distribution.

3. The system of claim 2, wherein: The driver distraction level is based on which portions of the gaze area fall within the attention zone and the inattention zone, respectively.

4. The system of claim 3, wherein: A high driver distraction level is obtained when a substantial portion of the gaze area falls within the inattention region, and a low driver distraction level is obtained when a substantial portion of the gaze area falls within the attention region.

5. The system of claim 1, wherein: The system is further configured to: One or more threshold signal values ​​are set for the driver distraction signal and the system degradation signal to indicate one or more levels of driver distraction and one or more levels of system degradation, respectively.

6. The system of claim 1, wherein: The system is further configured to: categorizing the driver distraction signal into one of the following categories: attention, distraction, degraded, and uncertainty; and The determined reliability is associated with the determined category.

7. The system of claim 1, wherein: The gaze region is determined using a gaze estimation algorithm based on machine learning.

8. The system of claim 1, wherein: The system is further configured to control a device of the vehicle based on the outputted driver distraction signal and / or the system degradation signal.

9. A method for providing a driver distraction signal and / or a system degradation signal indicating the reliability of the driver distraction signal based on a gaze of a driver of a vehicle, the method comprising: receiving a plurality of images of a driver of the vehicle, wherein the images include information indicative of a gaze of the driver; Determining a gaze area of ​​the driver for each image, wherein the gaze area includes a gaze point and a gaze uncertainty value; for each image, associating the fixation region with a region of a plurality of regions around the vehicle, wherein the plurality of regions include at least one predefined attention region and at least one predefined inattention region; determining a driver distraction level and / or a system degradation level indicative of reliability of the driver distraction level based on the determined gaze areas for the plurality of images, the determined gaze uncertainty values, and the associated area of ​​the vehicle; and A driver distraction signal and / or a system degradation signal are output, wherein the driver distraction signal indicates the determined driver distraction level and the system degradation signal indicates the determined system degradation level.

10. A computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to perform the method of claim 9.

11. An eye tracking system configured to provide a driver distraction signal based on a gaze of a driver of a vehicle according to the method of claim 9.

12. A vehicle comprising a system as claimed in any one of claims 1 to 8.

Images