Driver State Estimation Device

Abstract

To detect as early as possible a sign that a driver driving a mobile body falls into a driving disabled state.SOLUTION: A driver state estimating device is equipped with a head portion behavior detecting portion 21 that detects a behavior of a head portion of a driver, an eyeball behavior detecting portion 22 that detects a behavior of eyeballs of the driver, a first detecting portion 110 that detects an abnormality sign of the driver only from the heat portion behavior detected by the head portion behavior detecting portion 21, and a second detecting portion 120 that detects an abnormality sign of the driver from a correlation of the head portion behavior detected by the head portion behavior detecting portion 21 and an eyeball behavior detected by the eyeball behavior detecting portion 22.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The technology disclosed herein belongs to the technical field related to a driver state estimation device that estimates the state of a driver who drives a mobile object. [Background technology]

[0002] Recently, the development of automated driving systems has been promoted nationwide. The applicant of this application believes that, at present, there are two main directions for automated driving systems.

[0003] The first direction is a system in which the car takes the initiative and transports passengers to their destination without the need for driver operation, known as fully automated driving. On the other hand, the second direction is an automated driving system that assumes that a human will still be driving the car, for example, for the enjoyment of driving.

[0004] In the second direction of automated driving systems, it is assumed that the vehicle will automatically take over from the driver and continue driving automatically when, for example, the driver develops an illness or other condition that makes normal driving difficult. For this reason, from the perspective of improving the driver's survival rate and ensuring the safety of those around the vehicle, it is extremely important to be able to detect driver abnormalities, particularly functional disorders or illnesses, as early and accurately as possible.

[0005] In Patent Document 1, the timing of the driver's eye movement and the timing of the head movement are identified, the sequence of these identified timings is determined, and the state of the driver is estimated based on the determination result.

[0006] Furthermore, Patent Document 2 discloses a condition detection device that detects a driver's inability to drive based on the amount of shaking of the driver's head, or detects a driver's inability to drive based on the degree to which the driver's eyes are white. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6068964 [Patent Document 2] Patent No. 6361312 [Non-patent literature]

[0008] [Non-Patent Document 1] T. Nakamura, et al., “Multiscale Analysis of Intensive Longitudinal Biomedical Signals and its Clinical Applications”, Proceedings of the IEEE, Institute of Electrical and Electronics Engineers, 2016, vol.104, pp.242-261 [Non-Patent Document 2] Mizuta et al., "Fractal Analysis of Center of Gravity Sway," Equilibrium Research, Japan Society of Equilibrium Medicine, 2016, Vol. 75(3), pp. 154-161 Summary of the Invention [Problem to be solved by the invention]

[0009] As shown in Patent Documents 1 and 2, a technique for determining whether a driver is unable to drive or has an abnormality based on the movement of the driver's head and the state of the driver's eyes is already known. However, Patent Document 1 can detect a decrease in concentration and physical fatigue, but cannot predict in advance the illness that the driver is suffering from. Patent Document 2 detects a state in which the driver becomes unable to drive after the illness has appeared. When an abnormality occurs in the driver, in order to perform an emergency stop or the like more safely, it is preferable to quickly detect the signs before the driver becomes unable to drive.

[0010] The technology disclosed herein has been made in consideration of the above points, and its purpose is to detect, as early as possible, a sign that a driver operating a mobile object is about to become unable to drive. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems, the technology disclosed herein is directed to a driver state estimation device that estimates the state of a driver driving a moving object, and is configured to include a head behavior detection unit that detects the behavior of the driver's head, an eye behavior detection unit that detects the behavior of the driver's eyes, a first detection unit that detects signs of an abnormality in the driver solely from the head behavior detected by the head behavior detection unit, a second detection unit that detects signs of an abnormality in the driver from a correlation between the head behavior detected by the head behavior detection unit and the eye behavior detected by the eye behavior detection unit, and a judgment unit that judges whether or not an abnormality has occurred in the driver based on the detection results of at least one of the first detection unit and the second detection unit.

[0012] The present inventors have focused on the homeostasis function of humans and conducted research into the relationship between the decline of the homeostasis function and the head and eye behaviors of drivers. Homeostasis refers to the function of trying to keep a constant state against external disturbances, and in terms of head behavior, it refers to the tendency of a driver to maintain the head posture while driving. As a result of intensive research by the present inventors, it has been found that the coordination between eye behavior and head behavior is disrupted as the driver's homeostasis function declines. Specifically, it has been found that when a driver is in a normal state, when the driver moves his / her gaze, the head also follows the gaze movement in order to maintain the gaze in a stable state, whereas when the driver's homeostasis function declines, the head has difficulty following the gaze even if the gaze moves. From this, the present inventors have found that it is effective to detect abnormal signs from the correlation between head behavior and eye behavior.

[0013] Therefore, in the technology disclosed herein, in addition to detecting abnormal signs based only on head behavior, abnormal signs are also detected based on the degree of agreement between eye behavior and head behavior. This makes it possible to detect abnormal signs as early as possible even in driving situations where it is difficult to detect abnormal signs based only on head behavior, such as constant speed driving. As a result, it is possible to detect as early as possible the signs that a driver operating a moving object will become unable to drive.

[0014] In one embodiment of the driver state estimation device, the first detection unit calculates a periodic feature for time-series data indicating a head behavior of the driver, calculates a time-series fluctuation pattern for the periodic feature obtained by the calculation, and compares the time-series fluctuation pattern obtained by the calculation with a predetermined first threshold value to detect the presence or absence of an abnormality sign of the driver.

[0015] That is, basically, if the driver is in a normal state, the driver's head behavior becomes irregular according to the acceleration input to the vehicle. However, the inventors of the present invention have conducted extensive research and found that periodicity is often observed in the head behavior even when the driver is in a normal state. Therefore, the present inventors have calculated a time series variation pattern of the periodic feature amount, and compared the obtained time series variation pattern with a predetermined threshold value to determine whether or not there is a sign of abnormality in the driver. This makes it possible to detect the sign of abnormality in the driver at an earlier stage.

[0016] In the embodiment, the first detection unit further calculates coherence between the driver's head behavior and the lateral acceleration acting on the driver, and calculates a time-series variation pattern for the periodic feature amount and the coherence obtained by the calculation.

[0017] That is, in a driving scene where the lateral acceleration acts on the driver's head, such as when cornering, periodicity is likely to appear in the head behavior even if the driver is in a normal state. Therefore, in the technology according to the present disclosure, the coherence between the driver's head behavior and the lateral acceleration acting on the driver is calculated to calculate a time-series variation pattern. This makes it possible to detect abnormal signs by taking into account the correlation between the driver's head behavior and the lateral acceleration, which is an external factor, and therefore makes it possible to detect abnormal signs in the driver at an earlier stage.

[0018] In the driver state estimation device, the second detection unit may be configured to detect the presence or absence of signs of an abnormality in the driver by comparing the amount of eye displacement detected by the eye behavior detection unit with the amount of head displacement detected by the head behavior detection unit.

[0019] That is, when the eyeballs are slightly displaced, the head may not follow. If the abnormality sign is detected based on the correlation between the eyeball displacement amount and the head displacement amount, the abnormal state can be detected based on the tendency of the head behavior relative to the eyeball behavior, including the slight eyeball movement and the relatively large eyeball movement. This makes it possible to accurately detect the abnormality sign of the driver.

[0020] In a driver state estimation device that compares eye displacement amount and head displacement amount, the second detection unit may be configured to calculate a two-dimensional map in which the head displacement amount against the eye displacement amount detected at the same time is plotted, calculate an occurrence probability distribution of the plots in the calculated two-dimensional map, and detect the presence or absence of a sign of an abnormality in the driver by comparing a degree of agreement between the calculated occurrence probability distribution and a previously calculated occurrence probability distribution in the event of an abnormality.

[0021] This configuration makes it possible to appropriately determine the difference in the degree of coordination between eye behavior and head behavior between normal and abnormal conditions, thereby enabling early and accurate detection of a driver's abnormal behavior. Effect of the Invention

[0022] As described above, the technology disclosed herein makes it possible to detect, as early as possible, a sign that a driver of a mobile object is becoming unable to drive. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a conceptual diagram for explaining the positioning of the technology according to the present disclosure. [Diagram 2] FIG. 2 shows an example of time series data of head pitch and roll angles. [Diagram 3] FIG. 3 is a graph showing the distribution of autocorrelation indices of head pitch and roll angles. [Figure 4] FIG. 4 is a graph showing the time series fluctuation of the autocorrelation index. [Diagram 5] FIG. 5 is a graph showing the classification results of the time-series fluctuation patterns of the autocorrelation index. [Figure 6] FIG. 6 shows time series data of the head pitch angle. (a) shows the normal state when driving in a straight line, (b) shows the simulated abnormal state, and (c) shows the normal state when driving around a corner. [Figure 7] FIG. 7 shows data from a driving experiment, where (a) shows when entering a corner from a straight section, and (b) shows when transitioning from a normal state to a simulated abnormal state. [Figure 8] FIG. 8 is a graph showing the time series variation of the coherence between head motion and lateral acceleration. [Figure 9] FIG. 9 is a graph showing the classification results of the time series fluctuation patterns of the coherence and autocorrelation indices. [Figure 10] FIG. 10 shows examples of time-series data of head and eye movements, where (a) shows a normal state and (b) shows an abnormal state. [Figure 11] FIG. 11 is a diagram showing the process of creating data for comparing the correlation between head behavior and eye behavior. [Figure 12] FIG. 12 is a two-dimensional map in which the eyeball displacement amount and the head displacement amount are plotted, where (a) shows a normal state and (b) shows an abnormal state. [Figure 13] FIG. 13 is a distribution diagram obtained by calculating the occurrence probability distribution from the two-dimensional map of FIG. 12, where (a) shows a normal state and (b) shows an abnormal state. [Figure 14] FIG. 14 shows time-series data indicating head movement, head-eye coincidence, head displacement amount, and eye displacement amount, where (a) indicates a normal state, and (b) indicates an abnormal state. [Figure 15] FIG. 15 is a configuration diagram of an in-vehicle system including a driver state estimating device. [Figure 16] FIG. 16 is a flowchart showing a process for estimating a driver's state based only on the head behavior. [Figure 17] FIG. 17 is a flowchart showing a process of estimating a driver's state based on the coordination between the head behavior and the eye behavior. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Exemplary embodiments will now be described in detail with reference to the drawings.

[0025] FIG. 1 is a conceptual diagram showing the positioning of the technology according to the present disclosure. The state change that causes the driver to become unable to drive is summarized into three patterns. Case A is a pattern in which some of the functions of perception, judgment, and movement decline, Case B is a pattern in which the overall function declines gradually, and Case C is a pattern in which consciousness is suddenly lost. Of these, in the cases of Cases A and B, as shown in FIG. 1, the driver's driving ability level gradually declines after the onset of the disease, and eventually leads to a state in which the driver is unable to drive. Therefore, if this state of decline in driving ability can be detected, it is possible to detect the signs of the driver's inability to drive. If the signs of inability to drive can be detected, then, for example, the driver's intention can be confirmed and emergency measures such as pulling the vehicle to the shoulder by automatic driving control can be taken. Even in the case of Case C, if the signs before loss of consciousness can be detected, emergency measures can be taken.

[0026] The technology according to the present disclosure focuses on the homeostasis maintenance function of humans and detects a sign of inability to drive (hereinafter, referred to as an abnormality sign) from the driver's head behavior and the driver's eye behavior. Specifically, in this embodiment, the abnormality sign is detected based only on the driver's head behavior, and the abnormality sign is detected based on the coordinated movement of the driver's head behavior and eye behavior.

[0027] Humans have a function called homeostasis, which tries to keep a constant state against disturbances. Head behavior homeostasis refers to the tendency to maintain head posture while driving. In addition, when focusing on coordinated movement between the head and eyes, it refers to moving the head in the same direction as the eyes in order to maintain a stable line of sight.

[0028] Detecting signs of abnormality based solely on head behavior First, detection of an abnormality sign based only on head behavior will be described.

[0029] Various studies have been conducted on the above-mentioned human homeostasis, and for example, Non-Patent Document 1 describes that in a normal state, the human head moves irregularly to maintain homeostasis, whereas in a disease state, the head movement becomes smaller and more stable. Also, Non-Patent Document 2 describes that in a critical deceleration state in which the normal state transitions to a disease state, the head moves periodically.

[0030] The inventors of the present application have focused on the periodicity of head behavior and have conceived of detecting a sign of an abnormality in the driver by utilizing a periodic feature amount of head behavior.

[0031] FIG. 2 shows the results of an experiment conducted by the inventors of the present application on the behavior of the driver's head. In this experiment, the driver was made to drive on a test course, and the behavior of the driver's head while driving was measured. The driver was made to drive normally as usual (corresponding to a normal state), and while driving, he was given an abnormality simulation task (difficult mental arithmetic) as a task with a cue (corresponding to an abnormality prediction state). As data representing the behavior of the head, the pitch angle (angle in the forward and backward directions) and roll angle (angle in the left and right directions) of the head were measured from the image of a camera capturing the driver. Then, the autocorrelation index (scaling exponent α) was obtained from the time series data of the pitch angle and roll angle of the head by DFA (Detrended Fluctuation Analysis). DFA is a method for investigating scaling by removing components that change very slowly, so-called trends. The autocorrelation index (scaling exponent α) is an example of a feature that indicates the periodicity of data.

[0032] Figure 2 is an example of time series data on the pitch and roll angles of the head. Figure 3 is a graph showing the distribution of the autocorrelation index of the pitch and roll angles of the head. In Figure 3, the horizontal axis is the autocorrelation index of the roll angle, and the vertical axis is the autocorrelation index of the pitch angle. A smaller value of the autocorrelation index indicates a stronger autocorrelation, and a larger value indicates a weaker autocorrelation.

[0033] As shown in Figure 3, in abnormality precursor states, the distribution of autocorrelation indices is biased toward stronger autocorrelation than in normal states. This experimental result is consistent with the previously mentioned finding that in normal states, the head moves irregularly, whereas in critical deceleration states, the head moves periodically. However, as can be seen from Figure 3, the distribution of normal states is spread over a relatively wide range, and there is a large area that overlaps with the distribution of abnormality precursor states. In other words, it is not necessarily easy to identify abnormality precursor states using only autocorrelation indices.

[0034] Therefore, the inventors of the present application focused on the time-series fluctuation patterns of the autocorrelation index. As shown in Fig. 4, the time-series data of the autocorrelation index was extracted in chronological order, and this was used as a time-series fluctuation pattern for pattern classification. Here, a nonlinear dimensionality reduction method (UMAP: Uniform Manifold Approximation and Projection) was used as the pattern classification method.

[0035] Figure 5 shows the classification results of the time-series fluctuation patterns of autocorrelation indices. In Figure 5, the time-series fluctuation patterns are reduced to two dimensions by UMAP and mapped onto a two-dimensional map. In Figure 5, the distribution of normal states and the distribution of abnormality precursor states are more clearly separated than in Figure 3. The decision line LTH shown in Figure 5 was obtained by using a support vector machine on two-dimensional data. This decision line LTH achieved an error rate of 15% in discriminating abnormality precursor states.

[0036] From the above, it is considered that it is possible to detect signs of abnormality in the driver by calculating periodic feature values ​​for time series data representing the behavior of the driver's head, calculating a time series fluctuation pattern for the periodic feature values ​​obtained by the calculation, and comparing the time series fluctuation pattern obtained by the calculation with a predetermined threshold value.

[0037] However, as a result of continued investigations, the inventors of the present application have found that there are cases in which the behavior of the head resembles an abnormality precursor state even when the driver is in a normal state.

[0038] FIG. 6 shows time series data of head pitch angle obtained by an experiment by the inventors of the present application, where (a) is a case where the driver drives in a straight line under normal conditions, (b) is a case where the driver drives in an abnormal simulation condition, and (c) is a case where the driver drives around a corner under normal conditions. Each graph shows raw data and its moving average. As shown in FIG. 6(a), in the case where the driver drives in a straight line under normal conditions, the head fluctuates irregularly, while as shown in FIG. 6(b), in the case where the driver drives in an abnormal simulation condition, the head fluctuates regularly. However, as shown in FIG. 6(c), in the case where the driver drives around a corner under normal conditions, the head fluctuates regularly.

[0039] That is, in the case of Fig. 6(c), even though the driver is in a normal state, the autocorrelation index of the head pitch angle data becomes large, which makes it difficult to distinguish from the case of Fig. 6(b). It is presumed that the regular movement of the driver's head when cornering is caused by the lateral acceleration acting on the head.

[0040] Figure 7 shows data obtained from a driving experiment, where (a) is data when the vehicle enters a corner from a straight section of the circuit, and (b) is data when the driver changes from a normal state to a simulated abnormal state. In Figures 7(a) and (b), the upper graph shows the time series changes in vehicle lateral acceleration and head roll angle, and the lower graph shows the time series changes in coherence (cross-correlation index) between vehicle lateral acceleration and head roll. Coherence is found by taking the correlation between the power spectrum obtained from the time series changes in vehicle lateral acceleration and the power spectrum obtained from the time series changes in head roll angle. In the coherence graph, the horizontal axis is time and the vertical axis is frequency, with lighter colors indicating higher correlation and darker colors indicating lower correlation.

[0041] Looking at the data in the upper part of Figure 7(a), we can see that the head moves in conjunction with the change in lateral acceleration after the vehicle enters the corner. In other words, when the lateral acceleration acts and the head starts to shake, the driver corrects the head in the opposite direction to the lateral acceleration. Therefore, a correlation between the vehicle lateral acceleration and the head roll angle gradually begins to appear after the vehicle enters the corner. On the other hand, in the case of Figure 7(b), the head roll angle is not linked to the change in the vehicle lateral acceleration, and the coherence is almost unchanged.

[0042] From the above experimental results, it was found that periodicity is likely to appear in head behavior when the driver's head is subjected to large lateral acceleration, such as when cornering. The frequency at which correlation appears is mainly 2 Hz or less. This is thought to reflect the frequency response of the head to external forces.

[0043] Therefore, in order to further improve the accuracy of abnormality sign determination, the inventors of the present application have considered calculating feature quantities including cross-correlation indices with external factors (here, lateral acceleration) as well as the autocorrelation indices of head behavior, and classifying the patterns. That is, as shown in FIG. 8, the time series data of coherence was extracted in chronological order, and this was combined with the time series data of the autocorrelation indices described above to perform pattern classification. Here, as the time series data of coherence, data with a frequency of 1 Hz, which is likely to show correlation with vehicle lateral acceleration, and data with a frequency of 4 Hz, which is not likely to show correlation with vehicle lateral acceleration, were used. As the method of pattern classification, a nonlinear dimension reduction method (UMAP) was used.

[0044] Figure 9 shows the classification results of the time-series fluctuation patterns of the coherence and autocorrelation indices. In Figure 9, similar to Figure 5, the time-series fluctuation patterns are reduced to two dimensions by UMAP and mapped onto a two-dimensional map. The decision line LTH2 shown in Figure 9 was obtained by using a support vector machine on two-dimensional data. This decision line LTH2 made it possible to achieve an error rate of 1% in discriminating between abnormality precursor states.

[0045] From the above, in this embodiment, when detecting a driver's abnormality sign based only on the driver's head behavior, a periodic feature amount is calculated for the time-series data representing the driver's head behavior, and further, the coherence between the driver's head behavior and the lateral acceleration acting on the driver's head is calculated, a time-series fluctuation pattern is calculated for the periodic feature amount and the coherence obtained by the calculation, and the time-series fluctuation pattern obtained by the calculation is compared with a predetermined threshold value to detect the driver's abnormality sign. This makes it possible to make a judgment that takes into account the correlation between the head behavior and the lateral acceleration that is an external factor, and to detect the driver's abnormality sign with even greater accuracy.

[0046] (Detection of abnormal signs based on coordinated head-eye movements) Next, detection of an abnormality sign based on coordinated movement between the driver's head behavior and eye behavior will be described.

[0047] Figure 10 shows the simultaneous measurement of the driver's head behavior and eye behavior. Both the driver's head behavior and eye behavior were calculated from images captured by an in-vehicle camera. In Figure 10, (a) shows the behaviors when the driver is in a normal state, and (b) shows the behaviors when the driver is in an abnormal state. In each graph, the vertical axis represents the angle (hereinafter referred to as yaw angle) that the head and eyes have moved left and right from the reference position.

[0048] As shown in Fig. 10(a), when the yaw angle of the eyeball is small, the head hardly moves, but when the yaw angle of the eyeball is relatively large, the head moves to correspond to the movement of the eyeball. On the other hand, referring to Fig. 10(b), when the driver has a sign of abnormality, the head hardly moves even if the eyeball moves a lot. This is because the driver's homeostatic function is reduced, and the brain cannot keep up with the movement of the head, so the driver tries to see the object with only the eyes.

[0049] Based on the findings obtained from such experiments, the inventors of the present application have considered detecting a driver's abnormal sign by comparing the yaw angle of the head with the yaw angle of the eyeballs.The inventors of the present application have discovered that a driver's abnormal sign can be detected by making a two-dimensional map of the yaw angle of the driver's eyeballs and the yaw angle of the head when the yaw angle is measured, and calculating the degree of agreement between the two-dimensional map and the two-dimensional map when the driver is in an abnormal state.

[0050] In this embodiment, in order to calculate the degree of agreement between two-dimensional maps, the appearance probability distribution Q of plots on the two-dimensional map of the yaw angle of the driver's eyeballs and the yaw angle of the head is calculated by probability density estimation (KDE: Kernel density estimation) for the two-dimensional map.

[0051] Specifically, as shown in Fig. 11, first, a two-dimensional map 1101 is created by plotting the head yaw angle against the eye yaw angle of the driver detected at the same time from the time series data of the head yaw angle and the eye yaw angle for a predetermined time period. The predetermined time period is, for example, about 5 to 10 seconds.

[0052] Next, a probability distribution map 1102 showing the occurrence probability distribution Q is created for the created two-dimensional map 1101 by KDE. In calculating the probability distribution map 1102, the two-dimensional map 1101 is divided into meshes of a predetermined size, and the occurrence probability of a plot is calculated for each mesh. As shown in Fig. 11, in the probability distribution map 1102, the more densely the plots are in a mesh, the higher the occurrence probability. Note that when calculating the probability distribution map 1102 from the two-dimensional map 1101, a publicly known program can be used.

[0053] FIG. 12 shows a two-dimensional map 1101a (FIG. 12(a)) when the driver is in a normal state, and a two-dimensional map 1101b (FIG. 12(b)) when the driver is in an abnormal state. As shown in FIG. 12, when the driver is in a normal state, there are many plots with large head yaw angles. In particular, when the driver is in a normal state, it is understood that the larger the eye yaw angle, the more likely the head yaw angle is to also be large. This indicates that the head moves in accordance with the movement of the eyeballs due to human homeostasis. On the other hand, when the driver is in an abnormal state, it is understood that even if the eye yaw angle becomes large, the head yaw angle is small. This is because the head becomes difficult to move due to a decline in homeostasis function.

[0054] Fig. 13 shows a probability distribution map 1102a (Fig. 13(a)) calculated for the two-dimensional map 1101a in Fig. 12(a), and a probability distribution map 1102b (Fig. 13(b)) calculated for the two-dimensional map 1101b in Fig. 12(b). As shown in Fig. 13, when the driver is in a normal state, the occurrence probability is distributed even in the area where the head yaw angle is large, whereas when the driver is in an abnormal state, the occurrence probability is distributed only in the area where the head yaw angle is small.

[0055] In this embodiment, a probability distribution map 1102b showing an occurrence probability distribution P when the driver is in an abnormal state as shown in Fig. 13(b) is stored as data in advance, and a sign of abnormality in the driver is detected by determining the degree of agreement with an occurrence probability distribution Q calculated while the driver is driving. Specifically, the Kullback-Leibler distance (KL distance) between the latest occurrence probability distribution Q and the occurrence probability distribution P that is a model of the abnormal state is calculated. The KL distance is calculated by the following formula.

[0056]

number

[0057] In this formula, i represents the coordinate of the mesh.

[0058] The KL distance becomes smaller as the degree of agreement between the latest occurrence probability distribution Q and the occurrence probability distribution P during an abnormality increases, and becomes 0 when they completely match. Therefore, when the KL distance is equal to or smaller than a predetermined value, it is considered that the driver has a sign of an abnormality, and when the KL distance is larger than the predetermined value, it is considered that the driver is in a normal state.

[0059] FIG. 14 shows the results of detecting the head yaw angle and eye yaw angle of the driver, and calculating the KL distance between the occurrence probability distribution calculated from the detection result and the occurrence probability distribution in an abnormal state. In this experiment, the driver was made to drive using a driving simulator, and the head behavior and eye behavior of the driver while driving were measured. FIG. 14(a) shows the results detected from a driver in a normal state, and FIG. 14(b) shows the results detected from the above-mentioned case C patient. In both FIG. 14(a) and (b), the upper graph shows the KL distance, and the lower graph shows the detected head yaw angle and eye yaw angle. In FIG. 14(b), an abnormality occurs in the driver at the end of the graph.

[0060] As shown in FIG. 14(a), when the driver is normal, the driver's head and eyes move in roughly coordinated motion, and the KL distance remains relatively high. On the other hand, as shown in FIG. 14(b), when the driver shows signs of abnormality, the KL distance suddenly becomes smaller and changes to a value close to 0. This is thought to be because the homeostatic function is reduced as a sign of abnormality, and the coordinated movement function of the head and eyes is reduced. Therefore, the actual experimental results also prove that the driver's signs of abnormality can be detected by calculating the KL distance between the latest occurrence probability distribution Q and the occurrence probability distribution P in the abnormal state. In FIG. 14(b), the presence of an abnormal sign is detected at the time indicated by the dashed line after the KL distance becomes equal to or less than a predetermined value. In this way, even based on the coordinated movement of the head and eyes, an abnormal sign can be detected before an abnormality actually occurs. In particular, it was found to be effective for the patient in case C. By using this KL distance to determine the abnormal sign state, a misjudgment rate of 0.2% was achieved.

[0061] <Configuration of the Driver State Estimation Device> Fig. 15 shows an example of the configuration of an in-vehicle system including a driver state estimation device. In the in-vehicle system shown in Fig. 15, a camera 10, an acceleration sensor 11, a speaker 12, an information presentation unit 13, a switch 14, and a microphone 15 are mounted inside the vehicle. The information processing device 20 is composed of, for example, a single IC chip having a processor and a memory, or multiple IC chips having a processor and a memory. A vehicle stop control unit 40 receives an instruction from the information processing device 20 and performs control to automatically move the vehicle to the shoulder of the road and stop it.

[0062] The camera 10 is installed, for example, on the inside of the windshield and captures the situation inside the vehicle including the driver. The image captured by the camera 10 is transmitted to the information processing device 20, for example, via an in-vehicle network.

[0063] In the information processing device 20, the head behavior detection unit 21 detects the behavior of the driver's head from the image captured by the camera 10. For example, the head of the driver is recognized from the image, and the tilt angles of the head, such as the pitch angle, roll angle, and yaw angle, are obtained. The processing in the head behavior detection unit 21 can be realized by existing image processing technology. The processing by the head behavior detection unit 21 makes it possible to obtain time-series data of the head behavior as shown in Figs. 2, 10, and 14. The time-series data of the head behavior is sent to the first detection unit 110 and the second detection unit 120, which detect abnormal signs of the driver.

[0064] In the information processing device 20, the eye behavior detection unit 22 detects the behavior of the driver's eye from the image captured by the camera 10. For example, the driver's eye is recognized from the image, and the inclination angle of the iris of the eye, for example, the yaw angle, is obtained. The processing in the eye behavior detection unit 22 can be realized by existing image processing technology. The processing by the eye behavior detection unit 22 can obtain time series data of eye behavior as shown in Figs. 10 and 14. The time series data of eye behavior is sent to a second detection unit 120 that detects a sign of abnormality in the driver.

[0065] The first detection unit 110 detects a driver's abnormal sign only from the head behavior. The first detection unit 110 includes a periodic feature amount calculation unit 111, a time series variation pattern calculation unit 112, an abnormality determination unit 113, an abnormality determination threshold database 114, and a coherence calculation unit 115. The periodic feature amount calculation unit 111 calculates a periodic feature amount from the time series data of the head behavior obtained by the head behavior detection unit 21. Specifically, for example, an autocorrelation index (scaling exponent α) is obtained as the periodic feature amount by DFA. The periodic feature amount calculation unit 111 can obtain time series data of the periodic feature amount as shown in FIG. 4.

[0066] The coherence calculation unit 115 calculates the coherence between the driver's head behavior and the lateral acceleration acting on the driver's head using the output of the acceleration sensor 11 and the time-series data of the head behavior obtained by the head behavior detection unit 21. For example, the coherence calculation unit 115 regards the acceleration in the left-right direction of the vehicle recognized from the output of the acceleration sensor 11 as the lateral acceleration acting on the driver's head, and calculates a power spectrum from the time-series data of the lateral acceleration. The coherence calculation unit 115 also calculates a power spectrum from the time-series data of the head behavior, and calculates the coherence by taking a correlation between the power spectra. The coherence calculation unit 115 then calculates time-series data of, for example, 1 Hz as a frequency where the correlation with the lateral acceleration is likely to appear, and time-series data of, for example, 4 Hz as a frequency where the correlation with the lateral acceleration is not so apparent.

[0067] The time-series fluctuation pattern calculation unit 112 extracts, in chronological order, the time-series data of the periodicity feature obtained by the periodicity feature calculation unit 111 and the time-series data of the coherence obtained by the coherence calculation unit 115, calculates a time-series fluctuation pattern from the combination of the extracted time-series data, and performs pattern classification. Specifically, for example, the time-series data of the combination of the periodicity feature and the coherence is subjected to dimensionality reduction using UMAP, which is one of the nonlinear dimensionality reduction methods, and converted into two-dimensional data.

[0068] The abnormality determination unit 113 compares the data obtained by the time-series fluctuation pattern calculation unit 112 with the thresholds stored in the abnormality determination threshold database 114, and determines whether or not the driver has experienced an abnormality precursor. For example, the determination line LTH2 in the two-dimensional map of Fig. 9 corresponds to the thresholds stored in the abnormality determination threshold database 114. The abnormality determination unit 113 determines whether or not the driver has experienced an abnormality precursor depending on which side of the determination line LTH2 the data obtained by the time-series fluctuation pattern calculation unit 112 is located on the two-dimensional map.

[0069] The second detection unit 120 detects a driver's abnormal sign from the correlation between head behavior and eye behavior. The second detection unit 120 includes a head-eye displacement amount distribution calculation unit 121, an abnormality degree calculation unit 122, an abnormality determination unit 123, an abnormality probability distribution database 124, and an abnormality determination threshold database 125. The head-eye displacement amount distribution calculation unit 121 calculates a two-dimensional map 1101 of the eye displacement amount and the head displacement amount detected at the same time, based on the time series data of head behavior obtained by the head behavior detection unit 21 and the time series data of eye behavior obtained by the eye behavior detection unit 22.

[0070] The abnormality degree calculation unit 122 divides the two-dimensional map 1101 into a plurality of meshes and calculates the occurrence probability distribution for each mesh by, for example, KDE to create a probability distribution map 1102. In this way, it is possible to obtain the probability distribution map 1102 as shown in Fig. 13. Furthermore, the abnormality degree calculation unit 122 calculates the degree of agreement between the created probability distribution map 1102 and a probability distribution map in an abnormal state stored in the abnormality probability distribution database 124. The abnormality degree calculation unit 122 calculates the degree of agreement between the probability distribution maps by, for example, calculating the KL distance between the probability distribution maps.

[0071] The abnormality determination unit 123 compares the data obtained by the abnormality degree calculation unit 122 with a threshold stored in the abnormality determination threshold database 125, and determines whether or not the driver has experienced an abnormality sign. The threshold is set to, for example, 1. When the KL distance obtained by the abnormality degree calculation unit 122 is 1 or less, the abnormality determination unit 123 determines that the driver has an abnormality sign, whereas when the KL distance obtained by the abnormality degree calculation unit 122 is greater than 1, the abnormality determination unit 123 determines that the driver has no abnormality sign.

[0072] When the abnormality determination units 113, 123 determine that the driver has a sign of abnormality, the first detection unit 110 and the second detection unit 120 output to the interrogation unit 23 a request to interrogate the driver.

[0073] The interrogation unit 23 queries the driver when it receives an inquiry request from at least one of the first detection unit 110 and the second detection unit 120. This inquiry is for confirming the driver's intention as to whether or not it is acceptable to make the vehicle automatically drive to an emergency evacuation. The inquiry is made, for example, by voice through the speaker 12 or by display through the information presentation unit 13 such as a monitor.

[0074] The response detection unit 24 detects the driver's response to the question posed by the questioning unit 23. The driver responds, for example, by operating the switch 14 or by speaking via the microphone 15. When the driver's intention is confirmed or when there is no response from the driver, the information processing device 20 instructs the vehicle stop control unit 40 to automatically move the vehicle to the shoulder of the road and stop the vehicle.

[0075] The driver state estimating device according to the present embodiment includes at least a head behavior detecting unit 21, an eye behavior detecting unit 22, a first detecting unit 110, and a second detecting unit 120 in an information processing device 20. The driver state estimating device according to the present disclosure may also include a camera 10.

[0076] FIG. 16 is a flowchart showing the processing operation of driver state estimation based only on the head behavior.

[0077] First, in step S11, head behavior detection unit 21 recognizes the driver's head from the image captured by camera 10, and calculates the inclination angles, here the pitch angle and roll angle, of the recognized head. This calculation is performed, for example, every 100 ms.

[0078] Next, in step S12, first detection unit 110 determines whether or not a prescribed number of pieces of head tilt angle data have been accumulated. If the determined result is YES, that is, the prescribed number of pieces of head tilt angle data has been accumulated, first detection unit 110 proceeds to step S13. On the other hand, if the determined result is NO, that is, the prescribed number of pieces of head tilt angle data has not been accumulated, first detection unit 110 returns to step S11.

[0079] In step S13, the first detection unit 110 calculates a periodic feature amount for the time series data of the head tilt angle. The first detection unit 110 obtains an autocorrelation index (scaling exponent α) as the periodic feature amount by, for example, DFA. The calculation of the periodic feature amount is performed, for example, every 100 ms, using, for example, 256 pieces of tilt angle data while shifting the time range of the target tilt angle data.

[0080] Next, in step S14, first detection unit 110 uses the output of vehicle acceleration sensor 50 and the time-series data of the head tilt angle to calculate the coherence between the behavior of the driver's head and the lateral acceleration acting on the driver's head.

[0081] Next, in step S15, the first detection unit 110 determines whether or not the data of the periodicity feature and the coherence has been accumulated to a prescribed number or more. If the result is YES, that is, the data has been accumulated to a prescribed number or more, the first detection unit 110 proceeds to step S16. On the other hand, if the result is NO, that is, the data has not been accumulated to a prescribed number or more, the first detection unit 110 returns to step S11.

[0082] In step S16, the first detection unit 110 calculates a time series variation pattern for the combined data of the periodic feature and coherence. The first detection unit 110 reduces the dimension of the time series variation data of the periodic feature and coherence by, for example, using UMAP, which is one of the nonlinear dimensionality reduction techniques, and converts it into two-dimensional data. The calculation of the time series variation pattern is performed, for example, every 100 ms, using, for example, 256 pieces of periodic feature and coherence data while shifting the time range of the target periodic feature and coherence data.

[0083] Next, in step S17, the first detection unit 110 acquires the threshold value stored in the abnormality judgment threshold database 114. Specifically, for example, the first detection unit 110 acquires, as the threshold value, information on a judgment line in a two-dimensional map of the classification pattern, such as the judgment line LTH2 shown in Fig. 9.

[0084] Then, in step S18, the first detection unit 110 compares the two-dimensional data, which is the time-series fluctuation pattern obtained in step S16, with the judgment line acquired in step S17 to judge whether the two-dimensional data has crossed the judgment line. If the two-dimensional data has crossed the judgment line and is on the abnormality sign side (YES), the first detection unit 110 judges that the driver has a abnormality sign and proceeds to step S19. On the other hand, if the two-dimensional data has not crossed the judgment line and is on the normal side (NO), the first detection unit 110 judges that the driver has no abnormality sign and returns to step S11.

[0085] In step S19, first detection unit 110 outputs a request to inquire of the driver to inquiring unit 23. After step S19, the process ends.

[0086] FIG. 17 is a flowchart showing the processing operation of driver state estimation based on the correlation between head behavior and eye behavior.

[0087] First, in step S21, head behavior detection unit 21 performs image recognition of the driver's head from the image captured by camera 10, and calculates the head displacement amount (here, yaw angle) for the recognized head. Also, eyeball behavior detection unit 22 performs image recognition of the driver's eyes from the image captured by camera 10, and calculates the eyeball displacement amount (here, yaw angle) for the recognized eyeball. This calculation is performed, for example, every 100 ms.

[0088] Next, in step S22, the second detection unit 120 determines whether or not the head displacement amount data and eye displacement amount data have been accumulated to a prescribed number or more. If the head and eye displacement amount data have been accumulated to a prescribed number or more (YES), the second detection unit 120 proceeds to step S23. On the other hand, if the head or eye displacement amount data has not been accumulated to a prescribed number or more (NO), the second detection unit 120 returns to step S21.

[0089] In step S23, the second detection unit 120 calculates a two-dimensional map indicating the head-eye displacement amount distribution from the time-series data of the head and eye displacement amounts.

[0090] Next, in step S24, the second detection unit 120 calculates a probability distribution map indicating an occurrence probability distribution for the two-dimensional map calculated in step S23. The second detection unit 120 calculates the probability distribution map, for example, by dividing the two-dimensional map into a plurality of meshes and calculating the occurrence probability distribution for each mesh by KDE.

[0091] Next, in step S25, the second detection unit 120 compares the occurrence probability distribution calculated in step S24 with the occurrence probability distribution under abnormal conditions stored in the abnormal-condition probability distribution database. Specifically, the second detection unit 120 calculates the KL distance between the two occurrence probability distributions.

[0092] Next, in step S26, the second detection unit 120 determines whether the KL distance calculated in step S25 is equal to or less than a predetermined value. If the KL distance is equal to or less than the predetermined value (YES), the second detection unit 120 determines that the driver has a sign of abnormality and proceeds to step S27. On the other hand, if the KL distance is greater than the predetermined value (NO), the second detection unit 120 determines that the driver has no sign of abnormality and returns to step S21.

[0093] In step S27, second detection unit 120 outputs a request to inquire of the driver to inquiring unit 23. After step S27, the process ends.

[0094] In this way, in addition to detecting abnormal signs based on head behavior alone, abnormal signs can be detected based on the correlation between head behavior and eye behavior, making it possible to detect abnormal signs early and with high accuracy. That is, for example, in a scene of constant-speed driving where there is almost no acceleration on the head and the periodic feature amount of the driver's head behavior cannot be clearly calculated, the accuracy of detecting abnormal signs based on head behavior alone decreases. However, by detecting abnormal signs based on the correlation between head behavior and eye behavior, it is possible to appropriately detect abnormal signs of the driver even in such a driving scene.

[0095] In this embodiment, the first detection unit 110 that detects the driver's abnormal sign calculates a periodic feature amount for time-series data representing the driver's head behavior, calculates the coherence between the driver's head behavior and the lateral acceleration acting on the driver's head, calculates a time-series fluctuation pattern for the periodic feature amount and the coherence, and compares the obtained time-series fluctuation pattern with a predetermined threshold to determine the presence or absence of the driver's abnormal sign. This makes it possible to make a judgment that takes into account the correlation between the head behavior and the lateral acceleration that is an external factor, so that the driver's abnormal sign can be detected earlier and more accurately. In addition, the driver's head behavior can be detected from the image captured by the camera 10 installed in the vehicle, so that the present technology can quickly capture the signs of diseases that lead to driving inability using existing in-vehicle sensors.

[0096] In this embodiment, the second detection unit 120 that detects the driver's abnormal sign calculates a two-dimensional map in which the head displacement amount is plotted against the eye displacement amount detected at the same time, calculates the occurrence probability distribution Q of the plot in the calculated two-dimensional map, and detects the presence or absence of the driver's abnormal sign by comparing the degree of agreement between the calculated occurrence probability distribution Q and the occurrence probability distribution P in abnormality calculated in advance. That is, when the eyeballs are slightly displaced, the head may not follow. If the occurrence probability distribution of the plot in the two-dimensional map in which the eyeball displacement amount and the head displacement amount are plotted is calculated and the abnormal sign is detected based on the occurrence probability distribution, an abnormal state can be detected by the tendency of the head behavior relative to the eyeball behavior, including the slight movement of the eyeball and the relatively large movement. This makes it possible to detect the driver's abnormal sign early and with high accuracy.

[0097] (Other embodiments) The technology disclosed herein is not limited to the above-described embodiment, and may be substituted without departing from the spirit and scope of the claims.

[0098] For example, in the above embodiment, the first detection unit 110 obtains an autocorrelation index (scaling exponent α) as a periodic feature amount for the time series data of the head behavior by DFA. However, the method for obtaining the periodic feature amount is not limited to this, and for example, FFT (Fast Fourier Transform) may be used, or simple autocorrelation may be calculated.

[0099] In the above embodiment, the second detection unit 120 uses the KL distance to determine the degree of agreement between the probability distribution map at the time of an abnormality and the latest probability distribution map in determining the abnormality sign. However, without being limited to this, for example, the degree of agreement between the two-dimensional map at the time of an abnormality and the latest two-dimensional map may be compared using, for example, the chi-square method.

[0100] In the above-described embodiment, the interrogation unit 23 is configured to interrogate the driver when an interrogation request is received from either the first detection unit 110 or the second detection unit 120. However, the present invention is not limited to this, and the interrogation unit 23 may be configured to interrogate the driver only when an interrogation request is received from both the first detection unit 110 and the second detection unit 120. Alternatively, the information processing device 20 may be configured to request the interrogation unit 23 to interrogate the driver only when both the first detection unit 110 and the second detection unit 120 detect an abnormality sign.

[0101] In the above-described embodiment, the occurrence probability distribution P at the time of an abnormality is stored in the abnormality probability distribution database 124. This occurrence probability distribution P at the time of an abnormality may be updated based on the accumulated occurrence probability distribution Q by accumulating an occurrence probability distribution Q including the time of a driver's abnormality sign. In addition, the thresholds stored in the abnormality determination threshold database 125 may also be updated based on the accumulated occurrence probability distribution Q.

[0102] The application of the technology according to the present disclosure is not limited to automobiles, but is also effective for detecting abnormal signs in a driver of a moving object other than an automobile, such as a train.

[0103] In addition, the technology according to the present disclosure may be realized in a form other than a single information processing device. For example, the first detection unit 110 and the second detection unit 120 may be realized by an information processing device other than the head behavior detection unit 21 and the eye behavior detection unit 22. For example, an information processing device not mounted on a moving body, such as a smartphone or tablet held by a driver, may realize the functions of the first detection unit 110 and the second detection unit 120. Alternatively, a cloud may execute part or all of the arithmetic processing performed by the first detection unit 110 and the second detection unit 120.

[0104] The above-described embodiments are merely illustrative and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and variations within the scope of the claims are within the scope of the present disclosure. [Industrial Applicability]

[0105] The technology disclosed herein is useful as a driver state estimation device that estimates the state of a driver who drives a mobile object. [Explanation of symbols]

[0106] 21 Head behavior detection unit 22 Eye movement detection unit 110 First detection unit 120 Second detection unit

Claims

1. A driver state estimation device that estimates a state of a driver who drives a moving object, comprising: a head behavior detection unit that detects a behavior of the driver's head; An eye behavior detection unit that detects an eye behavior of the driver; a first detection unit that detects an abnormality sign of the driver only from the head behavior detected by the head behavior detection unit; a second detection unit that detects an abnormality sign of the driver based on a correlation between the head behavior detected by the head behavior detection unit and the eye behavior detected by the eye behavior detection unit, The first detection unit is Calculating a periodic feature amount for time series data indicating a driver's head behavior; A time series fluctuation pattern is calculated for the periodic feature obtained by the calculation; A driver state estimation device characterized by comparing a time-series variation pattern obtained by the calculation with a predetermined first threshold value to detect the presence or absence of a sign of abnormality in the driver.

2. 2. The driver state estimating device according to claim 1, The first detection unit is Further calculating the coherence between the driver's head behavior and the lateral acceleration acting on the driver; A driver state estimation device, comprising: a driver state estimation unit that calculates a time-series fluctuation pattern for the periodicity feature amount and coherence obtained by the calculation.

3. 3. The driver state estimating device according to claim 1, The second detection unit detects the presence or absence of signs of an abnormality in the driver by comparing the amount of eye displacement detected by the eye behavior detection unit with the amount of head displacement detected by the head behavior detection unit.

4. A driver state estimation device that estimates a state of a driver who drives a moving object, comprising: a head behavior detection unit that detects a behavior of the driver's head; An eye behavior detection unit that detects an eye behavior of the driver; a first detection unit that detects an abnormality sign of the driver only from the head behavior detected by the head behavior detection unit; a second detection unit that detects an abnormality sign of the driver based on a correlation between the head behavior detected by the head behavior detection unit and the eye behavior detected by the eye behavior detection unit, The second detection unit detects the presence or absence of signs of an abnormality in the driver by comparing the amount of eye displacement detected by the eye behavior detection unit with the amount of head displacement detected by the head behavior detection unit.

5. 5. The driver state estimating device according to claim 3, The second detection unit is A two-dimensional map is calculated by plotting the head displacement against the eye displacement detected at the same time. Calculating an occurrence probability distribution of plots in the calculated two-dimensional map; A driver state estimating device characterized in that the presence or absence of a sign of an abnormality in the driver is detected by comparing the degree of agreement between the calculated occurrence probability distribution and a pre-calculated occurrence probability distribution in the event of an abnormality.

Images