Human state estimation method and estimation system

Abstract

To estimate a human state of an evaluated person with high accuracy by calculations that are simple as much as possible, in estimation of the human state using a saliency map.SOLUTION: A saliency map is generated for an image contained in front view of an evaluated person. A first probability that a saliency at a position on the saliency map indicated by a line of sight of the evaluated person is equal to or higher than a predetermined threshold value is calculated. A probability that a saliency at a position on the saliency map indicated by a line of sight of a comparison person who is another person having attributes equivalent to those of the evaluated person is equal to or higher than the predetermined threshold value is calculated. It is estimated whether the evaluated person is in an abnormal state, based on the first and second probabilities.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The technology disclosed herein belongs to the technical field related to a human state estimation method and an estimation system. [Background technology]

[0002] It has been considered in the past to create a saliency map from an image showing the person's surroundings, which indicates the attractiveness to the person of objects in the person's field of vision, and to use the saliency map to estimate various states of the person being evaluated.

[0003] For example, in Patent Document 1, a saliency map is used to estimate an object to which the gaze of the person being evaluated is directed. Specifically, Patent Document 1 discloses that, based on the detection result of the gaze direction, a gaze area to which the gaze of the driver (person being evaluated) is directed is extracted in a captured image showing the periphery of the vehicle, objects included in the captured image are recognized, a top-down saliency map of the captured image is generated based on the captured image and vehicle information, objects having saliency in the top-down saliency map among the recognized objects are detected as candidates of the visual object to which the driver's gaze is directed, and the visual object to which the driver's gaze is directed is determined based on the extraction result of the gaze area and the detection result of the candidate visual object.

[0004] Furthermore, Patent Document 1 defines a top-down saliency map as a saliency map that absorbs individual differences such as the driver's viewing order, physique, and habits, ambiguity of the scene from which the captured image was obtained, and fluctuations in the gaze direction detected by the gaze detection unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-71528 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the inventors of the present application conducted extensive research and found that the state of the person being evaluated can be estimated by using a saliency map. In other words, when the person being evaluated is in a normal state and when he / she is absent-minded, there is a difference in the relationship between the saliency in the person's field of vision and the movement of the person's gaze, and by evaluating this difference, the state of the person being evaluated can be estimated.

[0007] In the evaluation using this saliency map, it is expected that the estimation accuracy will be improved if a univocally determined saliency map is used, such as the top-down saliency map in Patent Document 1. However, it is very difficult to define how to absorb individual differences such as the driver's habits and the ambiguity of the image, and in the worst case, there is a risk that even information necessary for estimating the state of the person being evaluated will be absorbed, which may actually lower the estimation accuracy. In addition, the amount of calculation required to generate the saliency map may become enormous, and the estimation time may become long.

[0008] The technology disclosed here has been developed in consideration of these points, and its purpose is to accurately estimate the human state of the person being evaluated using as simple calculations as possible when estimating the human state using a saliency map. [Means for solving the problem]

[0009] In order to solve the above problems, the technology disclosed herein is directed to a human state estimation method and includes a peripheral information acquisition process for acquiring peripheral information of the person being evaluated, a field of view image acquisition process for acquiring an image included in the forward field of view of the person being evaluated, a map generation process for generating a saliency map for the image acquired in the field of view image acquisition process, a first gaze information acquisition process for acquiring the gaze of the person being evaluated with respect to the forward field of view, a second gaze information acquisition process for acquiring the gaze of a comparator who is another person having attributes similar to those of the person being evaluated, a first probability calculation process for calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated acquired in the first gaze information acquisition process will be equal to or greater than a predetermined threshold, a second probability calculation process for calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the comparator acquired in the second gaze information acquisition process will be equal to or greater than the predetermined threshold, and an abnormality estimation process for estimating whether the person being evaluated is in an abnormal state based on the first probability calculated in the first probability calculation process and the second probability calculated in the second probability calculation process.

[0010] According to this configuration, unlike the conventional method, there is no need to absorb individual differences such as the habits of the person being evaluated when generating a saliency map, making it easier to generate the saliency map. Also, by comparing the relationship of the gaze of the person being evaluated to the saliency map with the relationship of the gaze of another person (comparator) who has attributes similar to the person being evaluated to the saliency map, it is possible to improve the estimation accuracy of the person being evaluated's human state without using a top-down saliency map. Therefore, the human state of the person being evaluated can be estimated with high accuracy by the simplest possible calculation.

[0011] The "attributes" referred to here include the space in which the comparator is located, the age of the comparator, the skill level of the comparator for the target job (driving a vehicle or sorting products in a factory), and the like.

[0012] In the human state estimation method, the abnormality estimation process may be configured to be a process of estimating that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value.

[0013] According to this configuration, when the subject is momentarily distracted, a deviation occurs between the first probability and the second probability, which can prevent the subject from being estimated to be in an abnormal state. This can improve the accuracy of estimating the subject's human state.

[0014] In the human state estimation method, the second gaze information acquisition process may be a process of acquiring the gazes of two or more of the comparators, and the second probability calculation process may be a process of obtaining saliency at positions on the saliency map indicated by each gaze of each of the comparators, calculating the probability that each saliency will be equal to or greater than the specified threshold, and setting the average value of each probabilistic probability or one probability selected from the respective probabilities as the second probability.

[0015] According to this configuration, slight differences between the person being evaluated and each comparison subject can be absorbed to some extent, thereby further improving the accuracy of estimating the human state of the person being evaluated.

[0016] In another aspect of the technology disclosed herein, the human state estimation method includes a peripheral information acquisition process for acquiring peripheral information including the space in which the person being evaluated is located, a field of view image acquisition process for acquiring an image included in the forward field of view of the person being evaluated, a map generation process for generating a saliency map for the image acquired in the field of view image acquisition process, a first gaze information acquisition process for acquiring the gaze of the person being evaluated with respect to the forward field of view, a first probability calculation process for calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated acquired in the first gaze information acquisition process will be greater than or equal to a predetermined threshold, a second probability calculation process for calculating the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated who was in the same space in the past will be greater than or equal to the predetermined threshold, and an abnormality estimation process for estimating whether the person being evaluated is in an abnormal state based on the first probability calculated in the first probability calculation process and the second probability calculated in the second probability calculation process.

[0017] According to this configuration, since the evaluation is compared with past evaluation subjects, it is possible to absorb as much of the individual differences, such as the subject's habits and physique, as possible without using a top-down saliency map. This makes it possible to accurately estimate the human condition of the subject with as simple a calculation as possible.

[0018] The techniques disclosed herein are also directed to human evaluation systems. Specifically, the human state estimation system is configured to include a peripheral information acquisition unit that acquires peripheral information of the person being evaluated, a field of view image acquisition unit that acquires an image included in the forward field of view of the person being evaluated, a map generation unit that generates a saliency map for the image acquired by the field of view image acquisition unit, a gaze information acquisition unit that acquires gaze information of the person being evaluated with respect to the forward field of view, a comparator gaze information acquisition unit that acquires gaze information of a comparator who is another person having attributes similar to that of the person being evaluated, a first probability calculation unit that calculates the probability that the saliency at a position on the saliency map indicated by the gaze information of the person being evaluated acquired by the gaze information acquisition unit will be greater than or equal to a predetermined threshold, a second probability calculation unit that calculates the probability that the saliency at a position on the saliency map indicated by the gaze of the comparator acquired by the comparator gaze information acquisition unit will be greater than or equal to the predetermined threshold, and an abnormality estimation unit that estimates whether the person being evaluated is in an abnormal state based on the first probability calculated by the first probability calculation unit and the second probability calculated by the second probability calculation unit.

[0019] Even with this configuration, it is not necessary to absorb individual differences such as the habits of the person being evaluated when generating a saliency map, as in the past, so that the generation of the saliency map becomes easy. In addition, by comparing the relationship of the gaze of the person being evaluated to the saliency map with the relationship of the gaze of the comparison person to the saliency map, it is possible to improve the estimation accuracy of the human state of the person being evaluated without using a top-down saliency map. Therefore, the human state of the person being evaluated can be estimated with high accuracy by the simplest calculation possible.

[0020] In the human state estimation system, the abnormality estimation unit may be configured to estimate that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value.

[0021] With this configuration, the accuracy of estimating the human state of the person being evaluated can be improved.

[0022] In the human state estimation system, the comparator gaze information acquisition unit may acquire the gazes of two or more of the comparators, and the second probability calculation unit may obtain the saliency at the positions on the saliency map indicated by each gaze of each of the comparators, calculate the probability that each saliency will be greater than or equal to the specified threshold, and set the second probability to be an average value of each probabilistic probability or one probability selected from the each probabilities.

[0023] With this configuration, the accuracy of estimating the human state of the person being evaluated can be further improved.

[0024] The human state estimation system may further include a memory unit that stores information regarding the gaze of the person being evaluated, and when the comparison person is not present around the person being evaluated, the second probability calculation unit may read out information regarding the gaze of the person being evaluated who was in the same space in the past from the memory unit, and calculate the second probability as the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated in the past will be greater than or equal to the specified threshold.

[0025] According to this configuration, even in a situation where a comparison person is unlikely to be nearby, such as driving a vehicle late at night, the human state of the person being evaluated can be estimated with high accuracy. In addition, since information about the evaluator's past line of sight is stored, the second probability can be easily calculated. Therefore, the human state of the person being evaluated can be estimated with high accuracy by the simplest possible calculation. Effect of the Invention

[0026] As described above, according to the technology disclosed herein, since the evaluation is made by comparing the person being evaluated with a comparator who has similar attributes to the person being evaluated, or with the person being evaluated when they were in the same space in the past, there is no need to use a top-down saliency map, and the human state of the person being evaluated can be accurately estimated with as simple a calculation as possible. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a schematic diagram of the inside of a vehicle when the human state of a driver of the vehicle is estimated by the human state estimating system according to the first embodiment. [Diagram 2] FIG. 2 is a front view of the vehicle as seen from the front side. [Diagram 3] FIG. 3 is a block diagram showing the configuration of the human state estimating system. [Figure 4] FIG. 4 is a schematic diagram showing the relationship between a vehicle and a cloud server in the human state estimation system. [Diagram 5] FIG. 5 is a diagram showing the external environment in front of the vehicle captured by a camera. [Figure 6] FIG. 6 is a diagram showing the distribution of saliency calculated based on images captured by a camera. [Figure 7] FIG. 7 is a graph showing the saliency of the driver's gaze point. [Figure 8] FIG. 8 is a graph showing the saliency of the gaze points of the comparison subjects. [Figure 9] FIG. 9 is a graph showing an ROC curve comparing the rate at which the comparison subject's gaze point exceeds the threshold with the rate at which the driver's gaze point exceeds the threshold. [Figure 10] FIG. 10 is a diagram showing the concept of calculating the AUC value from the ROC curve shown in FIG. [Figure 11] FIG. 11 is a flowchart of human state estimation. [Figure 12] FIG. 12 is a schematic diagram showing a state in a factory when the human state of workers in the factory is estimated by the human state estimating system according to the second embodiment. [Figure 13] FIG. 13 is a plan view illustrating an example of the arrangement of cameras and the like. [Figure 14] FIG. 14 is a block diagram showing the configuration of a human state estimating system according to the second embodiment. [Figure 15] FIG. 15 is a flowchart of human state estimation by the human state estimating system according to the second embodiment. [Figure 16]FIG. 16 is a block diagram showing a configuration when the human state estimating system according to the third embodiment is applied to a vehicle. [Figure 17] FIG. 17 is a flowchart of human state estimation by the human state estimating system according to the third embodiment. [Figure 18] FIG. 18 is a block diagram showing a configuration when the human state estimating system according to the fourth embodiment is applied to a vehicle. [Figure 19] FIG. 19 is a part of a flowchart of human state estimation by the human state estimation system according to the fourth embodiment. [Figure 20] FIG. 20 shows the remaining part of the flowchart of human state estimation by the human state estimation system according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] First Embodiment An exemplary embodiment 1 will be described in detail below with reference to the drawings. In the embodiment 1, a human state estimation system 50 (see FIG. 3) is applied to a driver of a car. In the following description of the embodiment 1, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. In addition, the left side when looking at the front side from the rear side is referred to as the left side, and the opposite is referred to as the right side.

[0029] 1 shows a schematic diagram of the interior of a vehicle cabin that employs a human state estimation system 50. This vehicle is a right-hand drive vehicle, with a steering wheel 8 disposed on the right side.

[0030] A windshield glass 1 is disposed in the vehicle interior at the front of the vehicle as viewed from the driver's seat. The windshield glass 1 is defined by a number of vehicle components as viewed from inside the vehicle interior. Specifically, the windshield glass 1 is defined by left and right front pillar trims 2, a roof trim 3, and an instrument panel 4.

[0031] The left and right front pillar trims 2 each constitute an outer boundary in the vehicle width direction of the front windshield glass 1. Each front pillar trim 2 is disposed along each front pillar 12. As shown in FIG. 1, each front pillar trim 2 is disposed at an angle so as to be spaced apart from each other toward the upper side.

[0032] The roof trim 3 forms the upper boundary of the front windshield glass 1. The roof trim 3 covers the interior side of the vehicle's roof panel 13. A rearview mirror 5 is attached to the center of the front windshield glass 1 in the vehicle width direction and slightly below the roof trim 3. An in-vehicle camera 101 (see FIG. 3) is provided in the roof trim 3 near the rearview mirror 5 to capture images of the interior of the vehicle, and in particular the face of the driver (hereinafter referred to as the person being evaluated). The in-vehicle camera 101 is composed of a high-performance camera that can detect changes in the pupils of the person being evaluated.

[0033] The instrument panel 4 constitutes the lower boundary of the front windshield glass 1. The instrument panel 4 is provided with a meter box and a display 7.

[0034] The vehicle also has side mirrors 6 located on the outside of the left and right front pillars 12 in the vehicle width direction. Each side mirror 6 is positioned so that the subject sitting in the driver's seat can see through the window of the side door.

[0035] The vehicle is provided with an exterior camera 100 for capturing images of the external environment in front of the vehicle, as shown in Fig. 2. The exterior camera 100 is disposed at the front end of the vehicle, slightly below the bonnet 9 of the vehicle.

[0036] The human state estimation system 50 according to this embodiment estimates the state of the person being evaluated by comparing the gaze movement of the person being evaluated with the gaze movement of other drivers (hereinafter referred to as comparators) who are present around the vehicle and have attributes similar to those of the person being evaluated. The "attributes" referred to here refer to the space in which the comparator is located, the age of the comparator, the proficiency of the comparator in the target job (driving a vehicle in this case), and the like. In other words, a comparator refers to, for example, a driver who is located in the same space as the person being evaluated, is at the same level as the person being evaluated, and has a driving level similar to that of the person being evaluated. The attributes of the comparator do not need to be strictly the same as those of the person being evaluated, and may be attributes that can be considered to be at the same level as those of the person being evaluated.

[0037] 3 is a schematic diagram of the configuration of the human state estimation system 50. The human state estimation system 50 uses a cloud server 70 for some of the calculations. The cloud server 70 acquires information about the assessee from the vehicle VA driven by the assessee, and acquires information about the comparator from another vehicle VB driven by the comparator.

[0038] The vehicle VA has a controller 60 for calculating gaze information of the subject. This controller 60 is a controller based on a known microcomputer, and includes a central processing unit (CPU) for executing a program, a memory configured of, for example, a random access memory (RAM) or a read only memory (ROM) for storing the program and data, and an input / output bus for inputting and outputting electrical signals.

[0039] The controller 60 is connected to various sensors as shown in FIG. An exterior camera 100 for capturing an image included in the forward field of view of the person being evaluated; An in-car camera 101 that captures the face of the person being evaluated; A vehicle speed sensor 102 for detecting the speed of the host vehicle VA; an acceleration sensor 103 for detecting a longitudinal acceleration input to the host vehicle VA; A yaw rate sensor 104 for detecting a lateral acceleration input to the host vehicle VA; a steering angle sensor 105 for detecting the steering angle of the host vehicle VA from the rotation angle of a steering shaft which rotates in conjunction with the steering wheel 8; A steering torque sensor 106 for detecting a steering torque input to the steering wheel 8 via the steering shaft; an accelerator opening sensor 107 that detects an accelerator opening from an amount of depression of an accelerator pedal of the host vehicle VA; A brake pressure sensor 108 that detects a brake pressure based on a depression amount of a brake pedal of the host vehicle VA; A GPS sensor 109 that detects the position of the host vehicle VA using a Global Positioning System (GPS); Includes.

[0040] Furthermore, the controller 60 is connected to a cloud server receiver 71 that acquires information from a cloud server 70. The cloud server receiver 71 and the cloud server 70 are capable of communicating with each other via wireless communication.

[0041] The controller 60 has a gaze calculation unit 61 that calculates the gaze direction of the person being evaluated from the eyeball of the person being evaluated photographed by the in-vehicle camera 101. The gaze calculation unit 61 calculates the gaze direction of the person being evaluated by detecting a change in the pupil of the person being evaluated, for example, based on the state in which the person being evaluated looks into the lens of the in-vehicle camera 101. The gaze direction may be calculated from either the left or right eye of the person being evaluated, or may be calculated as the average of the gaze directions (gaze vectors) obtained from both eyes of the person being evaluated. In addition, when it is difficult to calculate the gaze direction of the person being evaluated from the change in the pupil of the person being evaluated, the gaze direction may be calculated by further considering the direction of the face of the person being evaluated. The in-vehicle camera 101 and the controller 60 of the vehicle VA constitute a gaze information acquisition unit.

[0042] The controller 60 has a map generator 62 that calculates saliency, which indicates the degree to which a target in front of the vehicle attracts the subject's gaze, based on image data acquired by the exterior camera 100, and generates a saliency map. Saliency is a visual feature consisting of saliency that changes from moment to moment due to color, brightness, movement, etc. In other words, high saliency indicates high saliency, and a region with high saliency is a region that stands out from the surroundings. More specifically, a region that has a large color difference or brightness difference from the surrounding regions or that moves significantly from the surroundings has high saliency.

[0043] The vehicle VA has a cloud server transmitter 72 for transmitting information from the various sensors 100 to 109 to the cloud server 70. The cloud server transmitter 72 and the cloud server 70 are capable of communicating with each other via wireless communication.

[0044] The vehicle VA has a driving assistance device 110 that assists in driving the vehicle VA, such as steering and accelerating / decelerating, depending on the estimated state of the person being evaluated.

[0045] The configuration of the other vehicle VB is basically the same as that of the host vehicle VA, and therefore detailed description will be omitted. The gaze calculation unit 61 of the other vehicle VB constitutes a comparison person gaze information acquisition unit that acquires gaze information of a comparison person.

[0046] The cloud server 70 detects other vehicles VB located around the host vehicle VA based on information from the GPS sensors 109 of the host vehicle VA and the other vehicles VB. The GPS sensor 109 and the cloud server 70 constitute a surrounding information acquisition unit.

[0047] Based on information from various sensors 100-109 acquired from the vehicle VA and the other vehicle VB, the cloud server 70 determines whether the attributes of the driver of the other vehicle VB are equivalent to the attributes of the person being evaluated. As shown in FIG. 4, the cloud server 70 acquires information from each of the multiple other vehicles VB located around the vehicle VA, and compares the attributes of the driver of each of the other vehicles VB with the attributes of the person being evaluated. Then, the cloud server 70 selects a driver of each of the other vehicles VB who has attributes equivalent to the person being evaluated as a comparator. When there are two or more comparators, the cloud server 70 selects two or more comparators. The cloud server 70 determines whether the attributes of the driver of each of the other vehicles VB are equivalent to the attributes of the person being evaluated based on the travel distance, travel frequency, and acceleration / deceleration timing of each of the other vehicles VB.

[0048] Cloud server 70 calculates the probability (hereinafter referred to as the first probability) that the saliency at a position on the saliency map indicated by the gaze information of the person being evaluated will be equal to or greater than a predetermined threshold, based on the saliency map from host vehicle VA and the gaze direction of the person being evaluated. Cloud server 70 also calculates the probability (hereinafter referred to as the second probability) that the saliency at a position on the saliency map indicated by the gaze information of the comparator will be equal to or greater than a predetermined threshold, based on the saliency map from other vehicle VB and the gaze direction of the comparator. The calculation methods of the first probability and the second probability will be described later. Cloud server 70 constitutes a first probability calculation unit and a second probability calculation unit.

[0049] The cloud server 70 estimates whether the assessee is in an abnormal state based on the calculated first probability and second probability. The cloud server 70 transmits the estimation result to the host vehicle VA. The cloud server 70 constitutes an abnormality estimation unit.

[0050] (Generating saliency maps) Next, generation of the saliency map by the map generating unit 62 will be described.

[0051] 5 is an image captured by the vehicle exterior camera 100 and included in the forward field of view of the subject. The forward field of view shown in this image includes a roadway 150 and a white line 151 on the roadway 150. The external environment shown in this image D1 also includes another vehicle 161, a wall 162 formed on the right side of the roadway 150, a tree 163 planted in an area to the right of the wall 162, a hill 164 extending in an area to the right of the wall 162, and a forest 165 formed on the left side of the roadway 150. The external environment shown in this image D1 also includes a building 166 and a sky 167 extending above the building 166 and the forest 165.

[0052] The map generating unit 62 calculates saliency for the captured image to generate a saliency map. As described above, saliency varies depending on the color, brightness, movement, etc. of the target. The map generating unit 62 calculates saliency for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, generates a saliency map for each feature, and then adds them together to generate a final saliency map.

[0053] For example, for saliency based on color, the map generator 62 increases the saliency of nearby areas in areas with large color differences from the surroundings in the image, compared to areas with small color differences. Note that color difference is calculated by the following formula when the RGB of a pixel is (R1, G1, B1) and the RGB of another pixel is (R2, G2, B2). (color difference)={(R2-R1) 2 +(G2-G1) 2 +(B2-B1) 2} 1 / 2 The map generating unit 62 may calculate the saliency so that it continuously increases as the color difference increases, or may set a plurality of thresholds and calculate the saliency so that it increases by a certain value each time the threshold is exceeded.

[0054] Also, for example, with respect to saliency based on luminance, the map generating unit 62 increases the saliency of a neighboring region in a portion of the image having a large luminance difference from the surroundings, compared to a portion of the image having a small luminance difference. The map generating unit 62 may calculate the saliency so that it continuously increases as the luminance difference increases, or may set multiple thresholds and calculate the saliency so that it increases by a certain value each time the threshold is exceeded.

[0055] Furthermore, for saliency based on movement, the map generating unit 62 increases the saliency for an object that has a large relative movement with respect to the host vehicle VA. For example, when a bird flies across the sky 167, the map generating unit 62 increases the saliency of the area surrounding the bird.

[0056] For the calculation of saliency itself, a known computer program available on the Internet, etc. can be used. Also, for the calculation of a saliency map for each feature (here, a saliency map) and the integration of each saliency map, a known computer program can be used.

[0057] Fig. 6 is a saliency map in the case where saliency is calculated for an image captured by the exterior camera 100. As shown in Fig. 6, for example, the boundary between the trees 163 and the forest 165 and the sky 167, and the white line 151 on the left side of the roadway 150 are areas with high saliency (high saliency areas) because of high color difference and brightness difference. Conversely, the roadway 150 and the sky 167 are mostly similar and have low color difference, so there are many areas with low saliency.

[0058] Since the vehicle VA and the other vehicle VB are located at different positions, the saliency maps generated by the map generating units 62 do not strictly match. However, as long as the majority of the targets are the same, the difference between the saliency maps is not significant enough to cause a problem in estimating the human state, which will be described later.

[0059] (Estimation of the human condition) Next, estimation of a human state will be described with reference to FIGS.

[0060] Fig. 7 plots the saliency of the gaze point of the person being evaluated, that is, the point in the gaze of the person being evaluated. This graph is generated by determining the gaze direction of the person being evaluated by the gaze calculation unit 61, applying the gaze point in the gaze direction to a saliency map, and obtaining the saliency of the gaze point from the saliency map. Each point is obtained at a predetermined sampling period.

[0061] On the other hand, Fig. 8 plots the saliency of the gaze point of the comparator, that is, the point in the gaze of the comparator. This graph is generated by calculating the gaze direction of the comparator by the gaze calculation unit 61 of the other vehicle VB, applying the gaze point in the gaze direction to the saliency map generated by the map generation unit 62 of the other vehicle VB, and calculating the saliency of the gaze point from the saliency map. Each of these points is calculated at a predetermined sampling period.

[0062] After creating the graphs of Fig. 7 and Fig. 8, the cloud server 70 obtains a receiver operating characteristic (ROC) curve of the percentage of the gaze point of the comparator exceeding the threshold and the percentage of the gaze point of the assessee exceeding the threshold. Specifically, in the first step, the threshold is set to a value greater than the maximum value in the graphs of Fig. 7 and Fig. 8. Next, in the second step, the percentage of points exceeding the threshold is obtained for each threshold while decreasing the threshold. Then, the process of this second step is repeated until the threshold becomes equal to or less than the minimum value in the graphs of Fig. 7 and Fig. 8. The percentage of the gaze point of the assessee exceeding the threshold corresponds to a first probability, and the percentage of the gaze point of the comparator exceeding the threshold corresponds to a second probability.

[0063] The graphs shown in Fig. 7 and Fig. 8 are generated at intervals of a predetermined time. The predetermined time is set to a time that is sufficient to create an ROC curve and that allows the human state to be estimated as early as possible. The predetermined time is, for example, 30 seconds.

[0064] After the first and second steps, the cloud server 70, as a third step, creates an ROC curve as shown in Fig. 9 by plotting the second probability on the horizontal axis and the first probability on the vertical axis. Since the horizontal and vertical axes of the graph in Fig. 9 are both probabilities, the minimum value of the curve is 0 and the maximum value is 1.

[0065] In FIG. 9, the dashed line indicates that there is almost no difference between the gaze movement of the assessee and the gaze movement of the comparator. In contrast, when the curve spreads upward in a convex manner, as in the curve C1 in FIG. 9, it indicates that the gaze of the assessee exceeds the threshold more frequently than the gaze of the comparator. In other words, when a curve with a shape like the curve C1 in FIG. 9 is calculated, it means that the gaze of the assessee tends to look at a high saliency area compared to the gaze of the comparator. On the other hand, when the curve spreads downward in a convex manner, as in the curve C2 in FIG. 9, it indicates that the gaze of the assessee exceeds the threshold less frequently than the gaze of the comparator. In other words, when a curve with a shape like the curve C2 in FIG. 9 is calculated, it means that the gaze of the assessee tends to look at an area with low saliency (low saliency area) compared to the gaze of the comparator.

[0066] In this embodiment, the cloud server 70 obtains an AUC (Area Under Curve) value, which is the area under the ROC curve, as a fourth step. For example, when the ROC curve is the curve C1, the AUC value corresponds to the area of ​​the hatched region in FIG. 10. By calculating the AUC value, it is possible to evaluate the degree of deviation between the first probability and the second probability, that is, the degree of deviation between the gaze of the assessee and the gaze of the comparator. Specifically, when the gaze of the assessee tends to be more oriented toward the high saliency region than the comparator, the AUC value becomes larger, whereas when the gaze of the assessee tends to be more oriented toward the low saliency region than the comparator, the AUC value becomes smaller.

[0067] Next, as a fifth step, an estimated index value is calculated, which is the ratio of AUC values ​​that deviate from the reference value by a predetermined value or more among each AUC value for a preset measurement period. The measurement period is longer than the predetermined time, and includes a plurality of predetermined times. In this embodiment, the reference value is set to 0.5. That is, since the comparator has the same attributes as the assessee, if the assessee is in a normal state, there is not much difference between the first probability and the second probability. Therefore, if the assessee is in a normal state, the AUC value is 0.5. In this embodiment, the negative side of the predetermined value is set to 0.2, and the positive side is set to 0.1. The reason why the positive side is a stricter standard than the negative side is that when the assessee is in an abnormal state such as a absent-minded state, the assessee tends to direct his / her gaze toward a high saliency area. The estimated index value represents the deviation between the saliency of the assessee's gaze point and the saliency of the comparator's gaze point. Specifically, the larger the estimated index value, the greater the deviation between the saliency of the gaze point of the subject and the saliency of the gaze point of the comparison subject.

[0068] Then, in a sixth step, the cloud server 70 judges whether the estimated index value is equal to or greater than a specific value set in advance. Specifically, if the estimated index value is equal to or greater than the specific value, it is estimated that the assessee is in a different state from the comparator, that is, in an abnormal state. In this embodiment, the specific value is set to 60%.

[0069] In this way, by comparing the saliency of the areas gazed upon by the person being evaluated with the saliency of the areas gazed upon by the comparison person, it is possible to reduce dependency on the driving scene, such as the number and spread of high saliency areas.

[0070] In particular, the first probability based on the gaze of the assessee is compared with the second probability based on the gaze of the comparator to calculate the AUC value, and the human state of the assessee can be accurately estimated. Specifically, for example, if the rate at which the assessee looked at the high saliency area is calculated, even if the assessee looks at the entire field of view without bias, the assessee will be determined to look at the high saliency area frequently when the high saliency area is a driving scene that spreads over the entire saliency map. In addition, when the assessee is a driver with little driving experience, the field of view tends to be narrow, and the gaze may be biased to a specific location. As in this embodiment, by comparing with the saliency of the location gazed by the comparator, the dependency of the driving scene, such as the number and spread of high saliency areas, and the dependency of individual differences between assessees can be reduced. This makes it possible to accurately estimate the human state of the assessee based on the gaze information of the assessee.

[0071] In addition, when there is no comparison person around the subject, the cloud server 70 randomly specifies multiple coordinates on the saliency map generated by the controller 60 of the subject's vehicle VA, and calculates the saliency of each coordinate. Then, the second probability is calculated from the saliency of the random points. If the subject is in a normal state, the subject's line of sight is basically directed evenly to the entire field of vision without being pulled to a high saliency area. Therefore, even when the second probability calculated from the saliency of the random points is compared with the first probability, if the subject is in a normal state, the ACU will be close to 0.5. Therefore, even if the second probability is calculated from the saliency of the random points, the accuracy will not be significantly reduced.

[0072] (flowchart) Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 50 will be described with reference to the flowchart of FIG.

[0073] First, in step S101, the human state estimation system 50 acquires various data including peripheral information about the host vehicle VA and an image of the forward field of view of the person being evaluated.

[0074] Next, in step S102, the human state estimation system 50 acquires gaze information of the subject from the image captured by the in-vehicle camera 101 of the subject's vehicle VA. Although omitted in the flowchart, at the time of this step S102, the controller 60 of the subject's vehicle VA generates a saliency map based on the image data acquired in the step S101. The gaze information of the subject acquired in this step S102 is information including the saliency of the subject's gaze.

[0075] Next, in step S103, the human state estimation system 50 determines whether or not a driver of another vehicle VB who can be a comparator is present around the host vehicle VA. The human state estimation system 50 determines the presence or absence of a comparator based on the position information of the other vehicle VB and the detection results of various sensors 102-108 related to driving. If the result is YES, which indicates that a comparator is present, the human state estimation system 50 proceeds to step S104. On the other hand, if the result is NO, which indicates that a comparator is not present, the human state estimation system 50 proceeds to step S108.

[0076] In step S104, the human state estimation system 50 acquires gaze information of the comparison person from an image captured by the in-vehicle camera 101 of the other vehicle VB. Note that this step S104 is not performed after the judgment of step S103, but is actually performed in the other vehicle VB simultaneously with step S102. Although not shown in the flowchart, at the time of step S104, the controller 60 of the other vehicle VB also generates a saliency map based on the image data acquired by the other vehicle VB. The gaze information of the comparison person acquired in step S104 is information including the saliency of the gaze of the comparison person.

[0077] Next, in step S105, the human state estimation system 50 judges whether a predetermined time has elapsed, and in particular whether gaze information of the assessee and the comparator has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 50 proceeds to step S106. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 50 returns to step S101. The reason for returning to step S101 is that the situation around the assessee changes before the predetermined time has elapsed, and there is also a change in the presence or absence of a comparator, etc.

[0078] In step S106, the human state estimating system 50 calculates a first probability from the gaze information of the person to be evaluated.

[0079] Next, in step S107, the human state estimating system 50 calculates a second probability from the gaze information of the comparison person.

[0080] On the other hand, in step S108, which is reached when the determination in step S103 is NO, the human state estimation system 50 determines whether a predetermined time has elapsed, in particular whether the gaze information of the assessee and the comparison person has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 50 proceeds to step S109. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 50 returns to step S101.

[0081] In step S109, the human state estimating system 50 calculates a first probability from the gaze information of the person to be evaluated.

[0082] Next, in step S110, the human state estimation system 50 calculates a second probability from a random point selected from the saliency map generated by the controller 60 of the host vehicle VA.

[0083] Next, in step S111, the human state estimation system 50 calculates an ROC curve based on the first probability calculated in step S106 or step S109 and the second probability calculated in step S107 or step S110.

[0084] Next, in step S112, the human state estimation system 50 calculates an AUC value from the ROC curve calculated in step S111.

[0085] Next, in step S113, the human state estimation system 50 judges whether data for the measurement period, i.e., AUC values ​​for the measurement period, have been obtained. If the AUC values ​​for the measurement period have been obtained (YES), the human state estimation system 50 proceeds to step S114, whereas if the AUC values ​​for the measurement period have not been obtained (NO), the human state estimation system 50 returns to step S101 and repeats steps S101 to S112 until AUC values ​​for the measurement period are obtained.

[0086] Then, in step S114, the human state estimation system 50 judges whether or not the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a specific value set in advance. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 50 proceeds to step S115 and judges that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 50 proceeds to step S115 and judges that no abnormality has occurred in the person being evaluated.

[0087] After steps S115 and S116, the process returns.

[0088] The cloud server 70 transmits the estimation result to the controller 60 of the vehicle VA. When the controller 60 of the vehicle VA receives the result that an abnormality has occurred in the person being evaluated, the controller 60 of the vehicle VA supports the person being evaluated in driving the vehicle VA by the driving assistance device 110. When the condition of the person being evaluated is estimated to be such that the person being evaluated cannot continue driving the vehicle VA, the controller 60 of the vehicle VA moves the vehicle VA to the shoulder of the road by the driving assistance device 110 and then stops the vehicle VA. The controller 60 of the vehicle VA may transmit a control signal to an alarm device such as a speaker to ask the person being evaluated, "Are you okay?", "Would you like to take a break?", etc.

[0089] In step S103, when there are two or more comparators, the cloud server 70 selects two or more comparators. When the cloud server 70 selects two or more comparators, after the second probability is calculated for each comparator, the cloud server 70 obtains an average value of each second probabilities and calculates an ROC curve using the average value as the final second probability.

[0090] Therefore, the human state estimation system 50 according to the first embodiment acquires peripheral information of the person being evaluated (peripheral information acquisition process), acquires an image included in the forward field of view of the person being evaluated (field of view image acquisition process), generates a saliency map for the acquired image (map generation process), acquires gaze information of the person being evaluated (first gaze information acquisition process), acquires gaze information of a comparator who is another person having attributes similar to the person being evaluated (second gaze information acquisition process), calculates a first probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated is equal to or greater than a predetermined threshold (first probability calculation process), calculates a second probability that the saliency at a position on the saliency map indicated by the gaze of the comparator is equal to or greater than a predetermined threshold (second probability calculation process), and estimates whether the person being evaluated is in an abnormal state based on the first probability and the second probability. This makes it easier to generate the saliency map because it is not necessary to absorb individual differences such as the habits of the person being evaluated when generating the saliency map. In addition, by comparing the relationship of the gaze of the person being evaluated to the saliency map with the relationship of the gaze of the comparison person to the saliency map, it is possible to improve the estimation accuracy of the human state of the person being evaluated. Therefore, the human state of the person being evaluated can be estimated with high accuracy by the simplest calculation possible.

[0091] In addition, in the present embodiment 1, the human state estimation system 50 estimates that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a specific value set in advance. This makes it possible to prevent the person being evaluated from being estimated to be in an abnormal state due to a momentary distraction that causes a deviation between the first probability and the second probability. As a result, it is possible to improve the estimation accuracy of the person being evaluated's human state.

[0092] In addition, in the present embodiment 1, when there are two or more comparators, the human state estimation system 50 acquires the gaze information of each of the comparators, obtains the saliency at the position on the saliency map indicated by each gaze of each comparator, calculates the probability that each saliency is equal to or greater than the predetermined threshold, and sets the average of each probability as the final second probability. This makes it possible to absorb slight differences between the person being evaluated and each of the comparators to a certain extent. As a result, the estimation accuracy of the human state of the person being evaluated can be further improved.

[0093] Second Embodiment Hereinafter, the second embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals are used to designate the same parts as those in the first embodiment, and detailed description thereof will be omitted.

[0094] In the second embodiment, the human state estimation system 250 (see FIG. 14) is applied to factory workers. In the illustrated factory, as shown in FIG. 12, a worker as the assessee HA is a worker who inspects the product P flowing on the conveyor device 220. On the line where the assessee HA works, multiple workers including the assessee HA (two workers in FIG. 12 and FIG. 13) are working. In the following explanation, the conveyor device 220 side from the worker's point of view is referred to as the front side, and the opposite side of the conveyor device 220 from the worker's point of view is referred to as the rear side. In addition, the workers other than the assessee HA shown in FIG. 12 and FIG. 13 are described as comparators HB, who are other people with attributes equivalent to the assessee HA. The attributes referred to here are, for example, age, height, and work proficiency (years of work experience).

[0095] 13 and 14 show a schematic configuration of a human state estimation system 250 according to the second embodiment. The human state estimation system 250 includes a rear camera 200 that acquires images of the forward field of view of the assessee HA and the comparator HB. The rear camera 200 is a fixed camera and is located behind the assessee HA and the comparator HB. A plurality of rear cameras 200 (two in this example) are provided so that an image of the forward field of view of the assessee HA and an image of the forward field of view of the comparator HB can be acquired. The human state estimation system 250 includes a front camera 201 for acquiring the line of sight of the assessee HA and the comparator HB. The front camera 201 is a fixed camera and is located in front of the assessee HA and the comparator HB. The front camera 201 is configured as a high-performance camera capable of detecting changes in the pupils of the assessee HA and the comparator HB.

[0096] The human state estimation system 250 has a controller 260 for calculating the gaze information of the assessee HA and the comparator HB. The controller 260 is a controller based on a known microcomputer, and includes a central processing unit (CPU) for executing a program, a memory configured, for example, of a RAM (Random Access Memory) or a ROM (Read Only Memory) for storing programs and data, and an input / output bus for inputting and outputting electrical signals. Image data from the rear camera 200 and the front camera 201 are input to the controller 260. The controller 260 is also connected to a cloud server receiver 271 for acquiring information from a cloud server 270, which will be described later. The cloud server receiver 271 and the cloud server 270 can communicate with each other via wireless communication.

[0097] The controller 260 has a gaze calculation unit 261 that calculates the gaze directions of the assessee HA and the comparator HB from the eyeballs of the assessee photographed by the front camera 201. The gaze calculation method by the gaze calculation unit 261 can be the same as that of the gaze calculation unit 61 in the above-mentioned first embodiment, and therefore a detailed description thereof will be omitted.

[0098] The controller 60 has a map generating unit 262 that calculates saliency indicating the ease with which targets present in the forward field of view of the assessee HA and the comparator HB attract the gaze of the assessee HA and the comparator HB based on image data acquired by the rear camera 200, and generates a saliency map. The method of generating the saliency map by the map generating unit 262 can be the same as that of the map generating unit 262 in the above-mentioned embodiment 1, and therefore a detailed description thereof will be omitted.

[0099] The human state estimation system 250 of the second embodiment also uses the cloud server 270 for part of the calculation. The controller 260 has a cloud server transmitter 272 for transmitting information from the rear camera 200 and the front camera 201, the gaze information of the assessee HA and the comparator HB, and the saliency map to the cloud server 270. The cloud server transmitter 272 and the cloud server 270 can communicate with each other via wireless communication.

[0100] The cloud server 270 has a function of selecting a comparator HB from a plurality of workers. The cloud server 270 identifies the assessee HA and other workers from images captured by the rear camera 200 and the front camera 201. The cloud server 270 selects a comparator HB having attributes equivalent to those of the assessee HA from the identified other workers. In Figs. 12 and 13, there is one comparator HB, but when there are two or more comparators HB, the cloud server 70 selects two or more comparators HB.

[0101] The cloud server 270 calculates the probability that the saliency at the position on the saliency map indicated by the gaze information of the assessee HA will be equal to or greater than a predetermined threshold (i.e., a first probability) based on the information acquired from the controller 260. The cloud server 270 also calculates the probability that the saliency at the position on the saliency map indicated by the gaze information of the comparator HB will be equal to or greater than a predetermined threshold (i.e., a second probability) based on the information acquired from the controller 260. The calculation method of the first and second probabilities by the cloud server 270 can be the same as the calculation method in the first embodiment described above, and therefore a detailed description will be omitted. In the second embodiment, the cloud server 270 constitutes a first probability calculation unit and a second probability calculation unit.

[0102] The cloud server 270 estimates whether the assessee HA is in an abnormal state based on the calculated first probability and second probability. The method of estimating the abnormal state of the assessee HA is the same as that of the above-mentioned embodiment 1, and the reference value and the predetermined value that are the calculation criteria for the estimated index value, and the specific value that is the judgment criterion for the abnormal state can adopt the same values ​​as those of the above-mentioned embodiment 1.

[0103] The cloud server 270 transmits the estimation result to the controller 260 via the cloud server receiver 271. When the controller 260 receives an estimation result from the cloud server 270 indicating that the assessee HA is abnormal, the controller 260 has a notification device 210 for notifying the worker or the supervisor monitoring the worker of the estimation result. The notification device 210 is, for example, a display device such as a speaker or a display that can be confirmed by the supervisor. The notification device 210 asks the worker a question such as "Are you okay?" and displays on the display device that there is an abnormal person.

[0104] Moreover, the controller 260 is configured to control the conveyor device 220. The controller 260 temporarily stops the conveyor device 220 when it acquires an inferred result that the assessee HA is abnormal, and resumes the operation of the conveyor device 220 when the treatment for the assessee HA (removal from the line, etc.) is completed.

[0105] (flowchart) Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 250 will be described with reference to the flowchart of FIG.

[0106] First, in step S201, the human state estimation system 250 acquires various data including peripheral information of the person being evaluated HA and an image of the person being evaluated HA's forward field of vision.

[0107] Next, in step S202, the human state estimation system 250 acquires the gaze information of the assessee HA from the image captured by the front camera 201. Although omitted in the flowchart, at the time of this step S202, the controller 260 generates a saliency map based on the image data acquired in the step S101. The gaze information of the assessee HA acquired in this step S202 is information including the saliency of the gaze of the assessee HA.

[0108] Next, in step S203, the human state estimation system 250 acquires gaze information of the comparator HB from the image captured by the front camera 201. The gaze information of the comparator HB acquired in step S203 is information including saliency ahead of the gaze of the comparator HB.

[0109] Next, in step S204, the human state estimation system 250 judges whether a predetermined time has elapsed, in particular whether the gaze information of the assessee HA and the comparison person HB has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 250 proceeds to step S205. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 250 returns to step S201.

[0110] In step S205, the human state estimating system 250 calculates a first probability from the gaze information of the assessee HA.

[0111] Next, in step S206, the human state estimating system 250 calculates a second probability from the gaze information of the comparison person HB.

[0112] Next, in step S207, the human state estimation system 250 calculates an ROC curve based on the first probability calculated in step S205 and the second probability calculated in step S206.

[0113] Next, in step S208, the human state estimation system 250 calculates an AUC value from the ROC curve calculated in step S207.

[0114] Next, in step S208, the human state estimation system 250 judges whether data for the measurement period, i.e., AUC values ​​for the measurement period, have been obtained. If the AUC values ​​for the measurement period have been obtained (YES), the human state estimation system 250 proceeds to step S210, whereas if the AUC values ​​for the measurement period have not been obtained (NO), the human state estimation system 250 returns to step S201 and repeats steps S201 to S208 until AUC values ​​for the measurement period are obtained.

[0115] Then, in step S210, the human state estimation system 250 judges whether or not the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a specific value set in advance. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 250 proceeds to step S211 and judges that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 250 proceeds to step S212 and judges that no abnormality has occurred in the person being evaluated.

[0116] After steps S211 and S212, the process returns.

[0117] When there are two or more comparators HB, the cloud server 270 selects two or more comparators. When the cloud server 270 selects two or more comparators HB, the cloud server 70 calculates the second probability for each comparator HB, and then calculates the average value of each second probability and calculates the ROC curve with the average value as the final second probability, as in the first embodiment described above.

[0118] Therefore, like the human state estimation system 250 of the second embodiment, the technology disclosed herein can also be used as a monitoring system for workers in a factory.

[0119] Third embodiment Hereinafter, the third embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals are used to designate parts common to the first and second embodiments, and detailed description thereof will be omitted.

[0120] The third embodiment is common to the first embodiment in that a human state estimation system 350 is applied to a vehicle driver, but differs from the first embodiment in the estimation method.

[0121] In the third embodiment, as shown in Fig. 16, the human state estimation system 350 includes a storage device 380. The storage device 380 stores the past gaze information of the driver as the subject. This gaze information includes the saliency of the subject's gaze. The storage device 380 is configured to be able to communicate with the controller 360 of the subject's vehicle.

[0122] The controller 360 has a probability calculation unit 363 that calculates the probability (first probability) that the saliency at a position on the saliency map indicated by the gaze information of the assessee is equal to or greater than a predetermined threshold based on the gaze information of the assessee, in addition to the gaze calculation unit 361 and the map generation unit 362. The calculation method of the first probability by the probability calculation unit 363 can be the same as the calculation method described in the above-mentioned embodiments 1 and 2, and therefore a detailed description thereof will be omitted.

[0123] The probability calculation unit 363 reads out the gaze information of the person to be evaluated who was in the same space in the past from the storage device 380, and calculates the probability that the saliency at the position on the saliency map indicated by the gaze of the person to be evaluated in the past will be equal to or greater than a predetermined threshold. In this embodiment 3, the probability based on the gaze information of the person to be evaluated in the past corresponds to the second probability. The calculation method of the second probability is the same as the calculation method of the first probability, so a detailed explanation will be omitted. When the gaze information of the person to be evaluated who was in the same space in the past does not exist in the storage device 380, the controller 360 randomly specifies multiple coordinates on the saliency map generated by the map generation unit 362, and obtains the saliency of each coordinate. Then, the controller 360 calculates the second probability from the saliency of the random point.

[0124] The controller 360 also includes an abnormality determination unit 364 that estimates the human state of the person being evaluated based on the first probability and the second probability calculated by the probability calculation unit 363, and determines whether or not the person being evaluated has an abnormality. The abnormality determination unit 364 obtains an ROC curve from the first probability and the second probability, and calculates an AUC value for the ROC curve, as in the cloud servers 70 and 270 of the first and second embodiments described above. The abnormality determination unit 364 then calculates an estimated index value based on each AUC value for the measurement period, and determines whether or not the person being evaluated has an abnormality based on the estimated index value. The reference value and the predetermined value that are the calculation criteria for the estimated index value, and the specific value that is the judgment criterion for the abnormal state can adopt the same values ​​as those in the first embodiment described above.

[0125] (flowchart) Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 350 will be described with reference to the flowchart of FIG.

[0126] First, in step S301, the human state estimation system 350 acquires various data including information about the surroundings of the vehicle and an image of the forward field of vision of the person being evaluated.

[0127] Next, in step S302, the human state estimation system 350 acquires the current gaze information of the person being evaluated from the image captured by the in-vehicle camera 101 of the vehicle. Although omitted in the flowchart, at the time of this step S302, the controller 360 generates a saliency map based on the image data acquired in the step S301. The gaze information of the person being evaluated acquired in this step S302 is information including the saliency ahead of the gaze of the person being evaluated.

[0128] Next, in step S303, the human state estimation system 350 judges whether or not past gaze information of the person being evaluated in the same space is stored in the storage device 380. If the past gaze information is stored in the storage device 380 (YES), the process proceeds to step S304, whereas if the past gaze information is not stored in the storage device 380 (NO), the process proceeds to step S308. The past gaze information includes saliency of the gaze of the person being evaluated in the past.

[0129] In step S304, the human state estimation system 350 acquires past gaze information from the storage device 380.

[0130] Next, in step S305, the human state estimation system 350 judges whether a predetermined time has elapsed, in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 350 proceeds to step S306. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 350 returns to step S301.

[0131] In step S306, the human state estimating system 350 calculates a first probability from the current gaze information of the person to be evaluated.

[0132] Next, in step S307, the human state estimating system 350 calculates a second probability from the past gaze information of the person to be evaluated.

[0133] On the other hand, in step S308, which is reached when the judgment in step S303 is NO, the human state estimation system 350 judges whether a predetermined time has elapsed, in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 350 proceeds to step S309. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 350 returns to step S301.

[0134] In step S309, the human state estimating system 350 calculates a first probability from the current gaze information of the person to be evaluated.

[0135] Next, in step S310, the human state estimation system 350 calculates a second probability from the random point selected from the saliency map.

[0136] Next, in step S311, the human state estimation system 350 calculates an ROC curve based on the first probability calculated in step S306 or step S309 and the second probability calculated in step S307 or step S310.

[0137] Next, in step S312, the controller 360 calculates an AUC value from the ROC curve calculated in step S311.

[0138] Next, in step S313, the human state estimation system 350 judges whether data for the measurement period, i.e., AUC values ​​for the measurement period, have been obtained. If the AUC values ​​for the measurement period have been obtained (YES), the human state estimation system 350 proceeds to step S314, whereas if the AUC values ​​for the measurement period have not been obtained (NO), the human state estimation system 350 returns to step S301 and repeats steps S301 to S312 until AUC values ​​for the measurement period are obtained.

[0139] Then, in step S314, the human state estimation system 350 judges whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a specific value set in advance. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 350 proceeds to step S315 and judges that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 350 proceeds to step S316 and judges that no abnormality has occurred in the person being evaluated.

[0140] After steps S315 and S316, the process returns.

[0141] In this way, in the third embodiment, peripheral information including the space in which the person to be evaluated is located is acquired (peripheral information acquisition process), an image included in the forward field of view of the person to be evaluated is acquired (field of view image acquisition process), a saliency map is generated for the image of the forward field of view (map generation process), gaze information of the person to be evaluated for the forward field of view is acquired (gaze information acquisition process), the probability that the saliency at the position on the saliency map indicated by the gaze of the person to be evaluated is equal to or greater than a predetermined threshold is calculated (first probability calculation process), the probability that the saliency at the position on the saliency map indicated by the gaze of the person to be evaluated who was in the same space in the past is equal to or greater than a predetermined threshold is calculated (second probability calculation process), and based on the first probability and the second probability, it is estimated whether the person to be evaluated is in an abnormal state (abnormality estimation process). As a result, it is possible to absorb as much individual difference as possible, such as the habits and physique of the person to be evaluated, without creating a top-down saliency map as in the past, in order to compare with the person to be evaluated. As a result, it is possible to accurately estimate the human state of the person to be evaluated with as simple calculation as possible.

[0142] Fourth embodiment Hereinafter, the fourth embodiment will be described in detail with reference to the drawings. In the following description, the same reference numerals will be used to designate the same parts as those in the first to third embodiments, and detailed description thereof will be omitted.

[0143] The fourth embodiment is common to the first and third embodiments described above in that a human state estimation system 450 is applied to a vehicle driver, but differs from the first and third embodiments described above in the estimation method.

[0144] In the human state estimation system 450 according to the fourth embodiment, the vehicle VA includes a storage device 480. The storage device 480 stores the past gaze information of the driver as the person being evaluated. This gaze information includes the saliency of the gaze of the person being evaluated. The storage device 480 is configured to be able to communicate with the controller 460 of the vehicle VA. In FIG. 18, the other vehicle VB does not include a storage device, but the other vehicle VB may also include a storage device that stores the past gaze information of the comparison person.

[0145] In the human state estimation system 450, basically, as in the above-mentioned embodiment 1, the cloud server 70 selects another vehicle VB driven by a driver who is a comparator and exists in the vicinity of the vehicle VA based on the surrounding information of the vehicle VA. Then, the cloud server 70 acquires the gaze information of the comparator from the controller 460 of the other vehicle VB. However, in the human state estimation system 450, when the cloud server 70 determines that the other vehicle VB driven by the driver who is a comparator does not exist in the vicinity of the vehicle VA, the controller 460 of the vehicle VA acquires the past gaze information of the subject in the same space from the storage device 480 and transmits it to the cloud server 70. In addition, in the human state estimation system 450, when the other vehicle VB driven by the driver who is a comparator does not exist in the vicinity of the vehicle VA and the past gaze information of the subject in the same space does not exist in the storage device 480, the cloud server 70 randomly specifies multiple coordinates on the saliency map generated by the map generation unit 462, and obtains the saliency of each coordinate.

[0146] In the human state estimation system 450, the controllers 460 of the subject vehicle VA and the other vehicle VB generate a saliency map and acquire gaze information of the subject and the comparison subject, as in the first embodiment. Meanwhile, the cloud server 70 is used to calculate the first probability and the second probability, calculate the ROC curve and the AUC value, and estimate the human state of the subject. The reference value and the predetermined value that are the calculation criteria for the estimated index value, and the specific value that is the judgment criterion for the abnormal state can adopt the same values ​​as those in the first embodiment.

[0147] (flowchart) Next, the processing operation of estimating the human state of the person to be evaluated by the human state estimating system 450 will be described with reference to the flowcharts of FIGS.

[0148] First, in step S401, the human state estimation system 450 acquires various data including peripheral information about the host vehicle VA and an image of the subject's forward field of vision.

[0149] Next, in step S402, the human state estimation system 450 acquires the current gaze information of the subject from the image captured by the in-vehicle camera 101 of the subject's vehicle VA. Although omitted in the flowchart, at the time of this step S402, the controller 460 of the subject's vehicle VA generates a saliency map based on the image data acquired in the step S401. The gaze information of the subject acquired in this step S402 is information including the saliency ahead of the subject's gaze.

[0150] Next, the first probability and the second probability are calculated by a subroutine in step S500.

[0151] As shown in Fig. 20, in the subroutine of step S500, first, in step S501, the human state estimation system 450 determines whether or not a driver of another vehicle VB who can be a comparator is present around the host vehicle VA. The human state estimation system 450 determines the presence or absence of a comparator based on the position information of the other vehicle VB and the detection results of the various sensors 102-108 related to driving. If the result is YES, which indicates that a comparator is present, the human state estimation system 450 proceeds to step S502. On the other hand, if the result is NO, which indicates that a comparator is not present, the human state estimation system 450 proceeds to step S506.

[0152] In step S502, the human state estimation system 450 acquires gaze information of the comparison person from an image captured by the in-vehicle camera 101 of the other vehicle VB. Note that this step S502 is not performed after the judgment of the step S501, but is actually performed in the other vehicle VB simultaneously with the step S402. Although not shown in the flowchart, at the time of this step S502, the controller 460 of the other vehicle VB also generates a saliency map based on the image data acquired by the other vehicle VB. The gaze information of the comparison person acquired in this step S502 is information including the saliency of the gaze of the comparison person.

[0153] Next, in step S503, the human state estimation system 450 judges whether a predetermined time has elapsed, in particular whether the gaze information of the assessee and the comparison person has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 450 proceeds to step S504. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 450 returns to step S401.

[0154] In step S504, the human state estimating system 450 calculates a first probability from the gaze information of the person to be evaluated.

[0155] Next, in step S505, the human state estimation system 450 calculates a second probability from the gaze information of the comparison person.

[0156] On the other hand, in step S506, which is reached when the judgment in step S501 is NO, the human state estimation system 450 judges whether or not past gaze information of the person being evaluated in the same space is stored in the storage device 480. If the judgment is YES, that is, past gaze information is stored in the storage device 480, the human state estimation system 450 proceeds to step S507. On the other hand, if the judgment is NO, that is, past gaze information is not stored in the storage device 480, the human state estimation system 450 proceeds to step S511. Note that the past gaze information is information that includes the saliency of the gaze of the person being evaluated in the past.

[0157] In step S507, the human state estimation system 450 acquires past gaze information from the storage device 480.

[0158] Next, in step S508, the human state estimation system 450 judges whether a predetermined time has elapsed, in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 450 proceeds to step S508. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 450 returns to step S401.

[0159] In step S509, the human state estimating system 450 calculates a first probability from the current gaze information of the person to be evaluated.

[0160] Next, in step S510, the human state estimating system 450 calculates a second probability from the past gaze information of the person to be evaluated.

[0161] On the other hand, in step S511, which is reached when the judgment in step S506 is NO, the human state estimation system 450 judges whether a predetermined time has elapsed, in particular whether the current gaze information of the person being evaluated has been acquired for a predetermined time. This predetermined time is about 30 seconds. If the predetermined time has elapsed (YES), the human state estimation system 450 proceeds to step S512. On the other hand, if the predetermined time has not elapsed (NO), the human state estimation system 450 returns to step S401.

[0162] In step S512, the human state estimating system 450 calculates a first probability from the current gaze information of the person to be evaluated.

[0163] Next, in step S513, the human state estimation system 450 calculates a second probability from the random point selected from the saliency map.

[0164] After the step S505, the step S510, and the step S513, the process proceeds to step S403 shown in Fig. 19. In this step S403, the human state estimation system 450 calculates an ROC curve based on the first probability calculated in the step S504, the step S509, or the step S512, and the second probability calculated in the step S505, the step S510, or the step S513.

[0165] Next, in step S404, the human state estimation system 450 calculates an AUC value from the ROC curve calculated in step S403.

[0166] Next, in step S405, the human state estimation system 450 judges whether data for the measurement period, i.e., AUC values ​​for the measurement period, have been obtained. If the AUC values ​​for the measurement period have been obtained (YES), the human state estimation system 450 proceeds to step S406, whereas if the AUC values ​​for the measurement period have not been obtained (NO), the human state estimation system 450 returns to step S401 and repeats steps S401 to S404 and S500 until AUC values ​​for the measurement period are obtained.

[0167] Then, in step S406, the human state estimation system 450 judges whether the estimated index value calculated from the AUC value for the measurement period is equal to or greater than a specific value set in advance. If the estimated index value is equal to or greater than the specific value (YES), the human state estimation system 450 proceeds to step S407 and judges that an abnormality has occurred in the person being evaluated. On the other hand, if the estimated index value is less than the specific value (NO), the human state estimation system 450 proceeds to step S408 and judges that no abnormality has occurred in the person being evaluated.

[0168] After steps S407 and S408, the process returns.

[0169] In step S501, when there are two or more comparators, the cloud server 70 selects two or more comparators, similar to the above-described embodiment 1. When the cloud server 70 selects two or more comparators, after the second probability is calculated for each comparator, the cloud server 70 obtains an average value of each second probabilities and calculates an ROC curve using the average value as the final second probability.

[0170] Therefore, in the fourth embodiment, even in a situation where a comparison person is unlikely to be nearby, such as driving a car late at night or driving in the suburbs, the human state of the person being evaluated can be estimated with high accuracy. In addition, since the second probability is calculated based on information about the past gaze of the person being evaluated, the second probability can be calculated easily and with high accuracy. Therefore, the human state of the person being evaluated can be estimated with high accuracy by the simplest calculation possible.

[0171] (Other embodiments) The technology disclosed herein is not limited to the above-described embodiment, and may be substituted without departing from the spirit and scope of the claims.

[0172] In the above-described first, second, and fourth embodiments, the first probability and the second probability are calculated by the cloud server 70, 270. However, the first probability and the second probability may be calculated by the controller 60, 260, 460 in the configurations of the first, second, and fourth embodiments. Also, the human state estimation of the subject may be performed by the controller 60, 260, 460 in the configurations of the first, second, and fourth embodiments. In this case, in the first and fourth embodiments, the controller 60, 460 of the host vehicle VA needs to obtain information on the second probability from the controller 60, 460 of the other vehicle VB.

[0173] In the above-mentioned embodiment 2, the second probability is calculated from the gaze information of the comparison person HB. However, even if the human state estimation system is applied to a factory as in the embodiment 2, the second probability may be calculated based on the gaze information of the past evaluation subject HA as in the embodiments 3 and 4.

[0174] In addition, in the above-mentioned first to fourth embodiments, the ROC curve is calculated, and then the AUC value is calculated, and the human state of the person to be evaluated is estimated based on the AUC value. Not limited to this, for example, the human state of the person to be evaluated may be estimated by calculating the rate at which the ROC curve deviates from the reference value. In this case, for example, the reference value may be a value on the straight line formed when the first probability and the second probability are the same.

[0175] The above-described embodiments are merely illustrative and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and variations within the scope of the claims are within the scope of the present disclosure. [Industrial Applicability]

[0176] The technology disclosed herein is useful in estimating the human state of a person being evaluated using a saliency map with high accuracy and with as simple calculations as possible. [Explanation of symbols]

[0177] 50,250,350,450 Human Status Estimation System 60,260,360,460 Controller 61,261,361,461 Line of sight calculation unit (first line of sight information acquisition unit, second line of sight information acquisition unit) 62,262,362,462 Map Generation Unit 363 Probability Calculation Unit (First Probability Calculation Unit, Second Probability Calculation Unit) 364 Abnormality determination section 70,270 Cloud server (peripheral information acquisition unit, first probability calculation unit, second probability calculation unit, abnormality judgment unit)

Claims

1. A human state estimation method, comprising: A peripheral information acquisition step of acquiring peripheral information of the person being evaluated; A visual field image acquisition process for acquiring an image included in the subject's forward visual field; a map generation step of generating a saliency map for the image acquired in the field of view image acquisition step; A first gaze information acquisition step of acquiring gaze information of the subject with respect to the forward field of view; A second gaze information acquisition step of acquiring gaze information of a comparator who is another person having attributes equivalent to the assessee; A first probability calculation step of calculating a probability that the saliency at a position on the saliency map indicated by the gaze of the subject acquired in the first gaze information acquisition step is equal to or greater than a predetermined threshold; a second probability calculation step of calculating a probability that the saliency at a position on the saliency map indicated by the gaze of the comparison person acquired in the second gaze information acquisition step is equal to or greater than the predetermined threshold; and an abnormality estimation step of estimating whether the subject is in an abnormal state based on the first probability calculated in the first probability calculation step and the second probability calculated in the second probability calculation step, The human state estimation method, characterized in that the abnormality inference process is a process of inferring that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value.

2. The human state estimation method according to claim 1, The second gaze information acquisition step is a step of acquiring gaze information of two or more of the comparators, The human state estimation method, characterized in that the second probability calculation process is a process of determining saliency at positions on the saliency map indicated by each gaze of each of the comparators, calculating the probability that each saliency will be greater than or equal to the specified threshold, and setting the average value of each probabilistic probability or one probability selected from the respective probabilities as the second probability.

3. A human state estimation system, comprising: A peripheral information acquisition unit that acquires peripheral information of the person being evaluated; A visual field image acquisition unit that acquires an image included in the subject's forward visual field; A map generation unit that generates a saliency map for the image acquired by the field of view image acquisition unit; A gaze information acquisition unit that acquires gaze information of the subject with respect to the forward field of view; A comparison person's gaze information acquisition unit that acquires gaze information of a comparison person who is another person having attributes equivalent to the evaluation person; A first probability calculation unit that calculates the probability that the saliency at a position on the saliency map indicated by the gaze information of the subject acquired by the gaze information acquisition unit is equal to or greater than a predetermined threshold; a second probability calculation unit that calculates a probability that the saliency at a position on the saliency map indicated by the gaze of the comparator acquired by the comparator gaze information acquisition unit is equal to or greater than the predetermined threshold; an abnormality estimation unit that estimates whether the subject is in an abnormal state based on the first probability calculated by the first probability calculation unit and the second probability calculated by the second probability calculation unit; The human state estimation system is characterized in that the abnormality estimation unit estimates that the person being evaluated is in an abnormal state when the rate at which the first probability and the second probability deviate by a predetermined value or more is equal to or greater than a predetermined specific value.

4. The human state estimation system according to claim 3, The comparator gaze information acquisition unit acquires gazes of two or more comparators, The second probability calculation unit obtains saliency at positions on the saliency map indicated by each gaze of each of the comparators, calculates the probability that each saliency will be greater than or equal to the specified threshold, and sets the average value of each probabilities or one probability selected from the respective probabilities as the second probability.

5. The human state estimation system according to claim 3 or 4, Further comprising a storage unit for storing information regarding the gaze of the person to be evaluated; The human state estimation system is characterized in that, when the comparison person is not present around the person being evaluated, the second probability calculation unit reads information regarding the gaze of the person being evaluated who was in the same space in the past from the memory unit, and calculates as the second probability the probability that the saliency at a position on the saliency map indicated by the gaze of the person being evaluated in the past will be greater than or equal to the specified threshold.

Images