Electronic device, information processing device, concentration degree calculation program, and concentration degree calculation method

Abstract

To improve the accuracy in estimating a subject's attentiveness to various situations.SOLUTION: An electronic apparatus 10 has an imaging unit 11, a line-of-sight detection unit 12, and a control unit 14. The imaging unit 11 creates an image corresponding to a scene through imaging. The line-of-sight detection unit 12 detects a subject's line of sight to the scene. The control unit 14 determines the reliability of the image as an estimation source of the subject's degree of concentration based on the image. The control unit 14 estimates the subject's degree of concentration based on the reliability, the image, and the line of sight.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an electronic device, an information processing device, a concentration degree calculation program, and a concentration degree calculation method. [Background technology]

[0002] Safe driving of a moving object requires the driver's attention. Therefore, it has been considered to observe the driver's attention and to issue a warning to the driver or provide driving support if the driver's attention decreases. As an observation of the attention, it has been proposed to calculate a cumulative visibility, which is a cumulative value of the degree of overlap of the line of sight with an object such as an oncoming vehicle around the vehicle, and to compare it with a reference value (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2008-029802 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, in order to calculate the cumulative visibility, the visibility at each hour is calculated using a table. However, the appropriate table varies depending on various driving conditions in the actual environment, and it is difficult to accurately observe the driver's attention in various driving conditions.

[0005] Therefore, in consideration of the problems with the conventional technology as described above, the purpose of the present disclosure is to provide an electronic device, an information processing device, a concentration calculation program, a concentration calculation method, and a computer learning method that accurately estimate a subject's attention in a variety of situations. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an electronic device according to a first aspect comprises: an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight. 、 The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as a time for which the estimation of the concentration level is stopped becomes longer. do. Or, the electronic device is an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight, The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are captured continuously, as the reliability becomes lower.

[0007] An information processing device according to a second aspect of the present invention comprises: An acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight; an output unit that outputs the concentration degree; 、 The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as a time for which the estimation of the concentration level is stopped becomes longer. do. Or, the information processing device An acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight; An output unit that outputs the concentration degree, The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are captured continuously, as the reliability becomes lower.

[0008] The concentration calculation program from the third perspective is Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; and a control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight. 、 The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as a time for which the estimation of the concentration level is stopped becomes longer. do. Or, the concentration calculation program is Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; a control unit that determines a reliability of the image as an estimation source of the subject's concentration level based on the image, and estimates the subject's concentration level based on the reliability, the image, and the line of sight; The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are captured continuously, as the reliability becomes lower.

[0009] The concentration calculation method according to the fourth aspect is as follows: The imaging unit, an imaging step of generating an image corresponding to the scene by imaging; The gaze detection unit, a gaze detection step of detecting a gaze of a subject relative to the scene; The control unit: determining a reliability of the image as a source of the estimation of the concentration level of the subject based on the image; The control unit: an estimation step of estimating a concentration level of the subject based on the reliability, the image, and the line of sight; a step of lowering the reliability threshold as the time for which the estimation of the concentration level is stopped becomes longer by the control unit; a stopping step of the control unit stopping the estimation of the concentration degree when the reliability is equal to or less than the reliability threshold; Equipped with. Or, the concentration calculation method is The imaging unit, an imaging step of generating an image corresponding to the scene by imaging; The gaze detection unit, a gaze detection step of detecting a gaze of a subject relative to the scene; The control unit: determining a reliability of the image as a source of the estimation of the concentration level of the subject based on the image; The control unit: and estimating a concentration level of the subject based on the reliability, the image, and the line of sight, In the estimation step, the control unit estimates the concentration level of the subject based on a larger number of frames of images that are captured continuously, as the reliability becomes lower. Effect of the Invention

[0010] According to the electronic device, information processing device, concentration level calculation program, and concentration level calculation method disclosed herein that are configured as described above, the accuracy of estimating a subject's attention in a variety of situations is improved. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of an electronic device according to a first embodiment. [Diagram 2] 4A and 4B are diagrams for explaining the relationship between a line of sight detected by a line of sight detection unit and an image. [Diagram 3] FIG. 13 is a diagram showing a specific example of a gaze prediction map estimated for an image. [Figure 4] 4 is a flowchart for explaining an estimation process executed by a control unit in FIG. 1 in the first embodiment. [Diagram 5] 10 is a flowchart for explaining an estimation process executed by a control unit in FIG. 1 in a second embodiment. [Figure 6] FIG. 1 is a block diagram showing a schematic configuration of an information processing device which is a modification of the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of an electronic device to which the present disclosure is applied will be described with reference to the drawings. Note that the following description also includes a description of an information processing device, a concentration degree calculation program, and a concentration degree calculation method to which the present disclosure is applied.

[0013] The electronic device according to the first embodiment of the present disclosure is provided in, for example, a moving object. The moving object may include, for example, a vehicle, a ship, and an aircraft. The vehicle may include, for example, an automobile, an industrial vehicle, a railroad vehicle, a vehicle for daily life, and a fixed-wing aircraft running on a runway. The automobile may include, for example, a passenger car, a truck, a bus, a motorcycle, and a trolley bus. The industrial vehicle may include, for example, an industrial vehicle for agriculture and construction. The industrial vehicle may include, for example, a forklift and a golf cart. The industrial vehicle for agriculture may include, for example, a tractor, a cultivator, a transplanter, a binder, a combine, and a lawnmower. The industrial vehicle for construction may include, for example, a bulldozer, a scraper, a shovel, a crane, a dump truck, and a road roller. The vehicle may include one that runs by human power. The classification of the vehicle is not limited to the above-mentioned example. For example, the automobile may include an industrial vehicle that can run on a road. The same vehicle may be included in multiple classifications. The watercraft may include, for example, a marine jet, a boat, a tanker, etc. The aircraft may include, for example, a fixed-wing aircraft, a rotorcraft, etc.

[0014] As shown in FIG. 1, an electronic device 10 according to the first embodiment of the present disclosure includes an imaging unit 11, a line-of-sight detection unit 12, a memory 13, and a control unit 14.

[0015] The imaging unit 11 is provided on the moving body so as to be able to capture, for example, a scene in the forward direction of the moving body. The imaging unit 11 is, for example, a camera capable of capturing images at a speed of 30 fps. The imaging unit 11 captures images to generate an image corresponding to the scene.

[0016] The line of sight detection unit 12 is provided in the moving body so as to be able to detect the line of sight of a subject seated in, for example, the driver's seat of the moving body. The line of sight detection unit 12 is, for example, either a contact-type eye tracker or a non-contact-type eye tracker, and detects the line of sight of the subject with respect to a scene. As shown in Fig. 2, the line of sight LS is indicated as, for example, the direction of the line of sight LS corresponding to a position PE in the same coordinate system as the image IM generated by the imaging unit 11 through imaging.

[0017] The gaze detection unit 12 may detect gaze data of a time series of the gaze LS as the gaze data. More specifically, the gaze detection unit 12 may detect the position PE of the gaze LS on the image IM at each time and output the series of the positions PE of the gaze LS in time as the gaze data. The gaze detection unit 12 may detect the gaze LS at a higher speed, in other words, at a higher frequency than the imaging unit 11, and may accumulate and output the series of the positions PE of the gaze LS in time that are well detected as the path of the gaze LS of one image IM.

[0018] The memory 13 includes any storage device, such as a random access memory (RAM) and a read only memory (ROM), etc. The memory 13 stores various programs that cause the control unit 14 to function and various information used by the control unit 14.

[0019] The control unit 14 includes one or more processors and a memory. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 14 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together. The control unit 14 controls the operation of each component of the electronic device 10.

[0020] The control unit 14 normally causes the imaging unit 11 to perform continuous imaging at a speed of, for example, 30 fps, and continuously acquires images IM as information. The control unit 14 causes the line of sight detection unit 12 to detect the line of sight LS of the subject when imaging is performed by the imaging unit 11, and acquires the line of sight LS at substantially the same time as the imaging time of the image IM as information. The time substantially the same as the imaging time of the image IM may include a single detection time, or may include multiple detection times between the imaging time immediately before the most recent imaging time of the image IM and the most recent imaging time. When the time substantially the same as the imaging time of the image IM includes a single detection time of the line of sight LS, it may not be the exact same time, but may include the detection time closest to the imaging time in line of sight detection with the same period as the image imaging. The control unit 14 associates the image IM with the line of sight LS detected at substantially the same time as the imaging time of the image IM and stores them in the memory 13.

[0021] The control unit 14 may perform image processing on the acquired image IM. In a configuration in which the learning of the estimation unit is performed using an image IM that has been subjected to a predetermined image processing, as described below, the control unit 14 performs the predetermined image processing on the acquired image IM and stores the image IM in the memory 13 in association with the line of sight LS. The predetermined image processing may be, for example, a process of generating a semantic segmentation image based on the acquired image IM. The semantic segmentation image is an image in which a label or category is associated with all pixels in the image IM.

[0022] The control unit 14 determines the reliability of an image IM based on the image IM. As described later, in this embodiment, the concentration level is estimated based on the line of sight LS for an arbitrary image IM. The estimation accuracy of the concentration level may be influenced by elements contained in the image. The reliability of an image IM is an index indicating the suitability of the image IM as a source of concentration level estimation.

[0023] The control unit 14 may determine the reliability by calculating the reliability based on at least one of the type, number, and position in the image IM of the elements included in the image IM detected by image recognition. The elements include, for example, at least one of a vehicle, a pedestrian, a road shape, a sign, and a traffic light. In a configuration in which the reliability is calculated based on a plurality of images IM, the elements may include at least one of the optical flow, in other words, the movement, of a vehicle, a pedestrian, a road shape, a sign, and a traffic light. The reliability may be calculated, for example, so that the reliability increases as the number of elements detected increases. The reliability may be calculated so that the reliability increases when the type of the detected element is an object whose behavior requires the driver's attention, such as a vehicle or a pedestrian. The reliability may be calculated so that the reliability decreases compared to a vehicle, for example, when the type of the detected element is an object whose behavior does not require attention, such as a sign or a traffic light, because it is fixed relative to the ground surface. The reliability may be calculated so that the reliability increases as the position of the detected element is closer to the moving path of the moving body and the moving body. The reliability may be calculated so that the reliability increases as the shape of the detected road becomes more complex.

[0024] Alternatively, the control unit 14 may determine the reliability by functioning as a reliability estimation unit that calculates the reliability based on the image IM. The reliability estimation unit may be constructed by learning data in which the relationship between the learning image and the information on the reliability is machine-learned. The information on the reliability may be the concentration degree calculated according to the above-mentioned rule based on the elements included in the learning image.

[0025] The control unit 14 estimates the concentration level of the subject based on the determined reliability, the image IM generated by the imaging unit 11, and the gaze detection unit 12. In the first embodiment, the control unit 14 determines whether or not the concentration level can be estimated based on the reliability.

[0026] When the reliability is equal to or less than the reliability threshold, the control unit 14 stops estimating the concentration level. The control unit 14 may lower the reliability threshold as the time for which the estimation of the concentration level is stopped becomes longer. After estimating the concentration level, the control unit 14 may return the lowered reliability threshold to the initial value.

[0027] The control unit 14 estimates the concentration level when the reliability exceeds the reliability threshold. The control unit 14 estimates the concentration level of the subject based on the image IM and the line of sight LS whose imaging time and detection time are substantially the same. The control unit 14 may estimate the concentration level by functioning as an estimation unit constructed from learning data obtained by machine learning. The estimation unit may be constructed from a multi-layered neural network.

[0028] The estimation unit may directly estimate the concentration level based on, for example, the image IM and the subject's gaze LS. The estimation unit may be constructed from learning data obtained by machine learning the relationship between the learning image, the actual gaze, and biological information related to the concentration level using a plurality of sets of learning images, the actual gaze of the learning subject at a scene corresponding to the learning image, and the concentration level of the learning subject.

[0029] Alternatively, the estimation unit may estimate a gaze prediction map based on an image IM, and estimate the concentration level based on the gaze prediction map and the target's gaze LS. As shown in FIG. 3, the gaze prediction map MP is a two-dimensional map that indicates the probability that a general target person will gaze at each position in a specific scene, i.e., the probability of overlapping with the gaze LS, at each position constituting the image IM corresponding to the scene. The estimation unit may include a first estimation unit that estimates the gaze prediction map MP based on the image IM, and a second estimation unit that estimates the concentration level based on the gaze prediction map MP and the target's gaze LS. The first estimation unit may be constructed from learning data that has been machine-learned to learn the relationship between the learning image and the actual gaze using a plurality of sets of the learning gaze prediction map estimated by the first estimation unit based on the learning image, the actual gaze of the learning target person with respect to the scene corresponding to the learning image, and the concentration level of the learning target person. The second estimation unit may be constructed from learning data that has been machine-learned to learn the relationship between the learning gaze prediction map, the actual gaze, and the concentration level using a plurality of sets of the learning gaze prediction map estimated by the first estimation unit based on the learning image, the learning target person's actual gaze with respect to the scene corresponding to the learning image, and the concentration level of the learning target person.

[0030] Alternatively, the estimation unit may estimate a gaze prediction map of the subject with a concentration level within a predetermined range based on the image IM, for example. The control unit 14 may calculate the concentration level of the subject based on the estimated gaze prediction map and the gaze LS of the subject. The control unit 14 may, for example, identify a position of the gaze prediction map corresponding to the gaze LS of the subject, or an area including the position. The control unit 14 may further calculate a concentration level according to the probability at the position of the gaze prediction map, or the average value or weighted average value of the probability in the area. The control unit 14 may calculate the concentration level by, for example, a function or a table so that the concentration level is lower as the probability increases. The control unit 14 may calculate the concentration level so that the concentration level of the subject approaches a specific value such as an average value, maximum value, or minimum value within a predetermined range as the probability increases. The estimation unit may be constructed by learning data that is machine-learned to learn the relationship between a learning image and the actual gaze of the learning subject when the learning subject's concentration level on the learning image is within the predetermined range.

[0031] Alternatively, the estimation unit may estimate a low concentration heat map for each of a plurality of load factors based on the image IM when a load of a specific load factor is applied. The low concentration heat map is a two-dimensional map showing the probability of gazing at each position on a specific scene, i.e., overlapping with the line of sight, at each position constituting the image IM corresponding to the scene when a load of a specific load factor is applied to a general subject. The load factors are various factors that put a load on the concentration of a general subject on an action such as driving, for example, conversation with a passenger, hands-free conversation, listening to a radio, contemplation, drowsiness, fatigue, etc. Furthermore, the estimation unit may estimate a high concentration heat map when all of the specific load factors on the subject are removed or the load is reduced, based on the image IM, for example. The high concentration heat map is a two-dimensional map showing the probability of gazing at each position on a specific scene, i.e., overlapping with the line of sight, at each position constituting the image IM corresponding to the scene when a general subject is removed from the load of all of the above specific load factors or reduced. The control unit 14 may read out the probability of the position PE corresponding to the line of sight LS of the subject in all the low concentration heat maps and the high concentration heat maps. The control unit 14 may select the heat map with the highest readout probability from among the low concentration heat maps and the high concentration heat map. The control unit 14 may calculate the concentration level based on a reference value determined for the selected heat map. For example, a reference value may be determined within a range of a concentration level of 0% or more and less than 50% for all the low concentration heat maps. For example, a reference value may be determined within a range of a concentration level of 50% or more and 100% or less for the high concentration heat map. When the heat map with the highest probability of the position PE corresponding to the line of sight LS is any of the low concentration heat maps, the control unit 14 may calculate the reference value determined for the low concentration heat map as the concentration level of the subject. When the heat map with the highest probability of the position PE corresponding to the line of sight LS is a high concentration heat map, the control unit 14 may calculate the reference value determined for the high concentration heat map as the concentration level of the subject. The control unit 14 may calculate the concentration level based not only on the type of selected heat map but also on the probability of the position PE corresponding to the line of sight LS in the selected heat map.For example, a function or table may be defined so that the concentration degree is calculated to a smaller value as the probability increases for a low concentration heat map. Also, for example, a function or table may be defined so that the concentration degree is calculated to a larger value as the probability increases for a high concentration heat map. The control unit 14 may calculate the concentration degree using a function or table defined for a selected heat map based on the probability of the position PE corresponding to the line of sight LS. The estimation unit may include a fourth estimation unit that estimates a low concentration heat map for each of a plurality of load factors, and a fifth estimation unit that estimates a high concentration heat map. The fourth estimation unit may be constructed from learning data that has been machine-learned for each load factor to determine the relationship between a learning image and the line of sight for the learning image when a load that reduces concentration is applied to the learning subject. The fifth estimation unit may be constructed from learning data that has been machine-learned to determine the relationship between a learning image and the line of sight for the learning image when the load on the learning subject is removed.

[0032] The control unit 14 may estimate the concentration level based on a plurality of images IM captured continuously and the line of sight LS corresponding to each image IM. For example, the control unit 14 may estimate the concentration level based on the image IM of frames within a range going back from the latest image IM at a predetermined time interval. The control unit 14 may estimate the concentration level by excluding an image IM in which the movement speed of the line of sight LS is equal to or greater than a speed threshold among the plurality of images IM captured continuously. More specifically, when the movement amount of the line of sight LS at any detection time from the line of sight LS at the immediately preceding detection time exceeds the movement amount threshold, the control unit 14 may exclude an image IM associated with the detection time of the line of sight LS at the any detection time. In other words, the control unit 14 may exclude information during a saccade. The reason for the exclusion is that a saccade is an action in the middle of moving to the next gaze point, and the position PE of the gaze LS during a saccade does not have a meaning as a gaze point. The control unit 14 may determine whether or not a saccade is in progress based on the movement speed of the position PE of the gaze LS, or the like.

[0033] The control unit 14 may output the calculated concentration level to the external device 15. The external device 15 is a device that performs a predetermined operation based on the concentration level. The external device 15 is, for example, a warning device that issues a warning to a target person when the concentration level is equal to or lower than a warning threshold, a driving assistance device that assists in driving a moving object based on the concentration level, and a driving device that drives a moving object based on the concentration level.

[0034] Next, the estimation process executed by the control unit 14 in the first embodiment will be described with reference to the flowchart of Fig. 4. The estimation process starts every time the control unit 14 acquires one frame of image IM and line of sight LS.

[0035] In step S100, the control unit 14 calculates the movement speed of the acquired line of sight LS by comparing it with the position PE of the line of sight LS acquired immediately before. After the calculation, the process proceeds to step S101.

[0036] In step S101, the control unit 14 determines whether the moving speed calculated in step S101 is equal to or greater than the speed threshold. If it is equal to or greater than the speed threshold, the estimation process ends. If it is not equal to or greater than the speed threshold, the process proceeds to step S102.

[0037] In step S102, the control unit 14 determines a reliability threshold based on the time during which estimation of the concentration level is stopped. After the determination, the process proceeds to step S103.

[0038] In step S103, the control unit 14 determines the reliability based on the image IM acquired at the start of the estimation process. After the determination, the process proceeds to step S104.

[0039] In step S104, the control unit 14 determines whether the reliability determined in step S103 is equal to or less than the reliability threshold determined in step S102. If it is equal to or less than the reliability threshold, the estimation process ends. If it is not equal to or less than the reliability threshold, the process proceeds to step S105.

[0040] In step S105, the control unit 14 estimates the concentration level based on the image IM and the line of sight LS acquired at the start of the estimation process. After the estimation, the process proceeds to step S106.

[0041] In step S106, the control unit 14 outputs the concentration degree calculated in step S105 to the external device 15. After the output, the estimation process ends.

[0042] The electronic device 10 of the first embodiment configured as described above determines the reliability of the image IM based on the image IM, and estimates the concentration level of the subject based on the reliability, the image IM, and the line of sight LS. For example, the object and direction of gaze generally change for various scenes such as highways, urban areas, suburbs, and residential areas. However, it is difficult to improve the estimation accuracy of the concentration level by simply detecting the movement of the line of sight LS. Also, even in various scenes, in situations with few vehicles and pedestrians, and in straight roads without intersections and curves, the movement of the line of sight becomes monotonous regardless of the concentration level of the subject, so it is difficult to improve the estimation accuracy of the concentration level. On the other hand, with the above-mentioned configuration, the electronic device 10 estimates the concentration level based on the reliability of the image IM of various scenes, so that it can estimate the concentration level with high accuracy. Since human attention is affected by the concentration level, the electronic device 10 can improve the estimation accuracy of the subject's attention in various situations.

[0043] Furthermore, the electronic device 10 of the first embodiment stops estimating the concentration level when the reliability is equal to or lower than the reliability threshold. With this configuration, the electronic device 10 can estimate only the concentration level with a reliable estimation accuracy.

[0044] Furthermore, the electronic device 10 of the first embodiment lowers the reliability threshold as the time for which estimation of the concentration level is stopped becomes longer. With this configuration, the electronic device 10 can prevent a long period during which the concentration level is not estimated.

[0045] Next, an electronic device 10 according to a second embodiment of the present disclosure will be described. In the second embodiment, a method of calculating a concentration degree is different from that of the first embodiment. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as the first embodiment.

[0046] 1, an electronic device 10 according to the second embodiment includes an imaging unit 11, a line-of-sight detection unit 12, a memory 13, and a control unit 14. The configurations and functions of the imaging unit 11, line-of-sight detection unit 12, and memory 13 in the second embodiment are the same as those in the first embodiment. The configuration of the control unit 14 in the second embodiment is the same as that in the first embodiment.

[0047] In the second embodiment, the control unit 14 determines the reliability of an image IM based on the image IM, as in the first embodiment. In the second embodiment, unlike the first embodiment, the control unit 14 estimates the concentration level of the subject based on a larger number of frames of images IM captured continuously, as the reliability becomes lower. For example, the control unit 14 may estimate the concentration level using image IM of frames that go back a longer time interval from the latest image IM as the reliability becomes lower, or image IM of a larger number of frames as the reliability becomes lower.

[0048] In the second embodiment, similar to the first embodiment, the control unit 14 may estimate the concentration level by excluding images IM in which the movement speed of the line of sight LS is equal to or greater than a threshold value from among multiple images IM captured continuously.

[0049] In the second embodiment, the control unit 14 may output the calculated concentration level to the external device 15, similarly to the first embodiment.

[0050] Next, the estimation process executed by the control unit 14 in the second embodiment will be described with reference to the flowchart of Fig. 5. The estimation process starts every time the control unit 14 acquires one frame of image IM and line of sight LS.

[0051] In step S200, the control unit 14 calculates the moving speed of the acquired line of sight LS by comparing it with the position PE of the line of sight LS acquired immediately before. After the calculation, the process proceeds to step S201.

[0052] In step S201, the control unit 14 determines whether the moving speed calculated in step S200 is equal to or higher than a speed threshold value. If it is not equal to or higher than the speed threshold value, the process proceeds to step S202. If it is equal to or higher than the speed threshold value, the process proceeds to step S203.

[0053] In step S202, the control unit 14 stores the image IM and the line of sight LS acquired at the start of the estimation process in the memory 13. After the storage, the process proceeds to step S203.

[0054] In step S203, the control unit 14 determines the reliability based on the image IM acquired at the start of the estimation process. After the determination, the process proceeds to step S204.

[0055] In step S204, the control unit 14 determines the number of frames of the image IM to be used for estimating the concentration based on the reliability determined in step S203. After the determination, the process proceeds to step S205. For example, the control unit 14 may increase the number of frames of the image IM to be used for estimating the concentration as the reliability determined in step S203 is lower. For example, when the reliability determined in step S203 is equal to or lower than a first threshold value, the number of frames of the image IM to be used for estimating the concentration may be set to N1. When the reliability determined in step S203 is greater than the first threshold value, the number of frames of the image IM to be used for estimating the concentration may be set to N2 (N2 < N1). Further, not only the above first threshold value but also a plurality of threshold values may be compared with the reliability, and the number of frames of the image IM to be used for estimating the concentration may be determined step by step.

[0056] In step S205, the control unit 14 reads out images IM going back in time, the number of frames determined in step S204 minus one frame, together with the corresponding lines of sight LS, from the memory 13. After reading, the process proceeds to step S206.

[0057] In step S206, the concentration level is estimated based on the image IM and line of sight LS acquired at the start of the estimation process and the image IM and line of sight LS read out in step S205. After the estimation, the process proceeds to step S207.

[0058] In step S206, the control unit 14 outputs the concentration degree calculated in step S206 to the external device 15. After the output, the estimation process ends.

[0059] The electronic device 10 of the second embodiment configured as described above estimates the concentration level of the subject based on the number of frames of images IM captured continuously as the reliability becomes lower. With this configuration, the electronic device 10 increases the combinations of images IM and gaze LS used to estimate the concentration level when the reliability is low, so that the concentration level can be estimated with high estimation accuracy regardless of the difference in reliability. Furthermore, the electronic device 10 may reduce the combinations of images IM and gaze LS used to estimate the concentration level when the reliability is high, so that the calculation processing load of the control unit can be reduced.

[0060] The contents of the present disclosure can be modified and amended in various ways by those skilled in the art based on the present disclosure. Therefore, these modifications and amendments are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to be logically inconsistent, or can be replaced with each functional unit, each means, each step, etc. of other embodiments. In each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. In addition, each embodiment of the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described above, and can be implemented by combining each feature or omitting a part as appropriate.

[0061] For example, in the first and second embodiments, the electronic device 10 includes the imaging unit 11 and the line-of-sight detection unit 12, and the control unit 14 acquires the image IM and line-of-sight LS used to estimate the concentration level, but the present invention is not limited to such a configuration. As shown in FIG. 6, for example, an information processing device 16 such as a cloud server may estimate the concentration level based on the image IM and line-of-sight LS in a manner similar to the electronic device 10. In such a configuration, the information processing device 16 may acquire the image IM and line-of-sight LS as information from a moving object 17 equipped with the imaging unit 11 and line-of-sight detection unit 12 via an acquisition unit 18. The information processing device 16 may output the estimated concentration level via an output unit 19 to an external device 15 on the moving object 17. [Explanation of symbols]

[0062] 10 Electronic equipment 11 Imaging unit 12 Line of sight detection unit 13. Memory 14 Control section 15 External equipment 16 Information processing equipment 17 Mobile 18 Acquisition Department 19 Output section IM Image LS line of sight MP Gaze Prediction Map PE Position corresponding to the line of sight

Claims

1. an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight, The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as the time for which estimation of the concentration level is stopped increases. electronic equipment.

2. an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight, The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are continuously captured as the reliability becomes lower. electronic equipment.

3. 3. The electronic device according to claim 1, The control unit determines the reliability by calculating the reliability based on at least one of a type, a number, and a position of an element included in the image. electronic equipment.

4. 4. The electronic device according to claim 3, The elements include at least one of a vehicle, a pedestrian, a road shape, a sign, a signal, and an optical flow. electronic equipment.

5. 3. The electronic device according to claim 1, The control unit determines the reliability by constructing a relationship between the learning image and information regarding the reliability based on learning data obtained by machine learning, and estimating the reliability based on the image. electronic equipment.

6. An acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight; an output unit that outputs the concentration degree, The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as the time for which estimation of the concentration level is stopped increases. Information processing device.

7. An acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that determines a reliability of the image as an estimation source of the concentration level of the subject based on the image, and estimates the concentration level of the subject based on the reliability, the image, and the line of sight; an output unit that outputs the concentration degree, The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are continuously captured as the reliability becomes lower. Information processing device.

8. Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; a control unit that determines a reliability of the image as an estimation source of the subject's concentration level based on the image, and estimates the subject's concentration level based on the reliability, the image, and the line of sight; The control unit stops estimating the concentration level when the reliability is equal to or less than a reliability threshold, and decreases the reliability threshold as the time for which estimation of the concentration level is stopped increases. Concentration calculation program.

9. Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; a control unit that determines a reliability of the image as an estimation source of the subject's concentration level based on the image, and estimates the subject's concentration level based on the reliability, the image, and the line of sight; The control unit estimates the concentration level of the subject based on a larger number of frames of the images that are continuously captured as the reliability becomes lower. Concentration calculation program.

10. an imaging step in which an imaging unit captures an image corresponding to a scene; a line-of-sight detection step in which a line-of-sight detection unit detects a line of sight of a subject with respect to the scene; A determination step in which the control unit determines a reliability of the image as an estimation source of the subject's concentration level based on the image; an estimation step of estimating a concentration level of the subject based on the reliability, the image, and the line of sight by a control unit; a step of lowering the reliability threshold as the time for which the estimation of the concentration level is stopped becomes longer by the control unit; A stop step of the control unit stopping the estimation of the concentration degree when the reliability is equal to or less than the reliability threshold. Concentration calculation method.

11. an imaging step in which an imaging unit captures an image corresponding to a scene; a line-of-sight detection step in which a line-of-sight detection unit detects a line of sight of a subject with respect to the scene; A determination step in which the control unit determines a reliability of the image as an estimation source of the subject's concentration level based on the image; An estimation step of estimating a concentration level of the subject based on the reliability, the image, and the line of sight by the control unit, In the estimation step, the control unit estimates the concentration level of the subject based on a larger number of frames of the images that are continuously captured as the reliability becomes lower. Concentration calculation method.

Images