Electronic device, information processing device, concentration level calculation program, concentration level calculation method, and computer learning method

Abstract

To improve the accuracy in estimating a subject's attentiveness to various situations and various states.SOLUTION: An electronic apparatus 10 has an imaging unit 11, a line-of-sight detection unit 12, and a control unit 14. The control unit 14 functions as a low-concentration heat map group estimation unit 15 and a high-concentration heat map estimation unit 16. The low-concentration heat map group estimation unit 15 can estimate a low-concentration heat map group based on an image. The high-concentration heat map estimation unit 16 can estimate a high-concentration heat map based on an image. The control unit 14 calculates the degree of concentration of a subject based on the low-concentration heat map group, the high-concentration heat map, and a line of sight. The low-concentration heat map group estimation unit 15 is constructed with learning data in which the relationship between an image for learning and the line of sight is machine-learned for every burden factor. The high-concentration heat map estimation unit 16 is constructed with learning data in which the relationship between the image for learning and the line of sight when the burden is removed or when the burden is reduced is machine-learned.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an electronic device, an information processing device, a concentration degree calculation program, a concentration degree calculation method, and a computer learning method. [Background technology]

[0002] Safe driving of a moving object requires the driver's attention. Therefore, it has been considered to observe the driver's attention and to issue a warning to the driver or provide driving support if the driver's attention decreases. As an observation of the attention, it has been proposed to calculate a cumulative visibility, which is a cumulative value of the degree of overlap of the line of sight with an object such as an oncoming vehicle around the vehicle, and to compare it with a reference value (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2008-029802 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, in order to calculate the cumulative visibility, the visibility at each hour is calculated using a table. However, the appropriate table differs depending on the various driving situations and driver states in the actual environment, and it is difficult to accurately observe the driver's attention in various driving situations and various driver states.

[0005] Therefore, in consideration of the problems with the conventional technology as described above, the object of the present disclosure is to provide an electronic device, an information processing device, a concentration calculation program, a concentration calculation method, and a computer learning method that accurately estimate a subject's attention in a variety of situations and conditions. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an electronic device according to a first aspect comprises: an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that estimates a concentration level of the subject based on the image and the line of sight, The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; A concentration level of the subject is estimated based on the low concentration heat map group, the high concentration heat map, and the gaze of the subject. death, Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded, and the concentration level of the subject is estimated. do.

[0007] An information processing device according to a second aspect of the present invention comprises: an acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that estimates a concentration level of the subject based on the image and the line of sight; an output unit that outputs the concentration degree, The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; A concentration level of the subject is estimated based on the low concentration heat map group, the high concentration heat map, and the subject's line of sight. death, Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded, and the concentration level of the subject is estimated. do.

[0008] The concentration calculation program from the third perspective is Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; a control unit that estimates a concentration level of the subject based on the image and the line of sight; The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; A concentration level of the subject is estimated based on the low concentration heat map group, the high concentration heat map, and the gaze of the subject. death, Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded, and the concentration level of the subject is estimated. do.

[0009] The concentration calculation method according to the fourth aspect is as follows: an imaging step of generating an image corresponding to the scene by imaging; a gaze detection step of detecting a gaze of a subject relative to the scene; and estimating a concentration level of the subject based on the image and the line of sight, The estimation step includes: A relationship between a learning image and gaze toward the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject is constructed by machine learning learning data, and a group of low concentration heat maps showing the probability of the gaze at each position constituting the image when the load is applied based on the image for each load factor are estimated, and a high concentration heat map is constructed by machine learning learning data about a relationship between a learning image and gaze toward the learning image when the load on the learning subject is removed or reduced, and showing the probability of the gaze at each position constituting the image when the load is removed or reduced based on the image, The concentration level of the subject is calculated based on the low concentration heat map group, the high concentration heat map, and the line of sight of the subject. Estimate, Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded, and the concentration level of the subject is estimated. do.

[0010] The fifth perspective of learning is: In order to estimate, based on an arbitrary image, a group of low concentration heat maps showing the probability of line of sight overlap at each position constituting the image in a load-applied state for each of a plurality of load factors, and a high concentration heat map showing the probability of line of sight overlap at each position constituting the image in a load-applied state, A relationship between a learning image and a line of sight to the learning image when a load that reduces concentration is applied to a learning subject according to a plurality of load factors is calculated. Using a plurality of learning images captured consecutively, Machine learning, The relationship between the learning image and the line of sight of the learning subject when the load on the learning subject is removed or reduced is , using the plurality of learning images Machine learning 、 The machine learning is performed by excluding, from among the plurality of learning images captured successively, learning images for which the speed of movement of the gaze with respect to the plurality of learning images exceeds a threshold. . Effect of the Invention

[0011] According to the electronic device, information processing device, concentration calculation program, concentration calculation method, and computer learning method disclosed herein, which are configured as described above, the accuracy of estimating a subject's attention is improved for a variety of situations and conditions. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a schematic configuration of an electronic device according to a first embodiment. [Diagram 2] 4A and 4B are diagrams for explaining the relationship between a line of sight detected by a line of sight detection unit and an image. [Diagram 3] 11A to 11C are diagrams showing specific examples of low-concentration heat map groups estimated by a low-concentration heat map group estimation unit for an image. [Figure 4]4 is a flowchart for explaining an estimation process executed by a control unit in FIG. 1 in the first embodiment. [Diagram 5] 10 is a flowchart for explaining an estimation process executed by a control unit in FIG. 1 in a second embodiment. [Figure 6] FIG. 2 is a block diagram showing a schematic configuration of an information processing device which is a modification of the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of an electronic device to which the present disclosure is applied will be described with reference to the drawings. Note that the following description also includes an information processing device, a concentration level calculation method, a learning method, and a concentration level calculation program to which the present disclosure is applied.

[0014] The electronic device according to the first embodiment of the present disclosure is provided in, for example, a moving object. The moving object may include, for example, a vehicle, a ship, and an aircraft. The vehicle may include, for example, an automobile, an industrial vehicle, a railroad vehicle, a vehicle for daily life, and a fixed-wing aircraft that runs on a runway. The automobile may include, for example, a passenger car, a truck, a bus, a motorcycle, and a trolley bus. The industrial vehicle may include, for example, an industrial vehicle for agriculture and construction. The industrial vehicle may include, for example, a forklift and a golf cart. The industrial vehicle for agriculture may include, for example, a tractor, a cultivator, a transplanter, a binder, a combine, and a lawnmower. The industrial vehicle for construction may include, for example, a bulldozer, a scraper, a shovel, a crane, a dump truck, and a road roller. The vehicle may include one that runs by human power. The classification of the vehicle is not limited to the above-mentioned example. For example, the automobile may include an industrial vehicle that can run on a road. The same vehicle may be included in multiple classifications. The watercraft may include, for example, a marine jet, a boat, a tanker, etc. The aircraft may include, for example, a fixed-wing aircraft, a rotary-wing aircraft, etc.

[0015] As shown in FIG. 1, an electronic device 10 according to the first embodiment of the present disclosure includes an imaging unit 11, a line-of-sight detection unit 12, a memory 13, and a control unit 14.

[0016] The imaging unit 11 is provided on the moving body so as to be able to capture, for example, a scene in the forward direction of the moving body. The imaging unit 11 is, for example, a camera capable of capturing images at a speed of 30 fps. The imaging unit 11 captures images to generate an image corresponding to the scene.

[0017] The line of sight detection unit 12 is provided in the moving body so as to be able to detect the line of sight of a subject seated in, for example, the driver's seat of the moving body. The line of sight detection unit 12 is, for example, either a contact-type eye tracker or a non-contact-type eye tracker, and detects the line of sight of the subject with respect to the scene. As shown in FIG. 2, the line of sight LS is indicated as the direction of the line of sight LS corresponding to, for example, a position PE in the same coordinate system as the image IM generated by the imaging unit 11 through imaging.

[0018] The gaze detection unit 12 may detect gaze data of a time series of the gaze LS as the gaze data. More specifically, the gaze detection unit 12 may detect the position PE of the gaze LS on the image IM at each time and output the series of the positions PE of the gaze LS in time as the gaze data. The gaze detection unit 12 may detect the gaze LS at a higher speed, in other words, at a higher frequency than the imaging unit 11, and may accumulate and output the series of the positions PE of the gaze LS in time that are well detected as the path of the gaze LS of one image IM.

[0019] The memory 13 includes any storage device, such as a random access memory (RAM) and a read only memory (ROM), etc. The memory 13 stores various programs that cause the control unit 14 to function, and various information used by the control unit 14.

[0020] The control unit 14 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 14 may be either a system-on-a-chip (SoC) or a system in a package (SiP) in which one or more processors work together. The control unit 14 controls the operation of each component of the electronic device 10.

[0021] The control unit 14 normally causes the imaging unit 11 to perform continuous imaging at a speed of, for example, 30 fps, and continuously acquires images IM as information. The control unit 14 causes the line of sight detection unit 12 to detect the line of sight LS of the subject when imaging is performed by the imaging unit 11, and acquires the line of sight LS at substantially the same time as the imaging time of the image IM as information. The time substantially the same as the imaging time of the image IM may include a single detection time, or may include multiple detection times between the imaging time immediately before the most recent imaging time of the image IM and the most recent imaging time. When the time substantially the same as the imaging time of the image IM includes a single detection time of the line of sight LS, it may not be the exact same time, but may include the detection time closest to the imaging time in line of sight detection with the same period as the image imaging. The control unit 14 associates the image IM with the line of sight LS detected at substantially the same time as the imaging time of the image IM and stores them in the memory 13.

[0022] The control unit 14 may perform image processing on the acquired image IM. In a configuration in which the learning of the estimation unit is performed using an image IM that has been subjected to a predetermined image processing, as described below, the control unit 14 performs the predetermined image processing on the acquired image IM and stores the image IM in the memory 13 in association with the line of sight LS. The predetermined image processing may be, for example, a process of generating a semantic segmentation image based on the acquired image IM. The semantic segmentation image is an image in which a label or category is associated with all pixels in the image IM.

[0023] The control unit 14 estimates the concentration level of the subject based on the image IM and the line of sight LS whose imaging time and detection time are substantially the same. More specifically, the control unit 14 functions as a low concentration heat map group estimation unit 15 and a high concentration heat map estimation unit 16 to estimate the concentration level of the subject.

[0024] The low-concentration heat map group estimation unit 15 estimates a first low-concentration heat map when a specific load factor is applied to the subject based on the image IM. The low-concentration heat map group estimation unit 15 estimates a low-concentration heat map for each of a plurality of load factors. A plurality of low-concentration heat maps corresponding to the plurality of load factors respectively constitute a low-concentration heat map group. The load factors are various factors that put a load on the concentration of a general subject on an action such as driving, for example, conversation with a passenger, hands-free conversation, listening to a radio, deep thinking, drowsiness, fatigue, etc.

[0025] As shown in Figure 3, the low concentration heat map MP lc is a two-dimensional map showing the probability that a typical subject will gaze at each position in a specific scene, i.e., the probability that the line of sight will overlap with the gaze of the subject, when a specific load factor is applied to the subject. lc is the first low concentration heatmap MP corresponding to contemplation lc1 ,The second low-concentration heat map MP corresponding to conversations with passengers lc2 ,The third low concentration heat map MP corresponding to drowsiness lc3,The fourth low concentration heat map MP corresponding to fatigue lc4 Includes.

[0026] The high concentration heat map estimation unit 16 estimates a high concentration heat map when all specific stress factors for the subject are removed based on the image IM. The high concentration heat map is a two-dimensional map showing the probability that a general subject will gaze at each position in a specific scene, i.e., the probability that the line of sight will overlap with the gaze, at each position constituting the image IM corresponding to the scene, when all of the above-mentioned specific stress factors are removed.

[0027] The low concentration heat map group estimation unit 15 and the high concentration heat map estimation unit 16 are configured, for example, by a multi-layered neural network. The low concentration heat map group estimation unit 15 and the high concentration heat map estimation unit 16 are constructed by implementing machine learning, as described below.

[0028] The control unit 14 may estimate a single low-concentration heat map group based on a plurality of images IM captured continuously in a configuration in which the low-concentration heat map group estimation unit 15 is constructed by learning using a plurality of learning images captured continuously, as described below. The control unit 14 may estimate a single high-concentration heat map based on a plurality of images IM in a configuration in which the high-concentration heat map estimation unit 16 is constructed by learning using a plurality of learning images captured continuously, as described below. The control unit 14 may estimate a single low-concentration heat map group and a single high-concentration heat map based on, for example, the image IM of a frame within a range going back a predetermined time interval from the latest image IM.

[0029] The control unit 14 may estimate a single low concentration heat map group and a single high concentration heat map by excluding images IM in which the movement speed of the line of sight LS is equal to or greater than a threshold value among a plurality of images IM captured continuously. More specifically, when the movement amount of the line of sight LS at any detection time from the line of sight LS at the immediately preceding detection time exceeds a threshold value, the control unit 14 may exclude an image IM associated with the detection time of the line of sight LS at the any detection time. In other words, the control unit 14 may exclude information during a saccade. The reason for the exclusion is that a saccade is an action in the middle of moving to the next fixation point, and the position PE of the line of sight LS during a saccade does not have a meaning as a fixation point. The control unit 14 may determine whether or not a saccade is in progress based on the movement speed of the position PE of the line of sight LS, or the like.

[0030] The control unit 14 calculates the concentration level of the subject based on the estimated low concentration heat map group and high concentration heat map, and the subject's line of sight LS, as will be described below with reference to the example.

[0031] The control unit 14 generates a first low concentration heat map MP lc1 The control unit 14 may read out the probability of the position PE corresponding to the line of sight LS of the subject in the first low concentration heat map MP lc1 Similar to the probability readout in the second low-concentration heat map MP lc2 , the third low concentration heat map MP lc3 , 4th low concentration heat map MP lc4 , and the probability of a position PE corresponding to the line of sight LS of the subject in the high-concentration heat map. Instead of reading out the probability, the control unit 14 may calculate an average value of the probability in a predetermined range based on the position PE of the line of sight LS of the subject, and use this instead of the probability in the following description. The control unit 14 may calculate the average value of the probability in a predetermined range based on the position PE of the line of sight LS of the subject, and use this instead of the probability in the following description. lc The integrated value of the probability along the path of the subject's line of sight LS in the high concentration heat map may be calculated and used in place of the probability in the following description.

[0032] The control unit 14 may select the heat map with the highest readout probability from the low concentration heat map group and the high concentration heat map group. The control unit 14 may calculate the concentration level based on a reference value determined for the selected heat map.

[0033] For example, all low concentration heatmap MPs that make up the low concentration heatmap group lc For a high concentration heat map, the reference value may be set within a range of concentration from 0% to less than 50%. For example, for a high concentration heat map, the reference value may be set within a range of concentration from 50% to 100%. The control unit 14 determines whether the heat map with the highest probability of the position PE corresponding to the line of sight LS is one of the low concentration heat maps MP lc If, then, low concentration heatmap MP lc The control unit 14 may calculate the concentration level of the subject by determining that the heat map with the highest probability of the position PE corresponding to the line of sight LS is the low concentration heat map MP lc If so, the control unit 14 may recognize the load factor corresponding to the low-concentration heat map. If the heat map in which the probability of the position PE corresponding to the line of sight LS is the highest is a high-concentration heat map, the control unit 14 may calculate a reference value determined for the high-concentration heat map as the concentration level of the subject.

[0034] The control unit 14 may calculate the concentration level based on not only the type of the selected heat map but also the probability of the position PE corresponding to the line of sight LS in the selected heat map. For example, in the case of a low concentration heat map MP lc For a high concentration heat map, a function or table may be defined so that the greater the probability, the smaller the calculated concentration level. Also, for example, for a high concentration heat map, a function or table may be defined so that the greater the probability, the larger the calculated concentration level. The control unit 14 may calculate the concentration level using the function or table defined for the selected heat map based on the probability of the position PE corresponding to the line of sight LS.

[0035] The control unit 14 may calculate the concentration level based on the probability in the high concentration heat map and the probability in the low concentration heat map with the highest probability among the low concentration heat maps. For example, the control unit 14 may calculate the concentration level based on the reference value for the high concentration heat map and the probability in the low concentration heat map MP with the highest probability. lc The concentration degree may be calculated by weighting the reference value for each probability and the average value for each probability.

[0036] The control unit 14 may output the calculated concentration level to the external device 17. The external device 17 is a device that performs a predetermined operation based on the concentration level. The external device 17 is, for example, a warning device that issues a warning to a target person when the concentration level is equal to or lower than a warning threshold, a driving assistance device that assists in driving a moving object based on the concentration level, and a driving device that drives a moving object based on the concentration level.

[0037] The control unit 14 determines that the heat map with the highest probability at the position PE corresponding to the line of sight LS of the target person is the low concentration heat map MP lc If so, the load factor recognized as described above may be output together with the concentration degree.

[0038] The low concentration heat map group estimation unit 15 is constructed from learning data that is machine-learned to learn the relationship between a learning image and the gaze of a learning subject when a load that reduces concentration is applied to the learning subject. The low concentration heat map group estimation unit 15 is also constructed from learning data for each load factor that is machine-learned for each load factor.

[0039] The learning images and gazes for the load of contemplation are collected, for example, by having the learning subject look at the learning images while performing single-digit arithmetic operations in his / her head. The learning images and gazes for the load of conversation with a passenger are collected, for example, by having the learning subject and the collaborator have daily conversations about the weather, hobbies, health, etc. while showing the learning images. The learning images and gazes for the load of drowsiness are collected, for example, by showing the learning images to the learning subject 20 hours after waking up. Alternatively, the learning images and gazes for the load of drowsiness are collected, for example, by showing the learning images to the learning subject while collecting biological information related to drowsiness, such as the number of blinks and PERCLOS (Percent of Eyelid Closure), and selecting the learning images and gazes when the concentration level based on the biological information corresponds to an inattentive state. The learning images and gazes for the load of fatigue are collected, for example, by showing the learning images to the learning subject immediately after one hour of circuit training. Alternatively, learning images and gazes in response to fatigue load are collected by showing learning images to a learning subject while collecting biometric information related to fatigue such as heart rate, brain waves, etc., and selecting a learning image and gaze when the fatigue level based on the biometric information corresponds to an inattentive state.

[0040] The low concentration heat map group estimation unit 15 estimates the probability that each pixel or each area consisting of multiple pixels constituting an arbitrary image IM overlaps with the line of sight LS based on the relationship between the learning image and the line of sight of the learning subject who is loaded with a specific load factor. The low concentration heat map group estimation unit 15 generates a low concentration heat map MP2 that shows the probability for each position constituting an arbitrary image IM in two dimensions when a specific load factor is applied. lc is generated for each load factor.

[0041] The low concentration heat map group estimation unit 15 may further be constructed by machine learning of a plurality of continuously captured learning images and the actual gaze of a learning subject who is loaded with a specific load factor. More specifically, the low concentration heat map group estimation unit 15 may estimate the probability that each pixel constituting an arbitrary single image IM, or each area composed of a plurality of pixels, overlaps with the gaze LS based on the gaze position of the learning subject who is loaded with a specific load factor for each of the plurality of continuously captured learning images. The low concentration heat map group estimation unit 15 may further be constructed by learning images and gazes in which the movement speed of the gaze for the plurality of continuously captured learning images is less than a threshold value.

[0042] The high concentration heat map estimation unit 16 is constructed from learning data that has been machine-learned to learn the relationship between a learning image and the line of sight of the learning image when the load on the learning subject is removed. The high concentration heat map estimation unit 16 estimates the probability that each pixel or each area composed of multiple pixels constituting an arbitrary image IM overlaps with the line of sight LS based on the relationship between the learning image and the line of sight LS of the learning subject who has removed or reduced the load. The high concentration heat map estimation unit 16 generates a high concentration heat map that two-dimensionally shows the probability for each position constituting an arbitrary image IM when the load is removed or reduced.

[0043] The high concentration heat map estimation unit 16 may further be constructed by machine learning of a plurality of continuously captured learning images and the actual gaze of a learning subject who has removed or reduced his / her load. More specifically, the high concentration heat map estimation unit 16 may estimate the probability that each pixel or each area composed of a plurality of pixels constituting any single image IM overlaps with the gaze LS based on the gaze position of the learning subject who has removed or reduced his / her load for each of the plurality of continuously captured learning images. The high concentration heat map estimation unit 16 may further be constructed by learning images and gazes in which the movement speed of the gaze for the plurality of continuously captured learning images is less than a threshold value.

[0044] The learning images and the line of sight may be collected, for example, by the following load test method. In the load test, the subject may be made to drive a moving object. The subject may be made to drive a specified section without the load, and then the same section with a load according to a factor to be exemplified later. The subject may be made to drive without the load and with the load twice in a row, and the scene in front of the moving object may be collected as the learning image, and the line of sight may be collected as the line of sight toward the learning image. The load in the test may be within a range that may occur as an action while driving. A high load that is not expected to occur while driving may not be adopted as the load in the test. A load that requires the gaze to be taken away from the front may not be adopted in the test. This is because a load that requires the gaze to be taken away is obviously dangerous, and therefore there is little need to use it for measuring the concentration level.

[0045] The loads in the load test may include single-digit operation, conversation, radio listening, two-back operation, drowsiness, and fatigue. In the load test, each of the loads exemplified above may be applied alone or in combination with a plurality of loads.

[0046] The single-digit operation may be at least one of single-digit addition and subtraction. The single-digit operation may be posed to the subject by a recorded voice. If the subject is unable to answer the single-digit operation within the time limit, the next question may be asked. The forward view and gaze may be collected for learning when the correct answer rate for the single-digit operation is equal to or higher than a threshold. By limiting the correct answer rate to a threshold or higher, gaze can be collected under a load that reduces concentration on the recognition of the forward view due to concentration on the operation. Operations of two or more digits may not be used in the load test because they are too heavy a load.

[0047] The conversation may be conducted by the subject and a collaborator such as an operator on the theme of work, hobbies, etc. For example, a program in which the contents of the conversation are broadcast may be applied to the radio listening. The 2-back calculation may be a calculation in which the subject answers the number two numbers before the number of consecutive single-digit numbers. The 2-back calculation for a two-digit number and the 3-back calculation in which the number three numbers before the number are answered may not be used in the stress test because the stress is the issue. The stress state with drowsiness may be a state in which the subject is aware of drowsiness, or a state in which the subject is recognized as having drowsiness based on general drowsiness indicators such as the size of the pupils and the degree of opening of the eyelids. The stress state with fatigue may be a state immediately after exercise.

[0048] Next, the estimation process executed by the control unit 14 in the first embodiment will be described with reference to the flowchart of Fig. 4. The estimation process starts every time the control unit 14 acquires one frame of image IM and line of sight LS.

[0049] In step S100, the control unit 14 calculates the movement speed of the acquired line of sight LS by comparing it with the position PE of the line of sight LS acquired immediately before. After the calculation, the process proceeds to step S101.

[0050] In step S101, the control unit 14 determines whether the moving speed calculated in step S101 is equal to or greater than a threshold. If it is equal to or greater than the threshold, the process proceeds to step S102. If it is not equal to or greater than the threshold, the process proceeds to step S103.

[0051] In step S102, the control unit 14 discards the image IM and the line of sight LS acquired at the start of the estimation process. After discarding, the process proceeds to step S104.

[0052] In step S103, the control unit 14 associates the image IM and the line of sight LS acquired at the start of the estimation process with each other and stores them in the memory 13. After storage, the process proceeds to step S104.

[0053] In step S104, the control unit 14 reads out the image IM in the combination of the image IM and the line of sight LS stored within a range going back a predetermined time interval from the memory 13. After reading out, the process proceeds to step S105.

[0054] In step S105, the control unit 14 estimates a single low-concentration heat map group and a single high-concentration heat map based on the multiple images IM included in the combination read out in step S104. After the estimation, the process proceeds to step S106.

[0055] In step S106, the control unit 14 selects a heat map with the highest probability of a position PE corresponding to the line of sight LS based on the low-concentration heat map group and the high-concentration heat map estimated in step S105 and the line of sight LS acquired at the start of the estimation process. The control unit 14 calculates a reference value determined for the selected heat map as the concentration level of the subject. After the calculation, the process proceeds to step S107.

[0056] In step S107, the control unit 14 outputs the concentration degree calculated in step S106 to the external device 17. After the output, the estimation process ends.

[0057] In the electronic device 10 of the first embodiment configured as described above, the control unit 14 functions as a low concentration heat map group estimation unit 15 capable of estimating a low concentration heat map group showing the probability of the line of sight LS at each position constituting the image IM when a load is applied for each of a plurality of load factors based on the image IM, and a high concentration heat map estimation unit 16 capable of estimating a high concentration heat map showing the probability of the line of sight LS at each position constituting the image IM when the load is removed, and estimates the concentration level of the subject based on the estimated low concentration heat map group and high concentration heat map and the line of sight LS. For example, it is common for the object and direction of gaze to change for various scenes such as highways, urban areas, suburbs, and residential areas. In addition, the tendency of the object that a person gazes at varies depending on the type of load. Therefore, it is difficult to improve the estimation accuracy of the concentration level by simply detecting the movement of the line of sight LS alone. On the other hand, with the above-mentioned configuration, the electronic device 10 has learned the tendency of objects that people under stress tend to gaze at for various scenes according to multiple stress factors, and the tendency of objects that people without stress tend to gaze at, so it can estimate the concentration level with high accuracy based on the image IM and line of sight LS corresponding to the scene at each time. Since human attention is affected by the concentration level, the electronic device 10 can improve the accuracy of estimating the subject's attention level in various situations.

[0058] In addition, in the electronic device 10 of the first embodiment, the heat map with the highest probability of the position PE corresponding to the line of sight LS of the subject is one of the low-concentration heat maps MP lc If so, the low concentration heat map MP lc With this configuration, the electronic device 10 can provide useful information to the external device 17 that can execute various operations based on the state of the subject.

[0059] Moreover, the electronic device 10 of the first embodiment calculates the concentration level of the subject based on a plurality of images IM captured continuously. With this configuration, the electronic device 10 has learned the line of sight LS of the person in a load factor or no load state for a fluctuating scene, and therefore can calculate the concentration level with higher accuracy based on the images IM corresponding to the scene in the fluctuating state.

[0060] Moreover, the electronic device 10 of the first embodiment calculates the concentration level of the subject by excluding images IM in which the movement speed of the line of sight LS exceeds a threshold value from among a plurality of images IM captured continuously. It is considered that the object that the subject is gazing at in the scene is a factor that greatly affects the concentration level. Therefore, it is considered that the influence of the line of sight LS on the concentration level is low while the line of sight LS is changing from an arbitrary object to another object at high speed, such as during a saccade. Therefore, the electronic device 10 of the above configuration can calculate the concentration level with higher accuracy because it excludes images IM that are considered to have a low influence on the concentration level.

[0061] Next, an electronic device 10 according to a second embodiment of the present disclosure will be described. In the second embodiment, a method of calculating a concentration degree is different from that of the first embodiment. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that the same reference numerals are used to designate parts having the same configuration as the first embodiment.

[0062] 1, an electronic device 10 according to the second embodiment includes an imaging unit 11, a line-of-sight detection unit 12, a memory 13, and a control unit 14. The configurations and functions of the imaging unit 11, line-of-sight detection unit 12, and memory 13 in the second embodiment are the same as those in the first embodiment. The configuration of the control unit 14 in the second embodiment is the same as that in the first embodiment.

[0063] In the second embodiment, the control unit 14 estimates a low-concentration heat map group and a high-concentration heat map based on an image IM, similar to the first embodiment. In the second embodiment, unlike the first embodiment, the control unit 14 estimates a plurality of low-concentration heat maps MP each estimated based on a plurality of images IM captured continuously. lc In the above, the control unit 14 may calculate an integrated value of the probability of the position PE corresponding to the line of sight LS for the multiple images IM for each load factor. In multiple high-concentration heat maps estimated based on multiple images IM captured continuously, the control unit 14 may calculate an integrated value of the probability of the position PE corresponding to the line of sight LS for each of the multiple images IM. For example, the control unit 14 may calculate an integrated value for a low-concentration heat map group and a high-concentration heat map corresponding to images IM of frames within a range going back a predetermined time interval from the latest image IM.

[0064] In the second embodiment, similar to the first embodiment, the control unit 14 may exclude images IM in which the movement speed of the line of sight LS is equal to or greater than a threshold value from among multiple images IM captured continuously, and calculate an integrated value for each of the low concentration heat map group and the high concentration heat map.

[0065] In the second embodiment, unlike the first embodiment, the control unit 14 may calculate the concentration level based on the integrated values ​​calculated for the low concentration heat map group and the high concentration heat map, as illustrated and described below.

[0066] The control unit 14 may select the heat map with the largest integrated value from the low concentration heat map group and the high concentration heat map. In the second embodiment, the control unit 14 may calculate the concentration level based on a reference value set for the selected heat map, similar to the first embodiment. The reference value may be the same as or different from the first embodiment.

[0067] In the second embodiment, the control unit 14 may perform calculations using a function or a table such that, when the selected heat map is a low-concentration heat map, the concentration degree becomes smaller as the calculated integrated value increases, similar to the first embodiment. In the second embodiment, the control unit 14 may perform calculations using a function or a table such that, when the selected heat map is a high-concentration heat map, the concentration degree becomes larger as the calculated integrated value increases, similar to the first embodiment.

[0068] The control unit 14 may calculate the concentration level based on the integrated value in the high concentration heat map and the integrated value in the low concentration heat map having the maximum integrated value among the low concentration heat maps. For example, the control unit 14 may calculate the concentration level based on the reference value for the high concentration heat map and the low concentration heat map MP having the maximum integrated value. lc The concentration level may be calculated by weighting the respective integrated values ​​of the reference value for each of the above-mentioned integrated values ​​according to the respective integrated values.

[0069] In the second embodiment, the control unit 14 may output the calculated concentration level to the external device 17, similarly to the first embodiment.

[0070] Next, the estimation process executed by the control unit 14 in the second embodiment will be described with reference to the flowchart of Fig. 5. The estimation process starts every time the control unit 14 acquires one frame of image IM and line of sight LS.

[0071] In step S200, the control unit 14 calculates the movement speed of the acquired line of sight LS by comparing it with the position PE of the line of sight LS acquired immediately before. After the calculation, the process proceeds to step S201.

[0072] In step S201, the control unit 14 determines whether the moving speed calculated in step S200 is equal to or greater than a threshold. If it is equal to or greater than the threshold, the process proceeds to step S202. If it is not equal to or greater than the threshold, the process proceeds to step S203.

[0073] In step S202, the control unit 14 discards the image IM and the line of sight LS acquired at the start of the estimation process. After discarding, the process proceeds to step S206.

[0074] In step S203, the control unit 14 estimates a low concentration heat map group and a high concentration heat map based on the image IM acquired at the start of the estimation process. After the estimation, the process proceeds to step S204.

[0075] In step S204, the control unit 14 reads out the probability of the position PE corresponding to the line of sight LS acquired at the start of the estimation process in the low concentration heat map group and the high concentration heat map estimated in step S203. After reading out, the process proceeds to step S205.

[0076] In step S205, the control unit 14 stores the probabilities for the low concentration heat map group and the high concentration heat map identified in step S204 in the memory 13. After storage, the process proceeds to step S206.

[0077] In step S206, the control unit 14 reads out the probabilities of the low-concentration heat map group and the high-concentration heat map group based on the multiple images IM within a predetermined time interval going back from the start of the estimation process from the memory 13. After reading out, the process proceeds to step S207.

[0078] In step S207, the control unit 14 calculates an integrated value by integrating the probabilities read in step S206 for each of the low concentration heat maps and each of the high concentration heat maps included in the low concentration heat map group. After the integration, the process proceeds to step 208.

[0079] In step S208, the control unit 14 calculates the concentration level based on the integrated value calculated in step S207. After the calculation, the process proceeds to step S209.

[0080] In step S209, the control unit 14 outputs the concentration degree calculated in step S208 to the external device 17. After the output, the estimation process ends.

[0081] The contents of the present disclosure can be modified and amended in various ways by those skilled in the art based on the present disclosure. Therefore, these modifications and amendments are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to be logically inconsistent, or can be replaced with each functional unit, each means, each step, etc. of other embodiments. In each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. In addition, each embodiment of the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described above, and can be implemented by combining each feature or omitting a part as appropriate.

[0082] For example, in the first and second embodiments, the electronic device 10 includes the imaging unit 11 and the line-of-sight detection unit 12, and the control unit 14 acquires the image IM and line-of-sight LS used to estimate the concentration level, but the present invention is not limited to such a configuration. As shown in FIG. 6, for example, an information processing device 18 such as a cloud server may estimate the concentration level based on the image IM and line-of-sight LS, similar to the electronic device 10. In such a configuration, the information processing device 18 may acquire the image IM and line-of-sight LS as information from a moving object 19 equipped with the imaging unit 11 and line-of-sight detection unit 12 via the acquisition unit 20. The information processing device 18 may output the estimated concentration level via the output unit 21 to an external device 17 in the moving object 19. [Explanation of symbols]

[0083] 10 Electronic equipment 11 Imaging unit 12 Line of sight detection unit 13. Memory 14 Control section 15 Low concentration heat map group estimation part 16 High concentration heat map estimation part 17 External equipment 18 Information processing equipment 19 Mobile 20 Acquisition Department 21 Output section IM Image LS line of sight MP lc Low concentration heatmap PE Position corresponding to the line of sight

Claims

1. an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; A control unit that estimates a concentration level of the subject based on the image and the line of sight, The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; Estimating a concentration level of the subject based on the low concentration heat map group and the high concentration heat map and the subject's line of sight; Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded to estimate the concentration level of the subject. electronic equipment.

2. 2. The electronic device according to claim 1, When a heat map having a maximum probability of being located at a position corresponding to the line of sight of the subject among the low concentration heat map group and the high concentration heat map is a low concentration heat map in the low concentration heat map group, the control unit outputs a load factor corresponding to the low concentration heat map together with a concentration degree. electronic equipment.

3. an acquisition unit that acquires an image corresponding to a scene and a subject's line of sight with respect to the scene; A control unit that estimates a concentration level of the subject based on the image and the line of sight; an output unit that outputs the concentration degree, The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; Estimating a concentration level of the subject based on the low concentration heat map group and the high concentration heat map and the subject's line of sight; Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded to estimate the concentration level of the subject. Information processing device.

4. Computer, an imaging unit that captures an image corresponding to a scene; A gaze detection unit that detects a gaze of a subject with respect to the scene; a control unit that estimates a concentration level of the subject based on the image and the line of sight; The control unit is a low concentration heat map group estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject, and is capable of estimating a low concentration heat map group that shows the probability of the gaze at each position constituting the image when the load is applied, for each load factor, based on the image; and a high concentration heat map estimation unit that is constructed from learning data that is machine-learned of the relationship between a learning image and the gaze of the learning image when the load on the learning subject is removed or reduced, and is capable of estimating a high concentration heat map that shows the probability of the gaze at each position constituting the image when the load is removed or reduced, based on the image; Estimating a concentration level of the subject based on the low concentration heat map group and the high concentration heat map and the subject's line of sight; Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded to estimate the concentration level of the subject. Concentration calculation program.

5. An imaging step in which an imaging unit captures an image corresponding to a scene; a gaze detection step in which a gaze detection unit detects a gaze of a subject with respect to the scene; An estimation step in which the control unit estimates a concentration level of the subject based on the image and the line of sight, The estimation step includes: A relationship between a learning image and gaze toward the learning image when a load that reduces concentration for a plurality of load factors is applied to a learning subject is constructed by machine learning learning data, and a group of low concentration heat maps showing the probability of the gaze at each position constituting the image when the load is applied based on the image for each load factor are estimated, and a high concentration heat map is constructed by machine learning learning data about a relationship between a learning image and gaze toward the learning image when the load on the learning subject is removed or reduced, and showing the probability of the gaze at each position constituting the image when the load is removed or reduced based on the image, Estimating a concentration level of the subject based on the low concentration heat map group and the high concentration heat map and the subject's line of sight; Estimating a concentration level of the subject based on a plurality of the images captured successively; Among the plurality of images captured successively, images in which the speed of movement of the line of sight with respect to the plurality of images exceeds a threshold are excluded to estimate the concentration level of the subject. Concentration calculation method.

6. In order to estimate, based on an arbitrary image, a group of low concentration heat maps showing the probability of line of sight overlap at each position constituting the image in a load-applied state for each of a plurality of load factors, and a high concentration heat map showing the probability of line of sight overlap at each position constituting the image in a load-applied state, A relationship between the learning image and a line of sight of a learning subject when a load that reduces concentration according to a plurality of load factors is applied to the learning subject is machine-learned using a plurality of consecutively captured learning images; performing machine learning using the plurality of training images on a relationship between a training image and a line of sight of the training subject with respect to the training image when the load on the training subject is removed or reduced; The machine learning is performed by excluding, from among the plurality of learning images captured successively, learning images for which the speed of movement of the gaze with respect to the plurality of learning images exceeds a threshold. How computers learn.

Images