Estimation device, estimation method, program, and estimation system

Abstract

Provided are an estimation device and the like that make it possible to highly accurately estimate the auditory attention of a subject in real time, even when biological information for a neutral state cannot be acquired. An estimation device according to the present invention includes a storage unit that stores biological information that is time-series information for pupil diameter acquired while a subject is presented with an image that has different luminances, a region information acquisition unit that acquires region information that is index values for the brightness of regions of the image presented to the subject, an analysis unit that calculates a statistical trend for the time-series change in the pupil diameter from the biological information for the subject, and an estimation unit that uses the region information and the statistical trend for the time-series change in the pupil diameter to estimate the direction in which the attention of the subject is directed.

Description

Estimation device, estimation method, program, and estimation system

[0001] The present invention relates to a technique for estimating a subject's attention direction from changes in the subject's pupil diameter.

[0002] Conventional techniques are known that utilize the pupillary response to light (dilation or contraction of the pupil in response to light) to read visual or auditory attention (Non-Patent Documents 1 and 2).

[0003] For example, as a technique for estimating which information of various positions in space a subject is paying attention to, a method has been proposed that utilizes the fact that when images with different brightnesses are presented to the subject, the pupil diameter of the eye changes depending on the direction the subject is paying attention to. Patent Document 1 proposes a technique in which images with different brightnesses are presented to the subject while different sounds are presented to the left and right ears, and which sound, left or right, the subject is paying attention to is estimated from the change in pupil diameter.

[0004] Japanese Patent Application Laid-Open No. 2019-126423

[0005] Mathot et al., "The Pupillary Light Response Reveals the Focus of Covert Visual Attention", 2013.Liao et al., "Seeing an Auditory Object: Pupillary Light Response Reflects Covert Attention to Auditory Space and Object", 2022

[0006] However, in Patent Document 1, it is necessary to first acquire time-series information (first biological information) of pupil diameter when the subject is not paying attention to either the left or right (neutral state), and then acquire and estimate time-series information (second biological information) of pupil diameter when the subject is paying attention to either the left or right. Acquiring the first biological information in a neutral state is difficult, which limits the scope of use in the real world. Another issue is that real-time estimation is difficult in a situation where the subject's focus of attention changes from moment to moment within a given period of time.

[0007] The present invention aims to provide an estimation device, an estimation method, and a program that can estimate a subject's auditory attention with high accuracy in real time by using calculation results of trends in time-series changes in past biological information, even when biological information cannot be obtained in a neutral state. Note that "auditory attention" refers to the direction in which auditory attention is directed and whether or not auditory attention is directed (including when the subject is not hearing).

[0008] In order to solve the above problem, according to one aspect of the present invention, an estimation device includes a memory unit that stores biometric information, which is time-series information on pupil diameter acquired while presenting images of different brightness to a subject, an area information acquisition unit that acquires area information, which is an index value of the brightness of an area of ​​the image presented to the subject, an analysis unit that calculates the statistical trend of time-series changes in pupil diameter from the subject's biometric information, and an estimation unit that estimates the direction in which the subject is paying attention using the statistical trend of time-series changes in pupil diameter and the area information.

[0009] According to the estimation technology of the present invention, even if biometric information cannot be obtained in a neutral state, by using the calculation results of the trend of time-series changes in past biometric information, it is possible to estimate the subject's auditory attention in real time with high accuracy.

[0010] 1 is a diagram showing the situation of an experiment to explain the technical background; 2 is a diagram to explain a method for determining the trend of changes in human pupil size from time-series data of the subject's pupil size; 3 is a diagram showing examples of estimation results of actual pupil diameter and direction of attention; 4 is a diagram showing experimental estimation results and estimation accuracy; 5 is a block diagram showing the configuration of an estimation device according to a first embodiment; 6 is a diagram showing a flowchart illustrating the operation of the estimation device according to the first embodiment; 7 is a diagram showing an example of estimation results; 8 is a block diagram showing the configuration of an estimation device according to a second embodiment; 9 is a diagram showing a flowchart illustrating the operation of the estimation device according to the second embodiment; 10 is a diagram showing estimation results and estimation accuracy of a second estimation unit; 11 is a diagram showing estimation results and estimation accuracy of a third estimation unit; 12 is a diagram showing an example functional configuration of a computer.

[0011] Hereinafter, an embodiment of the present invention will be described in detail. Note that components having the same functions are assigned the same numbers, and duplicated explanations will be omitted. In the following description, the symbols " - " etc. should normally be written directly above the character immediately following it, but due to limitations in text notation, they are written immediately before the character in question. In formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on each element of a vector or matrix is ​​assumed to apply to all elements of that vector or matrix.

[0012] <Technical Background> An embodiment of the present invention, i.e., an estimation device for estimating human auditory attention, is based on the discovery of a natural law that there is a correlation between luminance information of an object to which auditory attention is directed and the trend of changes in human pupil size.

[0013] Therefore, the present invention is based on the discovery of a correlation between auditory attention and trends in changes in human pupil size.

[0014] Figure 1 shows the experimental setup. The experimental procedure is as follows: (1) A display 23 is placed in front of the subject 21, an image is presented, and speakers 25-R and 25-L are placed on either side of the display 23 to present different sounds. The images presented have different brightness levels in the right-hand region 23-R and the left-hand region 23-L, as shown in Figure 1. Images corresponding to the sounds presented from the speakers 25-R and 25-L are presented in the regions 23-R and 23-L of the display on the same side. In Figure 1, the sound of a saxophone is played from the left-hand speaker 25-L, and a video of a saxophone performance is played in the brightened left-hand region 23-L of the display 23. The sound of a bass guitar is played from the right-hand speaker 25-R, and a video of a bass guitar performance is played in the darkened right-hand region 23-R of the display 23. (2) The subject 21 is instructed to listen to the performance by alternating between different sound sources at any time, and to press the button corresponding to the sound to which the subject 21 is paying attention. The size of the subject's pupil (e.g., pupil diameter) is measured while the performance sounds and the video are being played.

[0015] FIG. 2 is a diagram for explaining a method for determining the trend of changes in human pupil size from time-series data of the pupil size of a subject.

[0016] The average pupil diameter for the analysis time width w1, which is set in advance depending on the attention target, is calculated by moving the discrimination interval w2 by increments of w3. i Mean pupil diameter in - p i is calculated as in equation (1).

[0017]

[0018] In addition, pupil t indicates the pupil diameter at time t. t0 represents the current time. i represents time t i where i = 1, 2, ..., L, and L = (w1 - w2) / w3. The average pupil diameter calculated from this - From p1 - p Land the vector T = (1, 2, ..., L) of the indexes of the times at which they were acquired, and using equations (2-1) to (2-3), β = (β0, β1, ..., β n ) is found.

[0019]

[0020] Note that n is the order of the coefficient β, and can be set to any value so that the coefficient β can be calculated from the average pupil diameter P and the vector T. Note that the vector T k is a vector whose elements are the k-th power of each element of vector T, that is, T k =(1 k ,2 k ,…,L k )

[0021] By inputting the index m of the current time t0 into the obtained formula, the predicted value of the pupil diameter p pred Calculate p pred =P(m)=β0+β1m+β2m 2 +…+β n m n (3) If the index m of the current time t0 is 0, then p pred =P(0)=β0. Furthermore, the current pupil diameter p current Ask for.

[0022]

[0023] Predicted value p pred and the current pupil diameter p current Compare with p current >p pred If so, we can assume that the pupil diameter is increasing and that the child is paying attention to the sound in the dark. current <p pred If so, it is assumed that there is a downward trend and that the person is paying attention to sounds in the brighter direction.

[0024] Figure 3 shows an example of the estimation results for the actual pupil diameter and the direction of attention, and Figure 4 shows the estimation results and estimation accuracy of the experiment. In the example of Figure 4, the estimation accuracy was about 60-70%.

[0025] These results show that the tendency for pupil size to change changes depending on the brightness of the direction in which auditory attention is directed (regardless of whether the eyes are actually facing that direction). The embodiment of the present invention utilizes the discovery of this natural law.

[0026] Instead of dividing the image into two, the image may be divided into three or more regions with different luminance levels.

[0027] Furthermore, although this experiment uses video of a performance, the images presented are not limited to moving images; they can also be still images or images of a real environment, as long as the brightness (e.g., luminance) that varies from position to position can be made similar in a certain area (i.e., each area can be made darker, brighter, etc.).

[0028] Hereinafter, still images, moving images, and the real environment are referred to as visual information.

[0029] First Embodiment An estimation device according to a first embodiment estimates the direction in which a subject's auditory attention is directed.

[0030] The estimation device 100 will be described below with reference to Figures 5 and 6. Figure 5 is a block diagram showing the configuration of the estimation device 100. Figure 6 is a flowchart showing the operation of the estimation device 100. As shown in Figure 5, the estimation device 100 includes a visual information presenting unit 110, an acoustic information presenting unit 120, a biometric information acquiring unit 130, a region information acquiring unit 140, an analyzing unit 150, an estimation unit 160, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for processing.

[0031] The operation of the estimation device 100 will be described below with reference to FIG.

[0032] [Visual Information Presentation Unit 110] In step S110, the visual information presentation unit 110 receives video data (also referred to as visual information a), plays the video data, and presents a video consisting of two or more regions with different index values ​​indicating brightness (e.g., luminance) to the subject. Therefore, the subject's field of vision includes two or more regions with different index values ​​indicating brightness. The visual information presentation unit 110 also outputs the visual information a to the region information acquisition unit 140. Note that a display device such as a monitor may be used to present the video.

[0033] [Acoustic Information Presentation Unit 120] In step S120, the acoustic information presentation unit 120 receives acoustic data corresponding to video image data, plays the acoustic data, and presents different sounds from two or more different directions to the subject. It also outputs information for identifying the sounds presented from each direction (also referred to as identification information b) to the estimation unit 160. For example, the acoustic information presentation unit 120 presents different sounds from two directions, one toward the right ear and one toward the left ear, and outputs information for identifying the sounds presented from each of the directions, the right ear and the left ear, (sound identification information) to the estimation unit 160. A sound playback device such as a speaker may be used to present the sounds.

[0034] [Biometric Information Acquisition Unit 130] In step S130, the biometric information acquisition unit 130 measures the pupil diameter while presenting images of different luminance to the subject, acquires biometric information c, which is time-series information on the measured pupil diameter, and outputs it to the analysis unit 150. Various conventional techniques can be used to measure the pupil diameter. For example, the biometric information acquisition unit 130 receives video image data of the subject's pupil as input, performs image analysis, and measures the pupil diameter.

[0035] [Region information acquisition unit 140] In step S140, the region information acquisition unit 140 receives visual information a as input, acquires region information d, which is an index value indicating the brightness of the region, for each region of the moving image presented in S110, i.e., each region included in the subject's field of view, and outputs the region information d to the estimation unit 160.

[0036] [Analysis unit 150] In step S150, the analysis unit 150 receives the subject's biometric information c as input, calculates the statistical trend of the time-series change in pupil diameter from the biometric information c, and outputs the calculated trend e to the estimation unit 160.

[0037] For example, the analysis unit 150 uses the biological information c (time-series information of the measured pupil diameter) to calculate the average pupil diameter for the analysis time width w1 while shifting the discrimination interval w2 by increments of w3. For example, the analysis unit 150 calculates the average pupil diameter for the analysis time width w1 by increments of w3 at time t i Mean pupil diameter in - p i is calculated using the following formula:

[0038]

[0039] The analysis unit 150 calculates L (L=(w1-w2) / w3) average pupil diameters. - p i Calculate the average pupil diameter of L pupils. - p i and the vector T of indexes of the time when they were acquired, and using equations (2-1) to (2-3), β = (β0, β1, ..., β n ) is found.

[0040]

[0041] Furthermore, the analysis unit 150 calculates the calculated β=(β0, β1, ..., β n ) and the index m of the current time t0, the predicted value p of the pupil diameter is calculated by the following formula: pred Calculate p pred =P(m)=β0+β1m+β2m 2 +…+β n m n (3) The analysis unit 150 calculates the pupil diameter p at the current time t0 using the following formula: current Ask for.

[0042]

[0043] Next, the analysis unit 150 calculates the predicted value p pred and the pupil diameter p at the current time t0 current Compare with p current >p pred If so, the change in pupil diameter is on the rise, and p current <ppred If so, it is determined that there is a downward trend, and an upward or downward trend is output as the statistical trend of the time-series change in pupil diameter.

[0044] [Estimation unit 160] In step S150, the estimation unit 160 receives the statistical trend e of the time-series change in pupil diameter and the area information d as input, and uses these values ​​to estimate the direction in which the subject is paying attention, and outputs the estimation result f.

[0045] Specifically, if the statistical trend of the time-series change in pupil diameter is an upward trend, the direction determined from relatively dark areas of the moving image presented by the visual information presentation unit 110, i.e., areas with small area information (e.g., low brightness), is estimated as the direction of auditory attention. On the other hand, if the statistical trend of the time-series change in pupil diameter is a downward trend, the direction determined from relatively bright areas of the moving image presented by the visual information presentation unit 110, i.e., areas with large area information (e.g., high brightness), is estimated as the direction of auditory attention. The reason for this estimation is that when auditory attention is directed toward relatively bright areas, the change in pupil size tends to downward, and when auditory attention is directed toward relatively dark areas, the change in pupil size tends to upward.

[0046] 3 and 7 show examples of estimation results, and FIG. 4 shows an example of estimation accuracy.

[0047] <Effects> With the above configuration, it is possible to estimate the direction of auditory attention of a subject in real time with high accuracy by using the tendency of time-series changes in biological information.

[0048] [Modification] For example, the present invention can be expanded to estimate which of three or more different objects the subject is paying attention to, such as other instrument players in addition to the bass and saxophone in Figure 1. In this case, a moving image consisting of three or more regions with different brightness index values ​​may be presented to the subject, three or more pieces of audio data corresponding to the moving image may be played back, and different sounds may be presented to the subject from three or more different directions, and it may be estimated which of the sounds presented from the three or more different directions the subject is paying auditory attention to. In this case, the statistical trend of the time-series change in pupil diameter may be divided into multiple levels (e.g., large upward trend, small upward trend, small downward trend, large downward trend). For example, multiple thresholds may be set, and the predicted value p pred and the pupil diameter p at the current time t0 current Difference p current -p pred θ is calculated, and based on the magnitude relationship between the difference and the threshold, it is estimated which sound to direct auditory attention to. For example, a threshold θ>0 is set, and θ <p current -p pred When , the trend is large and increases. <p current -p pred <θ, there is a small upward trend, -θ <p current -p pred When <0, it is assumed to have a small downward trend, and p current -p pred When <-θ, there is a large downward trend.

[0049] Alternatively, a model (classifier) ​​may be trained using training biometric information c and correct answer data, which inputs the subject's biometric information c and outputs a statistical trend e of time-series changes in pupil diameter from the biometric information c. The analysis unit 150 may then use this model to calculate the statistical trend e of time-series changes in pupil diameter from the subject's biometric information. The correct answer data may be obtained by pressing a button corresponding to a sound to which the subject is paying attention. By providing three or more sounds and buttons to which the subject is paying attention, it is possible to estimate which sound the subject will pay auditory attention to, presented from three or more different directions.

[0050] The estimated attention state may be fed back to the subject in real time, allowing the subject to determine their next action based on their own awareness of their attention state.

[0051] Furthermore, sounds may be converted and presented to the subject depending on the subject's attention state. For example, sounds in the direction of the subject's attention may be emphasized or reduced. This configuration also makes it possible to manipulate the subject's attention state.

[0052] Second Embodiment The following description will focus on the differences from the first embodiment.

[0053] In this embodiment, the auditory attention direction is estimated by incorporating not only the statistical tendency of the time-series change in pupil diameter but also other indices.

[0054] Other indices that can be used include gaze movement, minute gaze movement, microsaccades, etc. In this embodiment, gaze movement is used.

[0055] The estimation device of the second embodiment estimates the direction in which a subject's auditory attention is directed.

[0056] The estimation device 200 will be described below with reference to Figures 8 and 9. Figure 8 is a block diagram showing the configuration of the estimation device 200. Figure 9 is a flowchart showing the operation of the estimation device 200. As shown in Figure 5, the estimation device 100 includes a visual information presenting unit 110, an acoustic information presenting unit 120, a biometric information acquiring unit 130, a region information acquiring unit 140, an analyzing unit 150, an estimation unit 160, a second biometric information acquiring unit 230, a second estimation unit 270, a third estimation unit 280, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for processing.

[0057] The following description will focus on the second biometric information acquisition unit 230, the second estimating unit 270, and the third estimating unit 280, which are different from the first embodiment.

[0058] The operation of the estimation device 200 will be described below with reference to FIG.

[0059] [Second biometric information acquisition unit 230] In step S230, the second biometric information acquisition unit 230 measures the gaze movement of the subject while presenting images with different luminance, acquires second biometric information g that is time-series information on the measured gaze movement, and outputs the second biometric information g to the second estimation unit 270. Various conventional techniques can be used as a method for measuring the gaze movement.

[0060] [Second Estimation Unit 270] In step S270, the second estimation unit 270 receives the second biometric information g as input, estimates the direction in which the subject is paying attention using the second biometric information g, and outputs a second estimation result h. The second estimation unit 270 estimates that the direction of the gaze point determined from the movement of the gaze is the direction in which the subject is paying auditory attention. Figure 10 shows the estimation result and estimation accuracy of the second estimation unit 270. The estimation accuracy of the second estimation result h was approximately 60-70%.

[0061] [Third Estimation Unit 280] In step S270, the second estimation unit 270 receives the estimation result f and the second estimation result h as input, and uses these values ​​to determine and output the third estimation result p, which is the final estimation result. For example, if the estimation result f and the second estimation result h match, the matching estimation result is designated as the third estimation result p. If the estimation result f and the second estimation result h do not match, the third estimation result p is designated as a value indicating that an appropriate estimation was not performed. Figure 11 shows the estimation results and estimation accuracy of the third estimation unit 280. The estimation accuracy of the time when an appropriate estimation was performed was approximately 70-90%.

[0062] <Effects> With this configuration, it is possible to obtain the same effects as in the first embodiment, and further to improve the estimation accuracy.

[0063] [Modification] In this embodiment, eye movement is used as an index other than the tendency of change in pupil diameter, but other indices may be used. Also, three or more indices including the tendency of change in pupil diameter may be used. When three or more indices are used, the final estimation result may be determined, for example, by majority vote of the estimation results.

[0064] The system may also have a device (terminal) for using the device of the present invention or the method of the present invention via a network (telecommunications line). The "device (terminal) for use" may be equipped with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device of the present invention or the method of the present invention. Note that a configuration including a device (terminal) for using the device of the present invention or the method of the present invention via a network (telecommunications line) is also referred to as an estimation system.

[0065] [Processor, Program, Recording Medium] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0066] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0067] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0068] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 12, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0069] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0070] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0071] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0072] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. An estimation device comprising: a memory unit that stores biometric information, which is time-series information on pupil diameter obtained while presenting images of different brightness to a subject; an area information acquisition unit that acquires area information, which is an index value of the brightness of an area of ​​the image presented to the subject; an analysis unit that calculates the statistical tendency of time-series changes in pupil diameter from the biometric information of the subject; and an estimation unit that estimates the direction in which the subject is paying attention using the statistical tendency of time-series changes in pupil diameter and the area information.

2. An estimation device according to claim 1, wherein the memory unit stores second biometric information which is time-series information on eye movement acquired while presenting images of different brightness to the subject, and the estimation device further includes: a second estimation unit which estimates the direction in which the subject is paying attention using the time-series information on eye movement; and a third estimation unit which uses the estimation results of the estimation unit and the second estimation unit to obtain a final estimation result.

3. The estimation device according to claim 1, wherein the analysis unit uses the biological information to determine the average pupil diameter for the analysis time width while moving the discrimination interval by a predetermined interval, and calculates one or more average pupil diameters. - p i and a predetermined coefficient is calculated from the time when the data was acquired, a predicted value of the pupil diameter is calculated from the predetermined coefficient and the current time, and the predicted value is compared with the current pupil diameter to calculate the tendency of change in the pupil diameter.

4. The estimation device according to claim 1, wherein the analysis time width is w1, the average pupil diameter discrimination interval is w2, the movement width is w3, and the pupil diameter at time t is expressed as pupil t Let the current time be t0, and let i be the time t i where i=1, 2, ..., L, L=(w1-w2) / w3, and the analysis unit calculates the time t i Mean pupil diameter in - p i is calculated using the following formula: L average pupil diameters - p i and the vector T of indices of the time when they were acquired, β = (β0, β1, ..., β n ) and Using the β and the index m of the current time t0, the predicted value p of the pupil diameter is calculated. pred =β0+β1m+β2m 2 +…+β n m n and if the current pupil diameter is larger than the predicted value, determining that the change in pupil diameter is on an upward trend, and if the current pupil diameter is smaller than the predicted value, determining that the change in pupil diameter is on a downward trend.

5. An estimation method comprising: a storage step in which biometric information, which is time-series information on pupil diameter obtained while presenting images of different brightness to a subject, is stored in a storage unit; an area information acquisition step in which area information, which is an index value of the brightness of an area of ​​the image presented to the subject, is acquired; an analysis step in which the statistical trend of time-series changes in pupil diameter from the biometric information of the subject; and an estimation step in which the statistical trend of time-series changes in pupil diameter and the area information are used to estimate the direction in which the subject is paying attention.

6. A program for causing a computer to function as the estimation device of claim 1.

7. An estimation system comprising: the estimation device of claim 1; and a device for using the estimation device via a network.

Images