Auditory attention state estimation device, learning device, their methods, and programs

Abstract

The present invention: obtains a feature amount based on the strength of the correlation between the amount of change in user pupil diameter and each of mutually different multiple visual stimulation patterns that correspond to mutually different multiple sound sources; and estimates, using the feature amount, a point to which a user's auditory attention is directed regarding a sound from a sound source.

Description

[Technical field]

[0001] The present invention relates to a technique for estimating an auditory attention state. [Background technology]

[0002] Humans consciously or unconsciously select the information they want to perceive by selectively directing their attention to specific input stimuli for the senses, such as vision, hearing, and touch. In Non-Patent Document 1, the pupil diameter contraction / dilation (pupil oscillation, or PFT (Pupillary Frequency Tagging)) induced by turning a light source on / off is used to estimate where the user has directed their visual attention. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Naber, M., Alvarez, GA, and Nakayama, K., "Tracking the allocation of attention using human pupillary oscillations," [online], December 10, 2013, Frontiers in Psychology, 4., [Retrieved July 15, 2021], Internet<http: / / doi.org / 10.3389 / fpsyg.2013.00919> Summary of the Invention [Problem to be solved by the invention]

[0004] However, the relationship between where a user focuses their auditory attention and changes in pupil diameter is unknown, and there is no known method for estimating where a user focuses their auditory attention based on changes in pupil diameter.

[0005] The present invention provides a method for estimating a user's auditory attention focus based on changes in pupil diameter. [Means for solving the problem]

[0006] A feature is obtained based on the strength of correlation between each of a plurality of different visual stimulation patterns corresponding to a plurality of different sound sources and the change in pupil diameter of the user, and the feature is used to estimate where the user's auditory attention is directed in relation to the sound from the sound source. Effect of the Invention

[0007] In the present invention, it is possible to estimate where a user is directing their auditory attention based on changes in pupil diameter. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of an auditory attention state estimating system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating the mechanical configuration of the learning device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating a mechanical configuration of the auditory attention state estimating device according to the embodiment. [Figure 4] FIG. 4 is a conceptual diagram for explaining the contents of the experiment. [Diagram 5] FIG. 5 is a graph illustrating the experimental results. [Figure 6] FIG. 6 is a block diagram illustrating the hardware configuration of the learning device and the auditory attention state estimating device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [principle] First, the principle will be explained. The present invention is based on the discovery of a new natural law (physiological law) that a human will show a pupil response even if he or she is merely directing auditory attention to a sound from a sound source, without gazing at a visual stimulus corresponding to the sound source. First, the experimental results that led to this discovery will be presented.

[0010] <Experiment details> As illustrated in FIG. 4, in this experiment, four sound sources (speakers in this experiment) 140-1, ..., 140-4 were arranged, and a light source 130-i (LED light source in this experiment) was installed in each sound source 140-i (where i=1, ..., 4). The light sources 130-1, ..., 130-4 blink (ON / OFF) at different frequencies (blinking frequencies), thereby presenting visual stimuli with different visual stimulation patterns. In this experiment, the blinking frequencies of the light sources 130-1, 130-2, 130-3, and 130-4 were set to 1.50 Hz, 1.75 Hz, 2.00 Hz, and 2.25 Hz, respectively. Also, different sounds were simultaneously emitted from the four sound sources 140-1, ..., 140-4. The subject 100 performed a task of auditorily paying attention to a sound emitted from any of the sound sources 140-i. For example, sounds including a group of words based on a plurality of categories were randomly emitted from the sound sources 140-1, ..., 140-4, and the subject 100 performed a task of counting the number of words of a specified category in the sound emitted from any one of the sound sources 140-i. For example, the following sounds were simultaneously output from the four sound sources 140-1, ..., 140-4, and the subject 100 performed a task of answering the numbers (3, 8, 9, 6) or colors (white, pink, red, black) included in the sound emitted from the sound source 140-1. In this case, the subject 100 will direct his / her auditory attention to the sound emitted from the sound source 140-1. Audio Source 140-1: "Then the bears please go to the white 3." "Then the tigers please go to the pink 8." "Then the cats please go to the red 9." "Then the dog please go to the black 6." Audio 140-2: "Then, please move on to Rainbow 2" 4 times Audio 140-3: "Now, Tiger, please go to blue number 9" four times Audio 140-4: "Then the pigs go to pink number 7" four times The subject 100 sequentially performed a task of directing auditory attention to sounds emitted from each sound source 140-1, 2, 3, 4, and the pupil diameter of the subject 100 performing the task was measured by a pupil diameter acquisition device 150 (an eye tracker in this experiment). The task was performed and the pupil diameter was measured multiple times for multiple subjects 100, and the pupil diameter was measured when each subject 100 was directing auditory attention to the sound emitted from each sound source 140-i.

[0011] <Experimental Results> FIG. 5 illustrates a signal (frequency domain pupil diameter change signal) obtained by converting a time series signal representing the amount of change in pupil diameter of the subject 100 performing the above-mentioned task into the frequency domain. The horizontal axis of FIG. 5 represents frequency (Hz), and the vertical axis represents the power [au] of the frequency domain pupil diameter change signal. In FIG. 5, the difference between the sound sources 140-i (where i=1,...,4) to which the subject visually pays attention (i.e., the difference between the blinking frequencies of the light sources 130-i corresponding to the sound sources 140-i) is distinguished by the type of line. The thick line represents the average value of the power, and the thin line represents the distribution of the power.

[0012] 5, when subject 100 focuses his / her auditory attention on a sound emitted from sound source 140-i, the power peak of the frequency domain pupil diameter change signal based on the pupil diameter change of subject 100 is at or near the blinking frequency of light source 130-i provided in sound source 140-i. For example, when subject 100 focuses his / her auditory attention on a sound emitted from sound source 140-1 provided with light source 130-1 that blinks at a blinking frequency of 1.50 [Hz], the power peak p1 of the frequency domain pupil diameter change signal based on the pupil diameter change of subject 100 appears at or near 1.50 [Hz]. When subject 100 focuses his / her auditory attention on a sound emitted from sound source 140-2 provided with light source 130-2 that blinks at a blink frequency of 1.75 [Hz], a power peak p2 of the frequency domain pupil diameter change signal based on the amount of change in pupil diameter of subject 100 appears at or near 1.75 [Hz]. When subject 100 focuses his / her auditory attention on a sound emitted from sound source 140-3 provided with light source 130-3 that blinks at a blink frequency of 2.00 [Hz], a power peak p3 of the frequency domain pupil diameter change signal based on the amount of change in pupil diameter of subject 100 appears at or near 2.00 [Hz]. When the subject 100 directs his / her auditory attention to the sound emitted from the sound source 140-4 provided with the light source 130-4 that blinks at a blink frequency of 2.25 [Hz], the power peak p4 of the frequency domain pupil diameter change signal based on the pupil diameter change of the subject 100 appears at or near 2.25 [Hz]. That is, there is a correlation between the pupil diameter change when the subject 100 directs his / her auditory attention to the sound emitted from the sound source 140-i and the visual stimulation pattern emitted from the light source 130-1 corresponding to the sound source 140-i. Here, the subject 100 is not instructed to gaze at the light source 130-1 during the execution of the task. Therefore, it is expected that this correlation holds even if the subject 100 is not gazing at the light source 130-1. In addition, the visual stimulation pattern corresponding to the sound source is a time-varying pattern of visual stimulation, and may be, for example, a periodic time-varying pattern of visual stimulation, or a non-periodically time-varying pattern of visual stimulation.The visual stimulus pattern may be a time-varying luminance (brightness) pattern, a time-varying color pattern, a time-varying pattern, or a time-varying shape pattern. The visual stimulus pattern corresponding to the sound source is not limited to one created by a light source, but may be one displayed or projected by a display or projector, or may be one created by the sound source itself or the environment around the sound source (e.g., luminance change).

[0013] As described above, even if a person merely focuses their auditory attention on a sound emitted from a sound source, they will show a pupil response according to the visual stimulation pattern corresponding to the sound source. In each embodiment, this natural law is utilized to obtain a feature based on the strength of correlation between each of a plurality of different visual stimulation patterns corresponding to a plurality of different sound sources and the amount of change in the user's pupil diameter, and the feature is used to estimate where the user has focused their auditory attention with respect to the sound from the sound source.

[0014] [First embodiment] Next, a first embodiment will be described. <Overall composition> 1, the auditory attention state estimation system 1 of this embodiment includes a learning device 11, an auditory attention state estimation device 12, a plurality of visual stimulus generating devices 13-1, ..., 13-N, a plurality of sound source devices 14-1, ..., 14-N, and a pupil diameter acquisition device 15, and estimates where a user 10 has directed his / her auditory attention based on the amount of change in pupil diameter of the user 10. Here, N is an integer of 1 or more, for example, N is an integer of 2 or more.

[0015] <Learning Device 11> 2, the learning device 11 of this embodiment has an input unit 111, a storage unit 112, a learning unit 113, an output unit 114, and a control unit 117. Although explanation will be omitted hereafter, the learning device 11 executes each process under the control of the control unit 117, and input data to the learning device 11 and data obtained in each process are stored in the storage unit 112 and are read out and used as necessary.

[0016] <Auditory Attention State Estimation Device 12> 3, the auditory attention state estimation device 12 of this embodiment has an input unit 121, a storage unit 122, a visual stimulation control unit 123, an auditory information control unit 124, a feature extraction unit 125, an estimation unit 126, and a control unit 127. Although explanations will be omitted hereafter, the auditory attention state estimation device 12 executes each process under the control of the control unit 127, and input data to the auditory attention state estimation device 12 and data obtained by each process are stored in the storage unit 122 and are read out and used as necessary.

[0017] <Sound source device 14-n> The sound source device 14-n (where n=1, ..., N) of this embodiment is a sound source that emits a sound SO (Info-n). An example of the sound source device 14-n is a speaker, but this does not limit the present invention. Any device that can arrange a sound source at a desired spatial position may be used as the sound source device 14-n. The sound source devices 14-1, ..., 14-N are different from each other, and the sound source devices 14-1, ..., 14-N arrange the sound source at a different position from each other. For example, the direction of the sound source devices 14-1, ..., 14-N from the user 10 or the direction of the sound source arranged by the sound source devices 14-1, ..., 14-N are different from each other. The sound source devices 14-1, ..., 14-N may simultaneously emit sounds SO(Info-1), ..., SO(Info-N), or some of the sound source devices 14-1, ..., 14-N may emit sounds SO(Info-n) at different timings from the other sound source devices. However, it is preferable that sounds emitted simultaneously from different sound source devices are different from each other. The sounds SO(Info-1), ..., SO(Info-N) emitted from the sound source devices 14-1, ..., 14-N may be any sounds such as voice, music, environmental sounds, ringtones, alarm sounds, etc.

[0018] <Visual Stimulation Device 13-n> The visual stimulus generating device 13-n (where n=1,...,N) of this embodiment is a device that presents (displays) a visual stimulus pattern VS(Sig-n) corresponding to the sound source device 14-n. As long as the user 10 can perceive the correspondence between the sound source device 14-n and the visual stimulus generating device 13-n, the visual stimulus generating device 13-n may be arranged or configured in any manner. For example, the visual stimulus generating device 13-n may be arranged near the sound source device 14-n, may be arranged in contact with the sound source device 14-n, may be fixed to the sound source device 14-n, or may be configured integrally with the sound source device 14-n. Each of the visual stimulus patterns VS(Sig-n) of this embodiment is a visual stimulus pattern that changes periodically over time. For example, the visual stimulus pattern VS(Sig-n) of the embodiment may be a pattern of luminance (brightness) that changes periodically over time, a pattern of blinking (ON / OFF) that repeats periodically, a color pattern that changes periodically over time, a pattern pattern that changes periodically over time, or a shape pattern that changes periodically over time. That is, the visual stimulus generating device 13-n may present luminance (brightness) that changes periodically over time, light that blinks periodically, a color that changes periodically over time, a pattern that changes periodically over time, or a shape that changes periodically over time. As long as such a visual stimulus pattern VS(Sig-n) can be visually presented, the visual stimulus generating device 13-n may be of any type. For example, the visual stimulus generating device 13-n may be an LED light source, a laser light generator, a display, or a projector. The visual stimulus patterns VS(Sig-1), ..., VS(Sig-N) presented from the N visual stimulus generating devices 13-1, ..., 13-N are different from each other. In this embodiment, the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) presented from the visual stimulation generating devices 13-1, ..., 13-N have different frequency distributions.For example, the peak frequencies (peak frequencies) of the signals (frequency domain visual stimulation signals) obtained by transforming (e.g., Fourier transform) time series signals (e.g., luminance time series signals, color time series signals, pattern time series signals, shape time series signals, etc.) representing the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) presented from the visual stimulation generating devices 13-1, ..., 13-N of this embodiment into the frequency domain are different from one another.

[0019] <Pupillary diameter acquisition device 15> The pupil diameter acquisition device 15 of this embodiment is a device that measures the pupil diameter Pub of the user 10. For example, the pupil diameter acquisition device 15 is a device that includes a camera that captures the eye movement of the user 10 and acquires and outputs the pupil diameter Pub of the user 10 from the image captured by the camera. An example of the pupil diameter acquisition device 15 is a commercially available eye tracker or the like.

[0020] <Overall process> 1, learning data T is input to a learning device 11, which obtains an estimated model M(θ) by a learning process using the learning data T and outputs a model parameter θ that specifies the estimated model M(θ). The model parameter θ is input to an auditory attention state estimation device 12. The auditory attention state estimation device 12 outputs output information Info-1, ..., Info-N to sound source devices 14-1, ..., 14-N, respectively, and the sound source devices 14-1, ..., 14-N output sounds SO(Info-1), ..., SO(Info-N) based on the output information Info-1, ..., Info-N, respectively. The auditory attention state estimation device 12 outputs the output information Sig-1, ..., Sig-N to the visual stimulus generating devices 13-1, ..., 13-N, respectively, and the visual stimulus generating devices 13-1, ..., 13-N present visual stimulus patterns VS(Sig-1), ..., VS(Sig-N) based on the output information Sig-1, ..., Sig-N, respectively. The user 10 auditorily pays attention to the sound SO(Info-n) emitted from at least one of the sound source devices 14-n, and the pupil diameter acquisition device 15 acquires the pupil diameter Pub of the user 10 and sends it to the auditory attention state estimation device 12. The auditory attention state estimation device 12 obtains a feature value f based on the strength of correlation between each of the visual stimulus patterns VS(Sig-1), ..., VS(Sig-N) presented based on the output information Sig-1, ..., Sig-N and the pupil diameter change amount of the user 10 obtained based on the pupil diameter Pub. Furthermore, the auditory attention state estimation device 12 uses the feature amount f to obtain and output an estimation result E=M(θ;f) of the destination of the auditory attention of the user 10 with respect to the sound from the sound source, based on an estimation model M(θ) (machine learning model) specified by a model parameter θ. This will be described in detail below.

[0021] <Processing of learning device 11> Learning data T is input to an input unit 111 of the learning device 11 (FIG. 2), and the learning data T is stored in a storage unit 112. The learning data T is a training feature Tf based on the strength of correlation between each of N (plural) different training visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) corresponding to N (plural) different training sound sources and the training pupil diameter change TVPub.j and correct answer information Ta, which indicates where the auditory attention was directed for the sound from the learning sound source (learning sound). j The data (Tf1, Ta1),..., (Tf J , Ta J ), where J is a positive integer, j=1,...,J.

[0022] The learning sound may be any sound, such as voice, music, environmental sound, a call sound, an alarm sound, etc. Specific examples of the learning sound are the same as the sounds emitted from the sound source device 14-n described above.

[0023] A specific example of the N learning sound sources is the same as the above-mentioned sound source devices 14-1, ..., 14-N. However, the N learning sound sources may be any devices that generate learning sounds, and the N learning sound sources may be devices different from the above-mentioned sound source devices 14-1, ..., 14-N.

[0024] The N learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) are respectively the same as the above-mentioned visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), and are patterns of visual stimulation that change periodically over time. For example, the N learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) are respectively the same as the patterns of visual stimulation presented from the above-mentioned visual stimulation generating devices 13-1, ..., 13-N, and specific examples of the learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) are respectively the same as the specific examples of the above-mentioned visual stimulation patterns VS(Sig-1), ..., VS(Sig-N).

[0025] The learning pupil diameter change TVPub is the change in pupil diameter when a learning user, to whom learning sounds and learning visual stimulus patterns TVS(Sig-1), ..., TVS(Sig-N) are presented, experiences any of the learning sounds TSO n (where n ∈ {1,...,N}) is the change in pupil diameter of the training user when the training user is auditorily paying attention to one of the training sounds TSO nThe time when the learning user is auditorily paying attention to is, for example, when the learning user is performing the above-mentioned task. The learning pupil diameter change amount TVPub is obtained, for example, based on the learning user's pupil diameter TPub measured by the pupil diameter acquisition device 15, but may be obtained based on the learning user's pupil diameter TPub measured by another device. An example of a method for obtaining the learning pupil diameter change amount from the pupil diameter TPub is given below. (1.1) First, the time series data of the pupil diameter TPub is pre-processed to obtain the time series data of the pre-processed pupil diameter TPub'. The pre-processing may be, for example, a process of interpolating missing parts of the time series data of the pupil diameter TPub due to the blinking of the learning user, using linear interpolation or quadratic spline interpolation. Furthermore, a low-pass filter (for example, a low-pass filter that passes a band including all blinking frequencies) according to the blinking frequencies of the learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) may be applied to the time series data of the pupil diameter TPub after the interpolation to reduce noise. (1.2) Next, the user selects one of the training sounds TSO n From the time series data of pupil diameter TPub' when auditory attention is directed to the subject, the average pupil diameter TPub' before auditory attention is directed to the subject is subtracted, and standardization is performed using the z value to obtain the time series data of the learning pupil diameter change TVPub.

[0026] Learning feature Tf j can be any value as long as it is based on the strength of correlation between each of the N training visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) and the training pupil diameter change amount TVPub. j Here is an example:

[0027] (2.1) A training feature Tf representing the peak frequency of each of the multiple training frequency domain visual stimulus signals TFVS(Sig-1), ..., TFVS(Sig-N) and / or the magnitude of the training frequency domain pupil diameter change signal TFVPub in the vicinity of the peak frequency (e.g., the peak value of the magnitude of TFVPub) jmay be used. Here, each of the multiple learning frequency domain visual stimulation signals TFVS(Sig-1), ..., TFVS(Sig-N) is a signal obtained by converting (e.g., Fourier transforming) a time series signal representing each of the multiple learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) into the frequency domain. Also, the learning frequency domain pupil diameter change signal TFVPub is a signal obtained by converting a time series signal representing the learning pupil diameter change TVPub into the frequency domain. Also, the "magnitude of α" may be the absolute value of the amplitude of α, the power of α (the square of the amplitude of α), or any other value that monotonically increases with respect to the absolute value of α. For example, in the case of the example of FIG. 5, a learning feature Tf j may also be used.

[0028] Here, the larger the magnitude of the learning frequency domain pupil diameter change signal TFVPub at the peak frequency and / or in the vicinity of the peak frequency (e.g., the peak value of the magnitude of TFVPub), the stronger the correlation between the learning visual stimulation pattern TVS(Sig-n) corresponding to the peak frequency and the learning pupil diameter change TVPub.

[0029] (2.2) In addition to or instead of the magnitude of the training frequency domain pupil diameter change signal TFVPub at the peak frequency and / or in the vicinity of the peak frequency, other information may be added to the training feature Tf jFor example, (1) the magnitude (e.g., peak value of the magnitude) of the learning frequency domain pupil diameter change signal TFVPub at or near a double frequency of the peak frequency of each of the multiple learning frequency domain visual stimulation signals TFVS(Sig-1), ..., TFVS(Sig-N), (2) the synchronicity between the phase change of each of the N learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) and the phase change of the learning pupil diameter change TVPub, and (3) the maximum values ​​CCFmax(TSS(Sig-1), TPS), ..., CCFmax(TSS(Sig-N), TPS) of the cross-correlation function between each of the series TSS(Sig-1), ..., TSS(Sig-N) corresponding to the time series signals representing each of the N learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) and the series TPS corresponding to the time series signal representing the learning pupil diameter change TVPub, may be included in the learning feature Tf j Here, the "series corresponding to the time series signal" may be, for example, the time series signal itself, a series obtained by converting the time series signal into the frequency domain, or a series of function values ​​of other time series signals. Moreover, the maximum value of the cross-correlation function between the series α1 and the series α2 means the maximum value of the cross-correlation function values ​​between the series α1 and the series α2 for the variable delay amount τ.

[0030] Here, the larger the magnitude of the learning frequency domain pupil diameter change signal TFVPub at and / or near the double frequency of the peak frequency (e.g., the peak value of the magnitude of TFVPub), the stronger the correlation between the learning visual stimulation pattern TVS(Sig-n) corresponding to the peak frequency and the learning pupil diameter change amount TVPub. Also, the learning visual stimulation pattern TVS(Sig-n) having a phase change with a high degree of synchronization with the learning pupil diameter change amount TVPub has a stronger correlation with the learning pupil diameter change amount TVPub. Also, the larger the maximum value of the cross-correlation function CCFmax(TSS(Sig-n), TPS), the stronger the correlation with the learning pupil diameter change amount TVPub.

[0031] As mentioned above, the learning pupil diameter change TVPub is a learning sound TSO n The change in pupil diameter of the learning user when the learning user is auditorily paying attention to the correct answer information Ta j The learning sound TSO n That is, the correct answer information Ta j is a learning sound TSO corresponding to the learning pupil diameter change TVPub n For example, the correct answer information Ta j is a sound source (e.g., learning sound TSO n The information may be information (e.g., an index) representing a device that generates the learning sound TSO n The training visual stimulus pattern TVS(Sig-n) corresponding to the training sound TSO may be information (e.g., a blinking frequency) indicating the training visual stimulus pattern TVS(Sig-n) corresponding to the training sound TSO n As described above, the learning feature Tf j In order to correspond to the learning pupil diameter change TVPub, the correct answer information Ta j is the learning feature Tf j (Step S111).

[0032] The learning unit 113 obtains an estimated model M(θ) by a learning process (machine learning) using the learning data T read from the storage unit 112, and outputs a model parameter θ that specifies the estimated model M(θ). The estimated model M(θ) is a model for estimating the direction of the user's auditory attention (visual attention direction) with respect to a sound from a sound source, using as input a feature value f based on the strength of correlation between each of N (multiple) different visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) corresponding to N (multiple) different sound sources and the pupil diameter change amount VPub of the user. The configuration of the feature f is that the learning sound is replaced by sound, the learning sound source is replaced by the sound source, the learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) are replaced by visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), the learning user is replaced by the user, and the learning pupil diameter change amount TVPub of the learning user is replaced by the pupil diameter change amount VPub of the user, except that j The configuration is the same as that of

[0033] The estimation model M(θ) is configured to estimate the sound emitted from the sound source corresponding to the visual stimulation pattern VS(Sig-n) that has a high correlation with the pupil diameter change amount VPub as the target of the user's auditory attention. The "target of the user's auditory attention" estimated using such an estimation model M(θ) is, for example, at least one of the following (3.1), (3.2), and (3.3).

[0034] (3.1) The higher the correlation with the pupil diameter change VPub, the higher the frequency (probability) of a sound emitted from a sound source corresponding to the visual stimulation pattern VS(Sig-n) that is estimated to be the target of the user's auditory attention.

[0035] (3.2) When a visual stimulation pattern VS(Sig-n) (first visual stimulation pattern) included in the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) corresponds to a sound source AS1 (first sound source) and the strength of correlation between the visual stimulation pattern VS(Sig-n) and the user's pupil diameter change VPub is equal to or greater than a predetermined value, it is estimated that the user's auditory attention is directed to the sound source AS1 or near the sound source AS1.

[0036] (3.3) When the target N (multiple) sound sources include a sound source AS1 (first sound source) and a sound source AS2 (second sound source), the visual stimulation pattern VS(Sig-n1) (first visual stimulation pattern) included in the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) corresponds to the sound source AS1, the visual stimulation pattern VS(Sig-n2) (second visual stimulation pattern) corresponds to the sound source AS2, and the strength of correlation between the visual stimulation pattern VS(Sig-n1) and the pupil diameter change amount VPub is stronger than the strength of correlation between the visual stimulation pattern VS(Sig-n2) and the pupil diameter change amount VPub, it is estimated that the user's auditory attention is directed to the sound source AS1 or the vicinity of the sound source AS1. Here, n1, n2∈{1, ..., N}.

[0037] The "user's auditory attention" estimated using the estimation model M(θ) may be information (e.g., an index) representing a sound source (e.g., a device that emits sound) or may be information (e.g., an index) representing the sound source. n The information may be a visual stimulus pattern VS(Sig-n) corresponding to the sound SO n or may be information representing the direction of these. In addition, the estimation model M(θ) may estimate one "destination where the user auditorily directs attention", may estimate multiple "destinations where the user auditorily directs attention", or may estimate the probability of "destination where the user auditorily directs attention".

[0038] The estimation model M(θ) may be based on any method. For example, the estimation model M(θ) may be based on a k-nearest neighbor algorithm (K-NN), a support vector machine (SVM), deep learning, a hidden Markov model, or the like. A known method according to the method of the estimation model M(θ) may be used as a specific method of the learning process. In general, an initial value of an appropriate provisional model parameter θ' is first set, and then the learning feature Tf is added to the estimation model M(θ'). j The results obtained by applying j The process of updating the provisional model parameter θ′ is repeated so as to reduce the error with respect to the provisional model parameter θ′, and the provisional model parameter θ′ at the point in time when a predetermined termination condition is satisfied is set as the model parameter θ.

[0039] The model parameter θ output from the learning unit 113 is sent to the output unit 114, and the output unit 114 sends the model parameter θ to the auditory attention state estimation device 12 (step S113).

[0040] <Processing of the auditory attention state estimation device 12> The model parameter θ sent from the learning device 11 is input to the input unit 121 of the auditory attention state estimation device 12 (FIG. 3) and stored in the memory unit 122. The memory unit 122 also stores output information Sig-1, ..., Sig-N representing visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), and also stores output information Info-1, ..., Info-N representing sounds SO(Info-1), ..., SO(Info-N) (step S122).

[0041] The auditory information control unit 124 reads the output information Info-1, ..., Info-N from the storage unit 122 and sends it to the sound source devices 14-1, ..., 14-N, respectively. Each sound source device 14-n presents (outputs) a sound SO(Info-n) based on the sent output information Info-n (step S124).

[0042] The visual stimulus control unit 123 reads the output information Sig-1, ..., Sig-N from the storage unit 122 and sends it to the visual stimulus generation devices 13-1, ..., 13-N, respectively. Each visual stimulus generation device 13-n (n=1, ..., N) presents (outputs) a visual stimulus pattern VS(Sig-n) based on the sent output information Sig-n (step S123).

[0043] The user 10, who is presented with sounds SO(Info-1), ..., SO(Info-N) and visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), auditorily directs his / her attention to one of the sounds SO(Info-n). For example, the user 10 auditorily directs his / her attention to one of the sounds SO(Info-n) by performing the above-mentioned task. The pupil diameter acquisition device 15 measures the pupil diameter Pub of the user 10 and sends the time series data of the pupil diameter Pub to the feature extraction unit 125.

[0044] The feature extraction unit 125 further extracts output information Sig-1, ..., Sig-N corresponding to the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) from the storage unit 122. Using the time-series data of the pupil diameter Pub and the output information Sig-1, ..., Sig-N, the feature extraction unit 125 obtains and outputs a feature f based on the strength of correlation between each of the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the amount of change in pupil diameter VPub of the user 10. The configuration of the feature f is the same as that of the above-mentioned learning feature Tf except that the learning sound is replaced with a sound, the learning sound source is replaced with a sound source, the learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) are replaced with visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), the learning user is replaced with user 10, and the learning pupil diameter change amount TVPub of the learning user is replaced with the pupil diameter change amount VPub of user 10. j For example, the feature extraction unit 125 obtains the feature f as follows.

[0045] (4.1) The feature extraction unit 125 performs preprocessing on the time-series data of the pupil diameter Pub to obtain the time-series data of the pupil diameter Pub' after preprocessing. As an example of the preprocessing, a process of interpolating missing parts of the time-series data of the pupil diameter Pub due to blinking of the user 10, etc., using linear interpolation or quadratic spline interpolation, etc., can be used. Furthermore, a low-pass filter (e.g., a low-pass filter that passes a band including all blinking frequencies) according to the blinking frequencies of the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) may be applied to the time-series data of the pupil diameter Pub after interpolation to reduce noise.

[0046] (4.2) The feature extraction unit 125 extracts the feature of the training sound TSO n From the time series data of pupil diameter Pub' when auditory attention is directed to the subject, the average pupil diameter Pub' before auditory attention is directed to the subject is subtracted, and standardization is performed using the z-score to obtain time series data of pupil diameter change VPub.

[0047] (4.3) The feature extraction unit 125 obtains a feature f based on the time series data of the pupil diameter change VPub and the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N). The feature f is based on the strength of correlation between each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the pupil diameter change VPub. An example of the feature f is shown below.

[0048] (4.3.1) A feature f may be used that represents the peak frequency of each of the multiple frequency domain visual stimulation signals FVS(Sig-1), ..., FVS(Sig-N) and / or the magnitude of the frequency domain pupil diameter change signal FVPub in the vicinity of the peak frequency (e.g., the peak value of the magnitude of FVPub). Here, each of the multiple frequency domain visual stimulation signals FVS(Sig-1), ..., FVS(Sig-N) is a signal obtained by converting a time series signal representing each of the multiple visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) into the frequency domain. Also, the frequency domain pupil diameter change signal FVPub is a signal obtained by converting a time series signal representing the pupil diameter change VPub into the frequency domain.

[0049] (4.3.2) In addition to or instead of the peak frequency and / or the magnitude of the frequency domain pupil diameter change signal FVPub in the vicinity of the peak frequency, other information may be included in the feature f. For example, the feature f may include (1) the magnitude (e.g., peak value) of the frequency domain pupil diameter change signal FVPub at or near a multiple of the peak frequency of each of the multiple frequency domain visual stimulation signals FVS(Sig-1), ..., FVS(Sig-N), (2) the synchronicity between the phase change of each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the phase change of the pupil diameter change VPub, and (3) the maximum values ​​CCFmax(SS(Sig-1), PS), ..., CCFmax(SS(Sig-N), PS) of the cross-correlation function between each of the series SS(Sig-1), ..., SS(Sig-N) corresponding to the time series signals representing each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the series PS corresponding to the time series signal representing the pupil diameter change VPub.

[0050] Here, the greater the magnitude of the frequency domain pupil diameter change signal FVPub at and / or near the double frequency of the peak frequency (e.g., the peak value of the magnitude of FVPub), the stronger the correlation between the visual stimulation pattern VS(Sig-n) corresponding to the peak frequency and the pupil diameter change VPub. Also, the greater the phase change with which the degree of synchronization with the pupil diameter change VPub is, the stronger the correlation with the pupil diameter change VPub. Also, the greater the maximum value CCFmax(SS(Sig-n), PS) of the cross-correlation function is, the stronger the correlation with the pupil diameter change VPub. The feature value f is sent to the estimation unit 126 (step S125).

[0051] The estimation unit 126 reads the model parameter θ from the storage unit 122. The estimation unit 126 uses the feature f to obtain and output an estimation result E=M(θ;f) of a sound to which the user 10 auditorily focuses his / her attention, based on the estimation model M(θ) specified by the model parameter θ. That is, the estimation unit 126 applies the feature f to the estimation model M(θ) specified by the model parameter θ, and obtains and outputs an estimation result E corresponding to the feature f. As described above, the estimation result E may be information (e.g., an index) representing a sound source (e.g., sound source device 14-n), or may be information representing the sound source SO n The information may be a visual stimulus pattern VS(Sig-n) corresponding to the sound SO n or may be information representing the direction of these. In addition, the estimation result E may represent one "destination where the user auditorily directs his / her attention", may represent multiple "destinations where the user auditorily directs his / her attention", or may represent the probability of "destination where the user auditorily directs his / her attention" (step S126).

[0052] [Second embodiment] The visual stimulation pattern VS(Sig-n) in the first embodiment is a visual stimulation pattern that changes periodically over time. However, the visual stimulation pattern VS(Sig-n) may be a stimulation pattern that changes non-periodically over time. The following description will focus on the differences from the first embodiment, and the same reference numbers will be used to omit or simplify the description of matters common to the first embodiment.

[0053] <Overall composition> As illustrated in Figure 1, the auditory attention state estimation system 2 of this embodiment has a learning device 21, an auditory attention state estimation device 22, a plurality of visual stimulus generating devices 13-1, ..., 13-N, a plurality of sound source devices 14-1, ..., 14-N, and a pupil diameter acquisition device 15, and estimates where a user 10 is directing his / her auditory attention based on the amount of change in the pupil diameter of the user 10.

[0054] <Learning Device 21> As illustrated in FIG. 2, the learning device 21 of this embodiment includes an input unit 111, a storage unit 112, a learning unit 213, an output unit 114, and a control unit 117.

[0055] <Auditory Attention State Estimation Device 22> As illustrated in FIG. 3, the auditory attention state estimating device 22 of this embodiment has an input unit 121, a memory unit 122, a visual stimulation control unit 223, an auditory information control unit 124, a feature extraction unit 225, an estimation unit 226, and a control unit 127.

[0056] <Processing of learning device 21> The input unit 111 of the learning device 21 (FIG. 2) receives learning data T={(Tf1, Ta1),..., (Tf J , Ta J )} is input, and the learning data T is stored in the storage unit 112. The difference from the first embodiment is that the learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) of this embodiment are different stimulation patterns that change non-periodically over time, and the learning feature quantity Tf j This is based on the strength of correlation between each of the learning visual stimulus patterns TVS(Sig-1), ..., TVS(Sig-N) and the learning pupil diameter change amount TVPub.

[0057] Although there is no limitation to the learning pupil diameter change amount TVPub, for example, the maximum values ​​CCFmax(TSS(Sig-1), TPS), ..., CCFmax(TSS(Sig-N), TPS) of the cross-correlation function between each of the series TSS(Sig-1), ..., TSS(Sig-N) corresponding to the time series signals representing the N learning visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) and the series TPS corresponding to the time series signal representing the learning pupil diameter change amount TVPub are the learning features Tf j In addition to or instead of this, the synchronization degree between the phase change of each of the N training visual stimulation patterns TVS(Sig-1), ..., TVS(Sig-N) and the phase change of the training pupil diameter change amount TVPub is included in the training feature Tf j (Step S211).

[0058] The learning unit 213 obtains an estimated model M(θ) by a learning process (machine learning) using the learning data T read from the storage unit 112, and outputs a model parameter θ that specifies the estimated model M(θ). The only difference between the processing of the learning unit 213 and the first embodiment is the learning data T. The output unit 114 sends the model parameter θ to the auditory attention state estimation device 22 (step S213).

[0059] <Processing of the auditory attention state estimation device 22> The model parameter θ sent from the learning device 11 is input to the input unit 121 of the auditory attention state estimation device 22 (FIG. 3) and stored in the memory unit 122. The memory unit 122 also stores output information Sig-1, ..., Sig-N representing visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), and stores output information Info-1, ..., Info-N representing sounds SO(Info-1), ..., SO(Info-N). Each of the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) is a stimulation pattern that changes non-periodically over time (step S222).

[0060] The process of the auditory information control unit 124 is the same as that in the first embodiment (step S124).

[0061] The visual stimulus control unit 223 reads the output information Sig-1, ..., Sig-N from the storage unit 122 and sends it to the visual stimulus generation devices 13-1, ..., 13-N, respectively. Each visual stimulus generation device 13-n (n=1, ..., N) presents a visual stimulus pattern VS(Sig-n) based on the sent output information Sig-n. However, each of the visual stimulus patterns VS(Sig-1), ..., VS(Sig-N) is a stimulus pattern that changes non-periodically over time (step S223).

[0062] The user 10, who is presented with sounds SO(Info-1), ..., SO(Info-N) and visual stimulation patterns VS(Sig-1), ..., VS(Sig-N), auditorily directs attention to any one of the sounds SO(Info-n). The pupil diameter acquisition device 15 measures the pupil diameter Pub of the user 10 and sends the time-series data of the pupil diameter Pub to the feature extraction unit 225.

[0063] The feature extraction unit 225 uses the time-series data of the pupil diameter Pub and the output information Sig-1, ..., Sig-N extracted from the storage unit 122 to obtain and output a feature f based on the strength of correlation between each of the visual stimulus patterns VS(Sig-1), ..., VS(Sig-N) and the pupil diameter change amount VPub of the user 10. The configuration of the feature f is such that the learning sound is replaced with a sound, the learning sound source is replaced with a sound source, the learning visual stimulus patterns TVS(Sig-1), ..., TVS(Sig-N) are replaced with visual stimulus patterns VS(Sig-1), ..., VS(Sig-N), the learning user is replaced with the user 10, and the learning pupil diameter change amount TVPub of the learning user is replaced with the pupil diameter change amount VPub of the user 10, and the configuration of the feature f is the same as that of the learning feature Tf described above. jThe difference from the first embodiment is that the visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) of this embodiment are different stimulation patterns that change aperiodically over time, and the feature value f of this embodiment is based on the strength of correlation between each of such visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the pupil diameter change amount VPub. For example, the feature extraction unit 225 executes the processes of (4.1) and (4.2) described in the first embodiment, and obtains a feature f in (4.3) including the degree of synchrony between the phase change of each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the phase change of the pupil diameter change amount VPub, and the maximum values ​​CCFmax(SS(Sig-1), PS), ..., CCFmax(SS(Sig-N), PS) of the cross-correlation function between the series SS(Sig-1), ..., SS(Sig-N) corresponding to the time-series signals representing each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the series PS corresponding to the time-series signal representing the pupil diameter change amount VPub. In addition to this, or instead of this, the feature f may include the degree of synchrony between the phase change of each of the N visual stimulation patterns VS(Sig-1), ..., VS(Sig-N) and the phase change of the pupil diameter change amount VPub, and the like. The feature value f is sent to the estimation unit 226 (step S225).

[0064] The estimation unit 226 reads the model parameter θ from the storage unit 122. Using the feature amount f, the estimation unit 226 obtains and outputs an estimation result E=M(θ;f) of a sound to which the user 10 auditorily pays attention, based on an estimation model M(θ) specified by the model parameter θ (step S226).

[0065] [Modification 1 of the first and second embodiments] In the first and second embodiments, the sound source device 14-n is a sound source that emits the n-th sound SO(Info-n). However, the multiple sound source devices 14-1, ..., 14-N emit the n-th (n=1, ..., N) sound SO(Info-n) at different positions γ n In this case, each position γ nis the sound source that emits the n-th sound SO(Info-n), and the visual stimulus generator 13-n is located at the sound source position γ n For example, the visual stimulus generating device 13-n is placed at a position corresponding to the position γ n or position γ n is placed in the vicinity of

[0066] [Third embodiment] In the first and second embodiments and their modified examples, the learning visual stimulation pattern and the visual stimulation pattern are presented from a dedicated device for presenting visual stimulation patterns such as a visual stimulation generating device (hereinafter referred to as a "dedicated visual stimulation device"). However, time-varying images of devices, scenery, machines, plants, animals, etc. other than the dedicated visual stimulation device may be used as the learning visual stimulation pattern and the visual stimulation pattern. In this case, the learning visual stimulation pattern is generated from an image obtained by photographing such images with a camera or the like. In addition, the visual stimulation pattern at the time of estimating the auditory attention state is a pattern that the user 10 directly perceives visually from devices, scenery, machines, plants, animals, etc. other than the dedicated visual stimulation device. In this example, the visual stimulation control units 123, 223 and the visual stimulation generating devices 13-1, ..., 13-N can be omitted.

[0067] In the first and second embodiments and their modified examples, the learning sounds and sounds are presented from a dedicated device such as a sound source device (hereinafter referred to as a "dedicated sound presentation device"). However, sounds emitted from devices, scenery, machines, plants, animals, etc. other than the dedicated sound presentation device may be used. The learning sounds can be generated from audio signals obtained by recording such sounds with a microphone or the like. Furthermore, the sounds presented to the user 10 when estimating the auditory attention state are sounds that the user 10 directly perceives auditorily from devices, scenery, machines, plants, animals, etc. other than the dedicated sound presentation device. In this example, the auditory information control unit 124 and the sound source devices 14-1, ..., 14-N can be omitted.

[0068] The rest is as described in the first and second embodiments.

[0069] [Hardware configuration] The learning devices 11, 21 and the auditory attention state estimation devices 12, 22 in each embodiment are devices configured by a general-purpose or dedicated computer having a processor (hardware processor) such as a CPU (central processing unit) and a memory such as a RAM (random-access memory) and a ROM (read-only memory) executing a predetermined program. That is, the learning devices 11, 21 and the auditory attention state estimation devices 12, 22 in each embodiment have, for example, a processing circuit configured to implement each unit that each of them has. This computer may have one processor and memory, or may have multiple processors and memories. This program may be installed in the computer, or may be recorded in a ROM or the like in advance. In addition, some or all of the processing units may be configured using electronic circuits that realize processing functions independently, rather than electronic circuits that realize functional configurations by reading programs like a CPU. In addition, an electronic circuit that configures one device may include multiple CPUs.

[0070] FIG. 6 is a block diagram illustrating the hardware configuration of the learning device 11, 21 and the auditory attention state estimation device 12, 22 in each embodiment. As illustrated in FIG. 6, the learning device 11, 21 and the auditory attention state estimation device 12, 22 in this example have a CPU (Central Processing Unit) 10a, an input unit 10b, an output unit 10c, a RAM (Random Access Memory) 10d, a ROM (Read Only Memory) 10e, an auxiliary storage device 10f, and a bus 10g. The CPU 10a in this example has a control unit 10aa, a calculation unit 10ab, and a register 10ac, and executes various calculation processes according to various programs loaded into the register 10ac. The input unit 10b is an input terminal to which data is input, a keyboard, a mouse, a touch panel, etc. The output unit 10c is an output terminal to which data is output, a display, a LAN card controlled by the CPU 10a that has loaded a predetermined program, etc. The RAM 10d is a static random access memory (SRAM), a dynamic random access memory (DRAM), or the like, and has a program area 10da in which a predetermined program is stored, and a data area 10db in which various data is stored. The auxiliary storage device 10f is, for example, a hard disk, a magneto-optical disc (MO), a semiconductor memory, or the like, and has a program area 10fa in which a predetermined program is stored, and a data area 10fb in which various data is stored. The bus 10g connects the CPU 10a, the input unit 10b, the output unit 10c, the RAM 10d, the ROM 10e, and the auxiliary storage device 10f so that information can be exchanged. The CPU 10a writes the program stored in the program area 10fa of the auxiliary storage device 10f to the program area 10da of the RAM 10d according to the loaded OS (Operating System) program. Similarly, the CPU 10a writes various data stored in the data area 10fb of the auxiliary storage device 10f to the data area 10db of the RAM 10d. The addresses on the RAM 10d where these programs and data are written are stored in the register 10ac of the CPU 10a.The control unit 10aa of the CPU 10a sequentially reads out these addresses stored in the register 10ac, reads out programs and data from the areas on the RAM 10d indicated by the read addresses, causes the calculation unit 10ab to sequentially execute the calculations indicated by the programs, and stores the calculation results in the register 10ac. With this configuration, the functional configurations of the learning devices 11 and 21 and the auditory attention state estimation devices 12 and 22 are realized.

[0071] The above-mentioned program can be recorded on a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such a recording medium include a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, and the like.

[0072] The distribution of this program is performed, for example, by selling, transferring, lending, etc., a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network. As described above, a computer that executes such a program, for example, first temporarily stores in its own storage device the program recorded in the portable recording medium or the program transferred from the server computer. Then, when executing the process, the computer reads the program stored in its own storage device and executes the process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium and execute the process according to the program, and further, each time the program is transferred from the server computer to the computer, the computer may execute the process according to the received program. Furthermore, the server computer may not transfer the program to the computer, and the above-mentioned process may be executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by issuing an execution instruction and obtaining the result. In addition, the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (such as data that is not a direct instruction to a computer but has the nature of defining computer processing).

[0073] In each embodiment, the present device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0074] The present invention is not limited to the above-mentioned embodiments. For example, in each of the above-mentioned embodiments, an estimation model is obtained by learning, and the estimation model is used to estimate the location where the user's auditory attention is directed. However, as long as a feature based on the strength of correlation between each of a plurality of different visual stimulation patterns corresponding to a plurality of different sound sources and the change in the user's pupil diameter is used, the location where the user's auditory attention is directed may be estimated by any method. For example, a threshold value may be set by sampling a feature value obtained in the past, and the location where the user's auditory attention is directed may be estimated by comparing the threshold value with a newly obtained feature value.

[0075] In addition, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. Needless to say, other appropriate modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0076] 1,2 Auditory attention state estimation system 11,21 Learning Device 113,213 Learning Department 12,22 Auditory attention state estimation device 126,226 Estimation part

Claims

1. a feature extraction unit that obtains a feature based on a strength of correlation between each of a plurality of different visual stimulation patterns emitted from a plurality of different sound source positions and a change in pupil diameter of the user; an estimation unit that estimates a direction to which the user's auditory attention is directed with respect to a sound from the sound source by using the feature amount; An auditory attention state estimation device having the above-mentioned configuration.

2. 2. The auditory attention state estimation device of claim 1, Each of the plurality of visual stimulation patterns is a visual stimulation pattern that changes periodically over time; the feature amount represents at least a peak frequency of each of a plurality of frequency domain visual stimulation signals and / or a magnitude of a frequency domain pupil diameter change signal in the vicinity of the peak frequency; each of the plurality of frequency domain visual stimulation signals is a signal obtained by transforming a time series signal representing each of the plurality of visual stimulation patterns into a frequency domain; The frequency domain pupil diameter change signal is a signal obtained by converting a time series signal representing the pupil diameter change into a frequency domain.

3. 2. The auditory attention state estimation device of claim 1, each of the plurality of visual stimulation patterns is a time-varying pattern of visual stimulation; An auditory attention state estimation device, wherein the feature represents at least the maximum value of a cross-correlation function between a series corresponding to a time series signal representing each of the multiple visual stimulation patterns and a series corresponding to a time series signal representing the amount of change in pupil diameter.

4. 4. The auditory attention state estimation device according to claim 1, An auditory attention state estimation device, in which the estimation unit is more likely to estimate a sound source among the multiple sound sources as the destination of the user's auditory attention, the more strongly the sound source corresponds to a visual stimulation pattern that has a stronger correlation with the change in pupil diameter of the user.

5. 4. The auditory attention state estimation device according to claim 1, the plurality of sound sources includes a first sound source, the plurality of visual stimulus patterns includes a first visual stimulus pattern corresponding to the first sound source; An auditory attention state estimation device, wherein when the strength of correlation between the first visual stimulation pattern and the change in pupil diameter of the user is greater than or equal to a predetermined value, the estimation unit estimates that the user's auditory attention is directed to the first sound source or near the first sound source.

6. 4. The auditory attention state estimation device according to claim 1, the plurality of sound sources include a first sound source and a second sound source other than the first sound source, the plurality of visual stimulation patterns include a first visual stimulation pattern corresponding to the first sound source and a second visual stimulation pattern corresponding to the second sound source; An auditory attention state estimation device, wherein when the strength of correlation between the first visual stimulation pattern and the amount of change in pupil diameter of the user is stronger than the strength of correlation between the second visual stimulation pattern and the amount of change in pupil diameter of the user, the estimation unit estimates that the user's auditory attention is directed to the first sound source or near the first sound source.

7. A learning process is performed using learning data in which a learning feature based on the strength of correlation between each of a plurality of different learning visual stimulation patterns emitted from the positions of a plurality of different learning sound sources and a learning pupil diameter change amount is associated with correct answer information indicating a destination where auditory attention is directed with respect to the learning sound from the learning sound source, A learning device having a learning unit that receives as input a feature based on the strength of correlation between each of a plurality of different visual stimulation patterns emitted from the positions of a plurality of different sound sources and a change in pupil diameter of a user, and obtains an estimation model that estimates where the user's auditory attention is directed regarding sounds from the sound sources.

8. An auditory attention state estimation method using an auditory attention state estimation device, comprising: a feature extraction step in which a feature extraction unit obtains a feature based on a strength of correlation between each of a plurality of different visual stimulation patterns emitted from a plurality of different sound source positions and a change in pupil diameter of the user; an estimation step of estimating, by an estimation unit, a destination to which the user has auditory attention with respect to a sound from the sound source, using the feature amount; The auditory attention state estimation method includes:

9. A learning method using a learning device, comprising: A learning process is performed using learning data in which a learning feature based on the strength of correlation between each of a plurality of different learning visual stimulation patterns emitted from the positions of a plurality of different learning sound sources and a learning pupil diameter change amount is associated with correct answer information indicating a destination where auditory attention is directed with respect to the learning sound from the learning sound source. A learning method comprising a learning step of inputting features based on the strength of correlation between each of a plurality of different visual stimulation patterns emitted from a plurality of different sound source positions and the change in pupil diameter of a user, and obtaining an estimation model for estimating where the user's auditory attention is directed with respect to sounds from the sound sources.

10. A program for causing a computer to function as the auditory attention state estimating device according to any one of claims 1 to 6 or the learning device according to claim 7.

Images