User state determination device, user state determination method and computer program
The user state determination device accurately determines attention and immersion levels by measuring brain activity and blinking motions, addressing the limitations of existing systems in static viewing scenarios with enhanced accuracy and personalized criteria.
Patent Information
- Application Number
- JP2023074295
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing attention state determination systems struggle to accurately determine user states when the visual target or line of sight does not move, such as when a user is viewing a screen, and there is a need for further improvement in accuracy.
A user state determination device and method that measures brain activity and detects blinking motions to calculate the average potential difference between maximum and minimum values, using principal component analysis to determine user states based on brain potentials, and incorporates learning algorithms to adapt determination criteria for individual users and visual targets.
Enables high-accuracy determination of user states, including attention and immersion levels, even when the visual target or line of sight does not move, by utilizing brain activity measurements and blinking detection, reducing the need for additional sensors and improving accuracy through noise reduction and personalized criteria.
Smart Images

Figure 2024158777000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a user state determination device, a user state determination method, and a computer program. [Background technology]
[0002] Conventionally, an attention state determination system for determining a user's attention state has been proposed. For example, the following Patent Document 1 discloses an attention state determination system that measures a user's electroencephalogram signal and eye movement, and determines the user's attention state by using an eye fixation related potential (EFRP) based on the start time of eye fixation from the measured eye movement.
[0003] Furthermore, for example, Patent Document 2 discloses a technology for measuring a user's brain activity state in order to determine a user state including the user's attention state or degree of immersion, and when a blinking action of the user's eyes is detected, determining the user state based on the brain activity state at the time of the detected blinking action. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 5570386 [Patent Document 2] JP 2022-095462 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the technology disclosed in Patent Document 1, the timing for analyzing brain waves is when the eyes are fixed. Therefore, although it can be applied when the visual target is constantly moving, such as when a user is driving, it cannot be applied to continuous stimuli where the visual target or line of sight (eyes) is stationary, such as when watching a screen.
[0006] Furthermore, although the technology described in Patent Document 2 can accurately determine the user state even when the user's visual target or line of sight is not moving, further improvement in accuracy is desired.
[0007] Therefore, the present invention aims to provide a user state determination device, a user state determination method, and a computer program that can determine the user state more accurately even when the visual target or line of sight (eyeball) is not moving, such as when the user is watching a screen. [Means for solving the problem]
[0008] The present invention according to one aspect is a user state determination device that determines a user state including a user's attention state or a degree of immersion. The user state determination device includes a brain activity state measurement unit that measures the user's brain activity state, a blink detection unit that detects the blinking of the user's eyes, and a determination unit that determines the user state based on the brain activity state corresponding to the blinking detected by the blink detection unit. The determination unit calculates an average potential of the difference between the maximum and minimum values indicated by the brain activity state during a predetermined time period from the time the blinking is detected, and determines the user state based on the calculated average potential.
[0009] Another aspect of the present invention is a user state determination method for determining a user state including a user's attention state or a degree of immersion. The user state determination method includes measuring a brain activity state of the user, detecting a blinking action of the user's eyes, and determining the user state based on the brain activity state corresponding to the blinking action detected by the blink detection unit. The determining unit includes calculating an average potential of a difference between a maximum value and a minimum value indicated by the brain activity state during a predetermined time from the time when the blinking action is detected, and determining the user state based on the calculated average potential.
[0010] In addition, instead of calculating an average potential difference between the maximum and minimum values indicated by the brain activity state at a predetermined time from the time when the blinking action is detected, the present invention may calculate a maximum value of a potential component that has a relatively high influence by principal component analysis on the brain potential indicating the brain activity state corresponding to the blinking action, and judge the user state based on the calculated maximum value.
[0011] The present invention can also be understood as a computer program for executing the above-described method, or a computer-readable recording medium on which the computer program is non-transitoryly recorded.
[0012] In this specification, the term "means" does not simply mean physical means, but also includes cases where the functions of the means are realized by software. The functions of one means may be realized by two or more physical means, or the functions of two or more means may be realized by one physical means. Furthermore, the term "system" refers to a logical collection of multiple devices (or functional modules that realize specific functions), and it does not matter whether each device or functional module is contained within a single housing. Effect of the Invention
[0013] According to the present invention, even if the visual target or the line of sight is not moving, the user state can be determined with high accuracy. In particular, according to the present invention, even if the difference in the user state, such as the degree of immersion and / or the degree of concentration, is small, the user state can be determined with high accuracy.
[0014] Other technical features, objects, and operational effects or advantages of the present invention will become apparent from the following embodiments described with reference to the accompanying drawings. The effects described in this specification are merely examples and are not limited thereto, and other effects may also be present. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a user state determination device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing the relationship between the eyeblink-evoked potential (EEP) and the immersion state. [Diagram 3] FIG. 3 is a graph showing the fluctuation of the blink evoked potential after the blink in each of the conditions of high immersion, low immersion, and a reference condition. [Figure 4] FIG. 4 is a flowchart showing an example (embodiment 1) of a user state determination process in the user state determination device according to one embodiment of the present invention. [Diagram 5] FIG. 5 is a flowchart showing an example (embodiment 2) of a user state determination process in the user state determination device according to one embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart showing an example (Example 3) of a gazed target database construction process in a user state determination device according to an embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart showing an example (Example 3) of a user state determination process in the user state determination device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude various modifications and application of techniques not specified below. The present invention can be implemented by various modifications (for example, combining the respective embodiments) within the scope of the gist of the present invention. In addition, in the following description of the drawings, the same or similar parts are represented by the same or similar reference numerals. The drawings are schematic and do not necessarily correspond to actual dimensions, ratios, etc. The drawings may also include parts with different dimensional relationships and ratios.
[0017] (Configuration of the user state determination device) 1 is a block diagram showing an example of the configuration of a user state determination device according to an embodiment of the present invention. In the figure, components of the user state determination device 100 that are particularly related to this embodiment are conceptually shown. In addition, when assuming a user during automatic driving, the user state determination device 100 may be installed, for example, at an arbitrary location in the vehicle, or may be configured such that a control unit 102 and / or a storage unit 106 installed at a remote location communicate with a brain activity state measurement unit 112, a blink detection unit 114, etc. in the vehicle via a communication network or the like.
[0018] In the figure, the user state determination device 100 is configured to include, for example, a control unit 102 such as a CPU that comprehensively controls the entire user state determination device 100, a brain activity state measurement unit 112, a blink detection unit 114, and a storage unit 106, and these components are communicatively connected via an arbitrary communication path. The user state determination device 100 may be an ECU (Engine Control Unit), a microcomputer, a personal computer, a server computer, or the like.
[0019] The storage unit 106 stores various programs, various databases, tables, etc. As an example, the databases include a user information database 106a and a visually recognized target database 106b, etc. The storage unit 106 is formed of a secondary storage device such as a non-volatile memory or a hard disk drive.
[0020] The user information database 106a is a database that manages information on the brain activity state (e.g., brain potential, brain magnetic field, and brain waves, etc.) corresponding to the blinking action of the user for each user. As an example, the user information database 106a stores data showing the correlation between the brain activity state for a certain user and the user state (e.g., attention state, immersion level, etc.). For example, the correlation between the brain activity state and the user state may differ for each user in the vehicle. The determination unit 102a, which will be described later, accurately determines the user state based on the brain activity state corresponding to the blinking action of the user by referring to the user information database 106a. In this case, the determination unit 102a may determine the level of the user state (e.g., the attention state or immersion level (hereinafter sometimes simply referred to as "immersion level") is low, slightly low, slightly high, high, etc.) using multiple thresholds (determination criteria).
[0021] Furthermore, the learning unit 102b, which will be described later, may store the brain activity state and the user state for a certain user in the user information database 106a in association with each other. Furthermore, the learning unit 102b may refer to the user information database 106a to learn a judgment criterion for each user (for example, a plurality of thresholds that serve as a judgment criterion for the level of the user state), etc.
[0022] The visual target database 106b is a database that manages the brain activity state corresponding to the blinking action of the user for each visual target. For example, the visual target database 106b stores the brain activity state and the user state in association with each visual target. More specifically, the correlation between the brain activity state and the user state may be different when the user is watching a video or a car navigation screen while driving the vehicle and when the user is watching the environment around the vehicle. As will be described later, the determination unit 102a refers to the visual target database 106b to accurately determine the user state from the brain activity state corresponding to the blinking action of the user according to the visual target of the user. Also, as will be described later, the learning unit 102b may store the brain activity state and the user state in association with each visual target in the visual target database 106b, or may learn a determination criterion for each user (for example, a threshold value of the brain activity state for determining a high immersion state when watching a screen) by referring to the visual target database 106b.
[0023] The brain activity state measuring unit 112 is a brain measuring device that measures biological signals related to the user's brain activity state. The brain activity state measuring unit 112 may be, for example, at least one of a magnetic field measuring device, a radio wave measuring device (Freer), a scalp surface potential measuring device, and a cerebral blood flow measuring device, and may be what is called an electroencephalograph, a magnetoencephalograph, an electroencephalograph, or a cerebral blood flow meter. It is desirable for the brain activity state measuring unit 112 to be capable of measurement with high time resolution. The brain activity state measuring unit 112 is preferably a non-contact type, but may be a contact type.
[0024] For example, the Neurobiomonitor headrest manufactured by Freer Logic may be applied as the non-contact brain activity state measurement unit 112 (http: / / freerlogic.com / products / hardware). In addition, the non-contact brain activity state measurement unit 112 may be the following electroencephalograph (Patent No. 5981506) / (Patent Application No. 2019-201280) / (Takamasa Ando et al., Non-contact acquisition of brain function using a time-extracted compact camera,Scientific Reports | (2019) 9:17854 | https: / / doi.org / 10.1038 / s41598-019-54458-7). In addition, the in-vehicle brain activity state measuring unit 112 may be configured using the magnetoencephalography system "RICOH MEG" (magnetoencephalography system RICOH160-1 (medical device certification number: 22100BZX00914000), https: / / jp.ricoh.com / release / 2018 / 0709_1.html).
[0025] The blink detection unit 114 is a blink detector that detects the blinking of the user's eyes. The blink detection unit 114 is configured as a separate unit from the brain activity state measurement unit 112, but is not limited to this and may be configured integrally in terms of hardware and / or software. For example, the blinking can be detected based on an electrooculogram (EOG: Electro Oculo Graphy) extracted from a signal from the brain activity state measurement unit 112. The blink detection unit 114 may be realized as hardware and / or software. The blink detection unit 114 may be configured as a part of the control unit 102.
[0026] As an example, the blink detection unit 114 may apply a blink potential removal technique of an electroencephalogram analysis system (BESA Research 7.0) to estimate a potential due to a blink based on a model of what kind of potential on the scalp is generated by a blink, which model has been created in advance. The blink detection unit 114 may also detect the blinking of the user's eyes using the techniques listed below. ·Berg, P., and Scherg, M. A multiple source approach to the correction of eye artifacts, Electroenceph. clin. Neurophysiol., 1994, 90: 229-241. ·Ille N., Berg P., and SchergM. A Spatial Components Method for Continuous Artifact Correction in EEG and MEG. Biomed. Tech., 1997, 42 (suppl. 1): 80-83. ·Ille, N., Berg, P., Scherg, M. Artifact correction of the ongoing EEG using spatial filters based on artifact and brain signal topographies. J. Clin. Neurophysiol. 2002, 19: 113-124. ·Uusitalo, MA, Ilmoniemi, RJ Signal-space projection method for separating MEG or EEG into components. Med. Biol. Eng. Comput., 1997, 35: 135-140. ·http: / / wiki.besa.de / index.php?title=BESA_Research_Artifact_Correction
[0027] The control unit 102 is a chip circuit having a processor, control programs and drivers such as an OS (Operating System), programs defining various processing procedures, and an internal memory for storing various data, and executes various programs on the OS under the control of the processor to execute various information processing. The control unit 102 is conceptually configured to include a determination unit 102a and a learning unit 102b.
[0028] The determination unit 102a determines the user state (such as the user's attention state or the user's immersion level) based on the brain activity state corresponding to the blinking action detected by the blink detection unit 114. Here, the "blinking action" includes "immediately after blinking," "immediately after closing the eye for blinking," "immediately before blinking," "before and after blinking," "at the time of blinking," "the moment of blinking," "period of blinking," and "period from closing the eye for blinking to fully opening it," etc. The determination unit 102a extracts the brain activity state before and after the blinking action detected by the blink detection unit 114 and for a predetermined time after that point, and can determine the user state based on the extracted brain activity state.
[0029] In order to improve the accuracy of the determination, the determination unit 102a may calculate an average value of the magnitude of the brain potential corresponding to a plurality of blinks (average potential of the difference between the maximum value and the minimum value of the potential in a predetermined time) and determine the user state based on the average value. For example, the determination unit 102a may acquire an eyeblink-evoked potential (EEP) by averaging electroencephalograms using the blink detected by the blink detection unit 114 as a trigger. The EEP is a negative scalp potential that reflects a reset of the brain activity state.
[0030] Alternatively, the determination unit 102a may determine the user state based on the maximum value of the potential highly influenced by brain activity obtained by principal component analysis of the brain potential instead of the average value. Principal component analysis is known as a statistical data analysis method that summarizes data having many explanatory variables into fewer variables (principal components) to make the data easier to understand. In the present disclosure, the brain potential is an average of brain potential data measured at multiple brain measurement points, and is a value including different sources and factors of generation. Therefore, the main variables (principal components) that explain the brain potential data are extracted using principal component analysis. The potential highly influenced by brain activity is a potential that is correlated with the degree of immersion by statistical analysis among the potentials that are the principal components.
[0031] As an example, the determination unit 102a may determine the user state based on the magnitude of the activity amount (amount of brain activity) of the brain activity state in the blinking action detected by the blink detection unit 114. For example, when the determination unit 102a determines that the brain potential immediately after the blink is large, it determines that the user state is a low immersion state, and when the determination unit 102a determines that the brain potential immediately after the blink is small, it determines that the user state is a high immersion state. Fig. 2 is a diagram showing the relationship between the blink evoked potential (EEP) and the immersion state. Fig. 3 is a graph showing the fluctuation of the EEP after the blink (0 ms) in high immersion, low immersion, and another condition set as a reference.
[0032] It is known that the moment a user blinks, the activity of the default mode network (DMN), which is a network that is activated when the brain is in a resting state, increases, and thus blinking resets the brain activity state. Therefore, the EEP can be acquired by averaging brain potentials using the occurrence of a blink as a trigger. As shown in FIG. 2 and FIG. 3, when the user is immersed in the video being viewed, that is, when the degree of immersion is high, the degree of resetting the brain activity state by blinking becomes shallow, so the EEP becomes small, and when the degree of immersion is low, the degree of resetting the brain activity state by blinking becomes deep, so the EEP becomes large. Therefore, when the determination unit 102a determines that the amount of brain activity is large based on the magnitude of the amount of brain activity at the time of blinking (for example, the maximum value of the brain potential for a certain period immediately after the blink), it can determine that the user state is in a low immersion state, and when it determines that the amount of brain activity immediately after the blink is small, it can determine that the user state is in a high immersion state.
[0033] However, potentials caused by various factors other than the DMN that reflect the user's state are superimposed on the potentials generated and measured by using a blink as a trigger, and since these factors interfere with each other, it is necessary to exclude potentials caused by other factors that become noise in order to improve the accuracy of determining the user's state. For example, since the maximum potential immediately after a blink is likely to be superimposed by the electrooculogram (EOG) generated by the blink, such electrooculograms must be excluded from the measured potential.
[0034] Therefore, the determination unit 102a can determine the user state more accurately by calculating the average value (average potential) of the difference between the maximum value and the minimum value of the potential measured with the occurrence of a blink as a trigger for a certain period of time from immediately after the blink. The determination unit 102a may determine the user state based on the maximum value of the potential with a high influence obtained by principal component analysis of the brain potential, thereby making it possible to determine the user state more accurately.
[0035] The determination unit 102a may determine the user state based on a determination criterion for each user for the brain activity state corresponding to the blinking action. Since the brain activity state in the blinking action varies between individuals, the user state can be determined with high accuracy by using a determination criterion for each user. As an example, the determination unit 102a may determine the user state with respect to the brain activity state in the blinking action by referring to the user information database 106a. The determination unit 102a may determine the user state according to the learning result of the learning unit 102b based on the user information database 106a (determination criteria such as a threshold value of the brain activity state for determining a high immersion state, etc.).
[0036] The determination unit 102a may also determine the user state based on a determination criterion for each visual target for the brain activity state corresponding to the blinking action. Since the brain activity state varies depending on what the user is looking at when blinking, this allows the user state to be determined with high accuracy. As an example, the determination unit 102a may determine the user state by referring to the visual target database 106b for the brain activity state corresponding to the blinking action. The determination unit 102a may also determine the user state according to the learning result of the learning unit 102b based on the visual target database 106b (such as a determination criterion for the brain activity state threshold for determining the level of immersion state, etc.).
[0037] The learning unit 102b may be a machine learning model that has been subjected to machine learning so as to output a user state judgment criterion or the like for a brain activity state corresponding to a blinking action according to a predetermined machine learning algorithm. The learning unit 102b may associate a brain activity state with a user state for a certain user and store it in the user information database 106a. The learning unit 102b may also learn a judgment criterion for each user (for example, a threshold value of a brain activity state for judging a state as being highly immersive) or the like by referring to the user information database 106a. The learning unit 102b may also associate a brain activity state with a user state for each visual target and store it in the visual target database 106b. The learning unit 102b may also learn a judgment criterion for each user (for example, a threshold value of a brain activity state for judging a state as being highly immersive when viewing a screen) or the like by referring to the visual target database 106b.
[0038] The learning may be any of supervised learning, unsupervised learning, and reinforcement learning. Examples of machine learning algorithms include support vector machines (SVM), decision tree learning, neural networks (NN), genetic programming (GP), and Bayesian networks.
[0039] In this embodiment, it is not necessary to assume that the ECU or computer of a vehicle or the like functions as the user state determination device 100 from the beginning. For example, even if the storage unit 106 is not provided with the user information database 106a or the visually recognized target database 106b, the control unit 102 may create the user information database 106a or the visually recognized target database 106b by accumulating and storing data of the brain activity state measurement unit 112, the blink detection unit 114, and the like in the storage unit 106. In addition, even if the control unit 102 does not have the function of the determination unit 102a, the learning unit 102b may obtain a determination criterion for the user state by machine learning based on data stored in the storage unit 106, such as the user information database 106a and the visually recognized target database 106b, thereby controlling the control unit 102 to realize the function as the determination unit 102a. In addition, even if the brain activity state measurement unit 112 or the blink detection unit 114 is not connected to the control unit 102, the control unit 102 may be controlled to realize connection with these peripheral devices and the like by a predetermined program or the like. This makes it possible to create a user state determination device that can accurately determine the user state even when the visual target or line of sight (eyeball) is not moving, such as when viewing a screen.
[0040] (User status determination process) Next, an example of a user status determination process in the user status determination device 100 of this embodiment configured as described above will be described in detail below with reference to Fig. 4 to Fig. 6. Fig. 4 is a flowchart showing an example (Example 1) of a user status determination process in the user status determination device 100 of this embodiment.
[0041] Example 1 As shown in Fig. 4, first, the brain activity state measuring unit 112 starts measuring an electroencephalogram to determine the brain activity state of the user (S401). During the measurement of the electroencephalogram, the blink detection unit 114 monitors the occurrence of a blink of the user (S402). When the blink detection unit 114 detects a blink of the user (Yes in S402), the determination unit 102a extracts a brain potential reflecting the user state according to the blinking action from the signal of the brain activity state measuring unit 112 (S403). For example, the determination unit 102a acquires a brain potential for a predetermined time (for example, 250 ms) before and after the occurrence of a blink.
[0042] Next, the determining unit 102a extracts the maximum and minimum values from the acquired potential (S404), and then calculates the average value of the differences between the extracted maximum and minimum values (average potential) (S405).
[0043] Next, the determination unit 102a compares the calculated average potential with a plurality of thresholds (first to third thresholds in this example) in order to determine the level of immersion in the user state. Here, the first threshold>the second threshold>the third threshold.
[0044] That is, the judgment unit 102a first judges whether the calculated average potential is equal to or greater than the first threshold (S406), and if the judgment unit 102a judges that the average potential is equal to or greater than the first threshold (Yes in S406), the judgment unit 102a judges that the user's immersion level is "low" (S407). On the other hand, if the judgment unit 102a judges that the calculated average potential is not equal to or greater than the first threshold (No in S406), the judgment unit 102a next judges whether the average potential is equal to or greater than the second threshold (S408). If the judgment unit 102a judges that the average potential is equal to or greater than the second threshold (Yes in S408), the judgment unit 102a judges that the user's immersion level is "slightly low" (S409).
[0045] Furthermore, when the judgment unit 102a judges that the calculated average potential is not equal to or greater than the second threshold (No in S408), the judgment unit 102a further judges whether or not the average potential is equal to or greater than a third threshold (S410). When the judgment unit 102a judges that the average potential is equal to or greater than the third threshold (Yes in S410), the judgment unit 102a judges that the user's immersion level is "slightly high" (S411). On the other hand, when the judgment unit 102a judges that the average potential is not equal to or greater than the third threshold (No in S410), the judgment unit 102a judges that the user's immersion level is "high" (S412).
[0046] Next, the determination unit 102a outputs the determination result of the user's state of immersion (S413), and ends the measurement process.
[0047] As described above, since the brain potential reflecting the user state according to the blinking action is detected, the degree of immersion of the user in the visual target can be accurately determined. In addition, when the non-contact type brain activity state measuring unit 112 is used, it is not necessary to wear an electrooculography measuring device, so that the inconvenience can be reduced. In addition, it is possible to apply the method to visual targets where the line of sight does not move, such as when the user is looking at a screen installed inside the vehicle, not around the vehicle.
[0048] Furthermore, in this example, the user state is determined using multiple criteria based on the average potential of the peak potential values (maximum and minimum values) at a specified time in response to the blinking action, so that even slight differences in the user state can be accurately determined.
[0049] Example 2 A second embodiment will be described below in detail with reference to FIG. 5 as an example of the user state determination process in the user state determination device 100 of this embodiment.
[0050] 5 is a flowchart showing an example (Example 2) of a user state determination process in the user state determination device 100 of this embodiment. This example differs from Example 1 above in that the average potential of the peak value differences in a predetermined time is calculated a required number of times, and then the average potential (total average potential) is calculated. The following will mainly explain the differences from Example 1.
[0051] That is, in this embodiment, in order to improve the accuracy of the judgment, the judgment unit 102a calculates the average potential of the peak value differences at a specified time corresponding to each of multiple blinking actions a required number of times, and further judges the user state based on the total average potential.
[0052] More specifically, as shown in FIG. 5, the judgment unit 102a extracts a brain potential reflecting the user state corresponding to the blinking action from the signal of the brain activity state measurement unit 112, and further repeats the process of calculating the average potential of the difference between the maximum value and the minimum value in a predetermined time a required number of times to calculate a grand average potential based on the average potential of the brain potential reflecting the user state corresponding to multiple blinking actions (S501 to S506).
[0053] If the judgment unit 102a determines that the process has been repeated the required number of times (Yes in S506), as described above, it judges the user's immersion level based on multiple threshold values for the calculated total average potential (S507 to S513) and outputs the judgment result (S514).
[0054] As described above, according to this example, the average potential of the peak value difference at a specified time corresponding to each of multiple blinking actions is calculated the required number of times, and the level of immersion of the user is determined based on the total average potential calculated from these, thereby making it possible to make a more accurate determination.
[0055] Example 3 As an example of the user state determination process in the user state determination device 100 of this embodiment, Example 3 will be described in detail below with reference to Figs. 6 and 7. This example differs from the above-mentioned examples in that the immersion level is determined according to the user's gazed object. For this reason, in order to make it possible to determine the immersion level according to the user's gazed object, a gazed object database 106b is constructed by the learning unit 102b. The following description will focus on the differences from the above-mentioned examples.
[0056] That is, the visual target of a user driving a car is assumed to be, for example, the vehicle surroundings and an information presentation device (such as a display screen of a television or car navigation system). Therefore, the brain activity state caused by the blinking action may vary depending on which of the visual targets the user is visually identifying. In this embodiment, in order to improve the accuracy of determining the user state for each different visual target, the learning unit 102b constructs a visual target database 106b that associates the brain activity state with the user state, and the determining unit 102a accurately determines the user state by referring to the visual target database 106b.
[0057] First, construction of the visually recognized target database 106b by the learning unit 102b will be described. Here, it is assumed that the visually recognized target of the user is the vehicle surrounding environment. Fig. 6 is a flowchart showing an example of a visually recognized target database construction process in the user state determination device 100 of this embodiment.
[0058] As shown in FIG. 6, the judgment unit 102a extracts a brain potential reflecting a user state according to a blinking action from a signal of the brain activity state measurement unit 112, and further calculates an average potential of the difference between the maximum value and the minimum value for a predetermined period of time (S601 to S605).
[0059] Next, the learning unit 102b executes machine learning so as to input the calculated average potential and output the level of the user's immersion in the vehicle surroundings (S606). That is, the learning unit 102b adjusts the first to third thresholds used in the determination unit 102a according to a supervised machine learning algorithm so as to determine the immersion level of a specific visual target based on the average potential of the difference between the maximum and minimum values of the brain potential reflecting the user's state according to the blinking action at a predetermined time. Next, the learning unit 102b repeats the adjustment of the first to third thresholds by such machine learning a predetermined number of times (S607).
[0060] After performing machine learning a predetermined number of times (Yes in S607), the learning unit 102b stores the obtained first to third thresholds in the visual recognition target database 106b as determination criteria for the vehicle surroundings. This provides the first to third thresholds adapted to the determination of the immersion level for the vehicle surroundings.
[0061] Similarly, when the user's visual target is the information display device, the learning unit 102b executes machine learning and adjusts the first to third thresholds so as to output the level of the user's immersion in the vehicle surroundings using the calculated average potential as an input. This allows the first to third thresholds to be obtained that are suitable for determining the level of immersion in the information display device.
[0062] In this manner, the visually recognized object database 106b is constructed, which has a plurality of threshold values as a criterion for determining the immersion level for each visually recognized object.
[0063] Next, an example of a user state determination process in the user state determination device 100 using the visually recognized target database 106b will be described. Fig. 7 is a flowchart showing an example of a user state determination process in the user state determination device 100 of this embodiment. The following will mainly describe the points that are different from the above embodiment.
[0064] As shown in the figure, the judgment unit 102a extracts a brain potential reflecting the user state according to the blinking action from the signal of the brain activity state measurement unit 112, and further calculates an average potential of the difference between the maximum value and the minimum value for a predetermined period of time (S701 to S705).
[0065] Next, the determination unit 102a determines the user's visual target (S706). The user's visual target is determined, for example, by detecting the line of sight direction. If the determined visual target is the vehicle surroundings, the determination unit 102a refers to the visual target database 106b to determine the first to third thresholds for the vehicle surroundings (S707). Next, the determination unit 102a determines the immersion level for the vehicle surroundings using the first to third thresholds for the vehicle surroundings (S708). Note that the process of determining the immersion level for the vehicle surroundings (S708) is the same as the process of S406 to S412 shown in FIG. 4, except that the determination criteria for the vehicle surroundings (the first to third thresholds) are used.
[0066] On the other hand, when the determined visual target is an information presentation device, the determination unit 102a refers to the visual target database 106b to determine the first to third thresholds for the information presentation device (S709). Next, the determination unit 102a uses the first to third thresholds for the information presentation device to determine the immersion level for the vehicle's surrounding environment (S708). Note that the process of determining the immersion level for the vehicle's surrounding environment (S708) is the same as the processes of S406 to S412 shown in FIG. 4.
[0067] Next, the determination unit 102a outputs the determination result of the user's state of immersion (S711). Furthermore, the learning unit 102b associates the average potential calculated by the determination unit 102a with the determination result of the immersion for the identified visual target, and stores them in the visual target database 106b (S712). In this case, the learning unit 102b may execute machine learning and update the first to third thresholds as described above.
[0068] In this embodiment, an example has been described in which a database is constructed for each visual target and judgment criteria are generated for each visual target, but this is not limited to the above. A database (i.e., user information database 106a) may be constructed for each individual user in a similar procedure, and judgment criteria for each user may be generated.
[0069] As described above, according to this embodiment, even if there is variation in the relationship between the amount of brain activity during blinking and the degree of immersion for each visual object and / or each user, it is possible to accurately determine the user state.
[0070] (Effects of this embodiment) As described above, according to this embodiment, the following advantageous effects can be obtained.
[0071] In this embodiment, in order to determine the user state, including the user's attention state or the degree of immersion, the user's brain activity state (brain potential) is measured, and when the user's blinking action is detected, the user state is determined based on the brain activity state corresponding to the detected blinking action.
[0072] This makes it possible to determine the user state based on the brain activity state when the user blinks, without requiring eye movement as in conventional determination methods based on eye fixation related potentials (EFRPs). Therefore, the user state can be determined with high accuracy even when the viewed object or line of sight (eyes) does not move, such as when viewing a screen.
[0073] In particular, according to this embodiment, the maximum and minimum potential values at a predetermined time after the occurrence of a blinking action are extracted, and the user state is determined based on the average potential difference between the maximum and minimum values. Therefore, even when the difference in user state, such as the user's degree of immersion and / or concentration, is small, a more accurate determination can be made.
[0074] Furthermore, according to this embodiment, brain measurement detects brain activity using an electroencephalograph, a magnetoencephalograph, an electroencephalograph, and / or a cerebral blood flow measuring device, etc., so that the user's state can be accurately determined based on the electroencephalogram, the brain magnetic field, the brain potential, and / or the cerebral blood flow.
[0075] Furthermore, according to this embodiment, blinking is detected based on the electrooculogram (EOG) extracted from a brain measurement signal, so there is no need to provide a sensor for detecting blinking in addition to a sensor for brain measurement, and there is no need to wear an electrooculogram measuring device, which is less cumbersome and has excellent cost-effectiveness.
[0076] Furthermore, according to this embodiment, the user state including the user's attention state or the user's degree of immersion is determined based on the magnitude of brain activity (average potential) over a certain period of time corresponding to a blinking action, or the maximum value of the potential component that has a relatively high influence by principal component analysis on the brain potential indicating the brain activity state corresponding to a blinking action. Therefore, the user state can be determined with high accuracy based on the principle that the depth of the reset degree at the time of blinking changes depending on the degree of immersion.
[0077] Furthermore, according to this embodiment, the average potential of the difference between the maximum and minimum potential values at a specified time corresponding to each of multiple blinking actions is calculated, and the user state is determined based on these average potentials (total average potential). Therefore, even if the brain activity state corresponding to a single blinking action is affected by noise, etc., the user state can be determined with high accuracy.
[0078] Furthermore, according to this embodiment, the user state is judged for the brain activity state corresponding to the blinking action based on judgment criteria for each visual object and / or each user, so that the user state can be judged with high accuracy even if there is variation in the relationship between the amount of brain activity during the blinking action and the degree of immersion for each visual object and / or each user.
[0079] (Other embodiments)
[0080] The above-described embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. The present invention can be embodied in various forms without departing from the spirit of the present invention.
[0081] For example, in the methods disclosed herein, steps, operations, or functions may be performed in parallel or in a different order, provided that the results are consistent. The steps, operations, and functions described are provided merely as examples, and some of the steps, operations, and functions may be omitted or combined into one, or other steps, operations, or functions may be added, without departing from the spirit of the invention.
[0082] In addition, although various embodiments are disclosed in this specification, specific features (technical matters) in one embodiment can be added to or replaced with specific features in other embodiments, with appropriate modifications, and such forms are also included in the spirit of the present invention.
[0083] Furthermore, the specific form of distribution and integration of the equipment is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various additions, etc., or functional load.
[0084] In addition, for example, the user state determination device 100 has been described as being configured as an integrated device to perform processing in a stand-alone form, but this is not limited to this, and the device may function as an external server, etc., to perform processing in response to a request from an in-vehicle client terminal, and return the processing results to the in-vehicle client terminal.
[0085] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually or semi-automatically, or all or part of the processes described as being performed manually can be performed automatically.
[0086] In addition, the processing procedures, control procedures, specific names, registered data for each process, information including parameters such as search conditions, screen examples, and database configurations shown in the above documents and drawings may be changed as desired unless otherwise specified.
[0087] Moreover, with regard to the user state determination device 100 and the like, each of the components shown in the drawings is a functional concept, and does not necessarily have to be physically configured as shown in the drawings. For example, the processing functions of each device of the user state determination device 100, particularly the processing functions performed by the control unit 102, may be realized in whole or in part by a processor such as a CPU (Central Processing Unit) and a program interpreted and executed by the processor, or may be realized as a hardware processor using wired logic. The program is recorded on a non-transitory computer-readable recording medium including programmed instructions for causing a computer to execute the method according to the present invention, and is mechanically read by the user state determination device 100 as necessary. That is, a computer program for giving instructions to the CPU in cooperation with an OS (Operating System) and performing various processes is recorded in the storage unit 106 such as a solid state drive or a hard disk drive. The computer program is executed by being loaded into a memory used by the CPU, and cooperates with the CPU to form the control unit.
[0088] In addition, this computer program may be stored in an application program server connected to the user state determination device 100 via any network, and it is also possible to download all or a part of it as necessary.
[0089] The program according to the present invention may be stored in a computer-readable recording medium, or may be configured as a program product. Here, the term "recording medium" includes any "portable physical medium" such as a memory card, a USB memory, an SD card, a flexible disk, a magneto-optical disk, a ROM, an EPROM, an EEPROM, a CD-ROM, an MO, a DVD, and a Blu-ray (registered trademark) Disc.
[0090] Moreover, a "program" is a data processing method written in any language or description method, and may be in any format, such as source code or binary code. The "program" is not necessarily limited to a single one, but may be a distributed one composed of multiple modules or libraries, or may work with a separate program, such as an OS, to achieve its function. The specific configurations and reading procedures for reading the recording medium in each device shown in the embodiments, or the installation procedures after reading, may be well-known configurations and procedures. The present invention may be configured as a program product in which the program is recorded on a non-transitory computer-readable recording medium.
[0091] The various databases (user information database 106a, visual target database 106b, etc.) stored in the memory unit 106 are storage means such as memory devices such as RAM and ROM, fixed disk devices such as hard disks, flexible disks, and optical disks, and store various programs, tables, databases, and web page files used for various processes and providing websites.
[0092] The user state determination device 100 may be configured as a known information processing device such as a personal computer, ECU, or workstation, or may be configured by connecting any peripheral device to the information processing device. The user state determination device 100 may be realized by installing software (including programs, data, etc.) that realizes the method of the present invention in the information processing device. [Explanation of symbols]
[0093] 100...User state determination device 102...Control unit 102a...judgment section 102b…Learning Department 106...Storage section 106a...User information database 106b…Visual target database 112…Brain activity measurement unit 114...Blink detection unit
Claims
1. A user state determination device that determines a user state including a user's attention state or a degree of immersion, A brain activity state measurement unit that measures the brain activity state of the user; A blink detection unit that detects a blinking action of the user's eyes; A determination unit that determines the user state based on the brain activity state corresponding to the blinking action detected by the blink detection unit, the determination unit calculates an average potential of a difference between a maximum value and a minimum value indicated by the brain activity state during a predetermined time period from the time point when the blinking action is detected, and determines the user state based on the calculated average potential. User state determination device.
2. The brain activity state measuring unit detects the brain activity state by at least one of an electroencephalograph, a magnetoencephalograph, an electroencephalograph, and a cerebral blood flow meter. The user state determination device according to claim 1 .
3. The blink detection unit detects the blinking action based on an electrooculogram extracted from a signal output from the brain activity state measurement unit. The user state determination device according to claim 1 .
4. The determination unit determines the user state including the attention state or the immersion degree of the user based on the magnitude of the average potential corresponding to the blinking action. The user state determination device according to claim 1 .
5. The determination unit is When the average potential is equal to or greater than a first threshold value, it is determined that the user's attention state or immersion level is low; When the average potential is smaller than the first threshold value and is equal to or larger than a second threshold value, it is determined that the attention state or the immersion level of the user is somewhat low; When the average potential is smaller than the second threshold value and is equal to or larger than a third threshold value, it is determined that the attention state or the immersion degree of the user is somewhat high; When the average potential is smaller than the third threshold value, it is determined that the user's attention state or degree of immersion is high. The user state determination device according to claim 4 .
6. the determination unit determines the user state based on a total average potential based on the average potential corresponding to each of a plurality of blinking actions; The user state determination device according to claim 1 .
7. The determination unit determines the user state with respect to the brain activity state corresponding to the blinking action based on a determination criterion for each user. The user state determination device according to claim 1 .
8. The determination unit determines the user state based on a determination criterion for each visual target of the user with respect to the brain activity state corresponding to the blinking action. The user state determination device according to claim 1 .
9. A learning unit that performs machine learning according to a predetermined machine learning algorithm so that the judgment criterion for each visual target of the user is output. The user state determination device according to claim 8 .
10. A user state determination device that determines a user state including a user's attention state or a degree of immersion, A brain activity state measurement unit that measures the brain activity state of the user; A blink detection unit that detects a blinking action of the user's eyes; A determination unit that determines the user state based on the brain activity state corresponding to the blinking action detected by the blink detection unit, the determination unit calculates a maximum value of an electric potential component having a relatively high degree of influence by a principal component analysis of a brain electric potential indicating the brain activity state corresponding to the blinking action, and determines the user state based on the calculated maximum value. User state determination device.
11. A user state determination method for determining a user state including a user's attention state or a user's immersion level, comprising: Measuring the user's brain activity; Detecting an eye blinking action of the user; and determining the user state based on the brain activity state corresponding to the blinking action detected by the blink detection unit; The determining unit includes calculating an average potential of a difference between a maximum value and a minimum value indicated by the brain activity state during a predetermined time period from the time when the blinking action is detected, and determining the user state based on the calculated average potential. A method for determining a user state.
12. A program for causing a computer to execute a user state determination method for determining a user state including a user's attention state or a user's immersion level, A brain measurement step of measuring a brain activity state of the user; A blink detection step of detecting a blinking action of the user's eye; a determination step of determining the user state based on the brain activity state at the time of the blinking action detected in the blink detection step; A program for causing a computer to execute the following.
Citation Information
Patent Citations
Selective adsorption removal of phosphorous ion in water
JP1980070386A
User state determination device, user state determination method, program, computer, and creation method of user state determination device
JP2022095462A
