Status determination method and determination implementation device

Abstract

This invention provides a state determination method that enables real-time determination of the subject's state. [Solution] The method comprises the steps of measuring brain activity signals at multiple locations in the subject's brain (S101), preprocessing the measured brain activity signals (S103), calculating the time-varying characteristics of a two-dimensional distribution showing the relationship between multiple brain locations and the magnitude of the preprocessed signals at those locations (S104) based on the preprocessed signals, which are the brain activity signals after preprocessing, and determining the subject's state based on the calculated characteristics (S105).

Description

Technical Field

[0001] The present disclosure relates to a state discrimination method and a discrimination implementation device.

Background Art

[0002] Conventionally, before switching between manual driving and automatic driving, a device has been proposed that identifies the degree of driver anxiety based on the detection results of a brain activity sensor and determines whether to permit or prohibit the switching between manual driving and automatic driving based on the identified degree of anxiety (see, for example, Patent Document 1). In the device described in Patent Document 1, the distribution of brain activation sites is measured based on the detection results of the brain activity sensor, and the degree of driver anxiety is identified based on the measured distribution of brain activation sites.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, for example, when the driver's anxiety level (state) changes from state A to state B, due to the characteristics of cerebral blood flow, it takes time for the distribution of brain activation sites to change from the distribution indicating state A to the distribution indicating state B. Therefore, in the device described in Patent Document 1, there is a delay in discriminating the driver's state, and it becomes difficult to discriminate the driver's state in real time. An object of the present disclosure is to provide a state discrimination method and a discrimination implementation device capable of highly real-time discrimination of a subject's state.

Means for Solving the Problems

[0005] A state determination method according to one aspect of the present disclosure comprises the steps of: measuring brain activity signals at multiple locations in the subject's brain; pre-processing the measured brain activity signals; calculating the time-varying characteristics of a two-dimensional distribution showing the relationship between multiple locations in the brain and the magnitude of the pre-processed signals at those locations, based on the pre-processed signals, which are pre-processed brain activity signals for a predetermined period of time; and determining the state of the subject based on the calculated characteristics.

[0006] Furthermore, a discrimination implementer according to one aspect of the present disclosure comprises: a brain activity detection device that measures brain activity signals at multiple locations in the subject's brain; and an information processing device that preprocesses the brain activity signals measured by the brain activity detection device, calculates the characteristics of the time evolution of a two-dimensional distribution showing the relationship between multiple locations in the brain and the magnitude of the preprocessed signals at those locations based on the preprocessed signals which are preprocessed brain activity signals for a predetermined period of time, and discriminates the subject's state based on the calculated characteristics. [Effects of the Invention]

[0007] According to this disclosure, it is possible to provide a state determination method and determination device that can determine the state of a subject in real time. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows the overall configuration of the discrimination implementation device according to the embodiment. [Figure 2] This diagram shows the various functions that an information processing device can perform. [Figure 3] This figure shows brain activity signals before and after moving average processing. [Figure 4] This figure shows brain activity signals before and after first-order lag filtering. [Figure 5] This figure shows brain activity signals before and after difference processing. [Figure 6] This diagram shows how to obtain a two-dimensional distribution. [Figure 7] This figure shows a method for calculating the characteristics of the time evolution of a two-dimensional distribution. [Figure 8]This figure shows the BOLD signal when brain cell activation occurs once. [Figure 9] This figure shows the BOLD signal when brain cell activation occurs multiple times. [Figure 10] This is a flowchart showing the overall flow of the state determination method for the embodiment. [Figure 11] This figure shows the experimental results of the state determination method of the embodiment and the state determination method of the comparative example. [Modes for carrying out the invention]

[0009] The embodiments of this invention will be described in detail below with reference to the drawings. Note that the drawings are schematic and may differ from actual ones. Furthermore, the embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0010] Figure 1 shows the overall configuration of the discrimination device 1 according to this embodiment. The discrimination device 1 is a device that discriminates the state of a subject based on data obtained from the subject. Examples of subjects include people who drive or operate moving objects such as cars, trains, and airplanes. Examples of the subject's state include arousal level, immersion level, and anxiety level. As shown in Figure 1, the discrimination device 1 comprises a brain activity detection device 2 and an information processing device 3.

[0011] Brain activity detection device 2 is a device for detecting the brain activity of a subject. Brain activity detection device 2 comprises a brain activity signal measurement unit 4 and a brain activity signal extraction unit 5. The brain activity signal measurement unit 4 is attached to the subject's head and measures brain activity signals that indicate the activity level of various parts of the subject's brain. One method for measuring brain activity signals is to detect the activity of multiple regions of interest (ROIs) in the subject's brain and generate brain activity signals that indicate the activity levels of these regions. Examples of brain activity signals include fMRI data, fNIRS data, EEG (ElectroEncephalography) data, and MEG (MagnetoEncephalography) data. In this embodiment, the case where the BOLD (Blood Oxygenation Level Dependent) signal obtained by fMRI is used as the brain activity signal will be described. The BOLD signal is a signal generated by changes in cerebral blood flow (a decrease in deoxygenated hemoglobin in the bloodstream). Brain activity signals are continuously measured while the brain activity signal measurement unit 4 is attached to the subject's head. The measurement results (brain activity signals) are sequentially output to the brain activity signal extraction unit 5.

[0012] The brain activity signal extraction unit 5 extracts brain activity signals to be used to determine the state of the subject. As a method for extracting brain activity signals, for example, a method can be adopted in which brain activity signals of a predetermined fixed time T1 minutes (e.g., 5 seconds) are sequentially extracted from the brain activity signals measured by the brain activity signal measurement unit 4. The predetermined fixed time T1 is, for example, the current brain activity signal (hereinafter also called the "brain activity signal to be processed") and past brain activity signals from the present to the fixed time T1 before. The interval for extracting brain activity signals of the fixed time T1 minutes is a predetermined time (e.g., 1 second). The extraction results (brain activity signals) are sequentially output to the preprocessing unit 6.

[0013] The information processing device 3 includes a processor 3a and peripheral components such as a storage device 3b that stores a computer program and the like. As the processor 3a, for example, a CPU (Central Processing Unit) or an MPU can be adopted. Also, as the storage device 3b, for example, a semiconductor storage device, a magnetic storage device, or an optical storage device can be adopted. The storage device 3b may include memories such as registers, cache memories, a ROM and a RAM used as a main storage device. Each function of the information processing device 3 described below is realized, for example, when the processor 3a executes a computer program stored in the storage device 3b.

[0014] Next, each function of the information processing device 3 will be described in detail. As shown in FIG. 2, the information processing device 3 realizes the functions of a preprocessing unit 6, a feature extraction unit 7, and a discrimination execution unit 8. FIG. 2 is a diagram showing each function realized by the information processing device 3. The preprocessing unit 6 sequentially performs preprocessing in real time on each of the brain activity signals for a certain period of time T1 extracted by the brain activity signal extraction unit 5 (that is, the brain activity signals at a plurality of positions in the subject's brain). The preprocessing is performed, for example, in order from the oldest brain activity signal. Also, for brain activity signals at the same timing, it is performed simultaneously. As the preprocessing, for example, removal of noise components and removal of trend components are performed. The preprocessing unit 6 includes a noise removal unit 9 and a trend removal unit 10.

[0015] The noise removal unit 9 performs real-time removal of noise components for each of the brain activity signals for a certain period T1 extracted by the brain activity signal extraction unit 5. The removal of noise components is performed, for example, in order from the oldest brain activity signal. Also, for brain activity signals at the same timing, it is performed simultaneously. As a method for removing noise components, for example, as shown in FIG. 3, a method of performing a moving average process on the brain activity signal can be adopted. FIG. 3 is a diagram showing the brain activity signal before and after the moving average process. As the moving average process, for example, a backward moving average process using the current brain activity signal (the brain activity signal to be processed) and the brain activity signal before the brain activity signal to be processed can be adopted. Examples of the backward moving average process include a simple moving average process, a weighted moving average process, an exponential moving average process, a modified moving average process, a triangular moving average process, a restricted load moving average process, and a cumulative moving average process. For example, when adopting the simple moving average process, the brain activity signal y[i] after the moving average process is calculated according to the following formula (1) based on the brain activity signal x[i] to be processed, the brain activity signal x[i-1]... x[i-k] before the brain activity signal x[i], and the window size n.

[0016]

Equation

[0017] In addition to the method using the moving average process, as a method for removing noise components, for example, as shown in FIG. 4, a method of performing a first-order delay filter process on the brain activity signal can be adopted. FIG. 4 is a diagram showing the brain activity signal before and after the first-order delay filter process. The brain activity signal y[i] after the first-order delay filter process is calculated according to the following formula (2) based on, for example, the brain activity signal x[i] to be processed, the brain activity signal y[i] after the first-order delay filter process one before, the sampling period T, and the time constant τ. In FIG. 4, the dashed line shows the brain activity signal when the noise component is removed by a non-real-time data processing method. According to FIG. 4, it can be seen that the brain activity signal after the first-order delay filter process is a signal similar to the brain activity signal after the above data processing. y[i]=τ / (T+τ)*y[i-1]+T / (T+τ)*x[i] …(2)

[0018] The trend removal unit 10 removes trend components in real time from each brain activity signal (hereinafter also referred to as "de-noised signal") from which the noise removal unit 9 has removed noise components. Trend component removal is performed sequentially, for example, starting with the signals from which noise components have been removed. It is also performed simultaneously on signals from which noise components have been removed at the same time. As a method for removing trend components, for example, a method of performing difference processing on the de-noised signal can be employed, as shown in Figure 5. Figure 5 is a diagram showing brain activity signals before and after difference processing. In difference processing, the difference between the de-noised signal to be processed and the de-noised signal from a predetermined time T2 earlier is calculated. The predetermined time T2 can be arbitrarily set, for example, from 5 seconds to 60 seconds. When using a BOLD signal as the brain activity signal, 30 seconds is particularly preferable from the viewpoint of discrimination accuracy. The brain activity signal z[i] after difference processing is calculated, for example, based on the denoised signal y[i] to be processed and the denoised signal y[im] from a predetermined time T2 prior, according to equation (3) below. In Figure 5, the dashed line shows the brain activity signal when the trend component is removed using a non-real-time data processing method. As can be seen from Figure 5, the brain activity signal after difference processing is similar to the brain activity signal after the above data processing. z[i]=y[i]-y[im] …(3)

[0019] The feature extraction unit 7 calculates the characteristics of the time evolution of a two-dimensional distribution 12 (see Figure 6) that shows the relationship between multiple brain locations and the magnitude of the preprocessed signal at those locations, based on the brain activity signal after preprocessing by the preprocessing unit 6 (hereinafter also referred to as the "preprocessed signal"). In calculating the characteristics of the time evolution of the two-dimensional distribution 12, the feature extraction unit 7 first acquires a number of two-dimensional distributions 12 for a certain period of time T1 minutes based on the preprocessed signal 11 (preprocessed signal of multiple brain locations of the subject) generated by the trend removal unit 10 for a certain period of time T1 minutes, as shown in Figure 6. In Figure 6, an example is shown in which the two-dimensional distribution 12 is used in which, when the brain is viewed from a predetermined direction (for example, from above), each part of the brain is divided into multiple rectangular regions arranged in a two-dimensional array, and a distribution showing the relationship between the location of each divided region and the magnitude of the preprocessed signal at that brain region is used. That is, the "multiple brain locations" used in the two-dimensional distribution 12 do not have to be the same as the "multiple brain locations" where the brain activity signal measurement unit 4 measures brain activity signals. Here, pre-processed signals at locations where brain activity signals are not measured by the brain activity signal measurement unit 4 are interpolated using pre-processed signals generated by the pre-processing unit 6. Figure 6 shows the method for acquiring the two-dimensional distribution 12.

[0020] Next, as shown in Figure 7, the feature extraction unit 7 calculates the features of the time evolution of the two-dimensional distribution 12 by inputting a number of two-dimensional distributions 12 for a fixed time T1 minute into a trained model 13 that calculates the features of the time evolution of the two-dimensional distribution 12 based on information of a fixed number of two-dimensional distributions 12 arranged in the time direction for a fixed time T1 minute. Figure 7 is a diagram showing the method for calculating the features of the time evolution of the two-dimensional distribution 12. For example, a 3D-CNN (3D Conventional Neural Network) can be used as the trained model 13. As shown in Figure 7, the 3D-CNN is composed of multiple convolutional layers 14 that perform convolution calculations in the vertical axis, horizontal axis, and time axis directions, and pooling layers 15 that realize invariance to minute positional shifts of features, arranged alternately. The convolutional layers 14 and pooling layers 15 calculate the features (feature variables) of the time evolution of a fixed number of two-dimensional distributions 12 for a fixed time T1 minute by calculations using the nodes of each layer. In this embodiment, an example in which the trained model 13 is composed of a 3D-CNN is shown, but other configurations can also be used. For example, RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory) can be used as the pre-trained model 13. As shown in Figure 8, the BOLD signal fluctuates in the order of decrease, increase, decrease when brain cells are activated. Furthermore, if the subject's state changes, brain cell activation occurs multiple times in succession. Therefore, as shown in Figure 9, the overall BOLD signal becomes a signal in which the above-mentioned decrease, increase, decrease signals are superimposed, resulting in oscillations with small amplitude and high frequency. Consequently, the features extracted by 3D-CNN include these oscillation characteristics.

[0021] The discrimination unit 8 discriminates the state of the subject based on the time-varying characteristics of the two-dimensional distribution 12 extracted by the feature extraction unit 7. For example, as shown in Figure 7, if the trained model 13 of the feature extraction unit 7 is composed of a 3D-CNN, the discrimination unit 8 can be composed of the output layer 16 of that 3D-CNN. The output layer 16 converts the feature variables input from the convolutional layer 14 and the pooling layer 15 into probabilities using the softmax function, and outputs from each node the probability that the number of two-dimensional distributions 12 over a certain time T1 corresponds to each pre-set class (pre-set subject state). As pre-set subject states, for example, if the subject state is "awareness level", awareness levels 1 to 4 can be adopted. When using awareness levels 1 to 4, the output layer 16 outputs information such as the probability of corresponding to awareness level 1 (10%), awareness level 2 (80%), awareness level 3 (7%), and awareness level 4 (3%) as the subject discrimination result. Furthermore, when using a pre-trained model 13 (such as a 3D-CNN) for both extracting time-varying features of the two-dimensional distribution 12 and determining the state of the subject, the training of the pre-trained model 13 is performed using training data that includes a certain number of two-dimensional distributions 12 over a fixed time T1 and the state of the subject.

[0022] (operation) Next, we will explain the process for determining the subject's condition. First, brain activity signals are measured at multiple locations in the subject's brain (step S101 in Figure 10). Brain activity signals are measured by the brain activity signal measurement unit 4 and are continuously measured while the brain activity signal measurement unit 4 is attached to the subject's head. Figure 10 is a flowchart showing the overall flow of the state determination method in this embodiment. During the measurement of brain activity signals, the brain activity signal extraction unit 5 sequentially extracts brain activity signals for a fixed time T1 minutes (for example, 5 seconds) used to determine the subject's state. Specifically, it extracts brain activity signals for a fixed time T1 minutes from the brain activity signals measured by the brain activity signal measurement unit 4 (step S102 in Figure 10). For example, the brain activity signals for a fixed time T1 minute include the current brain activity signal and past brain activity signals from the present to a fixed time T1 minutes prior. The extracted brain activity signals are output to the preprocessing unit 6.

[0023] Furthermore, when the brain activity signal extraction unit 5 outputs brain activity signals for a certain period of time T1 minutes, the preprocessing unit 6 performs real-time preprocessing (removal of noise components, removal of trend components) on each output brain activity signal (step S103 in Figure 10). Preprocessing is performed, for example, starting with the oldest brain activity signals. The preprocessed brain activity signals (preprocessed signals) are output to the feature extraction unit 7. When the preprocessed signals are output from the preprocessing unit 6, the feature extraction unit 7 generates a two-dimensional distribution 12 (see Figure 6) of the number of elements for a certain period of time T1 minutes based on the output preprocessed signals (preprocessed signals of multiple locations in the subject's brain), and calculates the characteristics of the time change of the generated two-dimensional distribution 12 (step S104 in Figure 10). Subsequently, based on the characteristics of the time change of the extracted two-dimensional distribution 12 of the number of elements for a certain period of time T1 minutes, the discrimination unit 8 discriminates the state of the subject (step S105 in Figure 10). The discrimination result, for example, provides the probability of a subject falling into a pre-defined state (e.g., arousal level 1-4). Once the determination of the subject's state (step S105 in Figure 10) is complete, the process returns to step S102, and the flow from steps S102 to S105 is repeated, starting with the extraction of brain activity signals for a certain period of time T1 minutes, thereby repeatedly determining the subject's state at predetermined intervals (e.g., every second).

[0024] (Effects of this embodiment) (1) As a comparative example, consider a configuration in which the subject's state is determined from the two-dimensional distribution 12 by referring to a map showing the correspondence between the two-dimensional distribution 12 and the subject's state. Here, after the subject's state changes from state A to state B, time is required for the two-dimensional distribution 12 to change from the distribution representing state A to the distribution representing state B. Therefore, in the comparative example, there is a delay in determining the subject's state, making it difficult to determine the subject's state in real time.

[0025] In contrast, this embodiment includes the steps of: measuring brain activity signals at multiple locations in the subject's brain; pre-processing the measured brain activity signals; calculating the characteristics of the time evolution of a two-dimensional distribution showing the relationship between multiple brain locations and the magnitude of the pre-processed signals at those locations, based on the pre-processed brain activity signals (pre-processed signals) after a predetermined time T1 minutes; and determining the subject's state based on the calculated characteristics. That is, when the subject's state changes from state A to state B, the subject's state is determined using the "characteristics of the time evolution of the two-dimensional distribution" that appear during the time when the two-dimensional distribution 12 changes from a distribution showing state A to a distribution showing state B. Therefore, the subject's state can be determined before the two-dimensional distribution 12 changes to a distribution showing state B, enabling highly real-time determination of the subject's state.

[0026] As shown in Figure 11, experiments were conducted on the state determination method of this embodiment and the state determination method of the comparative example. Figure 11 shows the experimental results of the state determination method of this embodiment and the state determination method of the comparative example. In this experiment, subjects were asked to start a single task (autonomous driving monitoring) or a multitasking task (autonomous driving monitoring + spatial recognition / mental arithmetic), and the time required to determine that the subject's state had changed to the single-tasking state or the multitasking state with a probability of 70% or higher was investigated. The constant time T1 was set to 5 seconds. The subject's state was determined every second. In the comparative example, offline processing was used as preprocessing. As a result of this experiment, it was confirmed that the state determination method of this embodiment shortens the time required to determine the subject's state compared to the state determination method of the comparative example, in both single-tasking and multitasking cases.

[0027] (2) In this embodiment, the step of preprocessing the brain activity signal (S103) removes noise components and trend components from the brain activity signal in real time. Therefore, since the preprocessing is performed in real time, the real-time capability can be further enhanced.

[0028] (3) In this embodiment, a moving average process is performed on the brain activity signal to remove noise components. By using brain activity signals prior to the brain activity signal being processed, noise components can be removed from the brain activity signal being processed, and noise components can be removed in real time.

[0029] (4) In this embodiment, a first-order lag filter is applied to the brain activity signal to remove noise components. By using the output of past first-order lag filters, noise components can be removed from the brain activity signal being processed, and noise components can be removed in real time.

[0030] (5) In this embodiment, the trend component is removed by performing a difference operation on the brain activity signal (de-noised signal). In the difference operation, the difference between the brain activity signal to be processed and the brain activity signal from a predetermined time T2 earlier than the target brain activity signal is calculated. This makes it possible to remove the trend component of the brain activity signal (de-noised signal) in real time.

[0031] (6) In this embodiment, the step (S104) of calculating the characteristics of the time change of the two-dimensional distribution 12 is performed by inputting the two-dimensional distribution 12 for a certain period of time T1 minutes into a trained model 13 that calculates the characteristics of the time change of the two-dimensional distribution 12 based on information in which the two-dimensional distribution 12 is arranged in a certain number of minutes in the time direction, thereby calculating the characteristics of the time change of the two-dimensional distribution 12. This makes it possible to calculate the characteristics of the time change of the two-dimensional distribution 12 more appropriately, improve the accuracy of discriminating the state of the subject, and make it possible to discriminate the state of the subject at an earlier timing.

[0032] (modified version) In this embodiment, an example is shown in which brain activity signals are extracted before preprocessing of the brain activity signals, but other configurations can also be adopted. For example, the brain activity signals may be extracted after preprocessing of the brain activity signals. In this case, the brain measurement signals measured by the brain activity signal measurement unit 4 are sequentially preprocessed, and the brain activity signals used for state discrimination are extracted from the preprocessed brain activity signals. [Explanation of Symbols]

[0033] 1…Discrimination device, 2…Brain activity detection device, 3…Information processing device, 3a…Processor, 3b…Memory device, 4…Brain activity signal measurement unit, 5…Brain activity signal extraction unit, 6…Preprocessing unit, 7…Feature extraction unit, 8…Discrimination unit, 9…Noise removal unit, 10…Trend removal unit, 11…Preprocessed signal, 12…Two-dimensional distribution, 13…Trained model, 14…Convolutional layer, 15…Pooling layer, 16…Output layer

Claims

1. The steps include measuring brain activity signals at multiple locations in the subject's brain, The steps include preprocessing the measured brain activity signal, A step of calculating the time-varying characteristics of a two-dimensional distribution showing the relationship between multiple brain locations and the magnitude of the pre-processed signal at those locations, based on the pre-processed signal, which is the brain activity signal after pre-processing for a predetermined period of time; The system comprises the step of determining the state of the subject based on the calculated characteristics. A method for determining the state.

2. The step of preprocessing the brain activity signal involves removing noise components and trend components from the brain activity signal in real time. The method for determining a state according to claim 1.

3. In the removal of the noise component, a moving average process is performed on the brain activity signal. The state determination method according to claim 2.

4. In the removal of the noise component, a first-order lag filter is applied to the brain activity signal. The state determination method according to claim 2.

5. In removing the trend component, differential processing is performed on the brain activity signal. In the aforementioned difference processing, the difference between the brain activity signal to be processed and the brain activity signal from a predetermined time prior to the current brain activity signal is calculated. The state determination method according to claim 2.

6. The step of calculating the time-dependent characteristics of the two-dimensional distribution involves inputting the two-dimensional distribution for a certain period of time into a trained model that calculates the time-dependent characteristics of the two-dimensional distribution based on information arranged in a certain number of locations over a certain period of time in the time direction. The method for determining a state according to claim 1.

7. A brain activity detection device that measures brain activity signals from multiple locations in the subject's brain, The information processing device includes: preprocessing the brain activity signal measured by the brain activity detection device; calculating the characteristics of the time evolution of a two-dimensional distribution showing the relationship between multiple brain locations and the magnitude of the preprocessed signal at those locations, based on the preprocessed signal which is the brain activity signal after preprocessing for a predetermined period of time; and determining the state of the subject based on the calculated characteristics. Discrimination device.

Images