Perceptualization device, perceptualization method, and perceptualization program

Abstract

A visualization device (10) performs a main component analysis of the representational similarity matrix of brain wave data of a human on a predetermined time basis to extract the internal state of the human who shows the brain wave data as a parameter value that is used for the expression by RoV. The visualization device (10) also estimates parameter values of RoV respectively in the unit of frames at the above-mentioned predetermined times using the extracted parameter value. Subsequently, the visualization device (10) draws RoV's that express the internal state of the human using the extracted parameter value of RoV and the estimated parameter values of RoV in the unit of frames. Subsequently, the visualization device (10) presents, on a display, a series of RoVs that have been drawn.

Description

Technical Field

[0001] The present invention relates to a visualization device, a visualization method, and a visualization program for making a human's internal state perceivable.

Background Art

[0002] Estimation of a human's internal state, such as what the human is thinking and feeling and what state the human is in, is important in human communication. Here, for example, conventionally, a technique has been proposed for converting a recognition result obtained from brain waves into language information in order to estimate a human's internal state.

[0003] Also, conventionally, a technique has been proposed for estimating a human's emotional state from brain waves and visually feeding back the estimation result of the human's emotional state through an HMD (head-mounted display) (see Non-Patent Document 1). In this technique, a human's emotional state is evaluated in two dimensions of Arousal (arousal level) and Valence (emotional valence), and the evaluation result is expressed by a fractal (see FIG. 1).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] ​However, the technology that converts the recognition results obtained from EEG into linguistic information significantly reduces the amount of information from the original human internal state. Also, the technology that evaluates a human emotional state in two dimensions, arousal and valence, is insufficient for visualizing a human internal state in detail. Therefore, the object of the present invention is to solve the above-mentioned problems and visualize a human internal state in detail. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention is characterized by comprising a data acquisition unit that acquires a series of electroencephalogram data, a parameter extraction unit that, at predetermined time intervals, performs dimensional compression on feature quantities extracted from the electroencephalogram data and extracts the feature quantity values ​​obtained by the dimensional compression as parameter values ​​to be used for perceptualizing the internal state of a person indicated by the electroencephalogram data, a perceptualization unit that uses the extracted parameter values ​​to perceptualize the internal state of the person, and an output processing unit that outputs the perceptualized information. [Effects of the Invention]

[0007] According to the present invention, the internal state of a human can be visualized in detail. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining the prior art. [Figure 2] FIG. 2 is a diagram for explaining the Rose of Venus (RoV). [Figure 3] FIG. 3 is a diagram for explaining an overview of the visualization device. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a visualization device. [Figure 5] FIG. 5 is a flowchart showing an example of a procedure by which the visualization device renders a RoV. [Figure 6] FIG. 6 is a diagram illustrating a specific example of the process shown in FIG. [Figure 7]FIG. 7 is a flowchart illustrating in detail the post parameter extraction process shown in S5 of FIG. [Figure 8] FIG. 8 is a diagram showing an example of the PSD of electroencephalogram data. [Figure 9] FIG. 9 is a diagram showing an example of a representation similarity matrix (RSM) of the power spectral density (PSD) of electroencephalogram data. [Figure 10] FIG. 10 is a diagram for explaining parameters used by the visualization device to draw the RoV. [Figure 11] Figure 11 shows an example of RoV drawn from EEG data when a person talks about and listens to an emotional episode. [Figure 12] FIG. 12 is a diagram for explaining an outline of the perceptualization device of the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of a perceptualization device according to the second embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of a drawing parameter extraction process according to the second embodiment. [Figure 15] FIG. 15 is a diagram for explaining an outline of the perceptualization device of the third embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of the configuration of a perceptualization device according to the third embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of a process for extracting perceptualization parameters in the third embodiment. [Figure 18] FIG. 18 is a diagram for explaining an outline of the perceptualization device of the fourth embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of the configuration of a perceptualization device according to the fourth embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of a drawing parameter extraction process according to the fourth embodiment. [Figure 21] FIG. 21 is a diagram illustrating an example of the configuration of a perceptualization device according to the fifth embodiment. [Figure 22]FIG. 22 is a flowchart illustrating an example of processing executed by the perceptualization device of the fifth embodiment. [Figure 23] FIG. 23 is a diagram showing a computer that executes a perceptualization program. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0010] [First embodiment] The perceptualization device (visualization device) of this embodiment generates an image that expresses a human's internal state using geometric figures based on human electroencephalogram data, and displays the image on a display. For example, the visualization device generates an image that expresses a human's internal state using Rose of Venus (RoV), and displays the image on a display. The RoV will be described below.

[0011] [Rose of Venus (RoV)] RoV is a five-petal flower figure (see Figure 2) drawn by connecting the position of Venus and the position of the Earth with a line at predetermined intervals, utilizing the difference in the orbital periods of Venus and the Earth (for details, see reference 1 at the URL below).

[0012] Document 1: https: / / jp.mathworks.com / matlabcentral / fileexchange / 102885-the-rose-of-venus-matlab / ?s_tid=LandingPageTabfx

[0013] As shown in Figure 2, RoV is expressed using seven parameters: 1st dimension: size (Earth's orbital radius), 2nd dimension: difference in orbital radius, 3rd dimension: orbital period (Earth's orbital period), 4th to 6th dimensions: RGB (Red Green Blue), 7th dimension: plot interval.

[0014] The visualization device generates an image representing a person's internal state using RoV from data from the human brain at predetermined intervals and displays it on a display.

[0015] [overview] Next, an overview of the visualization device 10 will be described with reference to FIG. 3. When the visualization device 10 acquires human electroencephalogram data from an electroencephalograph, it calculates RoV parameter values ​​at predetermined time intervals (e.g., every u seconds) based on the acquired electroencephalogram data. Furthermore, based on the calculation results of the RoV parameter values ​​at the predetermined time intervals, the visualization device 10 interpolates (calculates) RoV parameter values ​​for each frame at the predetermined time interval. The visualization device 10 then draws a series of RoVs using the calculated RoV parameter values ​​and the interpolated RoV parameter values. Thereafter, the visualization device 10 displays the drawn series of RoVs on a display. For example, the visualization device 10 displays the drawn series of RoVs as an animation on a display.

[0016] By performing the above processing, the visualization device 10 can display the human's internal state as feedback from the RoV on the display in almost real time. For example, an animation is displayed on the display, in which changes in the human's emotions are expressed as changes in the color, shape, and size of the RoV.

[0017] Furthermore, the visualization device 10 interpolates the parameter values ​​of the RoV for each frame at the predetermined time based on the calculation results of the parameter values ​​of the RoV for each predetermined time, thereby reducing the calculation load when drawing the RoV from EEG data. This makes it possible to reduce processing delays, for example, when the visualization device 10 feeds back the internal state of a person in almost real time using the RoV.

[0018] [Configuration example] Next, a configuration example of the visualization device 10 will be described with reference to Fig. 4. The visualization device 10 includes a communication unit 11, an input / output unit 12, a storage unit 13, and a control unit 14. Note that, in the following, a case will be described where the geometric figure that the visualization device 10 draws based on parameters calculated from electroencephalogram data is an RoV, but the present invention is not limited to this.

[0019] The communication unit 11 is realized by a NIC (Network Interface Card) or the like, and controls communication with an external device such as an electroencephalograph via a network.

[0020] The input / output unit 12 is an interface that controls the input and output of various data. For example, the input / output unit 12 outputs the RoV drawn by the control unit 14 to a display.

[0021] The storage unit 13 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 13 stores data and programs referenced when the control unit 14 executes various processes. For example, the storage unit 13 stores electroencephalogram data acquired from an electroencephalograph, RoV parameter values ​​calculated from the electroencephalogram data, etc.

[0022] The control unit 14 is responsible for overall control of the visualization device 10. The control unit 14 realizes the functions of a data acquisition unit 141, a parameter extraction unit 142, a parameter estimation unit 143, a drawing unit (perceptualization unit) 144, and a display processing unit (output processing unit) 145, for example, by having a CPU (Central Processing Unit) execute a program stored in the storage unit 13.

[0023] The data acquisition unit 141 acquires a series of electroencephalogram data via the communication unit 11. The parameter extraction unit 142 extracts (calculates) parameter values ​​for drawing the RoV from the electroencephalogram data acquired by the data acquisition unit 141.

[0024] For example, first, the parameter extraction unit 142 calculates the power spectral density of a series of electroencephalogram data at predetermined time intervals. Next, the parameter extraction unit 142 calculates a representation similarity matrix (RSM) using the calculated power spectral density of the electroencephalogram data. The RSM is a matrix that indicates the similarity of brain activity (see FIG. 9). This RSM can be found by calculating a correlation matrix of the power spectral density of the electroencephalogram data.

[0025] Next, the parameter extraction unit 142 performs principal component analysis on the RSM. Then, the parameter extraction unit 142 extracts the values ​​of a predetermined number of principal components (for example, seven) obtained by the principal component analysis of the RSM as parameter values ​​to be used when drawing the RoV.

[0026] The parameter estimation unit 143 uses the parameter values ​​extracted by the parameter extraction unit 142 to estimate the parameter values ​​of the RoV in frame units at the above-mentioned predetermined time.

[0027] For example, the parameter estimation unit 143 interpolates the parameter values ​​of the RoV on a frame-by-frame basis at the predetermined time using the parameter values ​​of the RoV extracted by the parameter extraction unit 142 (= the parameter values ​​of the RoV calculated from the electroencephalogram data for each predetermined time) and the number of frames of the RoV to be drawn at the predetermined time. Then, the parameter estimation unit 143 sets the interpolated parameter values ​​as the estimated values ​​of the parameter values ​​of the RoV on a frame-by-frame basis at the predetermined time.

[0028] The drawing unit 144 draws a series of RoVs using the parameter values ​​of the RoV for each specified time extracted by the parameter extraction unit 142 and the parameter values ​​of the RoV for each frame at the specified time estimated by the parameter estimation unit 143.

[0029] The display processing unit 145 displays the series of RoVs drawn by the drawing unit 144 on a display via the input / output unit 12. For example, the display processing unit 145 displays the series of RoVs drawn by the drawing unit 144 as an animation.

[0030] According to such a visualization device 10, it is possible to display on a display an image that expresses the internal state of a human being using a series of RoVs.

[0031] [Example of processing procedure] Next, an example of the procedure by which the visualization device 10 renders the RoV will be described with reference to Fig. 5. As mentioned above, the parameters used to render the RoV are seven-dimensional, and the parameter values ​​of the RoV calculated from the EEG data are updated every u seconds.

[0032] In the following description, fps is the frame rate at which the visualization device 10 draws the RoV. t is a parameter for counting drawing frames. The δ parameter is a parameter value per frame for u seconds estimated (interpolated) by the visualization device 10.

[0033] First, the visualization device 10 initializes each parameter (S1). Then, the parameter extraction unit 142 determines whether t is a multiple of fps*u (S2). In other words, the parameter extraction unit 142 determines whether it is time to update the parameter value.

[0034] In S2, if the parameter extraction unit 142 determines that t is a multiple of fps*u (Yes in S2), it calculates a parameter value from the electroencephalogram data for the most recent u seconds, and defines the parameter value as a Pre parameter (S3: Parameter update).

[0035] After S3, parameter extraction unit 142 reads the electroencephalogram data acquired by data acquisition unit 141 (S4). Then, parameter extraction unit 142 extracts post-parameters from the read electroencephalogram data (S5). Thereafter, parameter estimation unit 143 estimates parameter values ​​(δ parameters) in frame units for u seconds by dividing the difference between the post-parameters extracted in S5 and the pre-parameters extracted in S3 by the number of frames for u seconds (u*fps) (S6: δ parameter estimation).

[0036] After S6, the drawing unit 144 draws the RoV using the Pre parameters defined in S3 (S7). Then, the control unit 14 increments t by 1 (S8) and returns to S2.

[0037] On the other hand, if the parameter extraction unit 142 determines in S2 that t is not a multiple of fps*u (No in S2), that is, if it determines that it is not time to update the parameter value, it draws the RoV using the Pre parameter defined in S3 and the δ parameter estimated in S6 (S9). Then, it proceeds to S8.

[0038] By performing the above processing, the visualization device 10 extracts parameter values ​​of the RoV from the EEG data every u seconds and estimates the parameter values ​​on a frame-by-frame basis for those u seconds.The visualization device 10 can then draw a series of RoVs using the extracted parameter values ​​of the RoV and the estimated parameter values ​​on a frame-by-frame basis.

[0039] [Example of processing procedure] Next, a specific example of the processing shown in Fig. 5 will be described with reference to Fig. 6. The update interval of the parameter values ​​of the RoV is set to every 4 seconds (u=4). The length of the electroencephalogram data required to calculate the parameter values ​​of the RoV is set to 4 seconds. The frame rate when the visualization device 10 draws the RoV is set to 30 fps.

[0040] Here, the processing from 4 seconds to 12 seconds after the start will be described as an example. First, the parameter extraction unit 142 of the visualization device 10 reaches the timing for the first parameter update at 4 seconds after the start, calculates a parameter value from the electroencephalogram data at 0 seconds after the start (the 4 seconds immediately before the start 0 seconds, i.e., from -4 seconds to 0 seconds), and defines this parameter value as a Pre-parameter ((1): Calculate Pre-parameter).

[0041] Thereafter, the parameter extraction unit 142 calculates a parameter value from the electroencephalogram data for the first 4 seconds (0 to 4 seconds) and sets this parameter value as a post parameter ((2): calculate post parameter). Thereafter, the parameter estimation unit 143 estimates (calculates) the amount of change in the parameter (δ parameter) per frame by dividing the difference between the post parameter calculated in (2) and the pre parameter defined in (1) by the number of frames (120) ((3)).

[0042] Then, the drawing unit 144 draws the RoV of the 120th frame (4 seconds x 30 fps) from the beginning using the Pre parameter of (1) ((4)). The above is the process at 4 seconds from the start.

[0043] Note that the period from 4 seconds after the start until just before 8 seconds after the start (from frame 121 to frame 239) is not the timing for updating parameters, so the drawing unit 144 draws the RoV using the Pre parameters defined in (1) and the δ parameters calculated in (3) ((5)).

[0044] Thereafter, the processing at 8 seconds after the start is as follows: At 8 seconds after the start, the parameter extraction unit 142 reaches the timing for the second parameter update, and defines the parameters calculated from the electroencephalogram data at 4 seconds after the start calculated in (2) as Pre parameters ((6)).

[0045] Then, the parameter extraction unit 142 calculates a parameter value from the electroencephalogram data for the first 8 seconds (4 to 8 seconds) and sets this parameter value as a post parameter ((7)).The parameter estimation unit 143 then divides the difference between the post parameter calculated in (7) and the pre parameter defined in (6) by the number of frames (120) to calculate the amount of change in the parameter (δ parameter) per frame ((8)).

[0046] Then, the drawing unit 144 draws ((9)) the RoV from the beginning, 8 seconds x 30 fps = 240th frame, using the Pre parameters defined in (6).

[0047] Note that the period from 8 seconds after the start until just before 12 seconds after the start (from frame 241 to frame 359) is not the timing for updating parameters, so the drawing unit 144 draws the RoV using the Pre parameters defined in (6) and the δ parameters calculated in (8) ((10)).

[0048] In this way, the visualization device 10 can draw a series of RoVs from the EEG data. Furthermore, since the visualization device 10 interpolates the parameter values ​​of the RoV on a frame-by-frame basis, it is possible to reduce the calculation load when drawing the RoV from a series of EEG data.

[0049] [Extraction of RoV parameters from EEG data] Next, the post parameter extraction process shown in S5 of Fig. 5 will be described in detail with reference to Fig. 7. For example, when the parameter extraction unit 142 receives input of electroencephalogram data (S61), it performs preprocessing of the electroencephalogram data (S62).

[0050] The pre-processing includes, for example, filtering of the electroencephalogram data (1-40 Hz: Butterworth filter, 50 Hz-: Notch filter), removing outliers (noise), and the like.

[0051] Next, the parameter extraction unit 142 calculates the PSD (power spectral density) of the electroencephalogram data preprocessed in S62 and performs normalization (S63: PSD calculation). For example, the parameter extraction unit 142 calculates the PSD for each of the left and right electroencephalogram data (L, R) and the electroencephalogram data (Difference) that is the difference between them (see FIG. 8).

[0052] Returning to the explanation of Figure 7, the parameter extraction unit 142 calculates the RSM (see Figure 9) of the PSD of the electroencephalogram data calculated in S63 (S64). For example, the parameter extraction unit 142 obtains the RSM by calculating a correlation matrix for a predetermined time window (for example, 4 seconds) for the PSD of the electroencephalogram data calculated in S63. Note that the vertical axis of the RSM shown in Figure 9 indicates the PSD of the past electroencephalogram data, and the horizontal axis indicates the PSD of the electroencephalogram data to be processed.

[0053] Returning to the explanation of Fig. 7, the parameter extraction unit 142 performs principal component analysis (PCA) of the RSM calculated in S64 (S65). After that, the parameter extraction unit 142 extracts the first to seventh principal components obtained by the PCA as first- to seventh-dimensional parameters to be used for drawing the RoV (S66: extract parameters).

[0054] [δ parameter estimation process] Next, the estimation process of the delta parameter shown in S6 of Fig. 5 will be described in detail. Calculation of the RoV parameter value from EEG data requires a time window of a predetermined time (for example, 4 seconds). Therefore, the parameter estimation unit 143 estimates the amount of change in the parameter value for each frame in the time window by linear interpolation.

[0055] For example, the parameter estimation unit 143 estimates the amount of change in the parameter value on a frame-by-frame basis by dividing the difference between the post parameter and the pre parameter by the number of frames (u*fps).

[0056] Among the conventional RoV parameters, the plot interval of the RoV line segments (see the seventh dimension in Figure 2) was determined by "how many days apart each line segment should be plotted when connecting the position of Venus and the position of the Earth for n years." In other words, since the RoV was previously drawn using each parameter value when "connecting the position of Venus and the position of the Earth for n years," when performing linear interpolation on a frame-by-frame basis as described above, the position of the RoV line segments will be linearly interpolated.

[0057] As a result, the positions of Venus and Earth on the RoV after linear interpolation will deviate from their orbital periods. Therefore, if you continuously display (animate) RoVs that include linearly interpolated RoVs, the display may appear unnatural.

[0058] Therefore, the parameter estimation unit 143 estimates the plot interval of the RoV's line segments by fixing the number of RoV's line segments to a predetermined number (see FIG. 10). In other words, the parameter estimation unit 143 estimates the positions of Venus and the Earth based on the orbital radii and orientations of Venus and the Earth when the number of RoV's line segments is fixed to a predetermined number. This enables interpolation to be performed so that a smooth display is achieved when the display processing unit 145 continuously displays a series of RoVs drawn by the drawing unit 144.

[0059] [Drawing of RoV] Next, the drawing of the RoV by the drawing unit 144 will be described in detail with reference to Fig. 10. For example, the drawing unit 144 draws the RoV using the value of the first principal component obtained by principal component analysis of the RSM of the electroencephalogram data (and the estimated value of that value for each frame) as the orbital radius of the RoV around the Earth (first-dimensional parameter).

[0060] In addition, the drawing unit 144 draws the RoV using the value of the second principal component obtained by the above-mentioned principal component analysis (and the estimated value of that value for each frame) as the difference in orbital radius of the RoV (second-dimensional parameter).

[0061] In addition, the drawing unit 144 draws the RoV using the value of the third principal component obtained by the above-mentioned principal component analysis (and the estimated value of that value for each frame) as the Earth's orbital period (the third-dimensional parameter) of the RoV.

[0062] Furthermore, the drawing unit 144 draws the RoV using the values ​​of the fourth to sixth principal components (and the estimated values ​​of these values ​​for each frame) obtained by the above-mentioned principal component analysis as RGB (fourth to sixth dimensional parameters) in the RoV.

[0063] Furthermore, the drawing unit 144 draws the RoV using the value of the seventh principal component obtained by the above-mentioned principal component analysis (and the estimated value of that value for each frame) as the plot interval (seventh-dimensional parameter) of the RoV. As described above, the plot interval of the RoV is the plot interval when the number of line segments of the RoV is set to n (a predetermined number).

[0064] Examples of RoVs drawn from EEG data when a person talks about an emotional episode and when they listen to it are shown in Figure 11. The RoVs shown by symbols 1, 3, 5, 7, and 9 in Figure 11 are RoVs drawn from EEG data when a person talks about an emotional episode. The RoVs shown by symbols 2, 4, 6, 8, and 10 in Figure 11 are RoVs drawn from EEG data when a person listens to an emotional episode.

[0065] Such a visualization device 10 can express the internal state of a person in detail. As a result, for example, when a series of RoVs rendered by the visualization device 10 are used for communication between people, mutual understanding that was not possible with conventional communication becomes possible.

[0066] [Second embodiment] Next, we will explain the perceptualization device of the second embodiment. The perceptualization device of the second embodiment is characterized by calculating features that explain the internal state of a person from human electroencephalogram data and extracting (calculating) drawing parameters of geometric figures by performing dimensional compression.

[0067] [overview] An overview of the perceptualization device 10a of the second embodiment will be described with reference to FIG. 12. The perceptualization device 10a acquires human electroencephalogram data from an electroencephalograph, calculates feature amounts from the acquired electroencephalogram data at predetermined time intervals, and extracts (calculates) drawing parameters for a predetermined number of geometric figures by performing dimensional compression. The perceptualization device 10a then draws geometric figures from the calculated drawing parameter values. Thereafter, the perceptualization device 10a displays the series of drawn geometric figures on a display. For example, the perceptualization device 10a displays the series of drawn geometric figures as animation on a display. This allows the state of the human brain to be fed back in almost real time using the geometric figures.

[0068] This perceptualization device 10a performs dimensionality compression on high-dimensional features calculated from electroencephalogram data in order to extract drawing parameters for geometric figures that more accurately represent a person's internal state. Here, for example, dimensionality compression using simple principal component analysis can make it difficult to interpret each principal component, potentially reducing the interpretability of the drawn geometric figure. Therefore, the perceptualization device 10a extracts drawing parameters that enable the drawing of geometric figures with higher interpretability by using, for example, discriminant analysis or independent component analysis.

[0069] [Configuration example] Next, a configuration example of the perceptualization device 10a will be described with reference to Fig. 13. The perceptualization device 10a includes a communication unit 11a, an input / output unit 12a, a storage unit 13a, and a control unit 14a. Note that, below, an example of a method for dimensionally compressing the feature amount calculated from electroencephalogram data by the perceptualization device 10a will be described, but the present invention is not limited to this.

[0070] The communication unit 11a is realized by a NIC or the like, and controls communication with an external device such as an electroencephalograph via a network.

[0071] The input / output unit 12a is an interface that controls the input and output of various data. For example, the input / output unit 12a outputs geometric figures drawn by the control unit 14a to a display.

[0072] The storage unit 13a is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 13a stores data and programs referenced by the control unit 14a when executing various processes. For example, the storage unit 13a stores electroencephalogram data acquired from an electroencephalograph, drawing parameters of geometric figures calculated from the electroencephalogram data, and the like.

[0073] The control unit 14a controls the entire perceptualization device 10a. The control unit 14a realizes the functions of a data acquisition unit 141a, a parameter extraction unit 142a, a drawing unit (perceptualization unit) 144a, and a display processing unit 145a, for example, by the CPU executing a program stored in the storage unit 13a.

[0074] The data acquiring unit 141a acquires a series of electroencephalogram data via the communication unit 11a. The parameter extracting unit 142a extracts (calculates) parameters for drawing a geometric figure from the electroencephalogram data acquired by the data acquiring unit 141a.

[0075] For example, first, the parameter extraction unit 142a calculates the power spectral density of a series of electroencephalogram data at predetermined time intervals. Next, the parameter extraction unit 142a calculates a representation similarity matrix (RSM) using the calculated power spectral density of the electroencephalogram data. The RSM is a matrix that indicates the similarity of brain activity. This RSM can be obtained by calculating the correlation of the power spectral densities of the electroencephalogram data. Next, the parameter extraction unit 142a performs dimensional compression on the RSM and extracts the values ​​of a predetermined number of obtained feature quantities as parameters to be used when drawing a geometric figure.

[0076] The drawing unit 144a uses the values ​​of the drawing parameters of the geometric figures extracted by the parameter extraction unit 142 to draw a series of geometric figures.

[0077] The display processing unit 145a displays the series of geometric figures drawn by the drawing unit 144a on a display via the input / output unit 12. For example, the display processing unit 145a displays the series of geometric figures drawn by the drawing unit 144a as animation.

[0078] According to such a perceptualization device 10a, it is possible to display on a display an image that expresses the internal state of a person using a series of geometric figures.

[0079] [Example of processing procedure] Next, an example of the procedure by which the perceptualization device 10a renders a geometric figure will be described.

[0080] First, the perceptualization device 10a sets an update interval for updating the drawing parameters, and calculates the drawing parameters from the electroencephalogram data of an arbitrary length (time) immediately preceding each parameter update timing.

[0081] Thereafter, the drawing unit 144a draws a geometric figure using the drawing parameters. Note that, in order to draw the geometric figure more smoothly, the drawing unit 144a may estimate complementary parameters that complement the drawing parameters calculated from the electroencephalogram by the parameter extraction unit 142a, and draw the geometric figure using the drawing parameters and the estimated complementary parameters.

[0082] By performing the above processing, the perceptualization device 10a draws a series of geometric figures using drawing parameters calculated (extracted) from the electroencephalogram data.

[0083] [Extraction of drawing parameters from EEG data] Next, the drawing parameter extraction process will be described in detail with reference to Fig. 14. For example, when the parameter extraction unit 142a receives input of electroencephalogram data (S71), it performs preprocessing of the electroencephalogram data (S72). The preprocessing includes, for example, filtering of the electroencephalogram data (1-40 Hz: Butterworth filter, 50 Hz-: Notch filter), removing outliers (noise), etc.

[0084] Next, the parameter extraction unit 142a calculates the PSD (power spectral density) of the electroencephalogram data preprocessed in S72 and performs normalization (S73: PSD calculation).

[0085] Next, the parameter extraction unit 142a calculates the RSM of the PSD of the electroencephalogram data (S74). For example, the parameter extraction unit 142a calculates the correlation between the PSD of the electroencephalogram data calculated in S73 and the PSDs of any multiple pieces of electroencephalogram data previously acquired, thereby obtaining a correlation matrix RSM.

[0086] The parameter extraction unit 142a performs dimensional compression on the RSM calculated in S74 (S75). However, in order for the drawing unit 144a to draw a geometric figure that more appropriately represents a human internal state, the parameter extraction unit 142a needs to perform dimensional compression so as to extract electroencephalogram features specific to a particular internal state.

[0087] Therefore, the parameter extraction unit 142a uses, for example, discriminant analysis or independent component analysis as a dimensionality reduction technique. Alternatively, when using principal component analysis, the parameter extraction unit 142a may use a principal component specific to a specific internal state other than a baseline signal common to all electroencephalograms by excluding the first principal component. Alternatively, the perceptualization device 10a may create a transformation matrix by machine learning or the like, which the parameter extraction unit 142a uses to perform dimensionality reduction. This allows the parameter extraction unit 142a to extract features that result in a geometric figure with higher interpretability.

[0088] Next, the parameter extraction unit 142a extracts the values ​​of the feature amounts obtained by the dimensional compression as a predetermined number of drawing parameters to be used for drawing the geometric figure (S76: Extract drawing parameters).

[0089] The perceptualization device 10a can express a person's internal state in detail using electroencephalogram features specific to a particular internal state. As a result, for example, when a series of geometric figures drawn by the perceptualization device 10a is used in communication between people, mutual understanding that was not possible with conventional communication becomes possible.

[0090] [Third embodiment] Next, a description will be given of a perceptualization device according to a third embodiment. The perceptualization device according to the third embodiment makes the internal state of a person perceptible by sound, vibration, temperature, and the like.

[0091] [overview] An overview of the perceptualization device 10b will be described with reference to Fig. 15. For example, when the perceptualization device 10b acquires human electroencephalogram data from an electroencephalograph, it calculates (extracts) feature amounts from the acquired electroencephalogram data at predetermined time intervals and extracts (calculates) a predetermined number of perceptualization parameters by performing dimensional compression.

[0092] The perceptualization device 10b then makes the human brain state (internal state) perceptible based on the calculated perceptualization parameter values. Thereafter, the perceptualization device 10b feeds back the perceptualized human internal state so that the user can perceive it naturally.

[0093] For example, the perceptualization device 10b provides feedback of the internal state of a person through hearing or touch. Auditory feedback may involve, for example, playing background music-like sounds with varying volume and pitch from a speaker, or ambient music. Haptic feedback may involve, for example, vibrating a device or changing its temperature. This allows feedback of the internal state of the brain in a natural manner in near real time.

[0094] In this way, the perceptualization device 10b feeds back the internal state of the person through hearing and touch. This allows the perceptualization device 10b to reduce the burden on the user when recognizing the internal state of the person. As a result, the perceptualization device 10b can feed back the internal state of the person to the user without interfering with the original communication. Note that the perceptualization device 10b may also feed back the internal state of the person through ambient visual information that blends into the background. This method also allows the perceptualization device 10b to reduce the burden on the user when recognizing the internal state of the person.

[0095] [Configuration example] Next, a configuration example of the perceptualization device 10b will be described with reference to Fig. 16. The perceptualization device 10b includes a communication unit 11b, an input / output unit 12b, a storage unit 13b, and a control unit 14b. Note that, in the following, an example will be described in which the control unit 14b makes an internal state estimated from electroencephalogram data perceptible as acoustic information and feeds it back, but the present invention is not limited to this.

[0096] The communication unit 11b is realized by a NIC or the like, and controls communication with external devices such as an electroencephalograph via a network.

[0097] The input / output unit 12b is an interface that controls input and output of various data. For example, the input / output unit 12b outputs acoustic information generated by the control unit 14b from a speaker.

[0098] The storage unit 13b is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 13b stores data and programs referenced by the control unit 14b when executing various processes. For example, the storage unit 13b stores electroencephalogram data acquired from an electroencephalograph, perceptualization parameters calculated from the electroencephalogram data, and the like.

[0099] The control unit 14b controls the entire perceptualization device 10b. The control unit 14b realizes the functions of a data acquisition unit 141b, a parameter extraction unit 142b, a perceptualization unit 144b, and a feedback unit (output processing unit) 145b, for example, by the CPU executing a program stored in the storage unit 13b.

[0100] The data acquiring unit 141b acquires a series of electroencephalogram data via the communication unit 11b. The parameter extracting unit 142b extracts (calculates) parameters for making a person's internal state perceptible from the electroencephalogram data acquired by the data acquiring unit 141b.

[0101] For example, first, the parameter extraction unit 142b calculates the power spectral density of a series of electroencephalogram data at predetermined time intervals. Next, the parameter extraction unit 142b calculates a representation similarity matrix (RSM) using the calculated power spectral density of the electroencephalogram data. The RSM is a matrix that indicates the similarity of brain activity. This RSM can be obtained by calculating the correlation of the power spectral densities of the electroencephalogram data.

[0102] Next, the parameter extraction unit 142b performs dimensional compression on the RSM, and extracts the values ​​of a predetermined number of obtained feature quantities as parameters to be used for perceptualization (perceptualization parameters).

[0103] The perceptualization unit 144b uses the perceptualization parameter values ​​extracted by the parameter extraction unit 142b to make a series of electroencephalogram data perceptible by acoustic information such as volume, musical scale, and note value.

[0104] The feedback unit 145b outputs the information (audio information) made perceptible by the perceptualization unit 144b via the input / output unit 12b. For example, the feedback unit 145b outputs a series of acoustic information, in which the internal state of a person is expressed by the perceptualization unit 144b, as music from a speaker.

[0105] According to such a perceptualization device 10b, it is possible to provide auditory feedback of a person's internal state.

[0106] [Example of processing procedure] Next, an example of the procedure by which the perceptualization device 10b feeds back acoustic information will be described.

[0107] First, the perceptualization device 10b sets an update interval for updating the perceptualization parameters, and calculates the perceptualization parameters from the electroencephalogram data of an arbitrary length (time) immediately preceding each parameter update timing.

[0108] Next, the perceptualization unit 144b generates acoustic information using the calculated perceptualization parameters. Note that, in order to more smoothly express changes in the internal state, the perceptualization unit 144b may estimate complementary parameters that complement the perceptualization parameters calculated from the electroencephalogram by the parameter extraction unit 142b, and generate acoustic information using the perceptualization parameters and the complementary parameters.

[0109] By performing the above-mentioned processing, the perceptualization device 10b generates a series of acoustic information using perceptualization parameters calculated (extracted) from the electroencephalogram data.

[0110] [Extraction of perceptual parameters from EEG data] Next, the process of extracting perceptual parameters will be described in detail with reference to Fig. 17. The processes from S81 to S84 in Fig. 17 are the same as the processes from S71 to S74 in Fig. 14 described above, so their description will be omitted and the description will start from S85 in Fig. 17.

[0111] The parameter extraction unit 142 performs dimensional compression on the RSM calculated in S84 (S85), and extracts the feature values ​​obtained by the dimensional compression as a predetermined number of perceptualization parameters to be used for perceptualization (S86: extract perceptualization parameters).

[0112] [Generation of acoustic information] Next, the generation of acoustic information by the perceptualization unit 144b will be described in detail.

[0113] For example, the perceptualization unit 144b applies the first feature obtained by the dimensionality compression of the RSM of the electroencephalogram data as the volume level of the sound. The perceptualization unit 144b also applies the second feature obtained by the dimensionality compression as the musical scale of the sound. The perceptualization unit 144b also applies the third feature obtained by the dimensionality compression as the note value (length) of the sound. The perceptualization unit 144b also applies the fourth feature obtained by the dimensionality compression as the BPM (speed) of the sound. The perceptualization unit 144b also applies the fifth to seventh feature obtained by the dimensionality compression as sound changes (decay time, attack time, release time).

[0114] The perceptualization device 10b described above can express a person's internal state in detail without interfering with the original communication. As a result, for example, when the music generated by the perceptualization device 10b is used for communication between people, it becomes easier to understand each other without imposing a cognitive load, which was not possible with conventional communication.

[0115] [Fourth embodiment] Next, a perceptualization device according to a fourth embodiment will be described. The perceptualization device according to the fourth embodiment is characterized in that it extracts features that explain the internal state of a person from electroencephalogram data, taking into account the baseline of the electroencephalogram data.

[0116] [overview] An overview of the perceptualization device 10c of the fourth embodiment will be described with reference to Fig. 18. The perceptualization device 10c uses a baseline of human electroencephalogram data to more accurately represent changes in the internal state of a human.

[0117] For example, when acquiring human electroencephalogram data from an electroencephalograph, the perceptualization device 10c calculates a gap between baseline electroencephalogram data and the acquired electroencephalogram data at predetermined time intervals.The perceptualization device 10c then calculates feature amounts from the calculated gap and performs dimensional compression to calculate parameters (perceptualization parameters) for making a human's internal state perceptible.The perceptualization device 10c then makes the internal state perceptible using the calculated parameter values.

[0118] Thereafter, the perceptualization device 10c feeds back the perceptualized internal state of the person to the user so that the user can perceive it naturally. For example, the perceptualization device 10c displays the internal state of the person as an image such as a geometric figure (visual feedback) or outputs it as a sound whose volume or pitch fluctuates (auditory feedback).

[0119] The baseline electroencephalogram may be normal electroencephalogram data measured in advance, or may be calculated from the immediately preceding electroencephalogram data and continuously updated each time the perceptualization device 10c acquires electroencephalogram data. This allows the user to receive feedback on the state of their brain, taking the baseline electroencephalogram into consideration, in almost real time.

[0120] [Configuration example] Next, a configuration example of a perceptualization device 10c according to the fourth embodiment will be described with reference to Fig. 19. The perceptualization device 10c includes a communication unit 11c, an input / output unit 12c, a storage unit 13c, and a control unit 14c. Note that, in the following, an example of a method in which the perceptualization device 10c applies pre-measured normal electroencephalogram data as a baseline electroencephalogram and provides visual feedback as a geometric figure will be described, but the present invention is not limited to this.

[0121] The communication unit 11c is realized by an NIC or the like, and controls communication with external devices such as an electroencephalograph via a network.

[0122] The input / output unit 12c is an interface that controls input and output of various data. For example, the input / output unit 12 outputs a geometric figure drawn by the control unit 14 to a display.

[0123] The storage unit 13c is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 13c stores data and programs referenced by the control unit 14c when executing various processes. For example, the storage unit 13c stores electroencephalogram data acquired from an electroencephalograph, perceptualization parameters calculated from the electroencephalogram data, and the like.

[0124] The control unit 14c controls the entire perceptualization device 10c. The control unit 14c realizes the functions of a data acquisition unit 141c, a parameter extraction unit 142c, a perceptualization unit 144c, and a feedback unit (output processing unit) 145c, for example, by the CPU executing a program stored in the storage unit 13c.

[0125] The data acquiring unit 141c acquires a series of electroencephalogram data via the communication unit 11. The parameter extracting unit 142c extracts (calculates) parameters for making the electroencephalogram data perceptible from the electroencephalogram data acquired by the data acquiring unit 141c.

[0126] For example, first, the parameter extraction unit 142c calculates the power spectral density of a series of electroencephalogram data at predetermined time intervals. Furthermore, the parameter extraction unit 142c calculates the power spectral density of a baseline electroencephalogram using electroencephalogram data at normal times stored in advance in the storage unit 13c as a baseline electroencephalogram. Then, the parameter extraction unit 142c calculates the difference (difference electroencephalogram power spectral density) between the power spectral density of the electroencephalogram data and the power spectral density of the baseline electroencephalogram. The baseline electroencephalogram may be calculated based on electroencephalogram data acquired immediately before the electroencephalogram data and updated each time.

[0127] Next, the parameter extraction unit 142c calculates a Representation Similarity Matrix (RSM) using the calculated differential EEG power spectral density. The RSM is a matrix that indicates the similarity of brain activity. This RSM can be obtained by calculating the correlation of the differential EEG power spectral density.

[0128] Next, the parameter extraction unit 142c performs dimensional compression on the RSM and extracts the values ​​of a predetermined number of obtained feature quantities as parameters to be used for perceptualization. Note that the parameter extraction unit 142c may perform dimensional compression on the differential electroencephalogram power spectral density as a feature quantity and use it as a perceptualization parameter without obtaining an RSM. Furthermore, the parameter extraction unit 142c may perform dimensional compression on the differential electroencephalogram power spectral density as a feature quantity and use it as a perceptualization parameter without using a baseline electroencephalogram.

[0129] The perceptualization unit 144c reflects the values ​​of the perceptualization parameters extracted by the parameter extraction unit 142c as, for example, the size, color, or shape of a geometric figure, and makes it perceptible.

[0130] The feedback unit 145c displays the series of geometric figures drawn by the perceptualization unit 144c on a display via the input / output unit 12. For example, the feedback unit 145c displays the series of geometric figures drawn by the perceptualization unit 144c as animation.

[0131] According to such a perceptualization device 10c, it is possible to display on a display an image that expresses the internal state of a person as a series of geometric figures, taking into consideration the baseline of the person's electroencephalogram data.

[0132] [Example of processing procedure] Next, an example of the procedure by which the perceptualization device 10c renders a geometric figure will be described.

[0133] First, the perceptualization device 10c sets an update interval for updating the drawing parameters, and calculates the drawing parameters from the electroencephalogram data of an arbitrary length (time) immediately preceding each parameter update timing.

[0134] The perceptualization unit 144c draws a geometric figure using the drawing parameters. Note that, in order to more smoothly express changes in the internal state, the perceptualization unit 144c may estimate complementary parameters that complement each other between the drawing parameters calculated from the electroencephalogram by the parameter extraction unit 142c, and draw the geometric figure using the drawing parameters and the complementary parameters.

[0135] By performing the above processing, the perceptualization device 10c draws a series of geometric figures using drawing parameters of the geometric figures calculated (extracted) from the electroencephalogram data.

[0136] [Extraction of drawing parameters from EEG data] Next, the drawing parameter extraction process will be described in detail with reference to Fig. 20. The process from S91 to S93 in Fig. 20 is the same as the process from S71 to S73 in 14 above, so the description will be omitted and the description will start from S94 in Fig. 20.

[0137] The parameter extraction unit 142c uses the normal electroencephalogram data measured in advance as the baseline electroencephalogram, and calculates the differential electroencephalogram PSD, which is the difference between the PSD calculated from the baseline electroencephalogram and the PSD of the input electroencephalogram data (S94: baseline gap calculation).

[0138] The parameter extraction unit 142c calculates the RSM of the calculated differential electroencephalogram PSD (S95). For example, the parameter extraction unit 142c calculates the correlation matrix RSM by calculating the correlation between the differential electroencephalogram PSD calculated in S94 and the PSDs of any multiple pieces of electroencephalogram data previously acquired under various conditions.

[0139] The parameter extraction unit 142c performs dimensional compression on the RSM calculated in S95 (S96), and extracts the feature values ​​obtained by the dimensional compression as a predetermined number of rendering parameters (S97: extract rendering parameters).

[0140] The perceptualization device 10c can express a person's internal state in detail by taking into account the baseline of electroencephalogram data. As a result, for example, when a series of geometric figures drawn by the perceptualization device 10c is used in communication between people, mutual understanding that was not possible with conventional communication can be facilitated.

[0141] In the above embodiment, the perceptualization device 10c has been described as making the amount of change in electroencephalogram data perceptible using a baseline electroencephalogram, but the present invention is not limited to this. For example, the perceptualization device 10c may make the amount of change in electroencephalogram data perceptible by time-differentiating the electroencephalogram data.

[0142] [Fifth embodiment] [overview] Next, an overview of the perceptualization device of the fifth embodiment will be described. The perceptualization device of the fifth embodiment is characterized by performing nonlinear filtering on features representing the internal state of a person obtained from human electroencephalogram data.

[0143] Because human brain waves are constantly fluctuating, the results of expression similarity analysis with pre-recorded brain wave data also fluctuate over time. Therefore, if a perceptualization device generates perceptual information using features obtained by linearly processing the results of expression similarity analysis, the perceptual information may change significantly over time. It is extremely difficult for users to accurately interpret a person's internal state from perceptual information that changes significantly over time. Furthermore, perceptual information with large fluctuations may attract more user attention than necessary, hindering communication.

[0144] Therefore, the perceptualization device suppresses temporal changes in perceptual information by applying nonlinear filtering to the feature quantities that represent the human internal state. This allows the perceptualization device to generate perceptual information that makes it easy to accurately read the human internal state. As a result, the perceptualization device can provide feedback on the human internal state without impeding communication.

[0145] [Configuration example] Next, an example of the configuration of a perceptualization device 10d according to the fifth embodiment will be described with reference to Fig. 21. The perceptualization device 10d includes a communication unit 11d, an input / output unit 12d, a storage unit 13d, and a control unit 14d.

[0146] The communication unit 11d is realized by an NIC or the like, and controls communication with external devices such as an electroencephalograph via a network.

[0147] The input / output unit 12d is an interface that controls input and output of various data. For example, the input / output unit 12d outputs perceptual information generated by the control unit 14d.

[0148] The storage unit 13d is realized by a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disk. The storage unit 13d stores data and programs referenced by the control unit 14d when executing various processes. For example, the storage unit 13d stores data obtained by processing human electroencephalogram data previously recorded under various internal conditions. The storage unit 13d also stores electroencephalogram data acquired from an electroencephalograph, perceptualization parameters calculated from the electroencephalogram data, and the like.

[0149] The control unit 14d controls the entire perceptualization device 10d. The control unit 14 realizes the functions of a data acquisition unit 141d, a parameter extraction unit 142d, a perceptualization unit 144d, and a feedback unit (output processing unit) 145d, for example, by the CPU executing a program stored in the storage unit 13d.

[0150] The data acquiring unit 141d acquires a series of electroencephalogram data via the communication unit 11a. The parameter extracting unit 142d extracts (calculates) parameters for generating perceptual information of a human internal state from the series of electroencephalogram data acquired by the data acquiring unit 141d.

[0151] For example, the parameter extraction unit 142d includes a feature calculation unit 1421 and a filter processing unit 1422. The feature calculation unit 1421 performs preprocessing such as noise removal and calculation processing of electroencephalogram feature amounts such as power spectral density estimation on the series of electroencephalogram data acquired by the data acquisition unit 141d. Then, the feature calculation unit 1421 extracts a similarity structure with the electroencephalogram data for each pre-recorded human internal state stored in the storage unit 13d by expression similarity analysis, and calculates a feature amount representing the human internal state.

[0152] The filter processing unit 1422 performs nonlinear filtering based on expression similarity on the features representing the internal state of a person calculated by the feature calculation unit 1421. For example, the filter processing unit 1422 suppresses components showing a weak similarity structure, emphasizes components showing a strong similarity structure, or selectively extracts components showing a relatively strong similarity structure in the features calculated by the feature calculation unit 1421. The parameter extraction unit 142d uses the features filtered by the filter processing unit 1422 to find parameters for making the internal state of a person perceptible (perceptibility parameters).

[0153] The perceptualization unit 144d makes a series of electroencephalogram data perceptible by using the perceptualization parameters obtained by the parameter extraction unit 142d. For example, the perceptualization unit 144d draws a geometric figure based on the perceptualization parameters.

[0154] The feedback section 145d outputs the information made perceptible by the perceptualization section 144d via the input / output section 12d. For example, the feedback section 145d displays the geometric figure drawn by the perceptualization section 144d on a display.

[0155] The perceptualization device 10d can suppress the temporal change in perceptual information caused by fluctuations in human brain activity, thereby enabling the perceptualization device 10d to provide feedback on the internal state of a person without impeding communication.

[0156] [Example of processing procedure] Next, an example of the procedure by which the perceptualization device 10d makes an internal state perceptible will be described with reference to FIG.

[0157] First, the perceptualization device 10d acquires electroencephalogram data using the data acquisition unit 141d (S91). The acquired electroencephalogram data is time-series data, and electroencephalograms measured at an arbitrary sampling rate are acquired sequentially. The acquired electroencephalogram data may be processed sequentially, or may be stored in the storage unit 13d and processed at required timing.

[0158] Next, the feature amount calculation unit 1421 of the parameter extraction unit 142d performs preprocessing on the electroencephalogram data from n seconds before to the current time in accordance with the timing for making the internal state perceptible (S92). The feature amount calculation unit 1421 may perform any preprocessing, but here performs filtering to remove various noise components contained in the electroencephalogram data.

[0159] For example, the feature calculation unit 1421 performs one or a combination of bandpass filtering to extract only signals in a specific band contained in the electroencephalogram data, bandstop filtering to remove hum noise originating from AC power supplies, noise reduction filtering to remove impulsive noise associated with body movements, and artifact removal filtering using independent component analysis (ICA).

[0160] In addition, if the feature calculation unit 1421 refers to the electroencephalogram data at the same time multiple times depending on the interval at which the internal state is made perceptible, the preprocessing may be omitted by storing the preprocessed electroencephalogram data in the memory unit 13d.

[0161] When the feature calculation unit 1421 completes the preprocessing of the electroencephalogram data, it calculates the power spectral density using the preprocessed electroencephalogram data from n seconds before to the current time (S93).

[0162] Next, the feature calculation unit 1421 calculates a feature representing the internal state of the person using the calculated power spectrum density (S94: Predict internal state).

[0163] For example, the feature calculation unit 1421 performs an expression similarity analysis between a plurality of power spectral densities collected in advance and the power spectral density calculated in S93. Then, the feature calculation unit 1421 extracts a similarity structure of human brain activity from the result of the expression similarity analysis, thereby predicting the human internal state.

[0164] For example, the feature calculation unit 1421 determines, for each pre-recorded power spectrum density, a similarity vector indicating the similarity with the power spectrum density calculated in S93 as an internal state vector (feature) representing the internal state of a person.

[0165] Thereafter, the filter processing unit 1422 corrects the internal state vector generated in S94 by applying nonlinear processing to the internal state vector (S95). For example, based on the similarity represented by the internal state vector, the filter processing unit 1422 performs one of the following processes: suppressing components with similarity lower than a predetermined level; emphasizing components with similarity higher than another predetermined level; and suppressing components other than those with relatively strong similarity, or a combination of these processes.

[0166] The suppression / enhancement process based on the similarity can be performed by applying an activation function to each component of the internal state vector. Any function can be used as the activation function. The activation function applied to each component can also be changed.

[0167] For example, based on the internal state vector obtained at the time when the internal state was made perceptible in the past, the filter processing unit 1422 uses an activation function with a low suppression effect for components that are continuously suppressed, and an activation function with a high suppression effect for components that change greatly.

[0168] The process of suppressing components other than those with relatively strong similarity in the internal state vector can be realized by converting the internal state vector into an M-hot vector for an integer M. M may be determined in advance, or may be variable depending on the state of the internal state vector. For example, the filter processing unit 1422 may determine M depending on the highest similarity among the internal state vectors, or may determine M so as to retain similarities within a predetermined range starting from the highest similarity.

[0169] Finally, the perceptualization device 10d makes the internal state perceptible using the internal state vector corrected in S95 (S96). For example, the parameter extraction unit 142d calculates perceptualization parameters using the internal state vector corrected in S95. The perceptualization unit 144d then draws a geometric figure based on the perceptualization parameters. Thereafter, the feedback unit 145d outputs the geometry drawn by the perceptualization unit 144d via the input / output unit 12d.

[0170] In addition to the above-mentioned drawing of a figure, the method of making a person's internal state perceptible may also be, for example, generation of sound, vibration, etc. using the internal state vector as a parameter. Note that the perceptualization device 10d may not make the internal state vector perceptible as it is, but may make it perceptible after converting it into a low-dimensional vector using a dimension reduction method such as principal component analysis or discriminant analysis.

[0171] By correcting the internal state vector based on the above processing flow, the perceptualization device 10d can suppress drastic temporal changes in the estimated internal state even when making perceptible electroencephalograms that constantly change over time. This allows the perceptualization device 10d to make the overall internal state perceptible in a manner that does not impede communication.

[0172] The perceptualization device 10d has been described as an example in which a person's internal state is made perceptible using geometric figures based on the person's electroencephalogram data, but the person's internal state may also be made perceptible as audio data, or as tactile data using a vibrator, Peltier element, or the like.

[0173] [System configuration, etc.] Furthermore, the components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0174] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0175] [program] The perceptualization devices 10, 10a, 10b, 10c, and 10d can be implemented by installing a program (visualization program) as package software or online software on a desired computer. For example, by causing an information processing device to execute the program, the information processing device can function as the perceptualization devices 10, 10a, 10b, 10c, and 10d. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0176] 23 is a diagram showing an example of a computer that executes a perceptualization program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0177] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0178] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the perceptualization devices 10, 10a, 10b, 10c, and 10d are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations of the perceptualization devices 10, 10a, 10b, 10c, and 10d are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0179] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed and executes them.

[0180] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN (Local Area Network) or WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0181] 10 Visualization device 10a, 10b, 10c, 10d Perceptualization device 11,11a,11b,11c,11d Communication Department 12,12a,12b,12c,12d Input / output section 13,13a,13b,13c,13d Storage section 14, 14a, 14b, 14c, 14d Control unit 141 Data Acquisition Unit 142, 142a, 142b, 142c, 142d Parameter extraction section 143 Parameter Estimation Unit 144,144a Drawing section 144b, 144c, 144d Perceptualization section 145,145a Display processing unit 145b, 145c, 145d Feedback section 1421 Feature Calculation Unit 1422 Filter processing section

Claims

1. a data acquisition unit for acquiring a series of electroencephalogram data; a parameter extraction unit that performs principal component analysis on the expression similarity matrix of the electroencephalogram data at predetermined time intervals and extracts values ​​of a predetermined number of principal components obtained by the principal component analysis as values ​​of parameters to be used when drawing a geometric figure; a parameter estimation unit that estimates the value of the parameter on a frame-by-frame basis during the predetermined time period using the extracted parameter value and the number of frames of the geometric figure to be drawn during the predetermined time period; a perceptualization unit that renders the geometric figure using the extracted parameter values ​​and the estimated parameter values ​​on a frame-by-frame basis; an output processing unit that displays the drawn geometric figure on a screen; A perceptualization device comprising:

2. the geometric figure is a Rose of Venus, The parameters are used when drawing the Rose of Venus: a parameter for the Earth's orbital radius, a parameter for the difference between the orbital radii of the Earth and Venus, a parameter for the Earth's orbital period, RGB (Red Green Blue) parameters, and a parameter for the plot interval of the line segments that make up the Rose of Venus.

2. The perceptualization device according to claim 1.

3. The parameter estimation unit The number of line segments that draw the Rose of Venus is fixed to a predetermined number, and the value of the parameter for the plot interval of the line segments is estimated.

3. The perceptualization device according to claim 2.

4. The perceptualization unit further generating acoustic information from the extracted parameter values; The output processing unit further The generated acoustic information is output from an acoustic device.

2. The perceptualization device according to claim 1.

5. A perceptualization method executed by a perceptualization device, comprising: acquiring a set of electroencephalogram data; a step of performing a principal component analysis on the expression similarity matrix of the electroencephalogram data at predetermined time intervals, and extracting values ​​of a predetermined number of principal components obtained by the principal component analysis as values ​​of parameters to be used when drawing a geometric figure; a step of estimating the value of the parameter on a frame-by-frame basis during the predetermined time period using the extracted value of the parameter and the number of frames of the geometric figure to be drawn during the predetermined time period; drawing the geometric figure using the extracted parameter values ​​and the estimated parameter values ​​on a frame-by-frame basis; a step of displaying the drawn geometric figure on a screen; A method for making a person perceptible, comprising:

6. A perceptualization program for causing a computer to function as the perceptualization device according to any one of claims 1 to 4.

Images