Brain response space generation device, evaluation device, and brain response space generation method

Abstract

To evaluate brain answering responding to multimodal information.SOLUTION: A brain answering space generating device comprises a feature quantity extraction part that extracts a feature quantity of modality data stimulating at least two or more kinds of modalities among sensation and language and a brain answering space generation part that generates a brain answering space common to two or more modalities, where the brain answering space represents an expression space of brain activity to the modality data based on a measurement result of the brain activity measured by a brain activity measurement part by giving each of the modality data to a subject and the feature quantity extracted by the feature quantity extraction part.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a brain response space generation device, an evaluation device, and a brain response space generation method.

Background Art

[0002] In recent years, a technique for predicting a brain response pattern generated by an arbitrary information input has been known (see, for example, Patent Document 1). In the technique described in Patent Document 1, using the predicted value of the brain response pattern for the word label given to the audiovisual content, the appeal content of the audiovisual content and the content actually felt by the user are compared, and for example, it is used for audiovisual content evaluation.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art such as Patent Document 1, the object handled in the brain response space is only language information. Therefore, in the prior art, for example, it does not correspond to a plurality of modalities (sensory types) indicating sensations such as vision, hearing, touch, smell, and taste, as well as language information, and it has been difficult to apply it to a plurality of modality information (multimodal information).

[0005] The present invention has been made to solve the above problems, and an object thereof is to provide a brain response space generation device, an evaluation device, and a brain response space generation method capable of evaluating a brain response corresponding to multimodal information.

Means for Solving the Problems

[0006] To solve the above problems, one aspect of the present invention is a brain response space generation device characterized by comprising: a feature extraction unit that extracts feature quantities of modality data that stimulates with respect to at least two or more modalities from among sensory and language; and a brain response space generation unit that generates a brain response space that represents the representation space of the brain response to the modality data, based on measurement results obtained by a brain activity measurement unit when each of the modality data is given to a subject and brain activity is measured, and the feature quantities extracted by the feature extraction unit, and which generates a brain response space common to the two or more modalities.

[0007] Furthermore, in one aspect of the present invention, the brain response space generation device described above is characterized in that the brain response space generation unit generates the brain response space based on the brain response data which is the measurement result by machine learning, and generates a predictive model that predicts the representation of the brain response space from the features such that the error between the representation data of the brain response space based on the brain response data which is the measurement result and the representation data of the brain response space based on the brain response pattern predicted from the features is minimized.

[0008] Furthermore, in one aspect of the present invention, in the brain response space generation device described above, the prediction model comprises a unimodal unit that uses a deep neural network to input the feature quantities for each modality, and a multimodal unit that converts the output of the unimodal unit into a representation of the brain response space, wherein the brain response space generation unit generates the prediction model by optimizing the parameters of the deep neural network in the unimodal unit and the multimodal unit through learning.

[0009] Furthermore, one aspect of the present invention is characterized in that the brain response space generation unit reduces the number of dimensions of the brain response space based on the measurement results of a plurality of subjects.

[0010] Furthermore, one aspect of the present invention is an evaluation device characterized by comprising an estimation processing unit that estimates the position in the brain response space corresponding to the modality data to be evaluated, based on the brain response space generated by the brain response space generation device described above, from the modality data to be evaluated.

[0011] Furthermore, one aspect of the present invention is a method for generating a brain response space, characterized in that it includes: a brain activity measurement step in which a brain activity measurement unit provides a subject with modality data that stimulates each of at least two or more modalities from among sensory and language, and measures brain activity; a feature extraction step in which a feature extraction unit extracts features from the modality data; and a brain response space generation step in which a brain response space that represents the representation space of the brain response to the modality data, and generates a common brain response space for the two or more modalities, based on the measurement results obtained by the brain activity measurement step in which each of the modality data is provided to the subject and the brain activity is measured, and the features extracted by the feature extraction step. [Effects of the Invention]

[0012] According to the present invention, it is possible to evaluate brain responses corresponding to multimodal information. [Brief explanation of the drawing]

[0013] [Figure 1] This is a functional block diagram showing an example of an evaluation system according to this embodiment. [Figure 2] This figure illustrates the overview of the brain response space generation process in this embodiment. [Figure 3] This figure shows an example of a prediction model using a deep neural network in this embodiment. [Figure 4] This flowchart shows an example of the operation of the evaluation system according to this embodiment. [Figure 5] This flowchart shows an example of the evaluation process performed by the evaluation device according to this embodiment. [Modes for carrying out the invention]

[0014] Hereinafter, a brain response space generation device, an evaluation device, and a brain response space generation method according to one embodiment of the present invention will be described with reference to the drawings. Figure 1 is a functional block diagram showing an example of the evaluation system 100 according to this embodiment.

[0015] As shown in Figure 1, the evaluation system 100 comprises an evaluation device 1 and an fMRI (functional Magnetic Resonance Imaging) 2. In this embodiment, the evaluation device 1 generates a common brain response space for multiple types of modalities using machine learning based on brain response data measured by providing subject S1 with modality data to stimulate each of multiple types of modalities, and uses this brain response space to evaluate new modality data to be evaluated.

[0016] Here, modality refers to sensations from various sensory organs such as sight, hearing, touch, smell, and taste, as well as the perception of language; it is also called a sensory species. In this embodiment, we will explain an example using five types of modalities: sight, hearing, touch, smell, taste, and language.

[0017] Furthermore, using multiple types of modalities (at least two or more) is called multimodal. Modality data, on the other hand, is data used to stimulate a modality. Modality data includes, for example, visual data, auditory data, tactile data, olfactory data, gustatory data, and linguistic data. Visual data includes, for example, video data and image data, while auditory data includes, for example, sound data. Tactile data includes, for example, pressure data, while olfactory data includes, for example, odor data. Gustatory data includes, for example, data related to taste, while linguistic data includes, for example, audio data of conversations and written data.

[0018] fMRI2 (an example of a brain activity measurement unit) measures brain activity by providing each of a plurality of types of modality data (modality data for training) to the subject S1. fMRI2 gives a stimulus to the subject S1 by the modality data, and measures the brain activity of the subject S1 in response to the stimulus as brain response data. fMRI2 outputs an fMRI signal (brain activity signal) that visualizes the hemodynamic response related to the brain activity of the subject S1. fMRI2 measures the brain activity of the subject S1 at a predetermined time interval (for example, at intervals of 2 seconds) as an expression (raw brain response pattern) using a measurement unit (for example, a voxel in the case of functional MRI), and outputs the measured result to the evaluation device 1 as an fMRI signal.

[0019] The evaluation device 1 is a computer device such as, for example, a server device or a personal computer. The evaluation device 1 includes a storage unit 11 and a control unit 12. The storage unit 11 stores various information used for various processes executed by the evaluation device 1. The storage unit 11 includes a modality data storage unit 111, a learning data storage unit 112, a brain response space information storage unit 113, and an estimation result storage unit 114.

[0020] The modality data storage unit 111 stores the modality data for training in advance. The learning data storage unit 112 stores the learning data of machine learning for generating a brain response space. The learning data storage unit 112 stores, for example, information indicating the type of modality, feature amounts extracted from the modality data, and the measurement results (brain response patterns) measured by fMRI2 in association with each other.

[0021] The brain response space information storage unit 113 stores information indicating the brain response space that is the learning result. The brain response space information storage unit 113 stores, for example, information indicating a prediction model that predicts a brain response pattern (expression of the brain response space) from the feature amounts of the modality data, and information for projecting the brain response pattern into the brain response space.

[0022] The estimation result storage unit 114 stores estimation results obtained by estimating the location of the corresponding brain response space from modality data (features) of an arbitrary target for evaluation.

[0023] The control unit 12 is a processor, such as a CPU (Central Processing Unit), and comprehensively controls the evaluation device 1. The control unit 12 executes various processes performed by the evaluation device 1. The control unit 12 also includes a feature extraction unit 121, a brain response space generation unit 122, an estimation processing unit 123, and an output processing unit 124.

[0024] The feature extraction unit 121 extracts features from the modality data. The features of the modality data include labels, statistics, sensor measurements, machine learning model features, etc., corresponding to each modality.

[0025] For example, if the modality data is image data or video data, the features might include object labels in the scene or hidden layer activation patterns of a convolutional neural network. Similarly, if the modality data is audio data, the features might include a spectrogram or hidden layer activation patterns. Furthermore, if the modality data is tactile data, the features might include pressure distribution or texture labels.

[0026] Furthermore, for example, if the modality data is olfactory data, the features would be the concentration of chemical components or the measurement values ​​from the odor sensor. Similarly, if the modality data is gustatory data, the features would be the concentration of chemical components. And if the modality data is linguistic data, the features would be word or sentence embeddings.

[0027] The feature extraction unit 121 associates information indicating the type of modality, the extracted features, and the brain response patterns (brain response data) measured by fMRI2, and stores them as training data in the training data storage unit 112.

[0028] The brain response space generation unit 122 generates a brain response space SP that represents the representation space of brain responses to modality data, based on measurement results obtained by fMRI2 when each modality data is given to subject S1 and brain activity is measured, and the features extracted by the feature extraction unit 121, and generates a common brain response space for multiple types (two or more types) of modalities. The brain response space generation unit 122 constructs a common brain response space for multimodal information by, for example, performing machine learning using the learning data stored in the learning data storage unit 112.

[0029] The brain response space generation unit 122 generates a brain response space based on the measured brain response data, for example, using machine learning. It also generates a predictive model that predicts the representation of the brain response space from the features, so as to minimize the error between the representation data of the brain response space based on the measured brain response data and the representation data of the brain response space based on the brain response pattern predicted from the features. Here, with reference to Figure 2, an overview of the brain response space generation process (construction process) by the brain response space generation unit 122 will be explained.

[0030] Figure 2 is a diagram illustrating the overview of the brain response space SP generation process in this embodiment. As shown in Figure 2, the brain response space generation unit 122 performs the processes of constructing the brain response space SP and constructing a predictive model. The brain response space generation unit 122 constructs a predictive model of brain response patterns (representation of brain response space) by integrating and transforming the feature quantities extracted for each modality during the prediction model construction process. Furthermore, the brain response space generation unit 122 constructs a brain response space SP in a common format for representing diverse modality information during the brain response space SP construction process. In the brain response space SP, the input data for each modality is transformed into a representation of brain response space by the prediction model, ensuring that all modalities are treated uniformly.

[0031] Furthermore, when constructing the brain response space SP, the brain response space generation unit 122 reduces the dimensionality of the brain response space based on the measurement results of multiple subjects S1. The brain response space generation unit 122 generates a low-dimensional representation space common to the population from brain response patterns, for example, by using machine learning data dimensionality reduction techniques (e.g., principal component analysis, independent component analysis) or functional alignment to obtain a common brain data coordinate system for the population (e.g., dimensionality reduction methods such as shared response models).

[0032] Furthermore, when constructing a prediction model, the brain response space generation unit 122 learns a model that predicts brain response patterns (representations of brain response spaces) based on data in which each modality input has been quantified. This prediction model is equivalent to a regression model in machine learning, and learning is equivalent to estimating the model's weight parameters based on paired data of features and measured brain response patterns.

[0033] Here, any regression method can be applied to the regression model, such as linear regression, support vector regression, or deep learning regression. The brain response space generation unit 122 may also train the prediction model for each modality, or it may train it as a single model that handles all modalities uniformly. When training a prediction model for each modality, it is necessary to predict a single brain response pattern from a single input set, so the brain response space generation unit 122 integrates the predicted brain response patterns for each modality using a (weighted) average or the like.

[0034] Next, referring to Figure 3, we will explain an example of training a unified predictive model across all modalities. Figure 3 shows an example of a prediction model using a deep neural network in this embodiment.

[0035] Figure 3 illustrates an example of a prediction model using a deep neural network, as described by the brain response space generation unit 122. The prediction model shown in Figure 3 has a unimodal unit NT1 that inputs feature data for each modality, and a common multimodal unit NT2 that converts this data into predicted values ​​of brain response patterns.

[0036] In this prediction model, feature-rich data for each modality is first input to the unimodal unit NT1, undergoes hierarchical computation, and then input to the multimodal unit NT2, which undergoes further hierarchical processing before being output as predicted values ​​of brain response patterns.

[0037] Here, unimodal part NT11 corresponds to vision, unimodal part NT12 corresponds to hearing, and unimodal part NT13 corresponds to touch. Furthermore, unimodal part NT14 corresponds to smell, unimodal part NT15 corresponds to taste, and unimodal part NT16 corresponds to language.

[0038] Furthermore, in this embodiment, unimodal sections NT11 to NT16 will be described as unimodal section NT1 when referring to any unimodal section, or when no particular distinction is made.

[0039] The brain response space generation unit 122 performs individual learning processing on the unimodal unit NT13 using brain response patterns (brain response data) corresponding to the input of each modality. Furthermore, the brain response space generation unit 122 performs learning processing on the multimodal unit NT2 using brain response patterns (brain response data) for all modal inputs as a common part.

[0040] Furthermore, when the brain response space generation unit 122 constructs the prediction model shown in Figure 3 using machine learning, multiple modalities may be input simultaneously, but it is not necessary for all modalities to be input at the same time. Also, when training the prediction model, the brain response space generation unit 122 uses feature-quantified data for each modality and corresponding brain response data, but the parameters of the unimodal unit NT1 are trained only from the input of each corresponding modality, while the parameters of the multimodal unit NT2 are trained for all modal inputs.

[0041] Thus, the prediction model may have a unimodal unit NT1 that uses a deep neural network to input features for each modality, and a multimodal unit NT2 that converts the output of the unimodal unit NT1 into a brain response pattern (representation of the brain response space). In this case, the brain response space generation unit 122 generates a prediction model by learning the parameters of the deep neural networks in the unimodal unit NT1 and the multimodal unit NT2 so as to minimize the error between the representation data of the brain response space of the measured values ​​and the representation data of the brain response space of the predicted values.

[0042] Returning to the explanation of Figure 2, the brain response space generation unit 122 stores the generated prediction model and information representing the brain response space SP in the brain response space information storage unit 113. In this embodiment, the modality data storage unit 111, the learning data storage unit 112, the feature extraction unit 121, and the brain response space generation unit 122 correspond to the brain response space generation device 10.

[0043] The estimation processing unit 123 estimates the position of the modality data to be evaluated in the brain response space SP based on the brain response space SP generated by the brain response space generation device 10. The estimation processing unit 123 causes the feature extraction unit 121 to extract the features of the modality data to be evaluated. The estimation processing unit 123 estimates the position in the brain response space SP from the features of the modality data to be evaluated based on the prediction model and information indicating the brain response space SP stored in the brain response space information storage unit 113. The estimation processing unit 123 stores the estimation results in the estimation result storage unit 114.

[0044] The output processing unit 124 causes the estimation results estimated by the estimation processing unit 123 to be output externally. For example, the output processing unit 124 causes the positional information corresponding to the modality data to be evaluated in the brain response space SP, which is the estimation result stored in the estimation result storage unit 114, to be output externally.

[0045] Next, the operation of the evaluation system 100 according to this embodiment will be described with reference to the drawings. Figure 4 is a flowchart showing an example of the operation of the evaluation system 100 according to this embodiment.

[0046] As shown in Figure 4, the fMRI2 provides subject S1 with each of several types of modality data to measure brain activity (brain response data) (step S101). Here, the modality data provided to subject S1 is the same as the training data stored in the modality data storage unit 111 of the brain response space generation device 10. The fMRI2 outputs the measured brain response data to the brain response space generation device 10.

[0047] Next, the feature extraction unit 121 of the brain response space generation device 10 extracts data from each modality. Feature extraction (Step S102). The feature extraction unit 121 acquires the training modality data stored in the modality data storage unit 111 and extracts features. The feature extraction unit 121 associates information indicating the type of modality, the extracted features, and the brain response patterns (brain response data) measured by fMRI2, and stores them as training data in the training data storage unit 112.

[0048] Next, the brain response space generation unit 122 of the brain response space generation device 10 generates a brain response space SP by machine learning based on each feature and the corresponding brain response data (step S103). The brain response space generation unit 122 generates (constructs) a prediction model and a brain response space SP as shown in Figures 2 and 3 above, using the training data stored in the training data storage unit 112. The brain response space generation unit 122 stores information indicating the generated prediction model and brain response space SP in the brain response space information storage unit 113. After the processing in step S103, the brain response space generation unit 122 terminates its processing.

[0049] Next, with reference to Figure 5, the evaluation process using the evaluation device 1 according to this embodiment will be described. Figure 5 is a flowchart showing an example of the evaluation process performed by the evaluation device 1 according to this embodiment.

[0050] As shown in Figure 5, the evaluation device 1 first extracts the features of the modality data to be evaluated (step S201). That is, the feature extraction unit 121 of the evaluation device 1 extracts the features of the modality data to be evaluated.

[0051] Next, the evaluation device 1 estimates the location of the brain response space SP corresponding to the feature quantity based on the brain response space information stored in the brain response space information storage unit 113 (step S202). The estimation processing unit 123 of the evaluation device 1 estimates the location in the brain response space SP from the feature quantity of the modality data to be evaluated, based on the prediction model and information indicating the brain response space SP stored in the brain response space information storage unit 113. The estimation processing unit 123 stores the estimation result in the estimation result storage unit 114.

[0052] Next, the evaluation device 1 outputs the estimated brain response space location information (step S203). The output processing unit 124 of the evaluation device 1 outputs to the outside the location information corresponding to the modality data of the target of evaluation in the brain response space SP, which is the estimated result stored in the estimation result storage unit 114. After the processing in step S203, the output processing unit 124 terminates its processing.

[0053] As described above, the brain response space generation device 10 according to this embodiment comprises a feature extraction unit 121 and a brain response space generation unit 122. The feature extraction unit 121 extracts feature quantities of modality data that stimulates each of at least two or more modalities from among sensory and language. The brain response space generation unit 122 generates a brain response space SP that represents the representation space of brain responses to modality data, based on the measurement results obtained by providing each modality data to the subject S1 and measuring brain activity by fMRI2 (brain activity measurement unit), and the feature quantities extracted by the feature extraction unit 121, and a brain response space SP common to two or more modalities.

[0054] As a result, the brain response space generation device 10 according to this embodiment can use a common brain response space SP across two or more modalities, thereby converting input data corresponding to a variety of modalities that the brain can handle into a common brain response representation. Therefore, the brain response space generation device 10 according to this embodiment can evaluate brain responses corresponding to multimodal information.

[0055] Furthermore, the brain response space generation device 10 according to this embodiment can be applied to a wider range of problems in real society than before by constructing a common representation (brain response space SP) of diverse modality information. In addition, the brain response space generation device 10 according to this embodiment enables numerical calculations such as comparison, integration, and conversion between data of different modalities, making it possible to use it as a general-purpose technology to solve multimodal recognition problems of a type that cannot be handled by conventional brain information technology or conventional machine learning technology. In particular, the brain response space generation device 10 according to this embodiment can be expected to improve recognition performance by utilizing the characteristics of brain information in multimodal recognition problems that estimate human cognition and behavior.

[0056] Furthermore, the brain response space generation device 10 according to this embodiment can quantify the relationship between linguistic data and diverse sensory data on a common representation space, thus demonstrating effectiveness in multimodal search. Specifically, conventional search techniques were limited to searching for images, videos, and music based on keywords. The brain response space generation device 10 according to this embodiment, by using the generated brain response space SP, makes it possible to search not only for modalities of conventional technology but also for taste, touch, and smell data based on the similarity of embedding vectors. Moreover, since the similarity of the brain response space generation device 10 according to this embodiment also takes into account the characteristics of brain information, it is expected to provide search results that are closer to human perception. Conversely, for taste, touch, and smell, which are generally difficult to verbalize, one effective use is to estimate similar linguistic data on a common representation space and add linguistic descriptions to these sensory data. As a result, the brain response space generation device 10 according to this embodiment can promote communication between individuals and enhance empathy.

[0057] For example, the brain response space generation device 10 according to this embodiment can be used in the following ways by using the brain response space SP. (Example of use 1) In the brain response space generation device 10 according to this embodiment, it becomes possible to search for a corresponding Scotch whisky using multimodal keywords such as "a Scotch with a rich, floral aroma and a subtle sweetness on the tip of the tongue."

[0058] (Example of use 2) The brain response space generation device 10 according to this embodiment can be used for data fit estimation between modalities, such as estimating the degree of fit of an interior image of a room + background music + scent.

[0059] (Example of use 3) The brain response space generation device 10 according to this embodiment can be used for cross-modal product recommendations, such as recommending books that the user likes (suitable for the user) based on the user's music purchase history.

[0060] Furthermore, in this embodiment, the brain response space generation unit 122 generates a brain response space SP based on the measured brain response data using machine learning, and also generates a predictive model that predicts the representation of the brain response space from the features so as to minimize the error between the representation data of the brain response space based on the measured brain response data and the representation data of the brain response space based on the brain response pattern predicted from the features.

[0061] As a result, the brain response space generation device 10 according to this embodiment can appropriately estimate the corresponding location of the brain response space SP from modality data (features) by generating a predictive model and a brain response space SP.

[0062] Furthermore, in this embodiment, the prediction model includes a unimodal unit that uses a deep neural network to input feature quantities for each modality, and a multimodal unit NT2 that converts the output of the unimodal unit NT1 into a brain response pattern (representation of the brain response space). The brain response space generation unit 122 optimizes the parameters of the deep neural network in the unimodal unit and the multimodal unit NT2 through learning to generate the prediction model.

[0063] As a result, the brain response space generation device 10 according to this embodiment can easily and appropriately generate a predictive model using a simple method by utilizing a deep neural network.

[0064] Furthermore, in this embodiment, the brain response space generation unit 122 reduces the number of dimensions of the brain response space based on the measurement results of multiple subjects S1. As a result, the brain response space generation device 10 according to this embodiment can reduce the number of dimensions from, for example, tens of thousands of dimensions to several hundred dimensions, while obtaining an effective brain response representation (brain response space SP) with low redundancy as an information representation.

[0065] Furthermore, the evaluation device 1 according to this embodiment includes an estimation processing unit 123 that estimates the position corresponding to the modality data to be evaluated in the brain response space SP based on the brain response space generated by the brain response space generation device 10.

[0066] As a result, the evaluation device 1 according to this embodiment achieves the same effects as the brain response space generation device 10 described above, and can evaluate brain responses corresponding to multimodal information. Furthermore, since the evaluation device 1 according to this embodiment can obtain the position in the brain response space SP corresponding to the modality data to be evaluated from the modality data to be evaluated, it can be used, for example, for similarity comparison between modalities or clustering. The evaluation device 1 according to this embodiment can universally use the brain response space SP as a vector representation of input data that has a common representation among various modalities.

[0067] Furthermore, the brain response space generation method according to this embodiment includes a brain activity measurement step, a feature extraction step, and a brain response space generation step. In the brain activity measurement step, fMRI2 provides subject S1 with modality data that stimulates at least two or more modalities from among sensory and language, and measures brain activity. In the feature extraction step, the feature extraction unit 121 extracts features from the modality data. In the brain response space generation step, the brain response space generation unit 122 generates a brain response space SP that represents the representation space of brain responses to modality data, based on the measurement results obtained by providing each of the modality data to subject S1 in the brain activity measurement step and the features extracted in the feature extraction step, and a brain response space SP common to two or more modalities. As a result, the brain response space generation method according to this embodiment has the same effect as the brain response space generation device 10 described above, and enables the evaluation of brain responses corresponding to multimodal information.

[0068] It should be noted that the present invention is not limited to the embodiments described above, and can be modified without departing from the spirit of the invention. For example, in the above embodiment, an example was described in which the brain response space generation device 10 and the evaluation device 1 are configured as a single device, but the invention is not limited to this, and the brain response space generation device 10 and the evaluation device 1 may be configured as separate devices. Furthermore, some of the components of the brain response space generation device 10 and the evaluation device 1 may be provided externally.

[0069] Furthermore, although the above embodiment describes an example where the brain activity measurement unit is fMRI2, it is not limited to this, and other brain activity measurement methods such as electroencephalography (EEG), magneto-encephalography (MEG), electroencephalography (ECoG), and functional near-infrared spectroscopy (fNIRS) may be used.

[0070] Furthermore, although the above embodiment describes an example in which the prediction model has a unimodal section NT1 and a multimodal section NT2, it is not limited to this, and any other prediction model may be used as long as it accepts feature data from each modality as input and outputs a common output for all modalities as a predicted value of the brain response pattern (representation of the brain response space). In addition, as a simpler implementation example, the prediction model may be constructed as a regression model that concatenates the feature data from each modality as a vector and uses that as input to predict the brain response pattern.

[0071] Furthermore, while the above embodiments described examples of using visual, auditory, tactile, olfactory, gustatory, and linguistic modalities, the invention is not limited to these. For example, mood or atmosphere may also be used as a modality. Also, while the invention described an example of using six types of modalities, the invention is not limited to this. Any number of modalities (e.g., two, three, etc.) is acceptable, as long as there are two or more types.

[0072] Furthermore, each component of the evaluation device 1 and the brain response space generation device 10 described above has a computer system inside. The processing in each component of the evaluation device 1 and the brain response space generation device 10 may be performed by recording a program for realizing the functions of each component of the evaluation device 1 and the brain response space generation device 10 onto a computer-readable recording medium, loading the program recorded on this recording medium into the computer system, and executing it. Here, "loading the program recorded on the recording medium into the computer system and executing it" includes installing the program into the computer system. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer system" may include multiple computer devices connected via a network, including communication lines such as the Internet, WAN, LAN, and dedicated lines. "Computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Thus, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM.

[0073] Furthermore, the recording medium also includes internal or external recording media accessible from the distribution server for distributing the program. The program may be divided into multiple parts, downloaded at different times, and then combined in the respective configurations of the evaluation device 1 and the brain response space generation device 10. The distribution servers for each of the divided programs may also be different. Moreover, "computer-readable recording media" includes volatile memory (RAM) within computer systems that act as servers or clients when a program is transmitted over a network, which retains the program for a certain period of time. The program itself may also be intended to implement only a portion of the functions described above. Furthermore, the program may be a so-called differential file (differential program) that can implement the functions described above in combination with a program already recorded in the computer system.

[0074] Furthermore, some or all of the above-mentioned functions may be implemented as integrated circuits such as LSIs (Large Scale Integrations). Each of the above-mentioned functions may be implemented as an individual processor, or some or all of them may be integrated into a single processor. In addition, the method of implementing integrated circuits is not limited to LSIs; they may also be implemented using dedicated circuits or general-purpose processors. Furthermore, if advances in semiconductor technology lead to the emergence of integrated circuit technologies that can replace LSIs, integrated circuits using such technologies may be used. [Explanation of symbols]

[0075] 1. Evaluation device 2 fMRI 10 Brain response space generation device 11 Storage section 12 Control Unit 111 Modality data storage unit 112 Learning Data Storage Unit 113 Brain Response Spatial Information Memory Unit 114 Estimation result storage unit 121 Feature Extraction Unit 122 Brain response space generation unit 123 Estimation Processing Unit 124 Output Processing Unit NT1, NT11~NT16 Unimodal Section NT2 Multimodal Unit S1 Subject SP Brain Response Space

Claims

1. A feature extraction unit that extracts feature quantities from modality data that stimulates at least two or more modalities from among sensory and language, A brain response space is generated which represents the representation space of the brain response to the modality data, based on the measurement results obtained by the brain activity measurement unit when each of the modality data is given to a subject and the brain activity is measured, and the feature quantities extracted by the feature quantity extraction unit, and a brain response space generation unit generates a common brain response space for two or more modalities. A brain response space generation device characterized by comprising the following features.

2. The brain response space generation unit generates the brain response space based on the measured brain response data using machine learning, and generates a predictive model that predicts the representation of the brain response space from the features so as to minimize the error between the representation data of the brain response space based on the measured brain response data and the representation data of the brain response space based on the brain response pattern predicted from the features. The brain response space generation device according to feature 1.

3. The prediction model comprises a unimodal unit that uses a deep neural network to input the feature quantities for each modality, and a multimodal unit that converts the output of the unimodal unit into a representation of the brain response space. The brain response space generation unit optimizes the parameters of the deep neural network in the unimodal and multimodal sections through learning to generate the predictive model. The brain response space generation device according to feature 2.

4. The brain response space generation unit reduces the number of dimensions of the brain response space based on the measurement results of multiple subjects. The brain response space generation device according to feature 1.

5. The brain response space generation device according to any one of claims 1 to 4 includes an estimation processing unit that estimates the position in the brain response space corresponding to the modality data to be evaluated, based on the brain response space generated by the brain response space generation device according to any one of claims 1 to 4. An evaluation device characterized by the following features.

6. A brain activity measurement step in which the brain activity measurement unit provides the subject with modality data that stimulates at least two or more modalities from among sensory and language, and measures brain activity, The feature extraction unit performs a feature extraction step of extracting features from the modality data, A brain response space generation unit generates a brain response space that represents the representation space of brain responses to the modality data, based on the measurement results obtained by providing each of the modality data to the subject in the brain activity measurement step and the features extracted in the feature extraction step, and a brain response space generation step that generates a common brain response space for two or more modalities. A method for generating a brain response space, characterized by including the following:

Images