Sensation estimation device, sensation transmission system, and sensation estimation method

Abstract

To properly estimate sensations of respective subjects in a case where there are individual differences among the subjects or there are differences in environments surrounding the subjects.SOLUTION: A sensation estimation device includes a storage unit that stores reference sensation information which is sensation information recalled in response to a perception, and a processing unit that estimates corresponding sensation information which is sensation information corresponding to reference sensation information concerning a subject on the basis of the reference sensation information stored in the storage unit.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a sensation estimation device, a sensation transmission system, and a sensation estimation method. [Background technology]

[0002] In recent years, non-invasive measurement of brain activation information, such as functional magnetic resonance and near-infrared spectroscopy, has become possible. As technology advances, the technology for brain-machine interfaces, which are interfaces between the brain and the outside world, is becoming increasingly sophisticated. As an example of the use of such technology, the brain activity of a subject going to sleep is The sleep state of the subject is determined by detecting the activation information, and the subject is determined to be in REM sleep. The present invention discloses a configuration for providing a dream-inducing stimulus to a subject when the subject is in a dream state (for example, Patent (See Reference 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-332251 Summary of the Invention [Problem to be solved by the invention]

[0004] The correspondence between the subject's brain activity and the measured brain activation information was It depends on the individual's personality and the environment surrounding the subject at the time of measurement. For example, as described in Patent Document 1 In this technology, the type of dream that a subject will have when given a dream-inducing stimulus varies depending on the subject or the stimulus. It depends on the timing of the granting, and it is difficult to control the content of the dream itself. In contrast, when determining or controlling brain activity based on the subject's brain activation information, Therefore, technology that takes into account factors such as individual differences and differences in the environment surrounding the subject is required.

[0005] The present invention has been made in view of the above, and takes into account individual differences between subjects or the circumstances of the subjects. It is possible to estimate the sensations between subjects appropriately when there are large differences in the environment. The object of the present invention is to provide a sensory estimation device, a sensory transmission system, and a sensory estimation method. [Means for solving the problem]

[0006] The sensation estimation device according to the present invention is configured to estimate reference sensation information, which is sensation information associated with perception. and a storage unit for storing the reference sensory information, and a processing unit for estimating corresponding sensory information, which is sensory information corresponding to all of the reference sensory information; do.

[0007] A sensation transmission system according to the present invention comprises the above sensation estimation device and a stimulation device that stimulates the subject so as to recall the estimated corresponding sensory information. can.

[0008] The sensation estimation method according to the present invention is to estimate reference sensation information, which is sensation information associated with perception. acquiring the reference sensory information from a storage unit that stores the reference sensory information; Based on this, corresponding sensory information, which is sensory information corresponding to the reference sensory information for the subject, is estimated. This includes determining the [Effects of the Invention]

[0009] According to the present invention, when there are individual differences between subjects or differences in the environment surrounding the subjects, In this case, it becomes possible to appropriately estimate sensations between subjects. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a sensation transfer system according to the first embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of a sensation transmission system according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a neural network. [Figure 4] FIG. 4 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the sensation transfer system according to the first embodiment. [Figure 6] FIG. 6 is a diagram schematically illustrating another example of the operation of the sensation transfer system according to the second embodiment. [Figure 7] FIG. 7 is a diagram schematically illustrating another example of the operation of the sensation transmission system according to the third embodiment. [Figure 8] FIG. 8 is a functional block diagram illustrating an example of a sensation transmission system according to the fourth embodiment. [Figure 9] FIG. 9 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the fourth embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the sensation transfer system according to the fourth embodiment. [Figure 11] FIG. 11 is a functional block diagram illustrating an example of a sensation transfer system according to the fifth embodiment. [Figure 12] FIG. 12 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the fifth embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the operation of the sensation transfer system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments are not limited to the above. This includes things that are possible and easy, or things that are substantially the same.

[0012] [First embodiment] FIG. 1 is a schematic diagram illustrating an example of a sensation transmission system 100 according to the first embodiment. 1 and 2 are functional block diagrams showing an example of a sensation transmission system 100. As described above, the sensation transfer system 100 includes a first device 10, an estimation device 20, and a second device 30. Prepare.

[0013] The first device 10 measures the brain activity of the first subject R1 when perceived by the first subject R1. The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, and a stimulator. It has a stimulus applying unit 14 and a memory unit 15.

[0014] The detection unit 11 detects brain activation information. The brain activation information may include, for example, the subject's Oxygenated hemoglobin, deoxygenated hemoglobin, and total hemoglobin concentrations in cerebral blood flow The detection unit 11 may be, for example, an fMRI (functional Magnetron Imaging) device. Functional Magnetic Resonance Imaging (fNIR) S(functional Near-Infrared Spectroscopy: Measurement devices that perform measurements based on the principles of functional near-infrared spectroscopy, etc., and invasive electrodes are used. A measuring device that places a micromachine inside the blood vessels of the brain and measures the The detector 11 is not limited to the above-mentioned device, and may be any other device. The brain activation information may be obtained by measuring the brain of the first subject R1 for example by measuring the brain activity for several milliseconds. When partitioned by a three-dimensional matrix consisting of voxels below 1, This can be expressed as the magnitude of activity for each

[0015] The communication unit 12 is an interface for performing wired or wireless communication. The brain activation information detected by the detection unit 11 is transmitted to the estimation device 20. The communication unit 12 For communicating with external devices via wireless LAN in accordance with the IEEE802.11 standard The communication unit 12 communicates with an external device under the control of the processing unit 22 and the like. The communication method is not limited to wireless LAN, and may be, for example, infrared. Wired communication method, Bluetooth (registered trademark) communication method, Wireless USB, etc. It can also include wireless communication methods. ), IEEE1394, Ethernet, or other wired connections may also be used.

[0016] The processing unit 13, the stimulus applying unit 14, and the storage unit 15 will be described later. In this case, the processing unit 13, the stimulus applying unit 14, and the storage unit 15 may not be provided.

[0017] The estimation device 20 includes a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is an interface for performing wired or wireless communication. The communication unit 21 receives the brain activation information transmitted from the device 10. The communication unit 21 transmits the corresponding sensory information, which will be described later, estimated by the above-mentioned method to the second device 30. The communication unit 12 may have the same configuration as the communication unit 12 described above.

[0018] The processing unit 22 is a processor such as a CPU (Central Processing Unit). The device and RAM (Random Access Memory) or ROM (Read Only Memory) The processing unit 22 has a storage device such as a memory device (memory device). It performs various processes including processing.

[0019] The storage unit 23 stores various information. The storage unit 23 may be, for example, a hard disk drive, The storage unit 23 has a storage such as a solid state drive. An external storage medium such as a bubble disk may also be used.

[0020] The processing unit 22 calculates the first subject R1's brain activation information based on the detected brain activation information of the first subject R1. The sensory information (reference sensory information) about the sensations recalled by the subject is estimated. The sense of smell is a sensation that involves at least one of the five senses, such as sight, hearing, touch, taste, and smell. The information may be information relating to sense of balance, or other somatic sensations. Specifically, when the sensory information is information related to vision, the sensory information is Perceived image data, but not limited to this, is not the image data itself but the image Image data sampled from the data or filtered to that image data The sensory information may be visual information. In this case, the information may be information about the light entering the eyeball of the first subject R1. For example, information about light may be obtained by a contact lens equipped with a light sensor. In addition, an artificial retina may be utilized to acquire information about light. The sensory information is information about the hearing. In some cases, the sensory information may be audio signal data perceived by the first subject R1. , when the sensory information is information about taste, the sensory information is the taste perceived by the first subject R1. It is sufficient if the data shows indicators of multiple chemical substances that reproduce the sense of touch. When the information is related to the sensory information, the sensory information is spread out on a plane over the entire body surface of the first subject R1, It is sufficient if the data indicates to what extent a stimulus occurred in which part of the development diagram. These sensory information are examples and are not limited to these. The type of brain activation that occurs when such sensory information is recalled remains to be determined. For example, the first subject R1 The brain activation information detected from the brain is associated with the sensory information corresponding to the brain activation information. A first learning model is created by performing machine learning on a set of learning datasets. The first learning model can be generated by storing it in the storage unit 23, for example. This can be done.

[0021] The processing unit 22 also calculates the reference sensory information of a second subject R1 different from the first subject R1 based on the estimated reference sensory information. Corresponding sensory information, which is sensory information corresponding to the reference sensory information for subject R2, is estimated. The specific example of the corresponding sensory information may correspond to the specific example of the sensory information. The subject to whom the sensations of the first subject R1 are transmitted. The relationship between the information can be determined by conducting experiments in advance. Corresponding sensory information between the subject R1 and the second subject R2 is used as a set of training data sets, A second learning model can be generated by performing machine learning on the learning dataset. The second learning model can be stored in the storage unit 23, for example.

[0022] The second device 30 provides a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information. The second device 30 includes a detection unit 31, a communication unit 32, a processing unit 33, and a stimulus applying unit 34. , and a storage unit 35.

[0023] The detection unit 31 will be described later. In the first embodiment, the detection unit 31 is not provided. It's okay.

[0024] The communication unit 32 is an interface for performing wired or wireless communication. The communication unit 32 receives corresponding sensory information transmitted from the estimation device 20. The communication unit 32 transmits the detected brain activation information to the estimation device 20. The communication unit 12 may have the same configuration as the communication unit 12 described above.

[0025] The stimulation unit 34 irradiates the target area of ​​the brain of the second subject R2 with an electromagnetic wave signal to stimulate the target area. By activating the second subject R2, the brain of the second subject R2 is stimulated. For example, a three-dimensional matrix consisting of voxels of a few millimeters or less is used to divide the image. The stimulus applying unit 34 applies electromagnetic waves to each voxel in the three-dimensional matrix. The electromagnetic wave can be irradiated based on stimulus image information indicating the intensity of the electromagnetic wave to be irradiated. A voxel in the three-dimensional matrix of stimulus image information can be used to represent, for example, a three-dimensional matrix of brain activation information. The dimensions and positions may correspond to those of the voxels in the 2-dimensional matrix. What kind of sensation would be produced if electromagnetic waves of a certain intensity were irradiated to which voxels in the brain of subject R2? To determine whether visual information is recalled, conduct experiments in advance to determine the correspondence. For example, stimulus image information for the second subject R2 and an electric potential based on the stimulus image information can be obtained. A set of learning data was created by associating the sensory information recalled by the second subject R2 when the magnetic waves were irradiated. The training dataset is used as a dataset, and a third learning model is generated by machine learning the training dataset. The third learning model may be stored in the storage unit 35 of the second device 30, for example. The processing unit 33 performs the processing based on the corresponding sensory information received by the communication unit 32 and the third learning model. Based on this, it is possible to calculate stimulation image information corresponding to the received corresponding sensory information.

[0026] The first, second and third learning models described above are, for example, VGG16. Generated using a neural network (convolutional neural network) represented as FIG. 3 is a diagram showing an example of a neural network. As shown, the neural network NW consists of 13 convolutional layers S1 and 5 pooling layers. The neural network has a neural network with a 3-layered fully connected layer S2 and a 3-layered fully connected layer S3. The information is processed in the convolutional layer S1 and the pooling layer S2 in order, and the processed results are combined in the fully connected layer S3. and output.

[0027] When generating the first learning model, the second learning model, and the third learning model, As shown in Fig. 1, a training data set containing the corresponding information (I1, I2) is used for the neural network. The correlations of the training data set are learned through machine learning such as deep learning. In other words, the neural network NW is optimized through learning to generate a learning model. For example, when one of the pieces of information that make up each learning data set is input, Students learn to solve problems that require one piece of information in a given situation.

[0028] When performing inference using the first, second, and third learning models, the bottom of Figure 3 As shown in the figure, each of the two pieces of information that make up the training data set is I1 is input to the neural network NW. The learning model extracts the input data based on the correlations of the information that make up the learning dataset. The other information I2 corresponding to the input information I1 is output. ,Generate a training model using a convolutional neural network represented by VGG16. Although examples are described, other types of neural networks may be used for learning. A training model may be generated.

[0029] Next, a method for transmitting sensation using the sensation transmission system 100 configured as described above will be described. FIG. 4 is a diagram illustrating an example of the operation of the sensation transmission system 100. As shown in the figure, the first subject R1 is made to perceive the reference sensory information. Let us take the example of subject R1 visually perceiving a cat's face and recalling it as visual information. do.

[0030] As shown in the upper part of FIG. 4, the detection unit 11 detects the cat's face and recalls sensory information 41. The communication unit 12 detects the brain activation information 42 of the subject R1. The brain activation information 42 is transmitted to the estimation device 20 .

[0031] In the estimation device 20, the communication unit 21 receives the brain activation information 4 transmitted from the first device 10. The processing unit 22 receives the reference sensory information 4 based on the received brain activation information 42. In this case, the processing unit 22 estimates the brain activation information 42 of the first subject R1 from the first learning The first learning model provides the brain activation information 42 and the reference sensory information 43. Based on the correlation learning results, the reference sensory information corresponding to the input brain activation information 42 is The processing unit 22 acquires the output reference sensory information 43 as an estimation result. do.

[0032] The processing unit 22 calculates the reference sensory information 43 based on the estimated reference sensory information 43. The processing unit 22 estimates the corresponding sensory information 44 corresponding to the quasi-sensory information 43. The reference sensory information 43 is input to the second learning model. Based on the learning result of the correlation between the input reference sensory information 43 and the corresponding sensory information 44, The processing unit 22 outputs the corresponding sensory information 44 corresponding to the output sensory information 43. The communication unit 21 transmits the acquired corresponding sensory information 44 to the second device 30. Send to.

[0033] In the second device 30, the communication unit 32 receives the corresponding sensory information 44 transmitted from the estimation device 20. The processing unit 33 receives the corresponding sensory information 44. The third learning model provides the correlation between the corresponding sensory information 44 and the stimulus image information. Based on the learning result of the relationship, stimulus image information 45 corresponding to the input corresponding sensory information 44 is output. The stimulation unit 34 applies a stimulus to the brain of the second subject R2 based on the output stimulation image information 45. The electromagnetic waves are applied to stimulate the second subject R2. The second subject R2, to whom the stimulus was given, imagines corresponding sensory information 46 corresponding to the stimulus image information 45. In other words, the second subject R2 recalls the visual information of the cat's face as the corresponding sensory information 46. do.

[0034] On the other hand, there are differences in personality between the first subject R1 and the second subject R2, so brain activity may differ. The correspondence between the movement state and the brain activation information is different. As shown in the figure (indicated by the dashed line), the brain activation information 42 of the first subject R1 is directly transmitted to the second device 30. and transmits the brain activation information 42 of the second subject R2 to the second device 30. When the stimulus was given to the second subject R2, the second subject R2 received visual information different from the cat's face as sensory information47. In this case, the sensory information of the first subject R1 is likely to be appropriate for the second subject R2. will not be transmitted to

[0035] In contrast, in the sensation transfer system 100 of this embodiment, the estimation device 20 To estimate the corresponding sensory information 44 of the subject R2, the first subject R1 is sent to the second subject R2. This allows the visual information of the cat's face to be transmitted appropriately.

[0036] FIG. 5 is a flowchart showing an example of the operation of the sensation transfer system 100. As shown in the figure, in the sensation transmission system 100, the first device 10 transmits the sensation perceived by the first subject R1. Then, brain activation information of the first subject R1 is detected (step S101). The estimation device 20 estimates the first subject R1's perception based on the brain activation information of the first subject R1. Next, the estimation device 20 estimates the reference sensory information recalled by the user (step S102). Based on the estimated reference sensory information, a reference sensory information is generated for a second subject R2, which is different from the first subject R1. The second device estimates corresponding sensory information corresponding to the quasi-sensory information (step S103). 30 provides a stimulus to the second subject R2 to recall the estimated corresponding sensory information (step Top S104).

[0037] As described above, the sensation transmission system 100 according to this embodiment transmits the sensation perceived by the first subject R1. a first device (10) for detecting brain activation information of a first subject (R1) when the first subject (R1) is The criterion is the sensory information recalled for the perception based on the activation information in the brain of the first subject R1. The sensory information is estimated, and a second subject R1 different from the first subject R1 is selected based on the estimated reference sensory information. An estimation device for estimating corresponding sensory information, which is sensory information corresponding to the reference sensory information for subject R2. and a second subject R2 is stimulated to recall the estimated corresponding sensory information. and a device 30.

[0038] The sensation transmission method according to this embodiment is a method for transmitting sensations to a first subject R1 when the first subject R1 perceives the sensations. Detecting the brain activation information of R1 and Based on this, the reference sensory information recalled by the first subject R1 for the perception is estimated, and the estimated reference sensory information is Based on the sensory information, the second subject R2, which is different from the first subject R1, is compared with the reference sensory information. and inferring corresponding sensory information corresponding to the second subject, and instructing the second subject to recall the estimated corresponding sensory information. and administering a stimulus to subject R2.

[0039] According to this configuration, the reference sensory information of the first subject R1 is directly recalled by the second subject R2. Instead of using the reference sensory information, the corresponding sensory information of the second subject R2 is estimated based on the reference sensory information. The first subject R2 was then stimulated to recall the estimated corresponding sensory information. If there is a difference in brain activity between subject R1 and subject R2 when recalling sensory information, Even if the sensory information is transmitted from the first subject R1 to the second subject R2, it is possible to transmit the sensory information appropriately. Cut.

[0040] In the sensory transmission system 100 according to this embodiment, the reference sensory information is brain activation information This is the sensory information recalled by the first subject R1 who detected the By directly associating the sensory information with the sensory information of the second subject R2, the sensory information is appropriately can be transmitted.

[0041] In the sensory transmission system 100 according to this embodiment, the stimulation is applied to the brain of the second subject R2. The method includes irradiating the target area with an electromagnetic wave signal to activate the target area. By directly activating the brain of subject R2, sensory information is more readily available to the second subject R2. It can be easily recalled.

[0042] [Second embodiment] Next, a second embodiment will be described. In the first embodiment, the sensation transmission system 100 Let us take the example of transmitting sensation in one direction from the first subject R1 to the second subject R2. In contrast to this, in the second embodiment, the sensation transfer system 100 transfers the sensation from the second subject R2 to the The sensation is transmitted from the first subject R1 to the first subject R2. This configuration allows sensations to be transmitted bidirectionally between the subject R1 and the second subject R2.

[0043] The overall configuration of the sensation transmission system 100 is the same as that of the first embodiment. 1 and 2, the configuration of the sensation transmission system 100 will be described from the side of the second device 30. explain.

[0044] The second device 30 includes a detection unit 31, a communication unit 32, a processing unit 33, a stimulus applying unit 34, and a storage unit. The processing unit 33, the stimulus applying unit 34, and the storage unit 35 are the same as those in the first embodiment. The detecting unit 31 is the same as the detecting unit 11 in the first embodiment. The communication unit 32 detects brain activation information of the subject R2. The characteristic information is transmitted to the estimation device 20.

[0045] The estimation device 20 includes a communication unit 21, a processing unit 22, and The communication unit 21 is capable of wired communication or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from the second device 30, for example. The communication unit 21 transmits, for example, corresponding sensory information estimated by the processing unit 22 (to be described later) to the first device. Send to 10.

[0046] The processing unit 22 calculates the brain activation information of the second subject R2 based on the detected brain activation information of the second subject R2. The second subject estimates the sensory information (reference sensory information) about the sensations recalled by the subject in response to the perception. For subject R2, what kind of sensory information was recalled and what kind of brain activation information was generated? The correspondence can be determined by conducting experiments in advance. For example, the brain activation information detected from the second subject R2 and the corresponding brain activation information are The sensory information is associated with the target object to create a set of training data sets, which are then used for machine learning. The fourth learning model can be generated by, for example, 23 can be stored.

[0047] Furthermore, the processing unit 22 calculates a reference value for the first subject R1 based on the estimated reference sensory information. In this case, the processing unit 22 estimates corresponding sensory information that is sensory information corresponding to the sensory information. The estimation can be performed based on the second learning model stored in the storage unit 23.

[0048] The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus applying unit 14, and a storage unit. The detecting unit 11 and the communicating unit 12 have the same configuration as in the first embodiment. The communication unit 12 receives the corresponding sensory information transmitted from the estimation device 20.

[0049] The processing unit 13 estimates stimulus image information according to the corresponding sensory information received by the communication unit 12. The image information is information indicating the content of the stimulus to be applied to the first subject R1 by the stimulus application unit 14. .

[0050] The stimulation unit 14 irradiates a target region of the brain of the first subject R1 with an electromagnetic wave signal to stimulate the target region. By activating the first subject R1, a stimulus is given to the first subject R1. As with the stimulation unit 34, the brain of the first subject R1 is represented by a voxel consisting of, for example, several millimeters or less. The stimulation unit 14 divides the area into three voxels by a three-dimensional matrix, and irradiates each voxel with electromagnetic waves. indicates the intensity of the electromagnetic waves to be irradiated to which voxel in the three-dimensional matrix. The electromagnetic waves can be emitted based on the stimulus image information. What kind of sensory information is evoked by irradiating a voxel with electromagnetic waves of what intensity? In this regard, the correspondence can be determined in advance by conducting experiments. For example, Stimulation image information for subject R1 and the results of irradiating electromagnetic waves based on the stimulation image information The sensory information recalled by one subject R1 is associated with the sensory information recalled by the subject R1 to form a set of learning data sets. By applying machine learning to the dataset, a 5th learning model can be generated. The model can be stored in the storage unit 15 of the first device 10, for example.

[0051] The fourth and fifth learning models mentioned above are the first to third learning models. Similarly, it can be generated using a neural network represented by, for example, VGG16. When generating the fourth and fifth learning models, the learning dataset is The correlation of the training data set is input into a neural network and is then analyzed using machine learning such as deep learning. In other words, the neural network is optimized by learning to generate a learning model. Note that this is not limited to convolutional neural networks represented by VGG16, but can also be used with other A variety of neural networks may be used to generate the learning model.

[0052] Next, a method for transmitting sensation using the sensation transmission system 100 configured as described above will be described. The following describes a method for transmitting sensation from the first subject R1 to the second subject R2. The method is the same as that of the first embodiment. The case of transmitting sensation to the subject R1 will be described.

[0053] FIG. 6 is a diagram schematically illustrating an example of the operation of the sensation transmission system 100. In this way, the second subject R2 is made to perceive the reference sensory information. This will be explained using an example in which a person 2 visually perceives a cat's face and recalls it as visual information. In the following examples, the cat's face is the same as in the first embodiment.

[0054] As shown in the upper part of FIG. 6, the detection unit 31 detects the cat's face and recalls sensory information 51. The communication unit 32 detects the brain activation information 52 of the subject R2. The brain activation information 52 is transmitted to the estimation device 20 .

[0055] In the estimation device 20, the communication unit 21 receives the brain activation information transmitted from the third device 130. The processing unit 22 receives the brain activation information 52. Based on the received brain activation information 52, the processing unit 22 calculates the reference sensory information In this case, the processing unit 22 estimates the brain activation information 52 of the second subject R2 as a fourth The fourth learning model inputs the activation information 52 and the reference sensory information 53 into the learning model. Based on the learning results of the correlation between the input brain activation information52 and the reference sensory information, The processing unit 22 acquires the output reference sensory information 53 as an estimation result. do.

[0056] The processing unit 22 calculates the reference sensory information 53 based on the estimated reference sensory information 53. The processing unit 22 estimates the corresponding sensory information 54 corresponding to the quasi-sensory information 53. The reference sensory information 53 is input to the second learning model. Based on the learning result of the correlation between the input reference sensory information 53 and the corresponding sensory information 54, The processing unit 22 outputs corresponding sensory information 54 corresponding to the output corresponding sensory information 53. The communication unit 21 transmits the acquired corresponding sensory information 54 to the first device 10. Send to.

[0057] As shown in the lower part of FIG. 6, in the first device 10, the communication unit 12 receives the signal transmitted from the estimation device 20. The processing unit 13 receives the transmitted corresponding sensory information 54. The corresponding sensory information is input to the fifth learning model stored in the memory unit 15. Based on the learning result of the correlation between 54 and stimulus image information 55, The corresponding stimulation image information 55 is output. The stimulation applying unit 14 Based on this, the first subject R1 is stimulated by irradiating the brain of the first subject R1 with electromagnetic waves. As a result, the first subject R1 to whom the stimulus was given by the stimulus giving unit 14 is The first subject R1 recalls the corresponding sensory information 56. In other words, the first subject R1 recalls the visual information of the cat's face. In this way, the second subject R2 recalls the information from the first subject R1 as corresponding sensory information 56. This transmits visual information about the cat's face.

[0058] As described above, in the sensory transmission system 100 according to this embodiment, the reference sensory information is This is the sensory information recalled by the second subject R2 from whom the brain activation information was detected. By directly associating the sensory information of the second subject R2 with the sensory information of the first subject R1, It is possible to appropriately transmit sensory information between subjects with differences in brain activity.

[0059] [Third embodiment] Next, a third embodiment will be described. In the first and second embodiments, The sensory information recalled by the subject who detected the activation information was explained as the reference sensory information. In contrast, in the third embodiment, the sensory information extracted based on the corresponding sensory information among multiple subjects is used. The following description will be given taking as an example a case where standard sensory information is used as reference sensory information. The overall configuration is the same as that of the first embodiment.

[0060] In the estimation device 20, when the corresponding sensory information is estimated from the reference sensory information, the reference sensory information is Standard sensory information is used as the information. For example, standard sensory information is used when multiple detectors detect the same image. When the same perception is performed, such as viewing an image, the brain activation information detected in multiple subjects is Therefore, the standard sensory information is based on acquired memory, etc. There is little individual variation, and it becomes sensory information for the average person.

[0061] Standard sensory information is extracted from the learning results when learning corresponding sensory information among multiple subjects. For example, a specific subject (e.g., first subject R1 or second subject R2) can be identified. ) and standard sensory information (average of brain activation information of multiple subjects) This is used as a set of training data sets, and the training data sets are used for machine learning to 6. When generating a learning model, the training dataset contains Standard sensory information is extracted from the multiple sensory information contained in the training data, and the extracted standard sensory information is compared with the training data. The brain activation information may be associated with each of the brain activation information included in the dataset. For example, sensory information about the visual sense when looking at an object, or the auditory sense when listening to a sound. sensory information about the sense of touch when touching an object, Correspondence between individual brain activation information and standard sensory information for multiple subjects for various sensory information The sixth learning model is generated by machine learning the relationship. can be stored in

[0062] Next, a method for transmitting sensation using the sensation transmission system 100 configured as described above will be described. FIG. 7 is a diagram illustrating an example of the operation of the sensation transmission system 100. For example, when transmitting sensory information 61 when a first subject R1 perceives something to a second subject R2, The first device 10 acquires brain activation information 62 of the first subject R1, and the information is sent to the estimation device 20. Send.

[0063] In the estimation device 20, the processing unit 33 receives the brain activation information 6 transmitted from the first device 10. 2 and the identification information of the second subject R2 to whom the sensory information is to be transmitted are stored in the storage unit 35. The sixth learning model inputs the standard sense corresponding to the brain activation information 62. The sensory information 63 of the second subject R2 is calculated, and the sensory information of the second subject R2 associated with the standard sensory information 63 is calculated. The information is output as corresponding sensory information 64. The communication unit 21 transmits the output corresponding sensory information 64 to Transmit to the second device 30.

[0064] In the second device 30, the corresponding sense transmitted from the estimation device 20 is Information 64 is received, and stimulation image information 65 is acquired based on the received corresponding sensory information 64. The stimulation applying unit 34 applies electromagnetic waves to the brain of the second subject R2 based on the output stimulation image information 65. The stimulus is applied to the second subject R2 by irradiating the stimulus. The second subject R2 recalls corresponding sensory information 66 corresponding to the stimulus image information 65.

[0065] As described above, in the sensory transmission system 100 according to this embodiment, the reference sensory information is This is standard sensory information 63 extracted based on corresponding sensory information among multiple subjects. In this configuration, standard sensory information6 is extracted based on corresponding sensory information among multiple subjects. 3 is used as the reference sensory information, so that sensory information can be transmitted appropriately among many subjects. It is possible.

[0066] [Fourth embodiment] FIG. 8 is a diagram showing an example of a sensation transmission system 200 according to the fourth embodiment. The sensory transmission system 100 is suitable for transmitting sensory information between different subjects. In contrast to this, in the sensation transmission system 200 described in the fourth embodiment, An example of the case where sensory information is transmitted between subjects will be described.

[0067] As shown in FIG. 8, the sensory transmission system 200 includes a detection and stimulation device (detection device, stimulation device). The detecting and stimulating device 110 includes, for example, the detecting and stimulating device described in the above embodiment. The first device 10 has the same configuration as the first device 10 described above, and includes a detection unit 11, a communication unit 12, a processing unit 13, and a stimulator. The device includes a stimulus applying unit 14 and a memory unit 15. The detecting unit 11 detects brain activation information. The receiving unit 12 performs wired or wireless communication and estimates the brain activation information detected by the detecting unit 11. The processing unit 13 then transmits the corresponding sensory information to the measuring device 20. The stimulus applying unit 14 calculates stimulus image information. Electromagnetic signals are irradiated to the target area of ​​R4's brain to activate the target area, and subject R4 is stimulated. The storage unit 15 stores various types of information.

[0068] The estimation device 120 includes a communication unit 21, a processing unit 22, and a storage unit 23. In this embodiment, the communication unit 21 is capable of wired or wireless communication. The communication unit 21 receives brain activation information transmitted from the detection and stimulation device 110. The corresponding sensory information estimated by the logic unit 22 is transmitted to the sensory stimulation device 110 as described below.

[0069] The processing unit 22 calculates a brain activation value based on the brain activation information of the target subject R4 detected at the first time point. The first time point is when the target subject R4 was a child. The time point may be at a young age, for example, the target subject R4 may be less than 3 years old.

[0070] The reference sensory information is, for example, the sensory information recalled by subject R4 at the first time point. In this case, the processing unit 22 uses the same sensory information as the first learning model described above. The reference sensory information can be estimated based on the seventh learning model. , can be stored in the storage unit 25, for example.

[0071] In addition, the reference sensory information can be extracted based on corresponding sensory information among multiple subjects. The standard sensory information may be, for example, the standard sensory information of the target subject R4. The information is extracted based on the sensory information recalled by multiple subjects whose brain growth states correspond to each other. Such a plurality of subjects can be, for example, a plurality of subjects of corresponding ages. Multiple subjects with corresponding growth environments (latitude, cultural environment, language used, etc.) and the same occupation For example, the processing unit 22 may process the target subject at the first time point. The eighth learning model can be used to estimate standard sensory information based on the brain activation information of R4. The eighth learning model can be stored in the storage unit 25, for example.

[0072] Furthermore, the processing unit 22 determines whether or not a time has elapsed since the first time point based on the estimated reference sensory information. The reference sensory information and corresponding sensory information for the target subject R4 at the second time point are For example, if the degree of brain growth of subject R4 is large between the first and second time points, For example, the relationship between the target subject R4 at the first time point and the target subject R4 at the second time point may be different. The corresponding sensory information between them is used as a set of training data sets, and the training data sets are used for machine learning. The ninth learning model can be generated by performing The information can be stored in the memory 23.

[0073] Next, a method for transmitting sensation using the sensation transmission system 200 configured as described above will be described. FIG. 9 is a diagram illustrating an example of the operation of the sensation transmission system 100. As shown in the upper part of the figure, the detection unit 11 of the detection and stimulation device 110 detects the cat's face and outputs sensory information 7. The communication unit 12 detects the brain activation information 72 of the target subject R4 who recalls the event 1. The brain activation information 72 detected in step 1 is transmitted to the estimation device 120. Then, the communication unit 21 receives the brain activation information 72 transmitted from the detecting and stimulating device 110 . The processing unit 22 inputs the received brain activation information into the seventh learning model or the eighth learning model. From the seventh learning model or the eighth learning model, the basic information corresponding to the input brain activation information 72 is obtained. The quasi-sensory information 73 is output. The processing unit 22 regards the output reference sensory information 73 as an estimation result. The storage unit 25 stores the acquired reference sensory information 73.

[0074] As shown in the lower part of FIG. 9, at a second time point after the first time point, for example, If subject R4 were to relive the perception of the cat face described above, subject R4 would use the detection stimulus device. The device 110 is prepared to receive stimulation from the stimulation unit 14. The processing unit 22 determines the target subject R4 based on the reference sensory information 73 stored in the memory unit 25. The reference sensory information 73 and the corresponding sensory information 74 are estimated. The unit 22 inputs the acquired reference sensory information 73 into the ninth learning model. The processing unit 22 outputs corresponding sensory information 74 corresponding to the input reference sensory information 73. The communication unit 21 acquires the outputted corresponding sensory information 74 as an estimation result. The sensory information 74 is transmitted to the sensory stimulation device 110 .

[0075] In the detection and stimulation device 110, the communication unit 12 receives the corresponding sensation transmitted from the estimation device 120. The processing unit 13 receives the corresponding sensory information 74. The processing unit 13 stores the received corresponding sensory information 74 in the storage unit 15. The third learning model then outputs the corresponding sensory information74 to the input. The stimulus applying unit 14 outputs the corresponding stimulus image information 75. Based on this, electromagnetic waves are irradiated onto the brain of subject R4 to stimulate subject R4. As a result, the target subject R4 to whom the stimulus was given by the stimulus giving unit 14 responded to the stimulus image information 75 In other words, subject R4 saw the cat's face at the first time point and recalled the corresponding sensory information 76. The visual information at the time of the first movement is recalled as corresponding sensory information 76 at the second time point. Visual information about the cat's face was transmitted from subject R4 at time point 1 to subject R4 at time point 2. This allows subject R4 to relive the experience of seeing the cat's face.

[0076] FIG. 10 is a flowchart showing an example of the operation of the sensation transfer system 200. As shown in FIG. 1, in the sensation transfer system 200, the detection and stimulation device 110 detects a first time point. When the target subject R4 perceives the 1). Next, the estimation device 20 estimates the brain activation information of the target subject R4 at the first time point. Next, the estimation device 20 estimates the reference sensory information based on the estimated reference sensory information (step S202). Reference sensory information about a second subject R2 that differs from a first subject R1 based on quasi-sensory information Then, the second device 30 estimates the corresponding sensory information corresponding to the estimated sensory information (step S103). A stimulus is given to the second subject R2 so as to recall the corresponding sensory information (step S10 4).

[0077] As described above, the sensation transmission system 200 according to this embodiment is capable of transmitting the sensation perceived by the target subject R4. A detection device (detection stimulation device 11) that detects activation information in the brain of the target subject R4 when 0) and the perception based on the brain activation information of the target subject R4 detected at the first time point. The sensory information that is recalled in response to the sensory information is estimated, and the sensory information is calculated based on the estimated sensory information. The baseline sensation for subject R4 at the second time point, which was later than the first time point, was An estimation device 120 that estimates corresponding sensory information corresponding to the information, and an estimation device 120 that estimates the estimated corresponding sensory information. A stimulation device (detection stimulation device 1) that stimulates target subject R4 at a second time point to induce a 10) and

[0078] In addition, the sensation transmission method according to this embodiment is a target sensation when perceived by the target subject R4. Detecting brain activation information of subject R4 and detecting the target subject R4 at the first time point Based on the activation information in the brain of R4, reference sensory information is the sensory information that is recalled for perception. Based on the estimated reference sensory information, at a second time point after the first time point, Estimating corresponding sensory information corresponding to the reference sensory information for target subject R4 in the At the second time point, subject R4 was given a stimulus to recall the estimated corresponding sensory information. This includes:

[0079] According to this configuration, the reference sensory information of the target subject R4 at the first time point is directly used as the target sensory information of the target subject R4 at the second time point. Rather than instructing the subject R4 to recall the same, the subject R4 at the second time point was instructed to recall the same based on the reference sensory information. The corresponding sensory information of 4 is estimated, and the subject is asked to recall the estimated corresponding sensory information. Stimulation is given to R4. Therefore, target subject R4 at the first time point and target subject R4 at the second time point Even if there are differences in brain activity when recalling sensory information between It can be transmitted.

[0080] In the sensory transmission system 200 according to this embodiment, the reference sensory information is In this configuration, the sensory information recalled by the target subject R4 at the first time point is The sensory information of the subject R4 at the second time point is directly associated with the sensory information of the target subject R4 at the second time point. This allows sensory information to be transmitted appropriately.

[0081] In the sensory transmission system 200 according to this embodiment, the reference sensory information is The standard sensory information is extracted based on the corresponding sensory information between the plurality of sensors. Standard sensory information extracted based on corresponding sensory information between subjects is used as reference sensory information. Therefore, sensory information can be efficiently transmitted between subject R4 at different times. can.

[0082] In the sensory transmission system 200 according to this embodiment, the standard sensory information is The brain growth state is extracted based on the sensory information recalled by multiple subjects. In this configuration, the sensory information recalled by multiple subjects whose brain growth state corresponds to that of the target subject R4 was Based on this, sensory information can be efficiently transmitted between target subjects R4 at different times. .

[0083] [Fifth embodiment] FIG. 11 is a diagram illustrating an example of a sensation transfer system 300 according to the fifth embodiment. As shown in FIG. 10, the sensation transfer system 300 according to the fifth embodiment includes a sensation estimation device 220, and a stimulation device 210.

[0084] The sensation estimation device 220 includes a communication unit 221, a processing unit 222, a storage unit 223, and an input unit 224. The communication unit 221 performs wired or wireless communication with the stimulation device 210. The storage unit 223 stores reference sensory information that is sensory information that is recalled in response to a perception. The memory unit 223 is a storage device such as a hard disk drive or a solid state drive. The storage unit 223 may be an external storage medium such as a removable disk. The reference sensory information may be used as the sensory information that subject R5 or a person different from subject R5 perceives. It may be sensory information recalled by a subject, or sensory information corresponding to multiple subjects. It may also be standard sensory information extracted based on the sensory information.

[0085] The processing unit 222 calculates the reference sensory information for the subject R5 based on the reference sensory information stored in the storage unit 223. For example, subject R estimates the corresponding sensory information, which is sensory information corresponding to the reference sensory information. The corresponding sensory information between the sensory information of 5 and the reference sensory information is a set of learning data sets; By performing machine learning on this learning dataset, a 10th learning model can be generated. The tenth learning model can be stored in the storage unit 223, for example. 2, for example, when reference sensory information is designated by the input unit 224 described later, the designated reference sensory information is Corresponding sensory information corresponding to the quasi-sensory information is estimated.

[0086] The input unit 224 allows a predetermined input operation for inputting information. For example, an input device such as a keyboard or a touch panel is used. 4. In addition to or instead of these, buttons, levers, dials, switches or other inputs The input unit 224 may be configured to receive, for example, a plurality of bases stored in the storage unit 223. From the quasi-sensory information, subject R5 can input the sensory information he or she wants to experience.

[0087] The stimulation device 210 includes a communication unit 212, a processing unit 213, a stimulation applying unit 214, and a storage unit 215. 15. The communication unit 212 is capable of wired communication or wireless communication. , and receives corresponding sensory information transmitted from the sensory estimation device 220.

[0088] The processing unit 213 receives the corresponding sensory information from the communication unit 212 and combines the corresponding sensory information with a stimulation image. Based on the 11th learning model that has learned the correspondence between the received sensory information and the sensory information, This eleventh learning model calculates stimulus image information corresponding to the first embodiment. The stimulus providing unit 214 can be a learning model similar to the third learning model in the previous example. By irradiating the target area of ​​the brain of subject R5 with an electromagnetic signal, the target area is activated. Give R5 some excitement.

[0089] Next, a method for transmitting sensation using the sensation transmission system 300 configured as described above will be described. FIG. 12 is a schematic diagram illustrating an example of the operation of the sensation transmission system 300 according to this embodiment. The subject R5 is given a stimulus from the stimulus applying unit 214 of the stimulus device 210. The subject R5 inputs the reference sense via the input unit 224. When the sensory information 81 is selected, the processing unit 222 in the sensory estimation device 220 Based on the reference sensory information 81, corresponding sensory information 82 for the subject R5 is estimated. The logic unit 222 inputs, for example, the selected reference sensory information 81 into the tenth learning model. The learning model outputs corresponding sensory information 82 corresponding to the input reference sensory information 81. The processing unit 222 acquires the output corresponding sensory information 82 as an estimation result. 21 transmits the acquired corresponding sensory information 82 to the stimulation device 210.

[0090] In the stimulation device 210, the communication unit 212 receives the corresponding sensation transmitted from the sensation estimation device 220. The processing unit 213 receives the corresponding sensory information 82. The processing unit 213 stores the received corresponding sensory information 82 in the storage unit 215. The eleventh learning model stores the corresponding sensory information. The stimulus applying unit 214 outputs the stimulus image information 83 corresponding to the stimulus image information 82. Based on information 83, stimulate subject R5 by irradiating his / her brain with electromagnetic waves. As a result, the subject R5 to whom the stimulus was given by the stimulus giving unit 214 responded to the stimulus image information 83. The subject R5 recalls the corresponding sensory information 84. The sensory information 84 is then recalled from the sensory estimation device 220 to the subject R5. , visual information of the cat's face is transmitted.

[0091] 13 is a flowchart showing an example of the operation of the sensation transfer system 300. As shown in FIG. 1, when the subject R5 selects the reference sensory information 81 via the input unit 224, In this case, in the sensation estimation device 220, the processing unit 222 acquires the selected reference sensation information 81. Based on the acquired reference sensory information 81 (step S301), The corresponding sensory information 82 is estimated (step S302). A stimulus is given to the fifth subject R5 so as to recall the corresponding sensory information 82 (step S30 3).

[0092] As described above, the sensation estimation device 220 according to this embodiment is capable of estimating sensations associated with perception. a memory unit 223 for storing reference sensory information, which is information; and a reference sensory information stored in the memory unit 223. Based on the information, corresponding sensory information is sensory information corresponding to the reference sensory information for subject R5. and a processing unit 222 for estimating the information.

[0093] Furthermore, the sensation estimation method according to this embodiment uses a reference sensory information that is associated with a perception. acquiring reference sensory information from a memory unit 223 that stores sensory information; Based on the information, corresponding sensory information is sensory information corresponding to the reference sensory information for subject R5. and estimating information.

[0094] According to this configuration, the sensory estimation device 220 estimates the sensory information corresponding to the subject R5 based on the reference sensory information. In order to estimate the corresponding sensory information, appropriate sensory information can be estimated for each subject R5. do.

[0095] In the sensation estimation device 220 according to this embodiment, the reference sensation information is different from that of the subject R5. This is the sensory information that a person recalls in response to perception. Appropriate sensory information can be estimated between subjects.

[0096] In the sensation estimation device 220 according to this embodiment, the reference sensation information is The standard sensory information is extracted based on the sensory information corresponding between the various sensors. It is possible to transmit sensory information appropriately between many subjects.

[0097] The sensation transmission system 300 according to this embodiment includes the sensation estimation device 220 and the sensation estimation device 220. Stimulation is given to subject R5 so as to recall the corresponding sensory information estimated by device 220. The stimulation device 210 is provided. According to this configuration, the reference sensory information stored in the storage unit 223 Based on this, appropriate sensory information can be estimated for each subject R5 and transmitted to the subject R5. do.

[0098] The technical scope of the present invention is not limited to the above-described embodiment, and any modifications may be made without departing from the spirit of the present invention. For example, in the above embodiments, the visual sensory information However, the present invention is not limited to this example, and brain activation information may be any detectable sensory information. Other sensory information may also be used if it is available. For example, information obtained by linking sensory information from the five senses may be used. It may also be detailed sensory information that is acquired.

[0099] When generating a learning model, the content presented to the subject is something like "Raise your right hand." The learning model may be configured to learn the response of the subject's motor cortex by using the instruction. When generating the content, the content presented to the subject is a movie-like viewing content, and the subject The configuration may be such that the response of the entire brain is learned.

[0100] In addition, IDs are assigned to the elements of standard sensory information obtained when generating the learning model, and The closer the correlation with the content, the closer the ID. For example, "dog", "cat", If there is a content of "paper", "dog" and "cat" are assigned similar IDs, and "dog" and For example, "cat" and "paper" may be assigned different IDs.

[0101] In addition, if there is no corresponding sensory information corresponding to the standard sensory information, Content with a similar ID may be used as corresponding sensory information, or the corresponding sensory information may not be transmitted. It may be composed of

[0102] In addition, we present the corresponding sensory information derived from a certain standard sensory information by replacing it with another sensory information. For example, the olfactory sense of a mandarin orange can be calculated based on standard sensory information corresponding to the visual information of the mandarin orange. Alternatively, corresponding sensory information, which has been replaced with information, taste information, or tactile information, may be transmitted to the destination.

[0103] In addition, when transmitting sensory information from the first subject R1 to the second subject R2, The command information "send this sensory information to the first subject R1" was deleted, and "the first subject R1 sends this sensory information to the second subject R1" was deleted. "The first subject R1 recalls sensory information like this to the second subject R2. This is a communication equivalent to a packet header such as "requesting the recall of sensory information." The communication control information may be added.

[0104] In addition, when extracting standard sensory information, online learning is performed to constantly update the content. The estimated sensory information may be stored in a storage unit, and after a predetermined period of time has elapsed, The information may be transmitted to the destination after being transmitted. [Explanation of symbols]

[0105] NW: neural network, R1: first subject, R2: second subject, R4: target subject R5...subject, S1...convolutional layer, S2...pooling layer, S3...connection layer, 10...th 1 device, 11, 31... detection unit, 12, 21, 32, 212, 221... communication unit, 13, 22 ,33,213,222...processing unit, 14,34,214...stimulation unit, 15,23,25 ,35,215,223...Storage unit, 20,120...Estimation device, 30...Second device, 41,4 7, 51, 61, 71...sensory information, 42, 52, 62, 72...brain activation information, 43, 5 3, 73, 81... Reference sensory information, 44, 46, 54, 56, 64, 66, 74, 76, 8 2, 84... Corresponding sensory information, 45, 55, 65, 75, 83... Stimulus image information, 63... Standard sensory Information, 100, 200, 300...Sensory transmission system, 110...Detection and stimulation device, 130... 3. Device: 210...stimulation device; 220...sensation estimation device; 224...input unit

Claims

1. a detection unit for detecting brain activation information of a first subject; a processing unit that estimates reference sensory information about the sensations that the first subject recalls in response to the perception based on the detected brain activation information of the first subject, and estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information about a second subject different from the first subject, based on the reference sensory information; A sensation estimation device comprising:

2. The sensation estimation device according to any one of claims 1 to 3; a stimulation device that stimulates the second subject so as to recall the corresponding sensory information estimated by the sensory estimation device; A sensory transmission system comprising:

3. detecting brain activation information of a first subject; acquiring reference sensory information from a storage unit that stores reference sensory information that is sensory information recalled in response to perception; Based on the detected brain activation information of the first subject, estimate reference sensory information regarding the sensation evoked by the subject in response to the perception, and based on the reference sensory information, estimate corresponding sensory information, which is sensory information corresponding to the reference sensory information of a second subject different from the first subject; A sensory estimation method comprising:

Images