Sensory transmission system and sensory transmission method

Abstract

To transmit properly sensation, when there is a difference of environment around each subject.SOLUTION: A sensory transmission system includes a detection device for detecting intracerebral activation information of a target subject, when being perceived by the target subject, an estimation device for estimating reference feeling information which is feeling information recalled to perception. based on the intracerebral activation information of the target subject detected at a first point of time, and for estimating corresponding feeling information corresponding to the reference feeling information about the target subject at a second point of time having a time elapsed from the first point of time, based on the estimated reference feeling information, and a stimulator for stimulating the target subject at the second point of time so as to recall the estimated corresponding feeling information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a sensory transmission system and a sensory transmission method. [Background technology]

[0002] In recent years, technologies for non-invasively measuring brain activation information, such as functional magnetic resonance and near-infrared spectroscopy, have been developed, and brain-machine interface technology, which is an interface between the brain and the outside, is becoming a reality. As an example of using such technology, a configuration has been disclosed in which brain activation information of a sleeping subject is detected to determine the subject's sleep state, and if the subject is determined to be in REM sleep, a dream-inducing stimulus is given to the subject (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-332251 Summary of the Invention [Problem to be solved by the invention]

[0004] The correspondence between the subject's brain activity state and the measured brain activation information varies depending on the subject's personality, the environment surrounding the subject at the time of measurement, etc. For example, with the technology described in Patent Document 1, the type of dream a subject will have when given a dream-inducing stimulus varies depending on the subject or the timing of the stimulus, making it difficult to control the content of the dream itself. In response to this, when determining or controlling the brain activity state based on the subject's brain activation information, a technology is needed that takes into account factors such as individual differences or differences in the environment surrounding the subject.

[0005] The present invention has been made in consideration of the above, and aims to provide a sensory transmission system and a sensory transmission method that can appropriately transmit sensations when there are differences in the environment surrounding the subject. [Means for solving the problem]

[0006] The sensory transmission system of the present invention comprises a detection device that detects activation information in the brain of a target subject when the target subject perceives something; an estimation device that estimates reference sensory information, which is sensory information that is evoked in response to the perception, based on the activation information in the brain of the target subject detected at a first point in time, and estimates corresponding sensory information that corresponds to the reference sensory information for the target subject at a second point in time that is later than the first point in time, based on the estimated reference sensory information; and a stimulation device that provides stimulation to the target subject at the second point in time so as to evoke the estimated corresponding sensory information.

[0007] The sensory transmission method of the present invention includes detecting brain activation information of a target subject when the target subject perceives something, estimating reference sensory information, which is sensory information evoked in response to the perception, based on the brain activation information of the target subject detected at a first time point, estimating corresponding sensory information corresponding to the reference sensory information for the target subject at a second time point that is later than the first time point based on the estimated reference sensory information, and providing a stimulus to the target subject at the second time point so as to evoke the estimated corresponding sensory information. [Effects of the Invention]

[0008] According to the present invention, it is possible to appropriately transmit sensations when there is a difference in the environment surrounding the subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a sensation transfer system according to the first embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of a sensation transmission system according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a neural network. [Figure 4] FIG. 4 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the sensation transfer system according to the first embodiment. [Figure 6] FIG. 6 is a diagram schematically illustrating another example of the operation of the sensation transfer system according to the second embodiment. [Figure 7] FIG. 7 is a diagram schematically illustrating another example of the operation of the sensation transmission system according to the third embodiment. [Figure 8] FIG. 8 is a functional block diagram illustrating an example of a sensation transmission system according to the fourth embodiment. [Figure 9] FIG. 9 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the fourth embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of the sensation transfer system according to the fourth embodiment. [Figure 11] FIG. 11 is a functional block diagram illustrating an example of a sensation transfer system according to the fifth embodiment. [Figure 12] FIG. 12 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the fifth embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the operation of the sensation transfer system according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially the same.

[0011] [First embodiment] Fig. 1 is a schematic diagram showing an example of a sensation transfer system 100 according to a first embodiment. Fig. 2 is a functional block diagram showing an example of the sensation transfer system 100. As shown in Figs. 1 and 2, the sensation transfer system 100 includes a first device 10, an estimation device 20, and a second device 30.

[0012] The first device 10 detects brain activation information of a first subject R1 when perceived by the first subject R1. The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus application unit 14, and a storage unit 15.

[0013] The detector 11 detects brain activation information. Examples of the brain activation information include the oxygenated hemoglobin concentration, deoxygenated hemoglobin concentration, and total hemoglobin concentration contained in the subject's cerebral blood flow. The detector 11 may be, for example, a measuring device that performs measurements based on the principles of fMRI (functional Magnetic Resonance Imaging) or fNIRS (functional Near-Infrared Spectroscopy), a measuring device using invasive electrodes, or a measuring device that performs measurements using a micromachine placed in the blood vessels of the brain. The detector 11 is not limited to the above devices, and other types of devices may also be used. The brain activation information can be expressed as the magnitude of activity for each voxel when the brain of the first subject R1 is partitioned into a three-dimensional matrix consisting of voxels of several millimeters or less.

[0014] The communication unit 12 is an interface for wired or wireless communication. The communication unit 12 transmits the brain activation information detected by the detection unit 11 to the estimation device 20. The communication unit 12 includes an interface for communicating with an external device via a so-called wireless LAN in accordance with the IEEE802.11 standard. The communication unit 12 may realize communication with an external device under the control of the processing unit 22 or the like. Note that the communication method is not limited to wireless LAN, and may include, for example, an infrared communication method, a Bluetooth (registered trademark) communication method, a wireless communication method such as Wireless USB, and the like. Furthermore, a wired connection such as a USB cable, HDMI (registered trademark), IEEE1394, or Ethernet may also be adopted.

[0015] The processing unit 13, the stimulus applying unit 14, and the storage unit 15 will be described later. In the first embodiment, the processing unit 13, the stimulus applying unit 14, and the storage unit 15 do not necessarily have to be provided.

[0016] The estimation device 20 includes a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is an interface that performs wired or wireless communication. The communication unit 21 receives brain activation information transmitted from the first device 10, for example. The communication unit 21 transmits corresponding sensory information (described below) estimated by the processing unit 22 to the second device 30, for example. Note that the communication unit 21 may have a configuration similar to that of the communication unit 12 described above.

[0017] The processing unit 22 has a processing device such as a CPU (Central Processing Unit) and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The processing unit 22 performs various processes including the estimation process described below.

[0018] The storage unit 23 stores various types of information. The storage unit 23 has storage such as a hard disk drive, a solid state drive, etc. Note that an external storage medium such as a removable disk may also be used as the storage unit 23.

[0019] The processing unit 22 estimates sensory information (reference sensory information) about the sensations evoked by the first subject R1 in response to the perception based on the detected brain activation information of the first subject R1. The sensory information may be information about at least one of the five senses, i.e., vision, hearing, touch, taste, and smell, or may be information about balance or other somatic sensations. Specifically, when the sensory information is visual information, the sensory information is image data perceived by the first subject R1. However, the sensory information is not limited to this. It may be image data obtained by sampling the image data or image data obtained by filtering the image data, rather than the image data itself. Furthermore, when the sensory information is visual information, the sensory information may be information about light entering the eyeball of the first subject R1. In this case, for example, the light information may be obtained using contact lenses equipped with optical sensors. Alternatively, the light information may be obtained using an artificial retina. Specifically, the light information from a CCD sensor provided in the artificial retina may be used. Furthermore, when the sensory information is auditory information, the sensory information may be audio signal data perceived by the first subject R1. Furthermore, if the sensory information is information about taste, the sensory information may be data indicating indicators of multiple chemical substances that reproduce the taste perceived by the first subject R1. If the sensory information is information about touch, the sensory information may be data indicating which part of the entire body surface of the first subject R1 is stimulated and to what extent in the unfolded view. Note that these sensory information examples are not limiting. Regarding the first subject R1, the correspondence between what kind of brain activation information is associated with what kind of sensory information is determined in advance by conducting experiments. For example, the brain activation information detected from the first subject R1 and the sensory information corresponding to the brain activation information are associated to form a set of training data sets, and the training data sets are subjected to machine learning to generate a first training model. The first training model may be stored, for example, in the memory unit 23.

[0020] Furthermore, the processing unit 22 estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information of a second subject R2 different from the first subject R1, based on the estimated reference sensory information. A specific example of the corresponding sensory information may correspond to a specific example of sensory information. The second subject R2 is a subject to whom the sensations of the first subject R1 are transmitted. The relationship between the reference sensory information and the corresponding sensory information can be determined by conducting experiments in advance. For example, the corresponding sensory information between the first subject R1 and the second subject R2 is used as a set of learning data sets, and a second learning model can be generated by machine learning the learning data sets. The second learning model can be stored, for example, in the memory unit 23.

[0021] The second device 30 applies a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information. The second device 30 has a detection unit 31, a communication unit 32, a processing unit 33, a stimulus application unit 34, and a storage unit 35.

[0022] The detection unit 31 will be described later. In the first embodiment, the detection unit 31 does not necessarily have to be provided.

[0023] The communication unit 32 is an interface that performs wired or wireless communication. The communication unit 32 receives corresponding sensory information transmitted from the estimation device 20. The communication unit 32 also transmits brain activation information detected by the detection unit 31 to the estimation device 20. The communication unit 32 may have the same configuration as the communication unit 12 described above.

[0024] The stimulation unit 34 stimulates the second subject R2 by irradiating a target region of the brain of the second subject R2 with an electromagnetic wave signal to activate the target region. In this case, the brain of the second subject R2 is partitioned into a three-dimensional matrix consisting of voxels, for example, several millimeters or less, and electromagnetic waves are irradiated to each voxel. The stimulation unit 34 can irradiate electromagnetic waves based on stimulation image information indicating which voxels in the three-dimensional matrix should be irradiated with electromagnetic waves and at what intensity. The voxels in the three-dimensional matrix of stimulation image information may correspond in size, position, etc. to the voxels in the three-dimensional matrix of brain activation information. The correspondence between which voxels in the brain of the second subject R2 are irradiated with electromagnetic waves and at what intensity will evoke what sensory information can be determined by conducting experiments in advance. For example, a third learning model can be generated by associating stimulus image information for the second subject R2 with sensory information evoked by the second subject R2 when irradiated with electromagnetic waves based on the stimulus image information to create a set of learning data sets, and then performing machine learning on the learning data sets. The third learning model can be stored in the memory unit 35 of the second device 30, for example. The processing unit 33 can calculate stimulus image information corresponding to the received corresponding sensory information based on the corresponding sensory information received by the communication unit 32 and the third learning model.

[0025] The first learning model, second learning model, and third learning model described above can be generated using a neural network (convolutional neural network) represented by, for example, VGG16. FIG. 3 is a diagram showing an example of a neural network. As shown in the upper part of FIG. 3, the neural network NW has 13 convolutional layers S1, 5 pooling layers S2, and 3 fully connected layers S3. The neural network processes input information in the convolutional layers S1 and pooling layers S2 in order, and the processing results are combined in the fully connected layer S3 and output.

[0026] When generating the first, second, and third learning models, as shown in the middle of Figure 3, learning data sets containing corresponding information (I1, I2) are input to the neural network NW, and the correlations between the learning data sets are learned by machine learning such as deep learning. In other words, the neural network NW is optimized through learning to generate learning models. For example, the neural network NW is trained to solve a problem that requires one piece of information when one piece of information constituting each learning data set is input.

[0027] When performing inference using the first learning model, the second learning model, and the third learning model, one piece of information I1 of two pieces of information constituting each learning data set is input to the neural network NW, as shown in the lower part of Figure 3. The first learning model, the second learning model, and the third learning model output the other piece of information I2 corresponding to the input piece of information I1 based on the learning results of the correlation between the pieces of information constituting the learning data set. Note that, although this embodiment describes an example in which a learning model is generated using a convolutional neural network represented by VGG16, the present invention is not limited to this, and learning models may be generated using other types of neural networks.

[0028] Next, a sensory transmission method using the sensory transmission system 100 configured as described above will be described. Fig. 4 is a diagram schematically illustrating an example of the operation of the sensory transmission system 100. As shown in Fig. 4, the first subject R1 is made to perceive reference sensory information so as to recall it. Below, an example will be described in which the first subject R1 visually perceives a cat's face and recalls it as visual information.

[0029] 4, the detection unit 11 detects brain activation information 42 of a first subject R1 who perceives a cat's face and recalls sensory information 41. The communication unit 12 transmits the brain activation information 42 detected by the detection unit 11 to the estimation device 20.

[0030] In the estimation device 20, the communication unit 21 receives brain activation information 42 transmitted from the first device 10. The processing unit 22 estimates reference sensory information 43 based on the received brain activation information 42. In this case, the processing unit 22 inputs the brain activation information 42 of the first subject R1 to a first learning model. The first learning model outputs reference sensory information 43 corresponding to the input brain activation information 42 based on the learning result of the correlation between the brain activation information 42 and the reference sensory information 43. The processing unit 22 acquires the output reference sensory information 43 as an estimation result.

[0031] The processing unit 22 estimates corresponding sensory information 44 for the second subject R2 based on the estimated reference sensory information 43. In this case, the processing unit 22 inputs the acquired reference sensory information 43 into the second learning model. The second learning model outputs corresponding sensory information 44 corresponding to the input reference sensory information 43 based on the learning result of the correlation between the reference sensory information 43 and the corresponding sensory information 44. The processing unit 22 acquires the output corresponding sensory information 44 as an estimation result. The communication unit 21 transmits the acquired corresponding sensory information 44 to the second device 30.

[0032] In the second device 30, the communication unit 32 receives the corresponding sensory information 44 transmitted from the estimation device 20. The processing unit 33 inputs the received corresponding sensory information 44 into a third learning model stored in the memory unit 35. Based on the learning result of the correlation between the corresponding sensory information 44 and the stimulus image information, the third learning model outputs stimulus image information 45 corresponding to the input corresponding sensory information 44. The stimulus application unit 34 applies a stimulus to the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulus image information 45. As a result, the second subject R2, who has been stimulated by the stimulus application unit 34, recalls corresponding sensory information 46 corresponding to the stimulus image information 45. In other words, the second subject R2 recalls the visual information of the cat's face as the corresponding sensory information 46.

[0033] On the other hand, due to differences in personality between the first subject R1 and the second subject R2, the correspondence between brain activity and brain activation information is different. For this reason, as shown in the lower part of Figure 4 (indicated by the dashed line), if brain activation information 42 of the first subject R1 is directly transmitted to the second device 30 and the second device 30 provides stimulation to the brain of the second subject R2 in accordance with the brain activation information 42, the second subject R2 is likely to recall visual information other than a cat's face as sensory information 47. In this case, the sensory information of the first subject R1 will not be properly transmitted to the second subject R2.

[0034] In contrast, in the sensory transmission system 100 of this embodiment, the corresponding sensory information 44 of the second subject R2 is estimated in the estimation device 20, so that visual information of the cat's face is appropriately transmitted from the first subject R1 to the second subject R2.

[0035] FIG. 5 is a flowchart showing an example of the operation of the sensory transfer system 100. As shown in FIG. 5, in the sensory transfer system 100, the first device 10 detects brain activation information of the first subject R1 when the first subject R1 perceives something (step S101). Next, the estimation device 20 estimates reference sensory information that the first subject R1 recalls in response to the perception based on the brain activation information of the first subject R1 (step S102). Next, the estimation device 20 estimates corresponding sensory information for a second subject R2, different from the first subject R1, that corresponds to the reference sensory information based on the estimated reference sensory information (step S103). Then, the second device 30 provides a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information (step S104).

[0036] As described above, the sensory transmission system 100 of this embodiment comprises a first device 10 that detects brain activation information of a first subject R1 when perceived by the first subject R1, an estimation device 20 that estimates reference sensory information, which is sensory information evoked in response to the perception based on the detected brain activation information of the first subject R1, and estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information for a second subject R2 different from the first subject R1, based on the estimated reference sensory information, and a second device 30 that provides stimulation to the second subject R2 so as to evoke the estimated corresponding sensory information.

[0037] The sensory transmission method of this embodiment includes detecting brain activation information of a first subject R1 when the first subject R1 perceives something, estimating reference sensory information that the first subject R1 recalls in response to the perception based on the detected brain activation information of the first subject R1, estimating corresponding sensory information corresponding to the reference sensory information for a second subject R2 different from the first subject R1 based on the estimated reference sensory information, and providing a stimulus to the second subject R2 to recall the estimated corresponding sensory information.

[0038] According to this configuration, instead of simply evoking the reference sensory information of the first subject R1 in the second subject R2, the corresponding sensory information of the second subject R2 is estimated based on the reference sensory information, and stimulation is given to the second subject R2 so as to elicit the estimated corresponding sensory information. Therefore, even if there is a difference in brain activity when recalling sensory information between the first subject R1 and the second subject R2, sensory information can be appropriately transmitted from the first subject R1 to the second subject R2.

[0039] In the sensory transfer system 100 according to this embodiment, the reference sensory information is the sensory information recalled by the first subject R1 from whom brain activation information was detected. In this configuration, the sensory information of the first subject R1 is directly associated with the sensory information of the second subject R2, thereby enabling appropriate sensory information transfer.

[0040] In the sensory transfer system 100 according to this embodiment, stimulation includes irradiating a target region of the brain of the second subject R2 with an electromagnetic wave signal to activate the target region. In this configuration, by directly activating the brain of the second subject R2, sensory information can be more directly evoked in the second subject R2.

[0041] [Second embodiment] Next, a second embodiment will be described. In the first embodiment, an example has been described in which the sensation transfer system 100 transfers sensation in one direction from the first subject R1 to the second subject R2. In contrast, in the second embodiment, the sensation transfer system 100 also transfers sensation from the second subject R2 to the first subject R1. In other words, the sensation transfer system 100 is configured to be able to transfer sensation in both directions between the first subject R1 and the second subject R2.

[0042] The overall configuration of the sensation transmission system 100 is the same as that of the first embodiment. Hereinafter, the configuration of the sensation transmission system 100 will be described from the second device 30 side with reference to Figs.

[0043] The second device 30 includes a detection unit 31, a communication unit 32, a processing unit 33, a stimulus application unit 34, and a storage unit 35. The processing unit 33, the stimulus application unit 34, and the storage unit 35 are the same as those in the first embodiment. The detection unit 31 detects brain activation information of the second subject R2, similar to the detection unit 11 in the first embodiment. The communication unit 32 transmits the brain activation information detected by the detection unit 31 to the estimation device 20.

[0044] The estimation device 20, like the estimation device 20 of the first embodiment, includes a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is capable of wired or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from, for example, the second device 30. The communication unit 21 transmits corresponding sensory information (described below) estimated by, for example, the processing unit 22 to the first device 10.

[0045] The processing unit 22 estimates sensory information (reference sensory information) about the sensations evoked by the second subject R2 in response to the perception based on the detected brain activation information of the second subject R2. Correspondences between what kind of brain activation information is associated with what kind of sensory information is achievable for the second subject R2 through preliminary experiments. For example, the brain activation information detected from the second subject R2 and the sensory information corresponding to the brain activation information are associated with each other to form a set of training data sets, and a fourth training model can be generated by machine learning the training data sets. The fourth training model can be stored, for example, in the storage unit 23.

[0046] Furthermore, the processing unit 22 estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information for the first subject R1, based on the estimated reference sensory information. In this case, the processing unit 22 can perform the estimation based on the second learning model stored in the memory unit 23.

[0047] The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus providing unit 14, and a storage unit 15. The detection unit 11 and the communication unit 12 have the same configuration as in the first embodiment. The communication unit 12 receives corresponding sensory information transmitted from the estimation device 20.

[0048] The processing unit 13 estimates stimulating image information according to the corresponding sensory information received by the communication unit 12. The stimulating image information is information indicating the content of the stimulus to be applied to the first subject R1 by the stimulus application unit .

[0049] The stimulus application unit 14 stimulates the first subject R1 by irradiating a target region of the brain of the first subject R1 with an electromagnetic wave signal to activate the target region. In this case, similar to the stimulus application unit 34 in the first embodiment, the brain of the first subject R1 is partitioned into a three-dimensional matrix consisting of voxels, for example, several millimeters or less in size, and electromagnetic waves are applied to each voxel. The stimulus application unit 14 can apply electromagnetic waves based on stimulus image information indicating the intensity of the electromagnetic waves to be applied to each voxel in the three-dimensional matrix. The correspondence between the intensity of the electromagnetic waves irradiated to each voxel in the brain of the first subject R1 and the type of sensory information evoked by the first subject R1 can be determined in advance through experiments. For example, a set of stimulus image information for the first subject R1 and the sensory information evoked by the first subject R1 when electromagnetic waves are applied based on the stimulus image information can be associated with each other to generate a learning dataset, and the learning dataset can be subjected to machine learning to generate a fifth learning model. The fifth learning model can be stored in the storage unit 15 of the first device 10, for example.

[0050] The fourth and fifth learning models described above can be generated using a neural network represented by, for example, VGG16, similar to the first to third learning models. When generating the fourth and fifth learning models, training data sets are input to the neural network, and correlations between the training data sets are learned by machine learning such as deep learning. In other words, the neural network is optimized through training to generate the learning models. Note that the learning models are not limited to convolutional neural networks represented by VGG16, and other types of neural networks may be used to generate the learning models.

[0051] Next, a sensation transmission method using the sensation transmission system 100 configured as described above will be described. The sensation transmission method for transmitting a sensation from the first subject R1 to the second subject R2 is the same as that in the first embodiment. In this embodiment, a case where a sensation is transmitted from the second subject R2 to the first subject R1 will be described.

[0052] 6 is a diagram schematically illustrating an example of the operation of the sensory transmission system 100. As shown in FIG. 6, the second subject R2 is made to perceive reference sensory information so as to recall it. Below, an example will be described in which the second subject R2 visually perceives a cat's face and recalls it as visual information. In the following example, the cat's face is assumed to be a cat's face, as in the first embodiment.

[0053] 6, the detection unit 31 detects brain activation information 52 of a second subject R2 who perceives a cat's face and recalls sensory information 51. The communication unit 32 transmits the brain activation information 52 detected by the detection unit 31 to the estimation device 20.

[0054] In the estimation device 20, the communication unit 21 receives brain activation information 52 transmitted from the third device 130. The processing unit 22 estimates reference sensory information 53 based on the received brain activation information 52. In this case, the processing unit 22 inputs the brain activation information 52 of the second subject R2 into a fourth learning model. The fourth learning model outputs reference sensory information 53 corresponding to the input brain activation information 52 based on the learning result of the correlation between the brain activation information 52 and the reference sensory information 53. The processing unit 22 acquires the output reference sensory information 53 as an estimation result.

[0055] The processing unit 22 estimates corresponding sensory information 54 for the first subject R1 based on the estimated reference sensory information 53. In this case, the processing unit 22 inputs the acquired reference sensory information 53 to the second learning model. The second learning model outputs corresponding sensory information 54 corresponding to the input reference sensory information 53 based on the learning result of the correlation between the reference sensory information 53 and the corresponding sensory information 54. The processing unit 22 acquires the output corresponding sensory information 54 as an estimation result. The communication unit 21 transmits the acquired corresponding sensory information 54 to the first device 10.

[0056] As shown in the lower part of FIG. 6 , in the first device 10, the communication unit 12 receives corresponding sensory information 54 transmitted from the estimation device 20. The processing unit 13 inputs the received corresponding sensory information 54 to a fifth learning model stored in the memory unit 15. Based on the learning result of the correlation between the corresponding sensory information 54 and the stimulus image information 55, the fifth learning model outputs stimulus image information 55 corresponding to the input corresponding sensory information 54. The stimulus application unit 14 applies a stimulus to the first subject R1 by irradiating the brain of the first subject R1 with electromagnetic waves based on the output stimulus image information 55. As a result, the first subject R1, who has been stimulated by the stimulus application unit 14, recalls corresponding sensory information 56 corresponding to the stimulus image information 55. In other words, the first subject R1 recalls visual information of a cat's face as corresponding sensory information 56. In this way, the visual information of the cat's face is transmitted from the second subject R2 to the first subject R1.

[0057] As described above, in the sensory transfer system 100 according to this embodiment, the reference sensory information is the sensory information recalled by the second subject R2 from which brain activation information was detected. In this configuration, by directly associating the sensory information of the second subject R2 with the sensory information of the first subject R1, sensory information can be appropriately transferred between subjects with different brain activities.

[0058] [Third embodiment] Next, a third embodiment will be described. In the above first and second embodiments, the sensory information recalled by the subject from whom brain activation information was detected was described as the reference sensory information. In contrast, in the third embodiment, a case will be described in which standard sensory information extracted based on corresponding sensory information among multiple subjects is used as the reference sensory information. The overall configuration of the sensory transmission system 100 is the same as that of the first embodiment.

[0059] When the estimation device 20 estimates corresponding sensory information from reference sensory information, standard sensory information is used as the reference sensory information. For example, when multiple detectors perform the same perception, such as viewing the same image, the standard sensory information can be the average value of brain activation information detected in multiple subjects. Therefore, the standard sensory information has little individual variation due to acquired memory, etc., and is sensory information for an average person.

[0060] The standard sensory information can be extracted from the learning results when learning corresponding sensory information among multiple subjects. For example, a sixth learning model can be generated by using a pair of learning data sets, each consisting of brain activation information of a specific subject (e.g., a first subject R1 or a second subject R2) and standard sensory information (the average value of the brain activation information of multiple subjects), and then performing machine learning on the learning data sets. When generating the sixth learning model, standard sensory information may be extracted from multiple pieces of sensory information included in the learning data sets, and the extracted standard sensory information may be associated with each piece of brain activation information included in the learning data sets. As a result, a sixth learning model is generated by machine learning the correspondence between the individual brain activation information of multiple subjects and standard sensory information for different types of sensory information, such as sensory information about vision when viewing an object, sensory information about hearing a sound, and sensory information about touching an object. The sixth learning model can be stored, for example, in the memory unit 25.

[0061] Next, a sensory transfer method using the sensory transfer system 100 configured as described above will be described. Fig. 7 is a diagram schematically illustrating an example of the operation of the sensory transfer system 100. For example, when sensory information 61 obtained when a first subject R1 perceives something is to be transferred to a second subject R2, brain activation information 62 of the first subject R1 is acquired in the first device 10 and transmitted to the estimation device 20.

[0062] In the estimation device 20, the processing unit 33 inputs the brain activation information 62 transmitted from the first device 10 and the identification information of the second subject R2 to whom the sensory information is to be transmitted to a sixth learning model stored in the memory unit 35. In the sixth learning model, standard sensory information 63 corresponding to the brain activation information 62 is calculated, and the sensory information of the second subject R2 associated with the standard sensory information 63 is output as corresponding sensory information 64. The communication unit 21 transmits the output corresponding sensory information 64 to the second device 30.

[0063] As in the first embodiment, the second device 30 receives the corresponding sensory information 64 transmitted from the estimation device 20, and acquires stimulation image information 65 based on the received corresponding sensory information 64. The stimulation application unit 34 applies a stimulation to the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulation image information 65. The second subject R2, to whom the stimulation is applied by the stimulation application unit 34, recalls corresponding sensory information 66 corresponding to the stimulation image information 65.

[0064] As described above, in the sensory transfer system 100 according to this embodiment, the reference sensory information is the standard sensory information 63 extracted based on corresponding sensory information among a plurality of subjects. In this configuration, the standard sensory information 63 extracted based on corresponding sensory information among a plurality of subjects is used as the reference sensory information, so that sensory information can be transferred appropriately among many subjects.

[0065] [Fourth embodiment] 8 is a diagram showing an example of a sensory transfer system 200 according to the fourth embodiment. The sensory transfer system 100 of the above embodiment has been described as a case where sensory information is transferred between different subjects. In contrast, the sensory transfer system 200 described in the fourth embodiment will be described as an example where sensory information is transferred between the same subjects.

[0066] As shown in FIG. 8, the sensory transfer system 200 includes a detection and stimulation device (detection device, stimulation device) 110 and an estimation device 120. The detection and stimulation device 110 has a configuration similar to that of the first device 10 described in the above embodiment, for example, and includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulation application unit 14, and a storage unit 15. The detection unit 11 detects brain activation information. The communication unit 12 performs wired or wireless communication and transmits the brain activation information detected by the detection unit 11 to the estimation device 20. The processing unit 13 calculates stimulation image information based on the corresponding sensory information received by the communication unit 12. The stimulation application unit 14 irradiates a target region of the brain of the target subject R4 with an electromagnetic wave signal based on the calculated stimulation image information to activate the target region and provide stimulation to the target subject R4. The storage unit 15 stores various information.

[0067] The estimation device 120 has a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is capable of wired or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from the detection and stimulation device 110, for example. The communication unit 21 transmits corresponding sensory information (described below) estimated by the processing unit 22, for example, to the detection and stimulation device 110.

[0068] The processing unit 22 estimates reference sensory information associated with the perception based on the brain activation information of the target subject R4 detected at a first time point. The first time point may be when the target subject R4 is a child, for example, when the target subject R4 is less than three years old.

[0069] The reference sensory information may be, for example, sensory information that the target subject R4 associates with the perception at the first time point. In this case, the processing unit 22 may estimate the reference sensory information based on a seventh learning model similar to the first learning model described above. The seventh learning model may be stored in the storage unit 25, for example.

[0070] The reference sensory information may also be standard sensory information extracted based on corresponding sensory information among multiple subjects. The standard sensory information may be information extracted based on sensory information recalled by multiple subjects whose brain growth states correspond to those of the target subject R4. Examples of such multiple subjects include multiple subjects with corresponding ages, multiple subjects who grew up in corresponding environments (latitude, cultural environment, language used, etc.), and multiple subjects with the same or corresponding occupations. The processing unit 22 may use an eighth learning model that estimates standard sensory information based on, for example, brain activation information of the target subject R4 at the first time point. The eighth learning model may be stored in the memory unit 25, for example.

[0071] Furthermore, based on the estimated reference sensory information, processing unit 22 estimates corresponding sensory information for target subject R4 at a second time point, which is later than the first time point. For example, the degree of brain growth of target subject R4 may be significantly different between the first and second time points. For example, corresponding sensory information between target subject R4 at the first time point and target subject R4 at the second time point may be used as a set of learning data sets, and machine learning may be performed on the learning data sets to generate a ninth learning model. The ninth learning model may be stored in storage unit 23, for example.

[0072] Next, a sensory transfer method using the sensory transfer system 200 configured as described above will be described. FIG. 9 is a diagram schematically illustrating an example of the operation of the sensory transfer system 100. As shown in the upper part of FIG. 9, the detection unit 11 of the detection and stimulation device 110 detects brain activation information 72 of a target subject R4 who perceives a cat's face and recalls sensory information 71. The communication unit 12 transmits the brain activation information 72 detected by the detection unit 11 to the estimation device 120. In the estimation device 120, the communication unit 21 receives the brain activation information 72 transmitted from the detection and stimulation device 110. The processing unit 22 inputs the received brain activation information into a seventh or eighth learning model. The seventh or eighth learning model outputs reference sensory information 73 corresponding to the input brain activation information 72. The processing unit 22 acquires the output reference sensory information 73 as an estimation result. The memory unit 25 stores the acquired reference sensory information 73.

[0073] As shown in the lower part of Figure 9, at a second time point after the first time point, for example, if target subject R4 attempts to relive the perception of the cat's face, subject R4 prepares to be able to receive a stimulus from the stimulus providing unit 14 of the detecting and stimulating device 110. Based on the reference sensory information 73 stored in the memory unit 25, the processing unit 22 estimates corresponding sensory information 74 for target subject R4. In this case, the processing unit 22 inputs the acquired reference sensory information 73 into a ninth learning model. The ninth learning model outputs corresponding sensory information 74 corresponding to the input reference sensory information 73. The processing unit 22 acquires the output corresponding sensory information 74 as an estimation result. The communication unit 21 transmits the acquired corresponding sensory information 74 to the detecting and stimulating device 110.

[0074] In the detection and stimulation device 110, the communication unit 12 receives the corresponding sensory information 74 transmitted from the estimation device 120. The processing unit 13 inputs the received corresponding sensory information 74 into a third learning model stored in the memory unit 15. The third learning model outputs stimulation image information 75 corresponding to the input corresponding sensory information 74. The stimulation application unit 14 applies stimulation to the target subject R4 by irradiating the brain of the target subject R4 with electromagnetic waves based on the output stimulation image information 75. As a result, the target subject R4, to whom the stimulation is applied by the stimulation application unit 14, recalls corresponding sensory information 76 corresponding to the stimulation image information 75. In other words, the target subject R4 recalls the visual information of seeing the cat's face at the first time point as the corresponding sensory information 76 at the second time point. In this way, the visual information of the cat's face is transmitted from the target subject R4 at the first time point to the target subject R4 at the second time point. This allows the target subject R4 to relive the experience of seeing the cat's face.

[0075] FIG. 10 is a flowchart showing an example of the operation of the sensory transfer system 200. As shown in FIG. 10, in the sensory transfer system 200, the detection and stimulation device 110 detects brain activation information perceived by the target subject R4 at a first time point (step S201). Next, the estimation device 20 estimates reference sensory information at the first time point based on the brain activation information of the target subject R4 (step S202). Next, the estimation device 20 estimates corresponding sensory information corresponding to the reference sensory information of a second subject R2 different from the first subject R1 based on the estimated reference sensory information (step S103). Then, the second device 30 provides a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information (step S104).

[0076] As described above, the sensory transmission system 200 of this embodiment comprises a detection device (detection and stimulation device 110) that detects brain activation information of the target subject R4 when perceived by the target subject R4, an estimation device 120 that estimates reference sensory information, which is sensory information evoked in response to perception, based on the brain activation information of the target subject R4 detected at a first point in time, and estimates corresponding sensory information corresponding to the reference sensory information for the target subject R4 at a second point in time that is later than the first point in time, based on the estimated reference sensory information, and a stimulation device (detection and stimulation device 110) that stimulates the target subject R4 at the second point in time so as to evoke the estimated corresponding sensory information.

[0077] In addition, the sensory transmission method of this embodiment includes detecting brain activation information of the target subject R4 when perceived by the target subject R4, estimating reference sensory information, which is sensory information evoked in response to the perception, based on the brain activation information of the target subject R4 detected at a first time point, estimating corresponding sensory information corresponding to the reference sensory information for the target subject R4 at a second time point that is later than the first time point, based on the estimated reference sensory information, and providing a stimulus to the target subject R4 at the second time point so as to evoke the estimated corresponding sensory information.

[0078] According to this configuration, instead of simply evoking the reference sensory information of the target subject R4 at the first time point in the target subject R4 at the second time point, corresponding sensory information of the target subject R4 at the second time point is estimated based on the reference sensory information, and stimulation is applied to the target subject R4 so as to elicit the estimated corresponding sensory information. Therefore, even if there is a difference in brain activity when recalling sensory information between the target subject R4 at the first time point and the target subject R4 at the second time point, the sensory information can be transmitted appropriately.

[0079] In the sensory transfer system 200 according to this embodiment, the reference sensory information is sensory information recalled by the target subject R4 at the first time point. In this configuration, the sensory information of the target subject R4 at the first time point is directly associated with the sensory information of the target subject R4 at the second time point, thereby enabling appropriate sensory information transfer.

[0080] In the sensory transfer system 200 according to this embodiment, the reference sensory information is standard sensory information extracted based on corresponding sensory information among a plurality of subjects. In this configuration, the standard sensory information extracted based on corresponding sensory information among a plurality of subjects is used as the reference sensory information, so that sensory information can be efficiently transferred between target subjects R4 at different times.

[0081] In the sensory transfer system 200 according to this embodiment, standard sensory information is extracted based on sensory information recalled by multiple subjects whose brain growth states correspond to those of the target subject R4. With this configuration, sensory information can be efficiently transferred between the target subjects R4 at different points in time based on the sensory information recalled by multiple subjects whose brain growth states correspond to those of the target subject R4.

[0082] [Fifth embodiment] 11 is a diagram showing an example of a sensation transfer system 300 according to the fifth embodiment. As shown in FIG. 10, the sensation transfer system 300 according to the fifth embodiment includes a sensation estimation device 220 and a stimulation device 210.

[0083] The sensation estimation device 220 has a communication unit 221, a processing unit 222, a storage unit 223, and an input unit 224. The communication unit 221 communicates with the stimulation device 210 via wired or wireless communication. The storage unit 223 stores reference sensory information, which is sensory information evoked in response to perception. The storage unit 223 has storage such as a hard disk drive or a solid state drive. Note that an external storage medium such as a removable disk may be used as the storage unit 223. The reference sensory information may be sensory information evoked in response to perception by subject R5 or a person other than subject R5, or may be standard sensory information extracted based on corresponding sensory information among multiple subjects.

[0084] The processing unit 222 estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information for the subject R5, based on the reference sensory information stored in the memory unit 223. For example, a tenth learning model can be generated by using a set of sensory information corresponding to the sensory information of the subject R5 and the reference sensory information as a set of learning data sets and performing machine learning on the learning data sets. The tenth learning model can be stored in the memory unit 223, for example. When the reference sensory information is specified by the input unit 224, which will be described later, for example, the processing unit 222 estimates corresponding sensory information corresponding to the specified reference sensory information.

[0085] The input unit 224 is capable of performing a predetermined input operation for inputting information. For example, an input device such as a keyboard or a touch panel is used as the input unit 224. Note that, in addition to or instead of these, a button, a lever, a dial, a switch, or other input device may be used as the input unit 224. The input unit 224 can input sensory information that the subject R5 wants to experience from a plurality of pieces of reference sensory information stored in the memory unit 223, for example.

[0086] The stimulation device 210 includes a communication unit 212, a processing unit 213, a stimulation applying unit 214, and a storage unit 215. The communication unit 212 is capable of wired or wireless communication. The communication unit 212 receives corresponding sensory information transmitted from the sensation estimation device 220.

[0087] The processing unit 213 calculates stimulation image information corresponding to the received corresponding sensory information based on the corresponding sensory information received by the communication unit 212 and an eleventh learning model that has learned the correspondence between the corresponding sensory information and stimulation image information. This eleventh learning model may be, for example, a learning model similar to the third learning model in the first embodiment described above. The stimulation application unit 214 applies a stimulation to the subject R5 by irradiating a target region of the brain of the subject R5 with an electromagnetic wave signal to activate the target region.

[0088] Next, a sensory transmission method using the sensory transmission system 300 configured as described above will be described. FIG. 12 is a diagram schematically illustrating an example of the operation of the sensory transmission system 300 according to this embodiment. The subject R5 prepares to be able to receive a stimulus from the stimulus application unit 214 of the stimulation device 210. When the subject R5 selects reference sensory information 81 via the input unit 224, the processing unit 222 in the sensation estimation device 220 estimates corresponding sensory information 82 for the subject R5 based on the selected reference sensory information 81. The processing unit 222 inputs, for example, the selected reference sensory information 81 into a tenth learning model. The tenth learning model outputs corresponding sensory information 82 corresponding to the input reference sensory information 81. The processing unit 222 acquires the output corresponding sensory information 82 as an estimation result. The communication unit 221 transmits the acquired corresponding sensory information 82 to the stimulation device 210.

[0089] In the stimulation device 210, the communication unit 212 receives the corresponding sensory information 82 transmitted from the sensation estimation device 220. The processing unit 213 inputs the received corresponding sensory information 82 into an eleventh learning model stored in the memory unit 215. The eleventh learning model outputs stimulation image information 83 corresponding to the input corresponding sensory information 82. The stimulation application unit 214 applies a stimulation to the subject R5 by irradiating the brain of the subject R5 with electromagnetic waves based on the output stimulation image information 83. As a result, the subject R5, who has been stimulated by the stimulation application unit 214, recalls the corresponding sensory information 84 corresponding to the stimulation image information 83. In other words, the subject R5 recalls the visual information of the cat's face as the corresponding sensory information 84. In this way, the visual information of the cat's face is transmitted from the sensation estimation device 220 to the subject R5.

[0090] Fig. 13 is a flowchart showing an example of the operation of the sensation transfer system 300. As shown in Fig. 13, when the reference sensory information 81 is selected by the subject R5 via the input unit 224, the processing unit 222 in the sensation estimation device 220 acquires the selected reference sensory information 81 (step S301), and estimates corresponding sensory information 82 for the subject R5 based on the acquired reference sensory information 81 (step S302). Then, the stimulation device 210 provides a stimulus to the fifth subject R5 so as to recall the estimated corresponding sensory information 82 (step S303).

[0091] As described above, the sensory estimation device 220 according to this embodiment includes a memory unit 223 that stores reference sensory information, which is sensory information associated with perception, and a processing unit 222 that estimates corresponding sensory information, which is sensory information corresponding to the reference sensory information for subject R5, based on the reference sensory information stored in the memory unit 223.

[0092] In addition, the sensory estimation method of this embodiment includes acquiring reference sensory information from a memory unit 223 that stores reference sensory information, which is sensory information that is evoked in response to perception, and estimating corresponding sensory information, which is sensory information that corresponds to the reference sensory information for subject R5, based on the acquired reference sensory information.

[0093] According to this configuration, the corresponding sensory information corresponding to the subject R5 is estimated based on the reference sensory information of the sensory estimation device 220, so that appropriate sensory information can be estimated for each subject R5.

[0094] In the sensation estimation device 220 according to this embodiment, the reference sensory information is sensory information that a person different from the subject R5 associates with perception. With this configuration, it is possible to estimate appropriate sensory information between subjects with different brain activities.

[0095] In the sensation estimation device 220 according to this embodiment, the reference sensory information is standard sensory information extracted based on corresponding sensory information among a plurality of subjects R5. This configuration allows sensory information to be appropriately transmitted among many subjects.

[0096] The sensation transfer system 300 according to this embodiment includes the sensation estimation device 220 described above, and a stimulation device 210 that applies a stimulus to the subject R5 so as to recall the corresponding sensory information estimated by the sensation estimation device 220. According to this configuration, it is possible to estimate appropriate sensory information for each subject R5 based on the reference sensory information stored in the storage unit 223 and transfer the information to the subject R5.

[0097] The technical scope of the present invention is not limited to the above embodiments, and appropriate modifications can be made within the scope of the present invention. For example, in the above embodiments, visual sensory information has been described as an example, but the present invention is not limited to this, and other sensory information may be used as long as it is sensory information that can detect brain activation information. Furthermore, for example, detailed sensory information obtained by linking sensory information from the five senses may be used.

[0098] When generating a learning model, the content presented to the subject may be an instruction such as "Raise your right hand" to learn the response of the subject's motor cortex.Alternatively, when generating a learning model, the content presented to the subject may be audiovisual content such as a movie to learn the response of the subject's entire brain.

[0099] In addition, an ID may be assigned to the elements of standard sensory information obtained when generating a learning model, and the closer the correlation with the content, the closer the ID. For example, if there are contents of "dog," "cat," and "paper," close IDs may be assigned to "dog" and "cat," and distant IDs may be assigned to "dog," "cat," and "paper."

[0100] In addition, if there is no corresponding sensory information corresponding to the standard sensory information, content with a similar ID set as described above may be used as the corresponding sensory information, or the corresponding sensory information may not be sent.

[0101] Alternatively, corresponding sensory information derived from a certain standard sensory information may be replaced with other sensory information and presented. For example, based on the standard sensory information corresponding to the visual information of a mandarin orange, corresponding sensory information replaced with olfactory information, taste information, and tactile information of the mandarin orange may be transmitted to the destination.

[0102] Furthermore, when transmitting sensory information from the first subject R1 to the second subject R2, the command information "send this sensory information to the second subject R2" may be deleted, and communication control information equivalent to a packet header such as "the first subject R1 is recalling the sensory information in this way" or "the first subject R1 is requesting the second subject R2 to recall the sensory information in this way" may be added.

[0103] Furthermore, when extracting standard sensory information, online learning may be performed so that the content is constantly updated. Furthermore, each estimated sensory information may be stored in a storage unit and transmitted to a destination after a predetermined period of time has elapsed. [Explanation of symbols]

[0104] NW... neural network, R1... first subject, R2... second subject, R4... target subject, R5... subject, S1... convolution layer, S2... pooling layer, S3... connection layer, 10... first device, 11, 31... detection unit, 12, 21, 32, 212, 221... communication unit, 13, 22, 33, 213, 222... processing unit, 14, 34, 214... stimulus application unit, 15, 23, 25, 35, 215, 223... memory unit, 20, 120... estimation device, 30... second device, 41, 47, 51, 61, 71...sensory information, 42, 52, 62, 72...brain activation information, 43, 53, 73, 81...reference sensory information, 44, 46, 54, 56, 64, 66, 74, 76, 82, 84...corresponding sensory information, 45, 55, 65, 75, 83...stimulus image information, 63...standard sensory information, 100, 200, 300...sensory transmission system, 110...detection stimulation device, 130...third device, 210...stimulation device, 220...sensory estimation device, 224...input unit

Claims

1. a detection device for detecting brain activation information of a target subject when perceived by the target subject; an estimation device that estimates reference sensory information, which is sensory information associated with the perception, based on brain activation information of the subject detected at a first time point, and estimates corresponding sensory information, based on the estimated reference sensory information, which corresponds to the reference sensory information for the subject at a second time point after the first time point, and is sensory information for reliving the sensory information perceived by the subject at the first time point at the second time point; a stimulation device that stimulates the target subject at the second time point so as to recall the estimated corresponding sensory information; the estimation device includes a learning model that sets corresponding sensory information between the target subject at the first time point and the target subject at the second time point as a set of learning data sets, and outputs the corresponding sensory information as an estimation result based on the reference sensory information input to the learning model. Sensory transmission system.

2. The reference sensory information is the sensory information that the target subject recalls in response to the perception at the first time point. The sensory transmission system according to claim 1 .

3. The reference sensory information is standard sensory information extracted based on the sensory information corresponding to each other among a plurality of subjects. The sensory transmission system according to claim 1 .

4. The standard sensory information is extracted based on the sensory information recalled by a plurality of subjects whose brain growth states correspond to those of the target subject. The sensory transmission system according to claim 3 .

5. Generating a learning model in which corresponding sensory information between a target subject at a first time point and the target subject at a second time point after the first time point is used as a set of learning data sets; Detecting brain activation information of the target subject when perceived by the target subject; Estimating reference sensory information, which is sensory information associated with the perception, based on the brain activation information of the target subject detected at the first time point, inputting the estimated reference sensory information into the learning model, and outputting corresponding sensory information as an estimation result, which is sensory information corresponding to the reference sensory information for the target subject at the second time point, and is sensory information for re-experiencing the sensory information perceived by the target subject at the first time point at the second time point; providing a stimulus to the target subject at the second time point so as to recall the output corresponding sensory information; A method of transmitting sensations, including:

Images