Sensory transmission system and sensory transmission method

Abstract

To provide a sensation transmission system and a sensation transmission method capable of properly protecting personal information of a subject.SOLUTION: A sensation transmission system comprises: a first device that detects intracerebral activation information of a first subject when the first subject makes a perception; an estimation device that, on the basis of the detected intracerebral activation information of the first subject, estimates recall sensation information, which is recalled by a second subject differing from the first subject with respect to the perception, and that encrypts the recall sensation information by using the intracerebral activation information of the first subject; and a second device that decodes the encrypted recall sensation information and applies a stimulus to the second subject so as to recall the decoded recall sensation information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a sensory transmission system and a sensory transmission method. [Background technology]

[0002] In recent years, technologies for non-invasively measuring activation information in the brain, such as functional magnetic resonance and near-infrared spectroscopy, have been developed, and brain-machine interface technologies, which are interfaces between the brain and the outside, are becoming a reality. As an example of the use of such technologies, a technology has been disclosed in which brain activation information is detected when a subject perceives something, and based on the detected brain activation information, recalled sensory information recalled in response to the subject's perception is converted into a message (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-115913 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, the subject's recalled sensory information corresponds to so-called personal information. From the viewpoint of protecting personal information, it is required to handle the subject's recalled sensory information appropriately.

[0005] The present invention has been made in view of the above, and aims to provide a sensory transmission system and a sensory transmission method that can appropriately protect the personal information of subjects. [Means for solving the problem]

[0006] The sensory transmission system of the present invention comprises a first device that detects brain activation information of a first subject when the first subject perceives something; an estimation device that estimates, based on the detected brain activation information of the first subject, the recollected sensory information that a second subject different from the first subject will recall in response to the perception and encrypts the recollected sensory information using the brain activation information of the first subject; and a second device that decrypts the encrypted recollected sensory information and provides stimulation to the second subject so as to recall the decrypted recollected sensory information.

[0007] The sensory transmission method of the present invention includes detecting brain activation information of a first subject when the first subject perceives something, estimating, based on the detected brain activation information of the first subject, the recollected sensory information that a second subject different from the first subject will recall in response to the perception, encrypting the recollected sensory information using the brain activation information of the first subject, decrypting the encrypted recollected sensory information, and providing a stimulus to the second subject so as to recall the decrypted recollected sensory information. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a sensory transmission system and a sensory transmission method that can appropriately protect the personal information of subjects. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a sensation transfer system according to the first embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of a sensation transmission system according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a neural network. [Figure 4] FIG. 4 is a diagram schematically illustrating an example of the operation of the sensation transfer system according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the operation of the sensation transfer system according to the first embodiment. [Figure 6] FIG. 6 is a diagram schematically illustrating another example of the operation of the sensation transfer system according to the second embodiment. [Figure 7] FIG. 7 is a diagram schematically illustrating another example of the operation of the sensation transmission system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially the same.

[0011] [First embodiment] Fig. 1 is a schematic diagram showing an example of a sensation transfer system 100 according to a first embodiment. Fig. 2 is a functional block diagram showing an example of the sensation transfer system 100. As shown in Figs. 1 and 2, the sensation transfer system 100 includes a first device 10, an estimation device 20, and a second device 30.

[0012] The first device 10 detects brain activation information of a first subject R1 when perceived by the first subject R1. The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus application unit 14, and a storage unit 15.

[0013] The detector 11 detects brain activation information. Examples of the brain activation information include the oxygenated hemoglobin concentration, deoxygenated hemoglobin concentration, and total hemoglobin concentration contained in the subject's cerebral blood flow. The detector 11 may be, for example, a measuring device that performs measurements based on the principles of fMRI (functional Magnetic Resonance Imaging) or fNIRS (functional Near-Infrared Spectroscopy), a measuring device using invasive electrodes, or a measuring device that performs measurements using a micromachine placed in the blood vessels of the brain. The detector 11 is not limited to the above devices, and other types of devices may also be used. The brain activation information can be expressed as the magnitude of activity for each voxel when the brain of the first subject R1 is partitioned into a three-dimensional matrix consisting of voxels of several millimeters or less. In this embodiment, the detection unit 11 separately detects brain activation information for estimating recalled sensory information to be transmitted to the second subject R2 and brain activation information for encrypting the recalled sensory information. When detecting brain activation information for encrypting recalled sensory information, the detection unit 11 can detect brain activation information generated in the primary visual cortex of the brain of the first subject R1, for example. Note that, as will be described later, when detecting brain activation information for encryption, the detection unit 11 can detect brain activation information in the primary visual cortex when the first subject R1 views an image of a monotone striped pattern with a predetermined direction (hereinafter referred to as a monotone striped pattern). Because the primary visual cortex is located on the surface of the brain, the detection unit 11 can obtain highly accurate brain activation information. Furthermore, the primary visual cortex is easily able to recognize the above-mentioned monotone striped pattern. Therefore, when viewing an image of a monotone striped pattern, many subjects will evoke approximately the same sensation.

[0014] The communication unit 12 is an interface for wired or wireless communication. The communication unit 12 transmits the brain activation information detected by the detection unit 11 to the estimation device 20. The communication unit 12 includes an interface for communicating with an external device via a so-called wireless LAN in accordance with the IEEE802.11 standard. The communication unit 12 may realize communication with an external device under the control of the processing unit 22 or the like. Note that the communication method is not limited to wireless LAN, and may include, for example, an infrared communication method, a Bluetooth (registered trademark) communication method, a wireless communication method such as Wireless USB, and the like. Furthermore, a wired connection such as a USB cable, HDMI (registered trademark), IEEE1394, or Ethernet may also be adopted.

[0015] The processing unit 13, the stimulus applying unit 14, and the storage unit 15 will be described later. In the first embodiment, the processing unit 13, the stimulus applying unit 14, and the storage unit 15 do not necessarily have to be provided.

[0016] The estimation device 20 includes a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is an interface that performs wired or wireless communication. The communication unit 21 receives brain activation information transmitted from the first device 10, for example. The communication unit 21 transmits corresponding sensory information (described below) estimated by the processing unit 22 to the second device 30, for example. Note that the communication unit 21 may have a configuration similar to that of the communication unit 12 described above.

[0017] The processing unit 22 has a processing device such as a CPU (Central Processing Unit) and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The processing unit 22 performs various processes including the estimation process described below.

[0018] The storage unit 23 stores various types of information. The storage unit 23 has storage such as a hard disk drive, a solid state drive, etc. Note that an external storage medium such as a removable disk may also be used as the storage unit 23.

[0019] The processing unit 22 estimates recalled sensory information (reference sensory information) about the sensations recalled by the first subject R1 in response to the perception, based on the detected brain activation information of the first subject R1. The recalled sensory information may be information about at least one of the five senses, i.e., vision, hearing, touch, taste, and smell, or may be information about the sense of balance or other somatic sensations. Specifically, when the recalled sensory information is visual information, the recalled sensory information is image data perceived by the first subject R1. However, the recalled sensory information is not limited to this, and may be image data sampled from the image data rather than the image data itself, or image data obtained by filtering the image data. Furthermore, when the recalled sensory information is visual information, the recalled sensory information may be information about light entering the eyeball of the first subject R1. In this case, for example, information about light may be acquired using contact lenses equipped with optical sensors. Alternatively, information about light may be acquired using an artificial retina. Specifically, information about light from a CCD sensor provided in the artificial retina may be used. Furthermore, when the recalled sensory information is auditory information, the recalled sensory information may be audio signal data perceived by the first subject R1. Furthermore, if the recalled sensory information is information about taste, the recalled sensory information may be data indicating indicators of multiple chemical substances that reproduce the taste perceived by the first subject R1. If the recalled sensory information is information about touch, the recalled sensory information may be data indicating which part of the entire body surface of the first subject R1 is stimulated and to what extent in the developed diagram. Note that these recalled sensory information examples are not limiting. Regarding the first subject R1, the correspondence between what kind of recalled sensory information corresponds to what kind of brain activation information can be determined in advance by conducting experiments. For example, the brain activation information detected from the first subject R1 and the recalled sensory information corresponding to the brain activation information can be associated with each other to create a set of learning data sets, and the learning data sets can be subjected to machine learning to generate a first learning model. The first learning model can be stored, for example, in the memory unit 23.

[0020] Furthermore, the processing unit 22 estimates corresponding sensory information, which is recollected sensory information corresponding to the reference sensory information of a second subject R2 different from the first subject R1, based on the estimated reference sensory information. A specific example of the corresponding sensory information may correspond to a specific example of recollected sensory information. The second subject R2 is a subject to whom the sensations of the first subject R1 are transmitted. The relationship between the reference sensory information and the corresponding sensory information can be determined by conducting a preliminary experiment. For example, the corresponding recollected sensory information between the first subject R1 and the second subject R2 is used as a set of learning data sets, and a second learning model can be generated by machine learning the learning data sets. The second learning model can be stored, for example, in the memory unit 23.

[0021] Furthermore, the processing unit 22 encrypts the estimated corresponding sensory information using brain activation information of the first subject R1. When performing encryption, the processing unit 22 can use brain activation information detected from the primary visual cortex of the first subject R1. More specifically, the processing unit 22 can use brain activation information detected from the primary visual cortex of the first subject R1 when the first subject R1 views, for example, the above-mentioned monotone striped pattern image. As described above, the primary visual cortex is located on the surface of the brain, so highly accurate brain activation information can be obtained by the first device 10. Furthermore, the primary visual cortex is easily able to recognize the above-mentioned monotone striped pattern. Therefore, when viewing an image of a monotone striped pattern, many subjects will evoke approximately the same sensation. This improves the reproducibility of the encryption.

[0022] The processing unit 22 can perform encryption using an encryption key generated using, for example, brain activation information. The processing unit 22 can generate the encryption key using a known encryption key generation method, such as public key encryption, symmetric key encryption, or key exchange. An example of generating an encryption key using brain activation information is to quantize the brain activation information detected from the primary visual cortex of the first subject R1 when the first subject R1 views a monotone striped pattern image to a degree that ensures reproducibility, and then use the quantized brain activation information to influence the factor information used to generate the encryption key. In this case, the detection unit 11 of the first device 10 detects brain activation information when the first subject R1 views a monotone striped pattern image, separately from the brain activation information used to estimate the recalled sensory information to be transmitted to the second subject R2. The processing unit 22 may also encrypt the estimated reference sensory information in addition to the corresponding sensory information using a similar procedure.

[0023] The second device 30 provides a stimulus to the second subject R2 so as to recall the estimated corresponding sensory information. The second device 30 includes a detection unit 31, a communication unit 32, a processing unit 33, a stimulus providing unit 34, and a storage unit 35.

[0024] The detection unit 31 will be described later. In the first embodiment, the detection unit 31 does not necessarily have to be provided.

[0025] The communication unit 32 is an interface that performs wired or wireless communication. The communication unit 32 receives corresponding sensory information transmitted from the estimation device 20. The communication unit 32 also transmits brain activation information detected by the detection unit 31 to the estimation device 20. The communication unit 32 may have the same configuration as the communication unit 12 described above.

[0026] The processing unit 33 decrypts the corresponding sensory information received by the communication unit 32. The processing unit 33 can decrypt the corresponding sensory information using, for example, a decryption key corresponding to the encryption key generated by the processing unit 22 of the estimation device 20. The decryption key is generated according to the encryption key generation method described above. The processing unit 33 may acquire the decryption key in advance and store it in the storage unit 35, or the encryption key and decryption key may be transmitted together from the estimation device 20 to the second device 30. The processing unit 33 can calculate stimulation image information corresponding to the received corresponding sensory information based on the decrypted corresponding sensory information and a third learning model described below.

[0027] The third learning model is a learning model in which stimulus image information for the second subject R2, which will be described later, is associated with corresponding sensory information, which is sensory information that the second subject R2 recalls when irradiated with electromagnetic waves based on the stimulus image information, to form a set of learning data sets, and the learning data sets are machine-learned. The third learning model can be stored in the storage unit 35 of the second device 30, for example.

[0028] The stimulation unit 34 stimulates the second subject R2 by irradiating a target region of the brain of the second subject R2 with an electromagnetic wave signal to activate the target region. In this case, the brain of the second subject R2 is partitioned into a three-dimensional matrix consisting of voxels, for example, several millimeters or less, and electromagnetic waves are irradiated to each voxel. The stimulation unit 34 can irradiate electromagnetic waves based on stimulation image information indicating which voxels in the three-dimensional matrix should be irradiated with electromagnetic waves and at what intensity. The voxels in the three-dimensional matrix of stimulation image information may correspond in size, position, etc. to the voxels in the three-dimensional matrix of brain activation information. The correspondence between which voxels in the brain of the second subject R2 are irradiated with electromagnetic waves and at what intensity will evoke what kind of evocative sensory information can be determined by conducting experiments in advance.

[0029] The first learning model, second learning model, and third learning model described above can be generated using a neural network (convolutional neural network) represented by, for example, VGG16. FIG. 3 is a diagram showing an example of a neural network. As shown in the upper part of FIG. 3, the neural network NW has 13 convolutional layers S1, 5 pooling layers S2, and 3 fully connected layers S3. The neural network processes input information in the convolutional layers S1 and pooling layers S2 in order, and the processing results are combined in the fully connected layer S3 and output.

[0030] When generating the first, second, and third learning models, as shown in the middle of Figure 3, learning data sets containing corresponding information (I1, I2) are input to the neural network NW, and the correlations between the learning data sets are learned by machine learning such as deep learning. In other words, the neural network NW is optimized through learning to generate learning models. For example, the neural network NW is trained to solve a problem that requires one piece of information when one piece of information constituting each learning data set is input.

[0031] When performing inference using the first learning model, the second learning model, and the third learning model, one piece of information I1 of two pieces of information constituting each learning data set is input to the neural network NW, as shown in the lower part of Figure 3. The first learning model, the second learning model, and the third learning model output the other piece of information I2 corresponding to the input piece of information I1 based on the learning results of the correlation between the pieces of information constituting the learning data set. Note that, although this embodiment describes an example in which a learning model is generated using a convolutional neural network represented by VGG16, the present invention is not limited to this, and learning models may be generated using other types of neural networks.

[0032] Next, a sensory transmission method using the sensory transmission system 100 configured as described above will be described. FIG. 4 is a diagram schematically illustrating an example of the operation of the sensory transmission system 100. As shown in FIG. 4, the first subject R1 is made to perceive reference sensory information so as to recall it. Below, an example will be described in which the first subject R1 visually perceives a cat's face and recalls it as visual information. In addition, the first subject R1 is made to visually perceive a monotone striped image IM1 for encryption.

[0033] As shown in the upper part of Figure 4, the detection unit 11 detects brain activation information 42 of the first subject R1 who perceives a cat's face and recalls recalled sensory information 41. The detection unit 11 also detects brain activation information 48 of the first subject R1 recalled by the monotone striped image IM1. The communication unit 12 transmits the brain activation information 42, 48 detected by the detection unit 11 to the estimation device 20.

[0034] In the estimation device 20, the communication unit 21 receives the brain activation information 42, 48 transmitted from the first device 10. The processing unit 22 estimates reference sensory information 43 based on the received brain activation information 42. In this case, the processing unit 22 inputs the brain activation information 42 of the first subject R1 to a first learning model. The first learning model outputs reference sensory information 43 corresponding to the input brain activation information 42 based on the learning result of the correlation between the brain activation information 42 and the reference sensory information 43. The processing unit 22 acquires the output reference sensory information 43 as an estimation result.

[0035] The processing unit 22 estimates corresponding sensory information 44 for the second subject R2 based on the estimated reference sensory information 43. In this case, the processing unit 22 inputs the acquired reference sensory information 43 into the second learning model. The second learning model outputs corresponding sensory information 44 corresponding to the input reference sensory information 43 based on the learning result of the correlation between the reference sensory information 43 and the corresponding sensory information 44. The processing unit 22 acquires the output corresponding sensory information 44 as an estimation result. In addition, the processing unit 22 encrypts the corresponding sensory information 44 using brain activation information 48 received separately from the brain activation information 42. The communication unit 21 transmits the encrypted corresponding sensory information 44 to the second device 30.

[0036] In the second device 30, the communication unit 32 receives the corresponding sensory information 44 transmitted from the estimation device 20. The processing unit 33 decodes the received corresponding sensory information 44 and inputs the combined corresponding sensory information 44 to a third learning model stored in the memory unit 35. Based on the learning results of the correlation between the corresponding sensory information 44 and the stimulus image information, the third learning model outputs stimulus image information 45 corresponding to the input corresponding sensory information 44. The stimulus application unit 34 applies a stimulus to the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulus image information 45. As a result, the second subject R2, who has been stimulated by the stimulus application unit 34, recalls corresponding sensory information 46 corresponding to the stimulus image information 45. In other words, the second subject R2 recalls the visual information of the cat's face as the corresponding sensory information 46.

[0037] On the other hand, due to differences in personality between the first subject R1 and the second subject R2, the correspondence between brain activity and brain activation information is different. For this reason, as shown in the lower part of Figure 4 (indicated by the dashed line), if brain activation information 42 of first subject R1 is directly transmitted to second device 30 and second device 30 provides stimulation to the brain of second subject R2 in accordance with brain activation information 42, second subject R2 is likely to recall visual information other than a cat's face as recalled sensory information 47. In this case, the recalled sensory information of first subject R1 will not be properly transmitted to second subject R2.

[0038] In contrast, in the sensory transmission system 100 of this embodiment, the corresponding sensory information 44 of the second subject R2 is estimated in the estimation device 20, so that visual information of the cat's face is appropriately transmitted from the first subject R1 to the second subject R2.

[0039] FIG. 5 is a flowchart showing an example of the operation of the sensory transfer system 100. As shown in FIG. 5, in the sensory transfer system 100, the first device 10 detects brain activation information for estimating the brain activation information of the first subject R1 when the first subject R1 perceives something, i.e., brain activation information for estimating recalled sensory information (hereinafter referred to as "for estimation") (step S101). The first device 10 also separately detects brain activation information of the first subject R1 for encryption (hereinafter referred to as "for encryption") (step S102). Next, the estimation device 20 estimates reference sensory information recalled by the first subject R1 in response to the perception based on the brain activation information for estimation of the first subject R1, and estimates corresponding sensory information corresponding to the reference sensory information of a second subject R2 different from the first subject R1 based on the estimated reference sensory information (step S103). The estimation device 20 also encrypts the estimated corresponding sensory information using the brain activation information for encryption and transmits it to the second device 30 (step S104). The second device 30 receives and decodes the encrypted corresponding sensory information (step S105), and provides a stimulus to the second subject R2 so as to recall the decoded corresponding sensory information (step S106).

[0040] As described above, the sensory transmission system 100 of this embodiment comprises a first device 10 that detects brain activation information of a first subject R1 when perceived by the first subject R1, an estimation device 20 that estimates the recollected sensory information that a second subject R2 different from the first subject R1 recalls in response to the perception based on the detected brain activation information of the first subject R1 and encrypts the recollected sensory information using the brain activation information of the first subject R1, and a second device 30 that decrypts the encrypted recollected sensory information and stimulates the second subject R2 to recall the decrypted recollected sensory information.

[0041] The sensory transmission method of this embodiment includes detecting brain activation information of a first subject R1 when the first subject R1 perceives something, estimating, based on the detected brain activation information of the first subject R1, the recollected sensory information that a second subject R2 different from the first subject R1 recalls in response to the perception, encrypting the recollected sensory information using the brain activation information of the first subject R1, decrypting the encrypted recollected sensory information, and providing a stimulus to the second subject R2 so as to recall the decrypted recollected sensory information.

[0042] According to this configuration, even if there is a difference in brain activity between the first subject R1 and the second subject R2, the recollected sensory information can be appropriately transmitted from the first subject R1 to the second subject R2. Furthermore, when the recollected sensory information is transmitted, the recollected sensory information is encrypted using the brain activation information of the first subject R1, so that the recollected sensory information, which is the personal information of the subjects, can be appropriately protected.

[0043] In the sensory transfer system 100 according to this embodiment, the estimation device 20 uses brain activation information detected from the primary visual cortex of the first subject R1 when performing encryption. The primary visual cortex is located on the surface of the brain. Therefore, this configuration allows for highly accurate brain activation information to be obtained.

[0044] In the sensory transfer system 100 according to this embodiment, the estimation device 20 performs encryption using brain activation information detected from the primary visual cortex of the first subject R1 when the first subject R1 views an image of a monotone striped pattern. The primary visual cortex is easily able to recognize the monotone striped pattern described above. Therefore, when viewing an image of a monotone striped pattern, many subjects will evoke a similar sensation. This ensures high reproducibility.

[0045] In the sensory transfer system 100 according to this embodiment, the estimation device 20 performs encryption using an encryption key generated using brain activation information. With this configuration, encryption using an encryption key generated using brain activation information can further improve security.

[0046] [Second embodiment] Next, a second embodiment will be described. In the first embodiment, an example has been described in which the sensation transfer system 100 transfers sensation in one direction from the first subject R1 to the second subject R2. In contrast, in the second embodiment, the sensation transfer system 100 also transfers sensation from the second subject R2 to the first subject R1. In other words, the sensation transfer system 100 is configured to be able to transfer sensation in both directions between the first subject R1 and the second subject R2.

[0047] The overall configuration of the sensation transmission system 100 is the same as that of the first embodiment. Hereinafter, the configuration of the sensation transmission system 100 will be described from the second device 30 side with reference to Figs.

[0048] The second device 30 includes a detection unit 31, a communication unit 32, a processing unit 33, a stimulus providing unit 34, and a memory unit 35. The processing unit 33, the stimulus providing unit 34, and the memory unit 35 are the same as those in the first embodiment. The detection unit 31 detects brain activation information of the second subject R2, similar to the detection unit 11 in the first embodiment. In this embodiment, the detection unit 31 separately detects brain activation information for estimating recalled sensory information to be transmitted to the first subject R1 and brain activation information for encrypting the recalled sensory information. When detecting brain activation information for encrypting recalled sensory information, the detection unit 31 can detect brain activation information generated in the primary visual cortex when the second subject R2 views an image with a monotone striped pattern, for example. The communication unit 32 transmits the brain activation information detected by the detection unit 31 to the estimation device 20.

[0049] The estimation device 20, like the estimation device 20 of the first embodiment, includes a communication unit 21, a processing unit 22, and a storage unit 23. The communication unit 21 is capable of wired or wireless communication. In this embodiment, the communication unit 21 receives brain activation information transmitted from, for example, the second device 30. The communication unit 21 transmits corresponding sensory information (described below) estimated by, for example, the processing unit 22 to the first device 10.

[0050] The processing unit 22 estimates the recollected sensory information (reference sensory information) about the sensations the second subject R2 evokes in response to the perception based on the detected brain activation information of the second subject R2. The correspondence between the recollected sensory information and the brain activation information of the second subject R2 can be determined in advance by conducting experiments. For example, the brain activation information detected from the second subject R2 and the recollected sensory information corresponding to the brain activation information are associated with each other to form a set of learning data sets, and a fourth learning model can be generated by machine learning the learning data sets. The fourth learning model can be stored, for example, in the memory unit 23.

[0051] Furthermore, the processing unit 22 estimates corresponding sensory information, which is recalled sensory information corresponding to the reference sensory information of the first subject R1, based on the estimated reference sensory information. In this case, the processing unit 22 can perform the estimation based on the second learning model stored in the memory unit 23.

[0052] Furthermore, the processing unit 22 encrypts the estimated corresponding sensory information using brain activation information of the second subject R2. When performing encryption, the processing unit 22 can use brain activation information detected from the primary visual cortex of the second subject R2. More specifically, as in the first embodiment, the processing unit 22 can use brain activation information detected from the primary visual cortex of the second subject R2 when the second subject R2 views an image with a monotone striped pattern. As in the first embodiment, the processing unit 22 can perform encryption using, for example, an encryption key generated using the brain activation information.

[0053] The first device 10 includes a detection unit 11, a communication unit 12, a processing unit 13, a stimulus providing unit 14, and a storage unit 15. The detection unit 11 and the communication unit 12 have the same configuration as in the first embodiment. The communication unit 12 receives corresponding sensory information transmitted from the estimation device 20.

[0054] The processing unit 13 decodes the corresponding sensory information received by the communication unit 12. The processing unit 13 can decode the corresponding sensory information using, for example, a decryption key corresponding to the encryption key by the processing unit 22 of the estimation device 20. The processing unit 13 estimates stimulation image information corresponding to the decoded corresponding sensory information. The stimulation image information is information indicating the stimulation content to be applied to the first subject R1 by the stimulation application unit 14.

[0055] The stimulus application unit 14 stimulates the first subject R1 by irradiating a target region of the brain of the first subject R1 with an electromagnetic wave signal to activate the target region. In this case, similar to the stimulus application unit 34 in the first embodiment, the brain of the first subject R1 is partitioned into a three-dimensional matrix consisting of voxels, for example, several millimeters or less in size, and electromagnetic waves are applied to each voxel. The stimulus application unit 14 can apply electromagnetic waves based on stimulus image information indicating the intensity of the electromagnetic waves to be applied to each voxel in the three-dimensional matrix. The correspondence between the intensity of the electromagnetic waves irradiated to each voxel in the brain of the first subject R1 and the type of evocative sensory information evoked by the first subject R1 can be determined in advance through experiments. For example, a set of stimulus image information for the first subject R1 and the evocative sensory information evoked by the first subject R1 when electromagnetic waves are applied based on the stimulus image information can be associated with each other to generate a learning dataset, and the learning dataset can be subjected to machine learning to generate a fifth learning model. The fifth learning model can be stored in the storage unit 15 of the first device 10, for example.

[0056] The fourth and fifth learning models described above can be generated using a neural network represented by, for example, VGG16, similar to the first to third learning models. When generating the fourth and fifth learning models, training data sets are input to the neural network, and correlations between the training data sets are learned by machine learning such as deep learning. In other words, the neural network is optimized through training to generate the learning models. Note that the learning models are not limited to convolutional neural networks represented by VGG16, and other types of neural networks may be used to generate the learning models.

[0057] Next, a sensation transmission method using the sensation transmission system 100 configured as described above will be described. The sensation transmission method for transmitting a sensation from the first subject R1 to the second subject R2 is the same as that in the first embodiment. In this embodiment, a case where a sensation is transmitted from the second subject R2 to the first subject R1 will be described.

[0058] FIG. 6 is a diagram schematically illustrating an example of the operation of the sensory transfer system 100. As shown in FIG. 6, the second subject R2 is made to perceive reference sensory information so as to recall it. Below, an example will be described in which the second subject R2 visually perceives a cat's face and recalls it as visual information. In the following example, the cat's face is assumed to be a cat's face, as in the first embodiment. In addition, the second subject R2 is made to visually perceive a monotone striped image IM2 for encryption.

[0059] As shown in the upper part of Figure 6, the detection unit 31 detects brain activation information 52 of the second subject R2 who perceives a cat's face and recalls recalled sensory information 51. The detection unit 11 also detects brain activation information 58 of the first subject R1 recalled by the monotone striped image IM2. The communication unit 32 transmits the brain activation information 52, 58 detected by the detection unit 31 to the estimation device 20.

[0060] In the estimation device 20, the communication unit 21 receives brain activation information 52 transmitted from the second device 30. The processing unit 22 estimates reference sensory information 53 based on the received brain activation information 52. In this case, the processing unit 22 inputs the brain activation information 52 of the second subject R2 to a fourth learning model. The fourth learning model outputs reference sensory information 53 corresponding to the input brain activation information 52 based on the learning result of the correlation between the brain activation information 52 and the reference sensory information 53. The processing unit 22 acquires the output reference sensory information 53 as an estimation result.

[0061] The processing unit 22 estimates corresponding sensory information 54 for the first subject R1 based on the estimated reference sensory information 53. In this case, the processing unit 22 inputs the acquired reference sensory information 53 to the second learning model. The second learning model outputs corresponding sensory information 54 corresponding to the input reference sensory information 53 based on the learning result of the correlation between the reference sensory information 53 and the corresponding sensory information 54. The processing unit 22 acquires the output corresponding sensory information 54 as an estimation result. In addition, the processing unit 22 encrypts the corresponding sensory information 54 using brain activation information 58 received separately from the brain activation information 52. The communication unit 21 transmits the encrypted corresponding sensory information 54 to the first device 10.

[0062] As shown in the lower part of FIG. 6 , in the first device 10, the communication unit 12 receives the corresponding sensory information 54 transmitted from the estimation device 20. The processing unit 13 decodes the received corresponding sensory information 54 and inputs the combined corresponding sensory information 54 to a fifth learning model stored in the memory unit 15. Based on the learning result of the correlation between the corresponding sensory information 54 and the stimulus image information 55, the fifth learning model outputs stimulus image information 55 corresponding to the input corresponding sensory information 54. The stimulus application unit 14 applies a stimulus to the first subject R1 by irradiating the brain of the first subject R1 with electromagnetic waves based on the output stimulus image information 55. As a result, the first subject R1, who has been stimulated by the stimulus application unit 14, recalls corresponding sensory information 56 corresponding to the stimulus image information 55. In other words, the first subject R1 recalls visual information of a cat's face as the corresponding sensory information 56. In this way, the visual information of the cat's face is transmitted from the second subject R2 to the first subject R1.

[0063] As described above, in the sensory transfer system 100 according to this embodiment, the reference sensory information is the recollected sensory information recalled by the second subject R2 from whom brain activation information was detected. In this configuration, by directly associating the recollected sensory information of the second subject R2 with the recollected sensory information of the first subject R1, it is possible to appropriately transfer the recollected sensory information between subjects with different brain activities. Furthermore, when the recollected sensory information is transferred, the recollected sensory information is encrypted using the brain activation information of the second subject R2, thereby appropriately protecting the recollected sensory information, which is the subject's personal information.

[0064] [Third embodiment] Next, a third embodiment will be described. In the above first and second embodiments, the recalled sensory information recalled by the subject from whom brain activation information was detected was described as the reference sensory information. In contrast, in the third embodiment, an example will be described in which standard sensory information extracted based on corresponding recalled sensory information among multiple subjects is used as the reference sensory information. The overall configuration of the sensory transmission system 100 is the same as that of the first embodiment.

[0065] When the estimation device 20 estimates corresponding sensory information from reference sensory information, standard sensory information is used as the reference sensory information. For example, when multiple detectors perform the same perception, such as viewing the same image, the standard sensory information can be the average value of brain activation information detected in multiple subjects. Therefore, the standard sensory information has little individual variation due to acquired memory, and is the sensory information recalled by an average person.

[0066] The standard sensory information can be extracted from the learning results when learning corresponding recollected sensory information among multiple subjects. For example, a sixth learning model can be generated by using a set of brain activation information of a specific subject (e.g., a first subject R1 or a second subject R2) and standard sensory information (the average value of the brain activation information of multiple subjects) as a learning data set, and then performing machine learning on the learning data set. When generating the sixth learning model, standard sensory information may be extracted from the multiple recollected sensory information included in the learning data set, and the extracted standard sensory information may be associated with each piece of brain activation information included in the learning data set. As a result, a sixth learning model is generated by machine learning the correspondence between the individual brain activation information of multiple subjects and standard sensory information for different types of recollected sensory information, such as visual recollected sensory information when viewing an object, auditory recollected sensory information when listening to a sound, and tactile recollected sensory information when touching an object. The sixth learning model can be stored, for example, in the memory unit 35.

[0067] Next, a sensory transfer method using the sensory transfer system 100 configured as described above will be described. Fig. 7 is a diagram schematically illustrating an example of the operation of the sensory transfer system 100. For example, when recalled sensory information 61 generated when a first subject R1 perceives something is to be transmitted to a second subject R2, brain activation information 62 of the first subject R1 is acquired in the first device 10 and transmitted to the estimation device 20. Furthermore, brain activation information 68 of the first subject R1 recalled by a monotone striped image IM3 is detected in the first device 10.

[0068] In the estimation device 20, the processing unit 22 inputs the brain activation information 62 transmitted from the first device 10 and the identification information of the second subject R2, who is the recipient of the recollected sensory information, into a sixth learning model stored in the memory unit 35. In the sixth learning model, standard sensory information 63 corresponding to the brain activation information 62 is calculated, and the recollected sensory information of the second subject R2 associated with the standard sensory information 63 is output as corresponding sensory information 64. The processing unit 22 encrypts the output corresponding sensory information 64 using brain activation information 68 received separately from the brain activation information 62. The communication unit 21 transmits the encrypted corresponding sensory information 64 to the second device 30.

[0069] As in the first embodiment, the second device 30 receives and decodes the corresponding sensory information 64 transmitted from the estimation device 20, and acquires stimulation image information 65 based on the decoded corresponding sensory information 64. The stimulation application unit 34 applies a stimulus to the second subject R2 by irradiating the brain of the second subject R2 with electromagnetic waves based on the output stimulation image information 65. The second subject R2, to whom the stimulus is applied by the stimulation application unit 34, recalls corresponding sensory information 66 corresponding to the stimulation image information 65.

[0070] As described above, in the sensory transfer system 100 according to this embodiment, the reference sensory information is the standard sensory information 63 extracted based on the corresponding recalled sensory information among multiple subjects. In this configuration, the standard sensory information 63 extracted based on the corresponding recalled sensory information among multiple subjects is used as the reference sensory information, so that the recalled sensory information can be appropriately transferred among many subjects. Furthermore, when the recalled sensory information is transferred, the recalled sensory information is encrypted using the brain activation information of the first subject R1, so that the recalled sensory information, which is the personal information of the subjects, can be appropriately protected.

[0071] The technical scope of the present invention is not limited to the above embodiments, and appropriate modifications can be made within the scope of the present invention. For example, in the above embodiments, visual recollection sensory information has been described as an example, but the present invention is not limited to this, and other recollection sensory information may be used as long as it is recollection sensory information that can detect brain activation information. Furthermore, for example, detailed recollection sensory information obtained by linking recollection sensory information of the five senses may be used.

[0072] The sensory transfer system 100 described in the above embodiment can also be used to cause the first subject R1 at a second time point to recall the recalled sensory information of the first subject R1 at a first time point. The first device 10 acquires reference sensory information of the first subject R1 at the first time point and transmits it to the estimation device 20. The estimation device 20 stores the reference sensory information in the storage unit 25. The estimation device 20 then estimates corresponding sensory information of the first subject R1 at a second time point based on the reference sensory information of the first subject R1 at the first time point and transmits the estimation result to the first device 10. In the first device 10, the stimulus application unit 14 applies a stimulus to the first subject R1 so as to recall the estimated corresponding sensory information. This allows the first subject R1 to relive the sensations of the first subject R1 at the first time point at the second time point. In this case, the sensory transfer system 100 transmits recalled sensory information between the first subject R1 at the past (first time point) and the first subject R1 at the time of stimulus application (second time point), i.e., between the same person. Although the above embodiment is premised on encryption, if there is no risk of unauthorized use such as eavesdropping, the estimation device 20 may transmit the recollection sensory information to the first device 10 without encrypting it. This allows for streamlining in terms of the processing time and power consumption required for encryption and decryption. [Explanation of symbols]

[0073] I1, I2...information, R1...first subject, R2...second subject, S1...convolutional layer, S2...pooling layer, S3...connection layer, IM1, IM2, IM3...image, NW...neural network, 10...first device, 11, 31...detection unit, 12, 21, 32...communication unit, 13, 22, 33...processing unit, 14, 34...stimulus application unit, 15, 23, 35...memory unit, 16...VGG, 20...estimation device, 30...second device, 41, 47, 51, 61...recalled sensory information, 42, 48, 52, 58, 62, 68...brain activation information, 43, 53...reference sensory information, 44, 46, 54, 56, 64, 66...corresponding sensory information, 45, 55, 65...stimulus image information, 63...standard sensory information, 100...sensory transmission system

Claims

1. a first device for detecting brain activation information of a first subject when perceived by the first subject; an estimation device that estimates recollected sensory information that a second subject, different from the first subject, recalls in response to the perception based on the detected brain activation information of the first subject, and encodes the recollected sensory information using the brain activation information of the first subject; a second device that decodes the encrypted recalled sensory information and provides a stimulus to the second subject so as to recall the decoded recalled sensory information; A sensory transmission system comprising:

2. When performing the encryption, the estimation device uses the brain activation information detected from the primary visual cortex of the first subject. The sensory transmission system according to claim 1 .

3. When performing the encryption, the estimation device uses the brain activation information detected from the primary visual cortex of the first subject when the first subject views a monotone image with a striped pattern having a predetermined direction. The sensory transmission system according to claim 2 .

4. The estimation device performs the encryption using a cryptographic key generated using the brain activation information. The sensory transmission system according to any one of claims 1 to 3.

5. detecting brain activation information of a first subject when perceived by the first subject; Based on the detected brain activation information of the first subject, estimating recalled sensory information that a second subject different from the first subject recalls in response to the perception, and encoding the recalled sensory information using the brain activation information of the first subject; decrypting the encrypted recalled sensory information and providing a stimulus to the second subject so as to recall the decrypted recalled sensory information; A method of transmitting sensations, including:

Images