Signal processing apparatus, signal processing method, and signal processing system

Abstract

The present disclosure relates to a signal processing device, a signal processing method, and a signal processing system capable of preventing out-of-control of BMI control. The determination unit determines whether or not to permit the interaction of the user on the basis of a neural signal corresponding to the user's intention acquired from the user's brain via the BMI. The present disclosure can be applied to a gateway device provided between the brain of a user and an external device or a network service interacting with the user in a system implemented using a BMI technology.

Description

Technical Field

[0001] The present disclosure relates to a signal processing device, a signal processing method, and a signal processing system, and more particularly, to a signal processing device, a signal processing method, and a signal processing system capable of preventing out-of-control of BMI (Brain-Machine Interface) control. Background Art

[0002] In recent years, as a mechanism for connecting to a machine, research and development of a new interface technology called Brain-Machine Interface (BMI) has been underway.

[0003] For example, Patent Document 1 discloses a system that filters data in a neural network environment including BMI to remove inappropriate content. Cited Reference List Patent Document

[0004] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-8359 Summary of the Invention Technical Problem to be Solved by the Invention

[0005] For example, according to the BMI technology, even a user with disabled hands and feet can control an external device only according to the user's intention, and in the future, such a user will be able to interact with the outside world through a network.

[0006] On the other hand, this BMI-based control (BMI control) has a risk of out-of-control behavior due to the user's consciousness, thoughts, or mental state.

[0007] The present disclosure is made in view of this situation, and thus an object of the present disclosure is to be able to prevent out-of-control of BMI control. Solution to the Technical Problem

[0008] The signal processing device of the present disclosure includes a determination unit that determines whether to allow the user's interaction based on neural signals corresponding to the user's intention acquired from the user's brain via a Brain-Machine Interface (BMI).

[0009] The signal processing method of the present disclosure includes causing a signal processing device to determine whether to allow the user's interaction based on neural signals corresponding to the user's intention acquired from the user's brain via a Brain-Machine Interface (BMI).

[0010] The signal processing system of the present disclosure includes a gateway device that determines whether to allow the user's interaction based on neural signals corresponding to the user's intention acquired from the user's brain via a Brain-Machine Interface (BMI).

[0011] According to the present disclosure, it is determined whether to permit the interaction of the user based on neural signals corresponding to the intention of the user acquired from the user's brain via BMI. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart showing the process of synesthesia formation and BMI activation. Figure 2 is a flowchart showing the reinforcement learning process. Figure 3 is a diagram showing an example of feedback in the initial stage of learning. Figure 4 is a block diagram showing an example of the configuration of the gateway device. Figure 5 is a flowchart showing the operation process of the gateway device. Figure 6 is a diagram showing an example of the configuration of the BMI control system according to an embodiment of the present disclosure. Figure 7 is a diagram for explaining an example of how to control an external device. Figure 8 is a diagram for explaining an example of how to use a network service. Figure 9 is a flowchart showing the process of the monitoring process executed during the execution of the AI collaboration task. Figure 10 is a flowchart showing the action-taking process executed by the monitor based on the evaluation result. Figure 11 is a flowchart showing the process of information dissemination processing via an avatar. Figure 12 is a block diagram showing an example of the configuration of the hardware of a computer. DETAILED DESCRIPTION

[0013] Hereinafter, modes for implementing the present disclosure (hereinafter referred to as embodiments) will be described. Note that the description will be made in the following order.

[0014] 1. BMI Technology and Its Challenges 2. Establishing Neural Pathways via BMI 3. Connection to an External Network via BMI and Its Challenges 4. Activating the Reward Mechanism of BMI 5. Formation of Artificial Synesthesia and Activation of BMI 6. Effects Produced by Using BMI 7. Overview of the Technology According to the Present Disclosure 8. BMI Control System According to an Embodiment of the Present Disclosure 8-1. System Configuration 8-2. Control of External Devices 8-3. Use of Network Services 8-4. Monitoring During Execution of AI Collaboration Tasks 8-5. Information Dissemination via Avatar 9. Example of the Configuration of the Hardware of a Computer

[0015] <1. BMI Technology and Its Challenges> Conventionally, a user perceives and understands the environment through his / her eyes, ears, etc., and operates control devices such as a steering wheel, accelerator, and brakes to operate a car (vehicle). Without properly perceiving the environment and operating the control devices, it is difficult to use the vehicle as a means of transportation. Therefore, a considerable number of people are unable to use a vehicle due to physical function limitations, and mobility becomes difficult.

[0016] Against this background, autonomous driving has the potential to be a powerful means of providing mobility services necessary for social participation to people with disabilities who have faced severe mobility restrictions. Many people with disabilities have difficulties in terms of accessibility, and their needs also vary. Therefore, it is difficult to uniformly determine which type of human-machine interface (HMI) should be introduced for autonomous driving.

[0017] On the other hand, a new interface technology called a brain-machine interface (BMI) has been proposed, which enables a user to convey his / her intention to an autonomous driving system only by thinking, thereby enabling the exchange of various information such as specifying a destination. The BMI can be considered the ultimate interface, which enables a user to convey his / her intention to an autonomous driving system in a manner that does not require any physical movement, so that the autonomous driving system reads the nerve signals transmitted from the user's brain according to the intention and reflects the intention in command and control. In other words, in a situation where the autonomous driving system cannot make an appropriate predictive judgment, the user issues necessary supplementary instructions to the system via the BMI by a kind of telekinesis to achieve movement.

[0018] However, as a mechanism that operates in a closed loop where the content of the intention conveyed through the BMI is not at all exposed to the outside world including third parties, etc., new challenges arise. Therefore, in the case of introducing the BMI as a means of accessibility into society, similar to the introduction of artificial intelligence (AI), measures need to be taken to obtain social acceptance of the new challenges that the interface may bring.

[0019] The above challenges are caused by the mechanism of using BMI, which is different from the conventional mechanism based on interacting with the outside world through language and physical movements (such as gestures, etc.), and through this mechanism, the brain can directly interact with the outside world (without being directly exposed to a third party). Generally, in the case of healthy people participating in society, they communicate in almost all situations through various means such as speech, text, gestures, and facial expressions. At this time, whether consciously or unconsciously, as behaviors associated with their own intentions, people interact with each other through observable communication methods that enable a third party to evaluate their intentions. That is to say, whether they like it or not, in the process of social life, people determine and appropriately predict how their own behaviors may affect the outside world and cause discomfort, harm, happiness, etc., and take corresponding actions by understanding the reactions and reflexive behaviors of others. Through this interaction with the outside world, normal social life is maintained, and it can be considered that this interaction is essential.

[0020] If a person tries to participate in society without being directly perceived by a third party, it is equivalent to influencing the outside world by acting as an invisible person. That is to say, it can be considered that the mechanism by which a person's behaviors and intentions are conveyed as visual and auditory information to the surrounding environment is very important in social activities. In the context of interacting with society, when a third party observes a person's behavior, or a person purchases a product or makes a request, these transactions are carried out through tangible procedures, and there will always be some records left, such as by issuing receipts or exchanging goods or physical objects. Therefore, in the process of social life and growth, people naturally acquire and learn behaviors that violate social morality and ethics as insurmountable boundaries. That is to say, people grow while unconsciously learning to act in a coexistent manner in society. Specifically, people do not act solely based on their desires at that time, but naturally acquire the form of self-restraint when making decisions and establish it as a code of conduct for survival in society.

[0021] However, in the case where such restrictive behaviors do not work repeatedly, people are unable to make appropriate decisions, which causes problems in daily life.

[0022] A medically known example is congenital insensitivity to pain. Patients who cannot feel pain physiologically do not feel pain even when injured, so they cannot generate a neural circuit network (hereinafter also referred to as a neural network) that triggers a preventive response to avoid injury. Even if a patient who does not feel pain can visually identify his / her own injury through bleeding or fracture, unless the patient can perceive the injury as "pain" combined with information about the factors that caused the injury, a neural circuit network that makes a preventive response / action to avoid injury cannot be generated, established, and strengthened.

[0023] Although details will be described later, the neural circuit networks of humans and other organisms are formed by the weak interconnection of many immature synapses in the early stage of life. When these synapses interact with each other and influence each other to take actions to eliminate the cause of the perceived pain, or enable the visual cortex to capture information about the factors that may cause pain, among the many neural circuits with weak connections generated in the initial stage, the neural circuits with lower participation are pruned, such as strengthening the neural circuit network that can promote the elimination of pain even without perceiving pain, and strengthening the neural circuits that lead to injury prevention measures, etc. In practice, before taking preventive measures to avoid injury, there are some actions aimed at causing injury, and these actions have both benefits and the risk of injury. There are also more complex mechanisms at work, such as neural circuits that transmit signals to inhibit neurons that desire benefits, and neural circuits that transmit signals to avoid the risk of injury, etc. It is believed that when success is achieved, that is, when a reward is obtained, an excitatory substance dopamine is released in the body within a limited period of time, and this release strengthens and stabilizes the synapses on the way of the neural circuit that emits or transmits nerve impulses representing thoughts, thus promoting local learning.

[0024] <2. Establishing Neural Pathways through BMI> In BMI technology, neural signals from neurons that convey "intentions" are read by probe electrodes. However, in practice, it is difficult to "interpret" and decode abstract intentions such as emotions with probe electrodes, that is, it is difficult to directly read the thoughts themselves as electrical signals via BMI, and it cannot be achieved even with the most advanced technology. Thinking information such as intentions is concretized into descriptive data including words and symbols at the thinking process stage of the brain, and the words and symbols need to be transmitted through a response mechanism similar to handwriting, keyboard input, touch panel, and voice input. One role of the BMI probe electrode (neural signal detection probe) is to detect the signals that have been converted into words, symbols, voices, or graphs. In addition, another role of the BMI probe electrode is to act as an interface as follows: in response to the variable (analog) output of the intention communication with the external environment, it detects and manipulates response displacements such as forward and backward, left and right, up and down, and push and pull. The response cycle read in the latter is not limited to a specific sense. Usually, many daily behaviors and actions of the human body are controlled by regulating the amount of exercise and the applied force by using feedback from a limited sensory response mechanism. On the other hand, it is difficult to communicate only through the above content, and people interact by concretizing their intentions using verbalized words.

[0025] Here, the focus is on the function of BMI to concretize intentions and thoughts by converting them into text or voice equivalent signals for keyboard input, rather than the direct reading of the thinking content or analog response by BMI.

[0026] In the case where a person conveys his / her intention by typing on a physical keyboard, the person first verbalizes their intention and then attempts to input the verbalized intention using the keyboard according to the spelling of the target language. At this time, when the touch-typing skill is well-developed, people do not have to consciously consider which finger to move to which key position on the keyboard. Instead, the fingers can move according to the spelling of the language for which the intention has been verbalized. In the case where touch-typing practice is insufficient, the line of sight follows the keys on the keyboard, and the keys are operated individually using, for example, the index finger. In order for BMI technology to be widely used as an established technology, it is necessary to establish an information path through a communication method using a relatively standardized interface (such as characters and symbols).

[0027] In the case where the intention is converted into words or characters that can be represented as text and the keyboard is used as the information input interface of the device, extended keys are deployed by combining the required number of keys with the CAPS key and function keys. In this case, some people follow the keys with their line of sight when typing, while others can type accurately using touch-typing. For people like the latter, through training, the neural circuits of the motor nerves required to control finger movements based on the input language are strengthened.

[0028] Communication methods such as pagers were once very popular. For a pager, it is only necessary to input all the language information to be transmitted as a katakana string using a numeric keypad (such as a public telephone). Inexperienced users intuitively check the feedback of the display characters on the form and the single-line operation panel and type while considering the input accuracy. Among experienced users, there are users who can naturally perform text input with the five fingers of one hand at an astonishing speed without confirming individual operation inputs. This can be said to be the result of the intention and the required operations of the fingertips being strengthened and established as the neural circuits for intention communication in the brain that operates the fingertips. That is, without visual feedback correction from the eyes, through repeated use, reinforcement learning is performed on the neural circuits for the intention of the input characters and the fingertip operation commands required to operate the numeric keypad, and a neural circuit network such as the fingertips and hands is constructed.

[0029] In addition, Morse code, which has long been used as a means of communication via a limited output method, can be considered as an option. In this case, instead of issuing physical commands with different numbers of characters, intentions are conveyed by detecting single signals that can be transmitted over a single-wire telephone line using BMI. The type of neural signal detection probe array used is determined by the number of signals of separable neural circuits that can perform the interface functions required for reinforcement learning. For example, as the number of neural signal detection probes, one signal (such as Morse code) or a combination within the range of the maximum number of letters in the alphabet is considered appropriate. Since the processing capacity of the human brain is limited, even if the number of electrodes in the neural signal detection probe array is increased to increase the number of simultaneous parallel processes, the brain may reach its processing limit, resulting in panic, or in the case where the brain cannot process information, it may prevent information transmission protectively. Therefore, increasing the number of electrodes in the neural signal detection probe array has the opposite effect. In daily life, only the fingers of both hands can be unconsciously manipulated simultaneously by allocating neural resources, and what the fingers of both hands can do is limited to playing the piano, touch typing on a computer keyboard, etc. It has been found that even if the hardware for manipulating all the toes of the feet is prepared and reinforcement learning is carried out, it is mechanically difficult to proficiently manipulate all the toes of the feet.

[0030] As a means of communication, BMI has the role of a bag auction that cannot be directly observed from the outside. Once the communication of intentions is determined, the necessary information can be transmitted without explaining the detailed actions of the hands and fingers in the bag, and during the process of getting familiar with the transaction, reinforcement learning of the neural circuits related to the interpretation of the actions of the hands and fingers is carried out on the information-sending side and the information-receiving side.

[0031] Here, considering only the simple connection between the brain and the outside world, it is also conceivable to integrate the sending and receiving of all information into a single signal, just like Morse code. In this case, it is necessary to verbalize intentions and ideas and further convert the results into time-modulated signals. This does not cause any problems to the external probe circuit. However, when a specific neural pathway continues to be responsible for this conversion within the brain, it causes local physiological over-neural utilization, resulting in stress, which can lead to self-protection dysfunction. Therefore, rather than focusing on specific neural systems, it is more desirable to carry out learning so that neural signals can be captured by a distributed array of approximately the number of fingers of one hand. Then, learning can be carried out to assign black-and-white responses such as yes / no, up / down, or left / right to specific neural signals.

[0032] Note that, for example, in a case where intentions are conveyed through a highly restricted signal output such as Morse code, it is entirely possible to read muscle movement commands that control body movements affected by nerve signals (such as movements of fingertips, hands, and facial expression muscles), or signals indicating the tension actually applied to the muscles being detected as the detection target, without directly reading signals of nerve circuits in the brain as in BMI. In such a case, signals can be read in a non-invasive manner.

[0033] <3. Connection of BMI to External Networks and Its Challenges> The system using BMI technology envisioned in the technology of the present disclosure is a mechanism in which a user wearing a BMI (i.e., a communication method that is completely invisible to third parties for communication with the outside world) balances between adverse factors that may cause harm to others, actions that are punished for violating social norms and laws or trigger negative reactions from others, and adverse factors that the user personally can accept, while acquiring and learning a range of behaviors. Then, a psychological state of behavioral decision-making is formed to actively participate in actions that increase personal interests and satisfaction, and a range of acceptable risks is learned and obtained from the accumulated experience.

[0034] Originally, healthy people have the ability to inhibit behavior, that is, as the main factor for inhibiting unrestricted behaviors that increase personal satisfaction, they perform self-consistent behavioral adjustments within the range of avoiding exposing the behavioral results to the outside world and becoming the object of social sanctions, and thus take goal-oriented behaviors that do not cause undue harm to others. Even when wearing a BMI, it is necessary to have this behavior inhibition ability from a social perspective.

[0035] Although details will be described later, simply by invasively inserting and implanting probe electrodes into cell groups in the brain, there will be no clear connection between the probe electrodes and the neurons around the probe electrodes, and it is impossible to expect the accelerated development of the neural circuit network between the neurons expressing intentions and the probe electrodes. Generally, infants and young children need several years to continuously learn through postnatal experiences and be able to freely convey their intentions when correctly connecting intentions and behaviors, that is, it takes several years to form a neural circuit network associated with behaviors. During the process of obtaining such experiences, the connections between neurons that contribute to beneficial experiences among the connections of immature neurons that are massively generated for a while during infancy are strengthened, and a specific neural circuit network is formed. On the other hand, after infancy, the connections of specific neural circuits become stable through experiences, making it difficult to generate new neural circuit networks. Therefore, in order to establish and activate the intention communication neural circuit of the probe electrodes implanted through invasive surgery, it is necessary to promote the generation of neural circuit networks.

[0036] In addition, in the part of the brain that controls intention, an important factor in accelerating the generation of an artificial neural circuit network among neurons around the probe electrode in a self-consistent manner is the reward for the intention communicator. When considering the mechanism of using rewards to generate neural circuits, as an example of linking two events through learning from a macroscopic perspective in psychology, Pavlov's dog experiment is well-known. This experiment uses the auditory input of a bell as trigger information and gives food as a reward, showing that as this repeated learning is carried out, even without food being given, in response to the stimulus of the bell, a signal to produce saliva is transmitted through the neural circuit. This experiment is carried out through a process where "a third party can observe and understand the auditory trigger that can strengthen the neural circuit, and in addition, a third party around can confirm rewards such as food through vision or smell". The reason for the production of saliva during the reflex process of receiving a reward even without food being given due to food reward learning is that the neural circuit connecting these two functions is gradually strengthened and becomes closely connected through repeated learning, and from a microscopic perspective, the neural circuit is strengthened through a complex process. It is believed that because a specific pathway of the neural circuit network is strengthened through repeated learning, useful neural pathways that produce benefits can be strengthened, and repeated learning makes the neural connections used to obtain rewards dominant in the interaction between excitatory synaptic stimulation and inhibitory stimulation caused by the sense of risk aversion that occurs during this process. As a mechanism to avoid congestion of unimportant information transmission that sends inhibitory neural signals to the target neural pathway due to the firing of unnecessary neural pathways with weak connections, the synapses that produce unnecessary neural pathways are pruned. However, even if the intention communication mechanism using BMI is arranged near the neuron group where the intention is expected to be communicated through invasive surgery, there is no direct means to confirm intention communication, so the timing of giving a reward as a result cannot be determined. The advantage of using BMI is in use cases where conventional interfaces such as prosthetic legs, prosthetic arms, or artificial vocal cords cannot be effectively used. That is to say, by adopting an artificial interface like BMI through medical intervention, the quality of life (QoL) can be improved. As a representative example, for patients in the late stage of amyotrophic lateral sclerosis (ALS), etc., BMI is considered a promising measure because they have many difficulties in using these traditional interfaces.

[0037] In cases where eye movements and the like can be used, a combination of arranging a keyboard on a monitor to display a character array is widely used, and an intention communication structure is executed by sequentially selecting characters using the line of sight. In the future, with further development of medicine, a mechanism such as BMI that directly reads intention communication through invasively inserted neural signal detection probes and transmits information as electrical signals can be considered. When such a system is established as an interface, information can be widely exchanged with the outside world through a network, and it is not necessarily required for a third party to observe and convert information during the intention communication process. At this time, due to insufficient feedback of the inhibitory function and overdevelopment of the neural circuits in the brain required for intention communication, side effects such as excessive and unregulated reward acquisition may occur during the activation of the BMI function. There are challenges associated with directly connecting the brain to the outside world through BMI, especially in terms of being able to communicate intentions without being exposed to a third party.

[0038] That is to say, it opens up the possibility of freely manipulating the world of Internet connections without being detected by a third party, just like through telekinesis.

[0039] <4. Reward Mechanism for Activating BMI> In the case of implementing a system using the above BMI technology at least with current technology, it is currently infeasible to precisely decode all neural circuits in each person's brain and accurately detect every electrical signal of individual neurons based on intention, and it may still be challenging in the future.

[0040] When the input information exceeds the decision level of a certain neuron, the neuron fires, and the firing signal propagates along the axon. It is well known that even if a part of the neural circuit network is damaged and the propagation along a specific axon is partially inhibited, the damaged network can be reconstructed through rehabilitation. For example, due to the self-repair ability that minimizes interference with daily life through early rehabilitation, the neural pathways in the brain can continuously and dynamically change.

[0041] Therefore, even if the probe electrode is accurately positioned and set on the neurons responsible for transmitting interactions with the outside world through invasive surgery, the individual connections of the neural circuit network in the brain are not assigned addresses, so it can be said that it is difficult to connect the probe electrode to the desired neurons. On the contrary, in the cerebral cortex area designated for controlling body movement, an arrangement where the neurons in the designated area can play a direct role can be found, and in the case of implanting and arranging electrodes one by one in the designated neurons through invasive surgery, the discharge signals of adjacent neural circuits can be captured. However, this alone cannot capture the signals based on the subject's intention; therefore, in the neural circuit network, it is necessary to reach a state where the neurons adjacent to the electrode become excited according to the intention, and the probe electrode that can detect the discharge of the neurons adjacent to the probe electrode implanted and arranged invasively detects the discharge.

[0042] The problem here is how to incorporate the subject's intention into specific neurons near the surgically implanted probe electrode. In addition, it is important to determine whether adjacent probe electrodes can direct the neural signals conveying the subject's intention into more accurate and dense active neural signals, and detect the intention signals that trigger the discharge with the probe electrode. Even if functional magnetic resonance imaging (fMRI) or other functional response evaluations of the brain are performed in advance, and the local three-dimensional arrangement information of the neural circuit network responsible for intention communication is specified as the area where neural signals are most easily captured through pre-surgical simulations, it is extremely difficult to connect the artificially implanted probe electrode to specific neurons in the same way as known physical connection cables in the real world. That is, even if the probe electrode is implanted targeting a specific area, the one-by-one arrangement as described above is extremely difficult and unrealistic. Needless to say, the following method can also be considered, where during the surgery with the subject remaining awake, by inducing fluorescence in the discharge of neurons or the neurotransmitters released from the ends of the surrounding neural network while the subject repeatedly expresses the intention, and using the visualization information as a marker to identify the location area or the signals detected by electrical detection, the arrangement of the neurons where intention communication occurs is identified. However, implementing this method takes a long time, and it is time-consuming, cumbersome, and full of uncertainties.

[0043] Therefore, as a mechanism for human neurons to record new knowledge and information, attention should be paid to inducing the migration and generation of neurons. Specifically, instead of precisely arranging the probe electrode array on specific neurons or neuron groups, the phenomenon of newly forming neural circuits in a self-consistent manner after electrode implantation can be used. Although there are still many unknown factors in the process of forming neural circuits in the medical field, it is well known that in cases where decisions can be made to obtain new knowledge or solutions to problems, the neural circuit connections of the neurons making the decisions gradually increase.

[0044] By developing a new neural circuit network that can obtain rewards, the following can be achieved. That is, for activities that can provide detection signals to artificially implanted probe electrodes, by providing certain rewards to the brain according to the detection signals, the connection with the neural network near the probe electrodes increases in a self-consistent manner, and the following situation can be achieved, as if the probe electrodes responsible for detecting intentions are directly connected to the neurons associated with intention communication.

[0045] However, the formation of a self-consistent network with respect to the probe electrodes in a self-aligning manner does not occur spontaneously, and a mechanism is required that artificially provides reward feedback to the subject's brain based on the signals amplified and waveform-analyzed by an external device via the probe electrodes. If the subject's behavior and the incentives as rewards are visible, like Pavlov's dog experiment, then the results of this learning can also be easily observed from the outside. However, it is not desirable to observe the behavior itself like that of Pavlov's dog, and as a BMI function, it is difficult to directly observe or control the growth of new neural networks on the probe electrodes. Generating a new neural network around the probe electrodes in a self-consistent manner requires repeated stimulation through reinforcement learning and learning the spontaneous behavior called operant conditioning (here, stimulating neural circuits). That is, a process of forming a neural network that stimulates a certain neural circuit to reach the neurons around the probe electrodes is required.

[0046] It cannot be expected that the signals captured only by electrodes implanted in an invasive manner can ensure that the brain reliably perceives rewards. Therefore, even with stimuli from weak neural circuits that may convey intentions, the formation of the neural circuit network responsible for outputting intentions can be promoted by providing well-responsive reward feedback to existing senses such as vision or hearing that have been developed.

[0047] On the other hand, regarding obtaining information from external devices, etc., it can be considered that the feedback of signals from electrodes inserted in an invasive manner to the brain is a developmental application. In particular, for ALS patients, there are many difficulties such as the difficulty of obtaining visual information from external devices through free movement of the neck and eyes; therefore, obtaining information and rewards by directly reading the electrical signals of external devices without relying on existing vision or hearing becomes a viable option. Currently, the most widely used technology for perceiving BMI is the cochlear implant. For cochlear implants, by directly applying electrical signals to the inner ear (cochlea), a pinpoint electrode arrangement of neurons in existing neural pathways can be achieved through implantation. Therefore, cochlear implants are being introduced as an efficient auditory rehabilitation therapy that does not require reinforcement learning to generate new neural networks for artificially inserted electrodes.

[0048] <5. Formation of Artificial Synesthesia and Activation of BMI> Since the information is extracted through the BMI which is the output port for thought-based information transmission, it is necessary to reinforce in some way the connection between the probe electrode array artificially implanted in the brain in an invasive manner and the neural network.

[0049] Currently, the validation of the BMI embedding function is still in the experimental stage technically. Among them, as a promising technology for the formation of the probe electrode array implanted through invasive surgery and the strengthening of the connection, functional activation, and network formation of the synapses of the neural circuit for actual thinking nerve connection, cord blood transplantation can be given as an example. Cord blood transplantation is one of the treatment methods for neural network regeneration and is used to treat patients suffering from neurological diseases due to nerve connection damage. Among the countless immature synapses connecting neurons shortly after an infant is born, through experiences (such as being exposed to rewards through successful experiences), many connections that have not experienced neurotransmission are gradually eliminated through pruning, while the neural pathways that have experienced signal transmission related to rewards are retained as useful neural pathways. As described above, even if the area around the probe electrode array is immature in the initial stage after implantation, feedback of rewards associated with intention communication is repeatedly provided to this area through vision, hearing, taste, and other possible senses via high-density neural synapses, thereby gradually strengthening the connection. Neurons distributed in a large number of intermediate areas connecting the brain region responsible for intention communication and the implantation site of the probe electrode that causes the BMI to operate transmit information through synaptic connections, which is necessary for activating the BMI function.

[0050] For example, as an example of the promotion means, the formation of neural circuits, the formation of newly formed immature synapses, and the formation and strengthening of neurons themselves are required. Although some attempts at regenerative medicine have also been made through induced pluripotent stem (iPS) cell transplantation, etc., in recent years, it has been reported that the damaged neural circuits of spinal cord injury patients can be regenerated by administering cord blood transplantation. Initially, it has been confirmed that the neural network that obtains beneficial results from the formation of synapses is gradually strengthened, and this synapse increases while remaining immature. Due to the recent development of real-time imaging under a microscope, this migration that leads to the formation of new connections between neurons has been visualized, and it is expected that even for artificially implanted electrode probes, similar neural circuit generation can be achieved by combining with the mechanism of providing rewards for learning.

[0051] Incidentally, synesthesia is considered an inborn superhuman ability of some people, and it is considered a unique sensory experience that can only be observed in a part of people. Although not fully understood, synesthesia is generally described as a sensory experience, and well-known examples involve numbers, letters, or shapes looking as if they have inherent colors in the visual area.

[0052] Synesthesia is thought to be caused by the networked development of neural circuits in specific and other sensory regions of the brain, which are densely generated in early childhood to transmit sensory information. By artificially networking neural circuits in the same way, the neural signals expected to be generated in the thinking region are connected to the detection signals of an artificially implanted electrode probe. In order to transmit the neural signals generated by thoughts to the nerves near the probe electrode and enable the probe electrode to accurately capture the discharges of these nerves, it is necessary to form such a neural circuit network. During the normal process of brain development, based on personal experiences, beneficial neural networks are activated and retained as memories, while useless or ineffective neural networks are only regarded as noise in neural transmission, resulting in delayed and chaotic judgments, and are known to gradually disappear through a process called "pruning".

[0053] The process of activating BMI requires the formation of networks that are strengthened and connected based on specific intention information for a probe electrode array arranged in a matrix form that does not exist in natural development. This process can also be regarded as a forced promotion of the formation of neural circuit networks corresponding to artificial synesthesia. Synesthesia that spontaneously appears in some people is a feeling in which numbers or sounds are perceived as specific colors, even though the information received from the senses was not originally directly perceived as color by the visual organ, and this feeling develops along with the strengthening process of the neural network acting on the visual region. Originally, if a neural network does not play a role in obtaining rewards, then it will be eliminated through spontaneous pruning, but in some people, they believe that the development of synesthesia is because the neural pathway is strengthened during development and retained as a feeling of obtaining beneficial rewards. Then, not only the information received by the visual and auditory organs, but in this feeling, even sounds or numbers may gradually directly affect the visual region.

[0054] BMI is similar to the original human neural network in terms of the formation and strengthening of neurons and networks involved in thinking that are connected to the probe electrode array, but is very different from the original human neural network in the following aspect: forcing the networking between the neural transmission circuit downstream of the neurons involved in thinking and the neural circuits around the probe electrode implanted in an invasive manner rather than with the sensory neurons involved in beneficial perception or the neural circuits of the visual region of the brain. On the other hand, by inducing synesthesia in the visual region while providing immediate rewards each time the neural pulses for a specific probe electrode array are repeatedly successful, in some cases, this will result in experiencing the results acting on BMI through synesthesia. That is to say, just as sensory information is perceived as another sense through synesthesia, when conveying intentions through BMI, even without visual feedback converted by an external device, fortunately, a closed-loop feedback circuit is obtained in the brain by perceiving the conveyed intentions through other sensory feelings.

[0055] That is to say, even if each probe electrode of the invasive implant is not originally color-coded, for example, learning of the recognition colors displayed on the monitor can be repeatedly performed according to the arrangement of the detection signals, and artificial synesthesia can be learned through reinforcement learning, so that information fragments to be conveyed can be obtained without waiting for the signal detection response feedback from an external device, and the intention can be output with less latency (time delay) for confirmation. Not everyone can acquire this ability through training; therefore, the usability and effectiveness vary depending on the situation.

[0056] Here, the above process of forming artificial synesthesia and activating BMI will be described with reference to Figure 1 the flowchart in

[0057] In process P1, the functional distribution in the brain is observed using fMRI or the like.

[0058] In process P2, the observed functional distribution in the brain is modeled as a three-dimensional model.

[0059] In process P3, the virtual space is used to identify the optimal neural structure for implanting the probe electrode array based on the modeled three-dimensional model.

[0060] In process P4, based on the neural structure identified using the virtual space, a surgical examination of the neurons around the area where the probe electrode array is to be implanted is performed in the actual brain of the subject.

[0061] In process P5, the probe electrode array is invasively implanted along the neurons with the desired function.

[0062] In process P6, reinforcement learning is performed. That is to say, according to the intention of the subject, a new neural network is generated from the existing neurons in the brain implanted with the probe electrode array.

[0063] Reference will be made to Figure 2 describe the reinforcement learning process of process P6.

[0064] In process P11, the migration of neurons and the formation of synapses are pharmacologically promoted, for example, by the administration of umbilical cord blood.

[0065] This promotes the generation of a neural network in process P12, through which neural signals are transmitted to the implanted (embedded) probe electrode array.

[0066] Thereafter, in process P13, the neural signals corresponding to the conveyed intention are detected and their waveforms are analyzed.

[0067] In process P14, in response to detecting a nerve signal (pulse signal) corresponding to an intention from the result of waveform analysis, a reward for gradually forming and stabilizing a neural network is provided. Here, the "reward" refers to various response results, including the instruction response expected by the subject. For example, as a result of the intention to make a choice, if the color can be displayed in response to the intention to display a color, the result of the intention will be manifested, and a reward will be provided in the form of a reward.

[0068] Learning is carried out by repeating this intention communication and reward provision. In the initial stage of learning, the reward is fed back via an external device such as a monitor or a speaker as a response to vision (color, shape, symbol, letter, etc.), hearing, touch, smell, etc. These are highly transparent information that can be detected by a third party.

[0069] Figure 3 It is a diagram showing an example of feedback in the initial stage of learning.

[0070] Figure 3 It shows a probe electrode array arranged in a matrix form, and the probe electrode array includes probe electrodes PE arranged in a 4×4 matrix form.

[0071] In the initial stage, as shown in Figure 3 A in, for example, a reward is provided as a result of detecting a nerve signal coming from any one of the probe electrodes PE in the 2×2 matrix surrounded by a dotted line. Based on the signal waveform that appears in response to the intention communication, a reward is provided in the form of lighting a lamp, audio output, etc. In addition, praise, fragrance, or a perceived exciting stimulus can also be provided to the subject.

[0072] As the next stage, in the case where the user has normal vision and hearing, as shown in Figure 3 B in, a reward is provided as a result of detecting a nerve signal obtained by recognizing each of the four parts (i.e., the upper, lower, left, and right parts) of the probe electrode array. For example, when a nerve signal is detected from any one of the four probe electrodes PE located in the upper left corner, a red light is turned on or flashed, when a nerve signal is detected from any one of the four probe electrodes PE located in the upper right corner, a blue light is turned on or flashed, when a nerve signal is detected from any one of the four probe electrodes PE located in the lower left corner, a white light is turned on or flashed, and when a nerve signal is detected from any one of the four probe electrodes PE located in the lower right corner, a green light is turned on or flashed.

[0073] In the subsequent stage, as shown in Figure 3 C in, a reward can be provided as a result of detecting a nerve signal obtained by recognizing each of the probe electrodes PE in the 4×3 matrix of the probe electrode array. In Figure 3In the example of C, the digits 0 to 9 and the symbols * and # are associated one-to-one with 12 probe electrodes PE, and the digit or symbol corresponding to the probe electrode PE that detects the nerve signal is displayed on, for example, a monitor.

[0074] In this way, the subdivision of the neural circuit continues to develop, enabling the acquisition of multiplexed nerve signals in response to the intended communication.

[0075] On the other hand, as learning progresses, as a response from an external device, signals are directly fed back to the neurons via the BMI. These are opaque messages that are difficult for third parties to detect.

[0076] As described above, after establishing a neural network for intention output through tangible rewards, a neural pathway is formed to acquire intangible information (information that is difficult for third parties to detect visually and auditorily) in response to feedback from an external device. That is, information from the outside is acquired through a new sense.

[0077] As described above, by receiving responses from the outside in forms different from those of conventional perceptual information such as characters, colors, touch, etc., artificial synesthesia is formed and the BMI is activated. However, the exchange of information that cannot be read by third parties can be carried out, and its transparency is lost, which increases the risk of excessive pursuit of rewards and benefits.

[0078] <6. Effects produced by using the BMI> Compared with conventional social activities based on human mobility, Internet communication provides a means to greatly expand the scope of activities. Then, the current popularity of Internet communication enables users with physical limitations (i.e., those who have difficulties in accessibility) to have many opportunities to participate in society. These users are not only restricted in their daily activities (such as walking like healthy people), but also due to diseases, etc., even with normal brain function, they are restricted in speaking, writing, or other forms of normal communication. In this case, the BMI is an option as an effective interface for interacting with the outside world.

[0079] In addition to enabling movement using, for example, an autonomous driving system by means of transmitting intention information using BMI, there are more possibilities for BMI itself. In the case of autonomous vehicles, although the actual steering itself may be difficult, there are possibilities to fully play roles such as destination instructions and acting as a remote monitor, and it is possible to use an instruction intervention type that is not level 4 of the Society of Automotive Engineers (SAE) autonomous driving. On the other hand, it is impractical to perform invasive surgery on BMI only in limited applications of autonomous driving, and actual implementation is only promising when there are many other additional benefits. One such application is to use the BMI function to conduct unrestricted ubiquitous information exchange with the outside world via the Internet or next-generation communication networks.

[0080] BMI includes the following characteristics.

[0081] 1. In the absence of reward feedback, the generation of the brain neural network will not proceed until BMI is fully activated as an interface. Therefore, a structure is adopted where feedback information is visible from the outside, that is, visible to a third party. The neural circuit network is constructed through a self-consistent learning process such as a process where rewards strengthen the neural network. 2. Through the process of verbalizing intentions and converting them into letters and symbols, repeated input-output learning progresses to response detection comparable to touch typing. 3. The neural signal detection device of BMI can be equipped with an error correction function. As learning reaches a certain stage, by combining penalties for repeated errors, the need for the process of converting into letters and symbols as a basic element for providing feedback is eliminated. 4. There is a minimal dependence on external connection establishment due to reasons such as relocation, and as long as the relocation destination is within the specified area, interconnections can be freely established as needed. 5. Although the function is associated with an individual, due to various privacy restrictions such as personal information protection, it is necessary to limit the identification of personal information transmission. 6. Even after starting to be used, BMI and the constructed neural circuit network remain in a state where continuation, reorganization, and reconstruction can occur on the network with the outside world. 7. Generally speaking, the information contains the transmitted personal intentions, so it is basically not processed as an object for third-party monitoring and viewing. 8. When staying in a certain private room, the user can freely establish a connection with the outside world. Since there are operation restrictions compared to computers or mobile terminals used by ordinary healthy people, even if all operations are restricted, it can excel in the ability to be converted into a language or symbol form that can be expressed through BMI.

[0082] On the other hand, since the pathway using BMI is limited, maximizing the use of this pathway leads to maximizing the rewards obtained. Therefore, there is a possibility of out-of-control behavior occurring during the pursuit of rewards.

[0083] Using BMI can establish a connection with the outside world without using audio or video texts. As a result, information can be exchanged without the conventional process of materializing the information (i.e., the information exchange in society is exposed to a third party). In the initial stage of the learning for activating BMI, during the formation and development of the brain neural circuits, while obtaining visual, auditory, and tactile feedback, through the repeated experience of obtaining rewards, the synaptic pathways connecting the specific intention neurons required to transmit information and the probe electrodes constituting BMI are gradually strengthened (become complex).

[0084] Through this process, even without direct feedback such as obtaining rewards, the required intention can be conveyed through the probe electrode array. When this learning process is fully developed, for communication, there is no longer a need for tangible feedback information that can be recognized by a third party such as audio and video. That is, the ability to directly act on the probe electrode array according to the intention of the brain is equivalent to obtaining the ability to directly convey intentions to an external device.

[0085] In many conventional input methods for immobile patients (such as voice, keyboard input, text-based input achieved through a combination of eye gaze and blinking, etc.), information is captured by an intention detection device, converted into display information for confirmation, and displayed on the screen in a format understandable by a third party, so that the third party can understand the information. On the other hand, when the BMI technology is established in the future, this series of processes of materializing information becomes unnecessary, and the relationship between the user and society also changes accordingly.

[0086] In addition, in the case of performing various tasks by supplementing a large number of work transactions by utilizing external AI processing capabilities, a large number of work transactions are carried out only by using the information transmitted on the network, which is an environment established in the dark. If the inhibition mechanism does not work, then combined with the trial learning for obtaining rewards and maximizing the rewards for developing AI functions, the unrestricted reward acquisition inherent in the brain where BMI is activated will cause a risk of out-of-control.

[0087] In the past, access to information involved physical movement and was carried out through tangible books and magazines. In the search operations of electronic terminal devices, the provision of unprecedented mobility and accessibility has brought about significant changes to the thinking mode in various forms such as the acquisition and memory of human information. However, using artificial intelligence for adaptive learning without being exposed to third parties also opens the door to the unknown in a certain way. That is to say, how to suppress the execution of selfish tasks that may occur due to the continuous process of obtaining rewards, which is inevitable for the activation of BMI, becomes a challenge. In typical social activities, the surveillance gaze of society naturally plays a role, and when the beneficial behavior of one party conflicts with moral, ethical, or social principles, individuals will learn the risk of sanctions, and self-discipline will be naturally exercised. In response to this challenge, even if the autonomous learning of beneficial tasks is carried out through cooperation between an individual's brain and the external AI via the network, as well as cooperation involving many individuals and AI entities, a mechanism can be introduced to create a situation that prevents learning for obtaining false rewards that lead to harmful behaviors to others, similar to behaviors in general society, through which the functions of surveillance and preventing runaway continuously function.

[0088] Computers are deeply involved in people's daily lives, and in simulations and various tasks, using computers to supplement individuals' thinking tasks serves as an aspect of undertaking socially beneficial processes. By equipping computers connected to an external network with artificial intelligence, these computers can be complementarily utilized according to instructions from the brain, while the brain can also directly control these computers. As seen in events such as the Paralympic Games, by using artificial tools to make up for physical disabilities, people can demonstrate extraordinary athletic abilities. Similarly, in intellectual tasks, the cooperation between the brain and AI enables the execution of tasks far beyond what an individual brain can accomplish alone. On the other hand, when overly engaging in tentative tasks for obtaining rewards to activate BMI, it is difficult to uniformly prevent and prohibit the acquisition of selfish rewards only through hardware.

[0089] As mentioned above, according to BMI technology, for example, even users with disabilities in hands and feet can control external devices only according to the user's intention, and in the future, they will also be able to interact with the outside world through the network.

[0090] On the other hand, this BMI-based control (BMI control) has a risk of runaway behavior due to the user's consciousness, thoughts, or mental state.

[0091] Therefore, it is necessary to construct a gateway responsible for maintaining social order to ensure that while protecting personal privacy and human rights and ensuring a situation that does not lead to personal surveillance or management related to a surveillance society, benefits or rewards can be obtained through interaction with the outside world via BMI, and underground activities can be prevented.

[0092] <7. Summary of the Technology According to the Present Disclosure> The technology according to the present disclosure can prevent the out-of-control of BMI control that may be caused by the user's consciousness, thoughts, or mental state. Note that the technology according to the present disclosure and its effects will be described on the premise that invasive surgery on the brain is required for using BMI, but the technology according to the present disclosure and its effects can also be applied to the interface function of obtaining a limited number of body surface signals outside the brain as intention signals in a non-invasive manner via a probe.

[0093] Figure 4 It is a block diagram showing a structural example of a gateway device as a signal processing device to which the technology according to the present disclosure can be applied.

[0094] Figure 4 The shown gateway device 10 may include a general computer such as a personal computer (PC), a mobile terminal such as a smartphone owned by the user, or a server or relay device on a network (cloud) owned by a telecommunications carrier, an Internet service provider (ISP), or a service provider providing various network services. The gateway device 10 can be set between the user's brain (BMI) and an external device or network service that interacts with the user in a system implemented using BMI technology.

[0095] The gateway device 10 includes a determination unit 11 and a storage unit 12.

[0096] The determination unit 11 determines whether to allow the user's interaction based on the nerve signal corresponding to the user's intention acquired from the user's brain via BMI. Specifically, the determination unit 11 acquires the intention signal obtained by decoding the nerve signal acquired from the user's brain via BMI, and determines whether to allow the user's interaction based on the intention signal.

[0097] The user's interaction based on the intention signal may correspond to the operation of an external device connected to the BMI in a wired or wireless manner, or may correspond to the use of a network service via the network. In the latter case, the user's interaction based on the intention signal may correspond to the execution of an AI collaboration task in the network service, or may correspond to the information dissemination using an avatar on the network.

[0098] The determination unit 11 may also use the determination information input from the outside or the determination information stored in the storage unit 12 to determine whether to allow the user's interaction based on the intention signal.

[0099] In the case where the user's interaction corresponds to the operation of an external device, the determination information indicates, for example, whether there is a risk due to a failure of the external device. In this case, the determination information may include the user's state of awareness obtained by monitoring the user, instruction input based on a third-party judgment, and the like. In addition, in the case where the user's interaction corresponds to the use of a network service, the determination information includes the user's mental state or thought content obtained by monitoring the user, the confirmation result of the user's feedback on the information disseminated by the user, and the like.

[0100] Based on the determination result of whether to permit the user's interaction, the determination unit 11 outputs a control signal to the external device that interacts with the user to operate (control) the external device, or outputs a request signal to the network service that interacts with the user to use the network service.

[0101] Reference will be made to Figure 5 the flowchart in

[0102] In step S11, the determination unit 11 of the gateway device 10 determines whether an intention signal based on a neural signal has been input.

[0103] Step S11 is repeated until it is determined that an intention signal has been input, and when it is determined that an intention signal has been input, the process proceeds to step S12.

[0104] In step S12, the determination unit 11 determines whether to permit the user's interaction based on the input intention signal. That is, it is determined whether to permit the interaction according to whether the interaction based on the user's intention will cause harm or adverse effects to a third party. In addition, it is determined whether to permit the interaction based on the user's intention according to whether the interaction violates moral, ethical, or social principles, or whether there is a risk of excessive pursuit of rewards and benefits.

[0105] The above processing can prevent the out-of-control of BMI control that may be caused by the user's awareness, thoughts, or mental state. As a result, social order can be maintained, that is, while protecting personal privacy and human rights and ensuring that there will be no personal surveillance or management related to a surveillance society, it is ensured that benefits or rewards can be obtained through interaction with the outside world via BMI, and underground activities are prevented.

[0106] <8. BMI Control System According to an Embodiment of the Present Disclosure> Hereinafter, the configuration and operation examples of a BMI control system including the above-described gateway device 10 will be described.

[0107] (8-1. System Configuration) Figure 6 is a diagram showing a configuration example of a BMI control system according to an embodiment of the present disclosure.

[0108] Figure 6 The BMI control system 100 shown includes a probe electrode array 101, a neural signal analysis unit 102, a decoding unit 103, a feedback signal generation unit 104, and a gateway device 110. At least one of the neural signal analysis unit 102, the decoding unit 103, and the feedback signal generation unit 104 may be combined and configured with the gateway device 110.

[0109] The probe electrode array 101 is configured as a probe electrode array including probe electrodes arranged in a matrix as described with reference to Figure 3 The probe electrode array 101 is implanted into the brain of the user U1, and the above-mentioned reinforcement learning is repeatedly performed to activate the BMI function of the user U1. The neural signal detected by the probe electrode array 101 is an analog signal including a time-series pulse signal and noise. The neural signal detected by the probe electrode array 101 is output to the neural signal analysis unit 102.

[0110] The neural signal analysis unit 102 includes a signal processing chip electrically connected to the probe electrode array 101 and the like. The neural signal analysis unit 102 removes the noise in the neural signal received from the probe electrode array 101, and detects and analyzes the waveform (pulse signal) corresponding to the intention of the user U1. A combination (neural signal) of the analyzed pulse signals is output to the decoding unit 103.

[0111] The decoding unit 103 converts the combination (neural signal) of the pulse signals received from the neural signal analysis unit 102 into a digital signal, and outputs the digital signal as an intention signal to the gateway device 110. In addition, the intention signal obtained by digital conversion may be output to the feedback signal generation unit 104 as needed. Note that the decoding unit 103 may include a single signal processing chip having an A / D conversion circuit, or may include a monolithic integrated circuit (IC) integrated with the neural signal analysis unit 102.

[0112] The feedback signal generation unit 104 generates a feedback signal for feeding back the intention of the user U1 to the user U1 based on the intention signal received from the decoding unit 103. The feedback signal may be at least one of a display signal for display, an audio signal for audio output, and a BMI signal input to the user's brain via the BMI (probe electrode array 101 or other probe electrode arrays).

[0113] Generally, when a person conveys a certain intention, for speech, he / she hears his / her own voice and uses feedback to produce correct speech, while for writing or typing, he / she visually checks for errors or inaccuracies and uses feedback to produce accurate text. There is a possibility that an error occurs in the intention output from the brain via BMI, etc., so some form of feedback loop is required. This feedback loop can be implemented through a third-party perceivable monitor or speaker or through other BMIs assigned to an external perception input system. In addition, feedback can be directly obtained through cochlear implants, etc., which have been widely used in actual applications. The function of the feedback loop for the intention conveyed by user U1 is essential. In the case where there is no feedback loop at all, the brain is continuously in a state where it cannot determine whether the information transmitted via BMI is accurate. For example, this is the same situation as when a hearing-impaired person has difficulty in normal conversations.

[0114] The gateway device 110 is an implementation example of the gateway device 10 described with reference to Figure 4 One or more external devices can be connected to the gateway device 110 in a wired or wireless manner. The gateway device 110 converts the intention signal (nerve signal) received from the decoding unit 103 into a control signal for operating (controlling) the external device based on the intention signal, and determines whether to allow the input of the control signal to the external device.

[0115] The user state monitoring device 120 and the control buttons 121 and 122 are connected to the gateway device 110.

[0116] For example, the user state monitoring device 120 includes a computer connected to a camera, a vital sign detection device, etc., monitors various states of the user U1, and outputs status information indicating the states as the above-described determination information to the gateway device 110. The states of the user U1 include the consciousness state (consciousness level, whether the user is in a sleeping state, whether the user is in a coma, etc.), the mental state (whether the user is in an excited state), and the thought content (whether there is malice).

[0117] The control buttons 121 and 122 can each include a mechanical analog button or a UI displayed on a touch panel. The control buttons 121 and 122 can be operated by a third party located near the user U1, for example.

[0118] The control button 121 is a button for controlling whether to make the status information output from the user state monitoring device 120 effective. For example, in the case where the status information is effective, the gateway device 110 uses the status information received from the user state monitoring device 120 to determine whether to allow the input of the control signal to the external device. Specifically, the gateway device 110 adds the effective status information to the control signal and supplies the resulting control signal to the external device.

[0119] The control button 122 is a button for controlling whether to enable or disable the input of a control signal from the gateway device 110 to an external device (i.e., the BMI function itself). In response to the operation of the control button 122, the gateway device 110 determines whether to allow the input of the control signal based on an instruction input for enabling / disabling the control signal input from a third party. For example, in the case of disabling the control signal input, the gateway device 110 does not input the control signal to the external device at all.

[0120] The gateway device 110 is connected to an emergency rescue request device 131, and user devices 132 and 133, which are external devices.

[0121] The emergency rescue request device 131 includes, for example, a call device in a nurse call system. The user device 132 is an external device worn and carried by the user U1 and located extremely close to the user U1, and includes, for example, a wearable device, a mobile device, a display device, an audio output device, a bed or chair with an electric tilting function, etc. The user device 133 is also an external device worn and carried by the user U1 and located extremely close to the user U1, and includes an automatically driving wheelchair as a kind of moving body, a water supply device for the user U1 to replenish water, etc.

[0122] Even in the case where the emergency rescue request device 131 and the user device 132 malfunction, the risk to the user U1 and the surrounding environment is extremely small, and safety can be maintained. Therefore, these external devices accept the input of the control signal from the gateway device 110 regardless of the status information added to the control signal.

[0123] In particular, the user device 132 is a device familiar to the user, such as a monitor or a speaker for transmitting information, and is used as a means for confirming a signal generated based on the user's own intention, or as a means for a user who cannot speak to convey his / her intention to a third party on the spot. The user device 132 has an extremely small impact on safety and does not require locking (cutting off the control signal), but can be disconnected, for example, during night sleep to avoid generating unnecessary information, such as information seen in a dream during sleep.

[0124] On the other hand, in the case where the user device 133 malfunctions, it will bring many risks to the user U1 and the surrounding environment, which may affect human life and safety. Therefore, these external devices determine whether to accept the input of the control signal based on the status information added to the control signal received from the gateway device 110.

[0125] In particular, among users who require a high degree of assistance, there are patients who even need the help of a third party to perform daily fluid replenishment, etc., or need to operate the device under BMI control. Therefore, depending on the safety and intended use of the external device, a function to cut off control due to excessive control time or abnormal control and access control based on the user's capabilities and characteristics (such as misoperation, etc.) are required.

[0126] In the case of the interaction target use of users who have difficulty in accessibility, when establishing a connection with a device that is always nearby using movement in daily life, movement to a nearby area, or communication without accompanying movement, the user interacts with a different world through this path. Interactions such as control and information acquisition on each connection are switched according to the current purpose. Here, when BMI is activated, trial and error is repeated in the formation of thinking nerves, and the idea that a reward can be obtained is retained and learned. As long as this reward-based learning habit remains as an instinct even after establishing a connection with the outside world and there is no functional degradation that makes the brain drowsy, the reward-based trial will continue to repeat as part of development and evolution. Therefore, in particular, as tasks are dynamically collaborated with a large number of unspecified external resources, a dynamic cut-off mechanism becomes indispensable to prevent the acceleration of harmful behaviors in the reward-based trial in the case of establishing a connection with the dynamically changing outside world.

[0127] Therefore, the gateway device 110 is connected to the gateway device 210 that can mainly be wirelessly connected to an external network, etc. The gateway device 110 supplies the intention signal received from the decoding unit 103 and the status information received from the user state monitoring device 120 to the gateway device 210 together.

[0128] The gateway device 210 also refers to Figure 4 the implementation example of the gateway device 10 described above. Similarly, one or more external devices can be connected to the gateway device 210 in a wired or wireless manner. The gateway device 210 converts the intention signal received from the gateway device 110 into a control signal and determines whether to allow the input of the control signal to the external device. The status information received from the gateway device 110 is added to the control signal as needed.

[0129] The peripheral devices 231 and 232 and the mobile body 233 are connected to the gateway device 210 as external devices.

[0130] The peripheral device 231 is an external device located near the user U1, and includes, for example, an indoor light, a television, a music player, etc. The peripheral device 232 is also an external device located near the user U1, and includes, for example, an air conditioner such as an indoor air conditioner, an appliance / device related to combustion or heat generation, etc. The mobile body 233 includes a vehicle or the like that can autonomously drive based on a control signal received from the gateway device 210. The mobile body 233 may include a wheelchair, a care / life assistance robot, etc. that can automatically travel based on a control signal received from the gateway device 210.

[0131] Even when a failure occurs in the peripheral device 231, the risk to the user U1 and the surrounding environment is extremely small, and safety can be maintained. Therefore, these external devices accept the input of the control signal received from the gateway device 210 regardless of the status information added to the control signal.

[0132] On the other hand, when a failure occurs in the peripheral device 232 or the mobile body 233, many risks are brought to the user U1 and the surrounding environment, which may affect human life and safety. Therefore, these external devices determine whether to accept the input of the control signal based on the status information added to the control signal received from the gateway device 210.

[0133] In addition, a mobile terminal 250 that can be connected to an external network such as the Internet can be wirelessly connected to the gateway device 210. The mobile terminal 250 includes a mobile phone such as a smartphone owned by the user U1. The user U1 can use network services as an interaction via the gateway device 110, the gateway device 210, and the mobile terminal 250.

[0134] The mobile terminal 250 is connected to the wide area network provider device 310 via a communication carrier (base station) 261 owned by a telecommunications carrier, a service provider 262, and an external proxy server 263.

[0135] In addition, the mobile terminal 250 can be connected to the wide area network provider device 310 via a third-party terminal 271 owned by a third party and an external proxy server 263, or can be connected to the wide area network provider device 310 via a public Wi-Fi (registered trademark) 272 and an external proxy server 263.

[0136] In addition, the mobile terminal 250 can be connected to the wide area network provider device 310 via a third-party terminal 271 or a public Wi-Fi (registered trademark) 272 and an external proxy server 273 different from the external proxy server 263.

[0137] The wide area network provider device 310 is configured as a relay device that connects the mobile terminal 250 (i.e., the brain of the user U1) to the wide area network or a virtual private network (VPN).

[0138] The wide area network provider device 310 is also an implementation example of the gateway device 10 described above. A network service 400 that provides various services can be connected to the wide area network provider device 310. The mobile terminal 250 supplies the intention signal received from the gateway device 210 to the wide area network provider device 310, and the wide area network provider device 310 determines whether it is necessary to restrict the use of the network service based on the intention signal received from the mobile terminal 250. Figure 4

[0139] The wide area network provider device 310 has functions such as monitoring abnormal traffic, anonymous transactions, intentional over-tunneling of information communication, and supplementary execution processing using external AI between the wide area network provider device 310 and the network service 400, as well as functions such as visualizing such monitoring or reporting such monitoring.

[0140] The network service 400 includes various services provided via the Internet. For example, the network service 400 includes search websites, food ordering websites, social network services (SNS), online shopping, financial transaction services, etc. In addition, the network service 400 can also include services for providing secure socket layer (SSL) processing to prevent tampering or impersonation in external transactions, and factory automation (FA) services capable of controlling and monitoring systems for automated production processes in factories.

[0141] In addition, the network service 400 includes a dark proxy that can be connected to malicious websites such as dark websites, etc., or includes an AI collaboration task execution service 411 that can execute AI collaboration tasks designed specifically for BMI via the monitoring gateway 410. In particular, the monitoring gateway 410 has functions of continuously monitoring malicious intentional information manipulation in AI collaboration tasks, market manipulation such as foreign exchange, false information generation, and assisting and abetting prohibited transactions. In addition, the monitoring gateway 410 also has functions similar to those of the gateway device 10 described above. Figure 4

[0142] In addition, the network service 400 can include a virtual world 450 that forms a virtual space on the Internet. In the virtual world 450, the user U1 can interact with other remote users via BMI, such as disseminating information using avatars.

[0143] The BMI control system 100 can include components other than the above components. Hereinafter, an example of BMI control in the BMI control system 100 will be described.

[0144] (8-2. Control of External Devices) Figure 7 ​​This is a diagram showing an example of how to control an external device in the BMI control system 100.

[0145] Figure 7 It shows an example where the gateway device 110 or the gateway device 210 determines whether to allow the input of a control signal converted from a nerve signal corresponding to the user's intention obtained from the brain BR of the user U1 via the BMI to an external device. The external devices include a device without a risk of failure and a device with a risk of failure. The device without a risk of failure includes the nurse call system 131a as an emergency rescue request device 131, the monitor / speaker 132a as a user device 132, and the peripheral device 231. The device with a risk of failure includes the peripheral device 231 and the vehicle / wheelchair 233a as a moving body 233. The device with a risk of failure has an interlock function LC to cut off the control signal based on the status information added to the control signal.

[0146] For example, in step S111, when a nerve signal corresponding to the intention of operating the peripheral device 232 or the vehicle / wheelchair 233a is obtained from the brain BR of the user U1 via the BMI, the intention signal converted from the nerve signal is supplied to the gateway device 210. At this time, the status information indicating that the user is in a normal conscious state output by the user status monitoring device 120 is supplied to the gateway device 210 together with the intention signal. Hereinafter, it is assumed that the status information is verified by the control button 121.

[0147] In step S112, the gateway device 210 converts the intention signal into a control signal and determines whether to allow the input of the control signal with the status information to the external device. Here, since the status information indicates that the user is in a normal conscious state, it is determined to allow the input of the control signal to the external device, and the peripheral device 231 or the vehicle / wheelchair 233a (interlock function LC) allows (accepts) the input of the control signal.

[0148] In step S121, when a nerve signal corresponding to the intention of operating the nurse call system 131a, the monitor / speaker 132a, or the peripheral device 231 is obtained from the brain BR of the user U1 via the BMI, the intention signal converted from the nerve signal is supplied to the gateway device 110. At this time, the user's consciousness is weakened, for example, the user's consciousness level is low, the user is sleeping, or the user is in a coma, and the status information indicating the weakening of the user's consciousness output by the user status monitoring device 120 is supplied to the gateway device 210 together with the intention signal.

[0149] In step S122, the gateway device 110 converts the intention signal into a control signal and determines whether to allow the input of a control signal with status information to an external device. Here, regardless of the status information, the input of the control signal to the external device is allowed, and the nurse call system 131a, the monitor / speaker 132a, or the peripheral device 231 accepts the input of the control signal.

[0150] Next, in step S123, when a nerve signal corresponding to the intention to operate the peripheral device 231 or the vehicle / wheelchair 233a is acquired from the brain BR of the user U1 via the BMI, the intention signal converted from the nerve signal is supplied to the gateway device 210. At this time, the user's consciousness also weakens, and the status information indicating the weakening of the user's consciousness output by the user status monitoring device 120 is supplied to the gateway device 210 together with the intention signal.

[0151] In step S124, the gateway device 210 converts the intention signal into a control signal and determines whether to allow the input of a control signal with status information to an external device. Here, since the status information indicates the weakening of the user's consciousness, it is determined to reject the input of the control signal to the external device, and the peripheral device 231 or the vehicle / wheelchair 233a (interlock function LC) cuts off (does not accept) the control signal.

[0152] This can reduce the risk to the user U1 and the surrounding environment due to the failure of the peripheral device 231 or the vehicle / wheelchair 233a, and can reduce the impact on human life and safety.

[0153] Thereafter, in the case where the user returns to the normal conscious state, in step S131, when a nerve signal corresponding to the intention to request the release of the state (restricted state) in which the control of the external device is restricted is acquired from the brain BR of the user U1 via the BMI, the intention signal converted from the nerve signal is supplied to the gateway device 110.

[0154] In step S132, upon receiving the intention signal, the gateway device 110 controls the monitor / speaker 132a to present a challenge-response test to check the consciousness of the user U1 (challenge-response test for consciousness). Then, in step S133, the monitor / speaker 132a presents, for example, CAPTCHA to the user U1 as a challenge-response test for the consciousness of the user U1.

[0155] Thereafter, in step S134, when the user status monitoring device 120 normalizes the status information upon receiving the deliberate response made by the user U1 when conscious as the status of the user U1 when presenting the challenge-response test for consciousness, and supplies a restriction release signal for releasing the restricted state to the gateway device 210 via the gateway device 110.

[0156] In step S135, the gateway device 210 determines, based on the restriction release signal, that input of the control signal to the external device is permitted, and the peripheral device 231 or the vehicle / wheelchair 233a (interlock function LC) permits (accepts) input of the control signal having the normalized status information. Note that, even for an external device that affects human life and safety, instead of going through a process of checking the awareness of the user U1 for all control signals, some pre-specified control signals (such as an emergency stop signal, etc.) are permitted to be input to the external device through a simple mechanism with fewer obstacles (such as a two-step command, etc.) to prevent accidents, etc. caused by operation delays due to this process.

[0157] As described above, in the BMI control system 100, it is possible to control an external device while preventing BMI control from getting out of control.

[0158] (8-3. Use of Network Services) Figure 8 FIG. is an example for explaining how network services are used in the BMI control system 100.

[0159] Figure 8 An example is shown in which the wide area network provider device 310 determines whether it is necessary to restrict the use of network services, or determines whether there is an abnormality in the AI cooperation task executed in the network services, based on the nerve signal corresponding to the user's intention acquired from the brain BR of the user U1 via BMI. The network services include a passive service 400a and an active service 400b. The passive service 400a includes services in which the user passively acquires information through processes such as searching and browsing. The active service 400b includes services in which the user actively exchanges information, such as payment procedures, financial transactions, and online shopping. The AI cooperation task is implemented by the above-mentioned monitoring gateway 410 and the AI cooperation task execution service 411.

[0160] For example, in step S311, when a nerve signal corresponding to the intention to use the passive service 400a or the active service 400b is acquired from the brain BR of the user U1 via BMI, the intention signal converted from the nerve signal is supplied to the wide area network provider device 310.

[0161] In step S312, the wide area network provider device 310 determines whether it is necessary to restrict the use of network services based on the intention signal. Here, based on the determination result that both the mental state and the thinking content of the user U1 are normal, it is determined that there is no need to restrict the use of network services, and the passive service 400a and the active service 400b are permitted to be used based on the request signal converted from the intention signal.

[0162] As a result, in steps S313 and S314, responses from the passive service 400a and the active service 400b are returned to the brain BR of the user U1.

[0163] On the other hand, in step S321, when neural signals corresponding to the intention to use the passive service 400a or the active service 400b are acquired from the brain BR of the user U1 via the BMI in a state of abnormal mental state or malicious thought content, the intention signal converted from the neural signals is supplied to the wide area network provider device 310.

[0164] In step S322, the wide area network provider device 310 determines whether it is necessary to restrict the use of network services based on the intention signal. Here, since the mental state of the user U1 is abnormal or the thought content is malicious, it is determined that it is necessary to restrict the use of network services, and the use of the passive service 400a and the active service 400b based on the request signal converted from the intention signal is rejected or restricted. For example, the use of the passive service 400a and the active service 400b is physically cut off, or viewing, transactions, etc. are restricted. Note that in the case of new risky transactions emerging with the widespread use of the BMI in society, such as attempting to access age-restricted information inappropriately, giving inappropriate instructions for drug transactions, instructions for information forgery, dissemination of false information, embedding of information causing social unrest, unauthorized access, impersonation transactions, assisting and abetting fraudulent information manipulation, etc., it is necessary to detect these transactions, cut off the use, or impose customized restrictions on the use.

[0165] As described above, it is possible to monitor the use of network services by users who may have abnormal mental states or malicious intentions.

[0166] For example, in step S331, when neural signals corresponding to the intention to use the AI collaborative task execution service 411 are acquired from the brain BR of the user U1 via the BMI, the control signal converted from the neural signals is supplied to the wide area network provider device 310. The AI collaborative task execution service 411 includes AI-assisted supplementary support services, self-development and evolution services, etc. specifically designed for the BMI.

[0167] In step S332, the wide area network provider device 310 requests the execution of an AI collaborative task based on the intention signal. Specifically, the wide area network provider device 310 supplies the intention signal to the monitoring gateway 410. The monitoring gateway 410 converts the intention signal received from the wide area network provider device 310 into a request signal for requesting the execution of an AI collaborative task, and supplies this request signal to the AI collaborative task execution service 411.

[0168] In step S333 , the AI ​​collaborative task execution service 411 executes the user execution task on the network (network service) with the support of AI.

[0169] At this time, the monitoring gateway 410 determines whether there is an abnormality in the task executed in the monitoring process during the execution of the AI ​​cooperative task, and outputs an alarm as the network service monitoring information if an abnormality in the task is detected (step S334 ).

[0170] (8-4. Monitoring during AI collaborative tasks) In the future, in order to use BMI-based network services, it is expected that interfaces capable of directly connecting to various services such as virtual worlds will be provided to allow information transmission from service provider companies without relay conversion. On the other hand, various services currently provided through the conventional Internet are based on interfaces designed for healthy users.

[0171] At this time, in the case where a mobile terminal owned by an individual wearing a BMI uses various services with communication content equivalent to that of a healthy person that are disclosed on the network, there is no need to perform processing such as monitoring and abnormality detection. However, in the future, as AI collaborative tasks continue to surpass human capabilities, there will be not only beneficial applications such as converting information indicating the intention of a user who has lost the ability to speak due to illness transmitted via BMI into natural speech, but also the risk of external AI resources being abused for supplementation, and even developing to the point where information can be manipulated at will. Therefore, in order to identify behaviors that may lead to antisocial information manipulation, it is necessary to effectively detect the occurrence of the following behaviors: performing inappropriate tasks or collaborative tasks; generating tasks that excessively obtain rewards; tunneling of transmission tasks; excessive concealment and evasion of surveillance; and tasks that flexibly evolve and are performed in response to countermeasures such as anti-money laundering operations.

[0172] As mentioned above, BMI can create new possibilities for human capabilities. On the other hand, using the processing power of an external artificial computer to perform operations that used to require human input through voice or keyboard operation can perform processing far beyond human capabilities by intention alone. In the case where such processing becomes the norm, society may be divided into BMI users who perform such processing and healthy people who live without BMI intervention, resulting in negative effects such as excessive competition in a coexisting society.

[0173] The above impacts bring about excessive competition in various aspects such as profits and ticket purchases affected by extremely small time differences and transactions in volatile markets, leading to the loss of rational human nature; therefore, it is necessary to implement a regulatory function to regulate human transactions. Even when conducting transactions to obtain legal rewards by performing AI collaborative tasks that supplement human capabilities according to command information based on human intentions transmitted via BMI, if ordinary people use AI collaborative tasks to achieve excessive productivity and rewards without BMI intervention, it will also result in unnecessary competition in social activities and lead to the loss of human nature. Therefore, as BMI becomes more widespread in society in the future, it is necessary to review and update the details of the regulatory function as needed.

[0174] On the other hand, in the technology according to the present disclosure, from the ethical perspective of society, filtering of artificially introduced processing and the assignment of necessity and permission flags for task transactions are performed.

[0175] For example, consider implementing the following functions.

[0176] • Establish a communication protocol dedicated to BMI information transmission and dynamic instructions for external AI, assign a permission flag for profit-making processing, assign an ethical permission flag, assign a detection flag for automatic replication and proliferation processing, and assign a verification flag for executing owner information • Use blockchain technology to manage the linkage of the command system to prevent the tampering of user information at the command source when malicious tasks are executed using external AI services • Detect abnormal communications and abnormal programs in traffic monitoring and notify the administrator • Detect excessive concealment processing of communication content and unauthorized information concealment and notify the administrator • Monitor information evasion programs and notify the administrator when detected • Detection report of web scraping processing • Detect anti-monopoly violations and notify the administrator • Detect unreported information processing and notify the administrator • Prevent and block task hoarding with the assistance of external AI • Detect information manipulation on SNS, etc. and notify the administrator • User soundness scoring • Introduce a positive score for sound transactions of BMI-intervened users

[0177] Here, the flowchart in Figure 9 will be referred to to illustrate the process of the monitoring processing performed during the execution of the AI collaborative task in step S334 in Figure 8 . Figure 9The processing in [ ] can be performed by the monitoring gateway 410 for each user using the AI collaboration task.

[0178] In step S411, the monitoring gateway 410 requests a transaction evaluation for each AI service based on its usage record.

[0179] In step S412, the monitoring gateway 410 updates the monitoring parameters for monitoring transmission information based on the excellent evaluations in the requested transaction evaluations.

[0180] In step S413, the monitoring gateway 410 detects signs of attempts to evade monitoring of transmission traffic.

[0181] In step S414, the monitoring gateway 410 monitors and filters transmission information. The monitoring and filtering can be performed in response to threats such as new forms of abuse and new usage trends that emerge over time.

[0182] In step S415, the monitoring gateway 410 detects the execution of inappropriate AI collaboration tasks in AI collaboration tasks beyond human capabilities.

[0183] In step S416, the monitoring gateway 410 detects information manipulation using an AI-driven SNS.

[0184] In step S417, the monitoring gateway 410 scores the user soundness indicating the soundness of various transactions performed by the user and evaluates whether to allow the continued use of various transactions.

[0185] In step S418, the monitoring gateway 410 notifies the monitor who monitors the AI collaboration task through, for example, the monitoring gateway 410 of the evaluation result of whether to allow the continued use of various transactions as monitoring information about the AI collaboration task. The evaluation result includes the above various flags, the scored user soundness, etc. However, here, the purpose is not to monitor the details of the information content transmitted by individuals. It is necessary to manage the information itself exchanged by individuals while ensuring its protection. Therefore, here, only progressive monitoring is required without directly reading and viewing the content of the transmission information. That is to say, abnormal or unnatural communication activities such as using jargon, accessing websites designated for monitoring, attempting unauthorized access, transmitting executable code, and using transaction websites or services with undisclosed operating status, fraudulent attempts, etc. are closely monitored. In the case of observing an abnormality, the monitor is notified of the abnormality, thus allowing people to flexibly track and confirm. Abnormal monitoring traps for detecting behaviors that evade monitoring make it easier to detect these behaviors.

[0186] It is necessary for the monitor to take actions on the user based on the evaluation result notified by the monitoring gateway 410 as needed.

[0187] Figure 10 This is a flowchart showing the action-taking process executed by the monitor based on the evaluation results.

[0188] In process P430, the monitor determines whether the evaluation results notified from the monitoring gateway 410 belong to "excellent", "need to correct", or "malicious".

[0189] When the evaluation result is "excellent", the process proceeds to process P431, and the monitor updates the user's health (user evaluation) scored by the monitoring gateway 410. The updated user health is saved for each user as the actual value for each AI service.

[0190] When the evaluation result is "need to correct", the process proceeds to process P432, and the monitor checks the user's transaction records based on various flags included in the evaluation results.

[0191] Next, in process P433, the monitor audits the transaction records and checks for any violations.

[0192] Then, in process P434, the monitor instructs the user and the service provider of the transaction service where the violation occurred to correct the violation through notifications, warnings, etc.

[0193] When the evaluation result is "malicious", the process proceeds to process P435, and the monitor temporarily or permanently prohibits transactions between the user and the service provider that provides the service used by the user.

[0194] Through the above processing, it is possible to prevent inappropriate tasks from being executed by monitoring various transactions of users who perform AI collaboration tasks via BMI.

[0195] (8-5. Information dissemination via an avatar) As described above, in the virtual world 450, interactions such as spreading information using an avatar can be performed through BMI. Interactions using an avatar can be achieved through natural language conversations using AI.

[0196] Incidentally, in a general chat function, the user's intention is electronically converted into text through a keyboard, and the user confirms that the intention has been accurately converted into text information and presses the send button, thereby sending the accurate intention to the other party.

[0197] On the other hand, when communicating with others, "retaining thoughts that may damage the receiver's impression of oneself rather than verbally expressing them" makes social activities smoother. However, directly transmitting the information of the expression intention via BMI can lead to social problems. The results of the intention to convey to the other party and the actually conveyed intention are completely different in social life. Therefore, in the case of converting the thinking result into information that can be sent by an external relay device via BMI, a process is required in which the thinking result is converted into information recognizable by the user and fed back, the user confirms that the information reflects what they actually want to convey to the other party, and then sends the information.

[0198] Here, a process of information dissemination processing through an avatar in the virtual world 450 will be described with reference to Figure 11 Describe the process of information dissemination processing through an avatar in the virtual world 450. Figure 11 The processing in is executed by the server that implements the virtual world 450 and starts when the user uses the avatar to send his / her intention for information dissemination. The server that implements the virtual world 450 also has a function equivalent to that of the gateway device 10 described with reference to Figure 4 The described function of the gateway device 10.

[0199] In step S451, the virtual world 450 verbalizes the intention content of the user represented by the intention signal sent via BMI or the Internet to generate audio data to be output by the avatar based on the language information.

[0200] In step S452, the virtual world 450 converts the language information obtained by verbalizing the user's intention into information recognizable by the user, and thus feeds back the generated audio data to the user. The user checks whether the content of the language information obtained by verbalizing the user's intention is inaccurate based on the feedback information. In the case where the content of the language information is inaccurate, the intention signal indicating the correction content is sent to the virtual world 450 via BMI or the Internet, and in the case where the content of the language information is accurate, the intention signal indicating whether to allow the transmission of the audio data (the verification result of the language information) is sent via BMI or the Internet.

[0201] In step S453, the virtual world 450 determines whether the verbalized content is inaccurate based on the intention signal transmitted via BMI or the Internet.

[0202] In the case where it is determined in step S453 that the verbalized content is inaccurate, the process proceeds to step S454, and the virtual world 450 corrects the audio data based on the correction content represented by the intention signal. Thereafter, the process returns to step S452, and the subsequent processing is repeated.

[0203] On the other hand, when it is determined in step S453 that the verbalized content is accurate, the process proceeds to step S455, where the virtual world 450 determines whether to allow the transmission (whether to allow information dissemination) based on whether to allow the transmission of the audio data (verification result of language information) represented by the intention signal.

[0204] When it is determined in step S455 that transmission is allowed, the process proceeds to step S456, where the virtual world 450 uses the avatar to send the generated audio data to the interaction partner.

[0205] On the other hand, when it is determined in step S455 that transmission is rejected, the process proceeds to step S457, where the virtual world 450 discards the language information converted from the intention information and the generated audio data.

[0206] Through the above processing, in a system capable of conveying intentions via BMI, it is possible to prevent thoughts that are not intended to be conveyed to the other party from being detected via BMI and conveyed to the other party.

[0207] Examples of BMI control in the BMI control system 100 have been described above. Note that any device constituting the BMI control system 100 may be equipped with the functions of the gateway device 10 described with reference to Figure 4 This can prevent the out-of-control of BMI control that may be caused by the user's awareness, thoughts, or mental state.

[0208] Note that the structure and function of the gateway device can be further improved. For example, it can evaluate the impact when the information transmitted by the user is conveyed to the recipient, and suggest modifications to the information transmitted to the recipient. In recent years, many information terminals are equipped with a function of presenting emojis or character stamps associated with the input content as a communication means. As an advanced use of this function, by providing feedback such as characters representing the possible impressions of the recipient for the information conveyed based on the context, accidental injuries caused by careless information transmission can be avoided. As described above, by visualizing the impact that may lead to social sanctions and providing feedback to the user before sharing information on SNS, etc., so that the user can identify and confirm careless information transmission, a mechanism to prevent the disclosure of such information can be realized.

[0209] Before the information is conveyed to the other party, there are temporal and physical steps in the process of actual pronunciation or the concretization of physical text input to the terminal. Similarly, in the case of conveying intentions via BMI, it is desirable for the gateway device to also provide the above-mentioned feedback to encourage the user to improve self-awareness, and in some cases, it is desirable for the gateway device to prevent the user from conveying his / her intentions.

[0210] <9. Structural Example of Computer Hardware> The above-described series of processes can be executed by hardware or software. In the case where the series of processes are executed by software, a program included in the software is installed from a program recording medium onto a computer incorporated with dedicated hardware, a general-purpose personal computer, or the like.

[0211] Figure 12 FIG. is a block diagram showing a configuration example of the hardware of a computer that executes the above-described series of processes according to a program. For example, the gateway device 10 and each device that constitutes the BMI control system 100 and corresponds to the gateway device 10 include a PC having a configuration similar to the Figure 12 configuration shown.

[0212] A central processing unit (CPU) 501, a read-only memory (ROM) 502, and a random access memory (RAM) 503 are interconnected via a bus 504.

[0213] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc. and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Further, a storage unit 508 including a hard disk, a non-volatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives a removable medium 511 are also connected to the input / output interface 505.

[0214] In the computer configured as described above, for example, the CPU 501 loads a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program to perform the above-described series of processes.

[0215] For example, a program executed by the CPU 501 is recorded in the removable medium 511, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and then installed in the storage unit 508.

[0216] A program executed by a computer may be a program that executes processes in time series in the order described in this specification, or may be a program that executes processes in parallel or at a necessary timing such as when making a call.

[0217] Note that, in this specification, a system refers to a collection of a plurality of components (devices, modules (parts), etc.), and it is not important whether all the components are in the same housing. Thus, a plurality of devices accommodated in separate housings and connected to each other via a network and a single device including a plurality of modules accommodated in a single housing are both systems.

[0218] The effects described in this specification are merely examples and are not restrictive, and other effects may also be provided.

[0219] Embodiments of the present disclosure are not limited to the above embodiments, and various modifications can be made without departing from the gist of the present disclosure.

[0220] In addition, for example, embodiments of the present disclosure can be configured as cloud computing in which multiple devices share a function and jointly execute processing via a network.

[0221] In addition, each step described in the above flowchart can be executed by one device, or can be executed by multiple devices in a shared manner.

[0222] In addition, in the case where a step includes multiple processes, the multiple processes included in the one step can be executed by one device or by multiple devices in a shared manner.

[0223] The effects described in this specification are merely examples and are not restrictive, and other effects can be provided.

[0224] In addition, the technology according to the present disclosure can have the following configuration. (1) A signal processing device, comprising: A determination unit that determines whether to permit the interaction of the user based on neural signals corresponding to the intention of the user acquired from the brain of the user via a brain-machine interface (BMI). (2) The signal processing device according to (1), wherein The interaction includes the operation of an external device connected in a wired or wireless manner, and The determination unit determines whether to permit the input of a control signal converted from the neural signals to the external device. (3) The signal processing device according to (2), wherein The determination unit determines whether to permit the input of the control signal to the external device according to whether there is a risk caused by a failure of the external device. (4) The signal processing device according to (3), wherein The determination unit determines whether to permit the input of the control signal to the external device based on the conscious state of the user. (5) The signal processing device according to (3), wherein The determination unit determines whether to permit the input of the control signal to the external device based on an instruction input by a third party. (6) The signal processing device according to any one of (2) to (5), wherein The external device includes at least one of a wearable device, a mobile device, a calling device, a display device, an audio output device, an air conditioner, and a moving body. (7) The signal processing device according to (6), wherein The moving body includes at least one of a wheelchair and a vehicle. (8) The signal processing device according to any one of (1) to (7) further includes: A generation unit that generates a feedback signal for feeding back the intention of the user to the user based on the intention signal based on the nerve signal. (9) The signal processing device according to (8), wherein The feedback signal includes at least one of a display signal for display, an audio signal for audio output, and a BMI signal input to the brain of the user via the BMI. (10) The signal processing device according to (1), wherein The interaction includes the use of a network service, and The determination unit determines whether it is necessary to restrict the use of the network service. (11) The signal processing device according to (10), wherein The determination unit determines whether it is necessary to restrict the use of the network service based on the mental state or thinking content of the user. (12) The signal processing device according to (11), wherein In the case where the user has an abnormal mental state or may have malicious intent, the determination unit refuses or restricts the use of the network service. (13) The signal processing device according to any one of (10) to (12), wherein The network service includes at least one of a social networking service (SNS), online shopping, and a financial transaction service. (14) The signal processing device according to (1), wherein The interaction includes performing a task in cooperation with artificial intelligence (AI) in a network service, and The determination unit determines whether there is an abnormality in the task to be performed. (15) The signal processing device according to (14), wherein In the case where an abnormality in the task is detected, the determination unit outputs an alarm as monitoring information regarding the network service. (16) The signal processing device according to (1), wherein the interaction includes information dissemination using an avatar on a network, and the determination unit determines whether to permit the information dissemination. (17) The signal processing device according to (16), wherein the information to be disseminated includes linguistic information obtained by verbalizing the intention of the user, and the determination unit determines whether to permit the information dissemination based on the verification result of the linguistic information fed back to the user. (18) The signal processing device according to any one of (1) to (17), wherein the neural signal includes a signal multiplexed using a probe electrode array arranged in a matrix form. (19) A signal processing method, comprising: causing a signal processing device to determine whether to permit interaction of the user based on a neural signal corresponding to the intention of the user acquired from the brain of the user via a brain-machine interface (BMI). (20) A signal processing system, comprising: a gateway device that determines whether to permit interaction of the user based on a neural signal corresponding to the intention of the user acquired from the brain of the user via a brain-machine interface (BMI). List of reference numerals

[0225] 10 Gateway device 11 Determination unit 12 Storage unit 100 BMI control system 110 Gateway device 210 Gateway device 310 Wide area network provider device 410 Monitoring gateway 450 Virtual world

Claims

1. A signal processing device, comprising: A determination unit that determines whether to permit the user's interaction based on neural signals corresponding to the user's intention obtained from the user's brain via a brain-machine interface (BMI).

2. The signal processing device according to claim 1, wherein The interaction includes the operation of an external device connected in a wired or wireless manner, and The determination unit determines whether to permit the input of a control signal converted from the neural signals to the external device.

3. The signal processing device according to claim 2, wherein The determination unit determines whether to permit the input of the control signal to the external device according to whether there is a risk caused by a failure of the external device.

4. The signal processing device according to claim 3, wherein The determination unit determines whether to permit the input of the control signal to the external device based on the user's state of consciousness.

5. The signal processing device according to claim 3, wherein The determination unit determines whether to permit the input of the control signal to the external device based on an instruction input by a third party.

6. The signal processing device according to claim 2, wherein The external device includes at least one of a wearable device, a mobile device, a calling device, a display device, an audio output device, an air conditioner, and a moving body.

7. The signal processing device according to claim 6, wherein The moving body includes at least one of a wheelchair and a vehicle.

8. The signal processing device according to claim 1, further comprising: A generation unit that generates a feedback signal for feeding back the user's intention to the user according to an intention signal based on the neural signals.

9. The signal processing device according to claim 8, wherein The feedback signal includes at least one of a display signal for display, an audio signal for audio output, and a BMI signal input to the user's brain via the BMI.

10. The signal processing device according to claim 1, wherein The interaction includes the use of a network service, and The determination unit determines whether it is necessary to restrict the use of the network service.

11. The signal processing device according to claim 10, wherein The determination unit determines whether it is necessary to restrict the use of the network service based on the user's mental state or thought content.

12. The signal processing device according to claim 11, wherein In the case where the user has an abnormal mental state or may have malicious intent, the determination unit rejects or restricts the use of the network service.

13. The signal processing device according to claim 10, wherein The network service includes at least one of a social networking service (SNS), online shopping, and a financial transaction service.

14. The signal processing device according to claim 1, wherein The interaction includes performing a task in cooperation with artificial intelligence (AI) in a network service, and The determination unit determines whether there is an abnormality in the task to be performed.

15. The signal processing device according to claim 14, wherein In the case where an abnormality in the task is detected, the determination unit outputs an alarm as monitoring information regarding the network service.

16. The signal processing device according to claim 1, wherein the interaction includes information dissemination using an avatar on a network, and the determination unit determines whether to permit the information dissemination.

17. The signal processing device according to claim 16, wherein the information to be disseminated includes language information obtained by verbalizing the intention of the user, and the determination unit determines whether to permit the information dissemination based on the verification result of the language information fed back to the user.

18. The signal processing device according to claim 1, wherein the neural signal includes a signal multiplexed using a probe electrode array arranged in a matrix form.

19. A signal processing method, comprising: causing a signal processing device to determine whether to permit an interaction of a user based on a neural signal corresponding to the intention of the user acquired from the brain of the user via a brain-machine interface (BMI).

20. A signal processing system, comprising: a gateway device that determines whether to permit an interaction of the user based on a neural signal corresponding to the intention of the user acquired from the brain of the user via a brain-machine interface (BMI).

Images