Device and method for performing human augmentation based on neuromorphic artifiial intelligence and neural interface

Abstract

The present invention relates to a method for generating a force augmentation signal from a biosignal using an AI algorithm and a neuromorphic chip, comprising: (a) receiving biosignals from a plurality of sensors connected to a plurality of parts of a human body; (b) preprocessing the biosignals; (c) performing training using the biosignals as training data to infer the type of movement intention and the probability value of the movement intention when the biosignals are input, using a force augmentation model including a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory); (d) inputting the biosignals of a human collected in real time into the trained force augmentation model to produce information about the movement intention as an output value; and (e) transmitting the output value to a wearable device worn by a human to apply force to the human's movement so that the wearable device can perform the movement intended by the human, wherein the wearable device includes a prosthetic arm or a robotic arm worn by a human.

Description

Technology Field

[0001] The present invention relates to a human body augmentation technology for increasing human cognitive or motor abilities based on neuromorphic AI and a neural interface.

[0002] Specifically, the present invention relates to an apparatus and method that overcomes the inherent limitations of existing Brain-Computer Interface (BCI) technology in the accuracy and response speed of neural signal interpretation and the difficulty of implementing direct body control and cognitive augmentation by strengthening connectivity between neurons, and precisely controls and enhances human cognitive and motor functions through ultra-low power (1 pJ or less) and ultra-high speed (within 50 μs) neural signal processing technology. Background Technology

[0003] BCI (Brain-Computer Interface)-based human augmentation technology has evolved with the goal of enhancing human memory, motor control, and sensory abilities. The U.S. Defense Advanced Research Projects Agency (DARPA) developed a BCI system to improve short-term memory, but it had limitations in practical cognitive enhancement. Additionally, the neurotechnology company Synchron demonstrated the potential for prosthetic arm control using its non-invasive Stentrode technology, but it faced limitations in realizing natural movements. Furthermore, Boston Dynamics experimented with applying neural controllers to robots to enhance physical capabilities, but faced issues with delays in real-time response.

[0004] Current BCI technology faces limitations due to low precision and response speed in neural signal processing. While EEG offers excellent temporal resolution, its spatial resolution is low, whereas EMG and fNIRS present challenges in measurement due to brain signal interpretation and real-time requirements, respectively. Furthermore, the cases of DARPA, Synchron, and Boston Dynamics all share a common problem of low accuracy and response delays exceeding tens of milliseconds.

[0005] Meanwhile, high-performance BCI systems are unsuitable for wearable devices due to high power consumption, and their application range is limited to short-term memory enhancement or limited motor assistance. In addition, security issues are serious, with tens of millions of neural data leaks occurring annually despite AES-128 level encryption.

[0006] Accordingly, there is a need to implement a next-generation BCI system capable of rapidly and accurately measuring and processing the user's neural signals while enhancing the user's cognitive or motor abilities based on this. The problem to be solved

[0007] The present invention aims to solve the problems of the aforementioned prior art, specifically the problem of low precision and response speed in neural signal processing inherent in conventional BCI systems.

[0008] Specifically, conventional BCI and neural interface technologies make it difficult to perform precise neural signal analysis and interpretation at the neuron level due to low spatial resolution (10 mm) and signal recognition rate (70% or less). In addition, conventional BCI and neural interface technologies have limitations that make them difficult to apply to wearables and real-time neural interfaces due to power consumption (tens of mW or more) and processing delay (10 ms or more). Furthermore, conventional BCI and neural interface technologies have weaknesses such as being limited to cognitive function enhancement (memory improvement 20%) and motor ability enhancement (50% naturalness), limited application fields, and vulnerability to data leakage or hacking.

[0009] Therefore, the goal is to overcome the limitations of the aforementioned existing BCI (Brain-Computer Interface) and neuromorphic computing technologies and to dramatically enhance human cognitive and physical capabilities. means of solving the problem

[0010] A method for generating a force augmentation signal from a biosignal using an AI algorithm and a neuromorphic chip to achieve the above-mentioned purpose comprises: (a) receiving biosignals from a plurality of sensors connected to a plurality of parts of a human body; (b) preprocessing the biosignals; (c) using a force augmentation model including a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory) to infer the type of movement intention and the probability value of the movement intention when the biosignal is input, using the biosignal as training data to perform training; (d) inputting the biosignal of a human collected in real time into the trained force augmentation model to produce information about the movement intention as an output value; and (e) transmitting the output value to a wearable device worn by a human to apply force to the human's movement so that the wearable device can perform the movement intended by the human, wherein the wearable device may include a prosthetic arm or a robotic arm worn by a human.

[0011] In addition, the biosignal includes at least one of a brainwave signal, an electromyogram signal, and a photoplethysmogram signal, and the brainwave signal or electromyogram signal can be collected by sampling at 250 to 1000 Hz.

[0012] In addition, the preprocessing process involves applying noise removal, normalization, and time-axis-based synchronization to the biosignals. Synchronization is performed so that the time axes of the brainwave signal, electromyogram signal, and photoplethysmogram signal are all matched identically. For noise removal, in the case of the brainwave signal, noise is removed using a 1-50Hz bandpass filter followed by defect removal using ICA / FFT (Independent Component Analysis in combination with FFT), and in the case of the electromyogram signal, noise is removed using a QRS detector and baseline drift technique. Normalization can be performed by normalizing the biosignals according to the time series using the z-score technique, and then matching the time series length values ​​of different biosignals using padding or slicing techniques.

[0013] Additionally, step (c) may include: (c-1) inputting the biosignal into a CNN model to calculate a spatial feature vector between the biosignals; (c-2) inputting the spatial feature vector into an LSTM model to learn the time pattern and calculate a temporal feature vector of the biosignal; (c-3) inputting the temporal feature vector into a linear transformation model that constitutes a Dense Layer and calculating a pre-set score for each body classification; and (c-4) inputting the score for each body classification into a Softmax and generating a human force augmentation model that infers the type of human movement intention and the probability value of the movement intention as output values.

[0014] In addition, the input value to the CNN model is a vector value of the biosignal, and the vector value may consist of the number of data samples of people participating in the biosignal collection, the sampling value in the time axis (unit is Hz), and the channel number of the biosignal collection sensor.

[0015] Additionally, step (c-3) is defined as a classification category for movements of each body part including hand movements, foot movements, and head movements, and the linear transformation model outputs how many points the data output by the LSTM model has in which classification category, and after step (c-4), the human augmentation model is evaluated by applying at least one of the Accuracy, F1 Score, and Confusion Matrix techniques, and may further include a step of evaluating the human augmentation model by comparing the output value produced when input data is constructed from biosignals actually measured from a person and input into the human augmentation model with the person's actual movement.

[0016] In addition, after step (d), the workforce augmentation model can be modified by fixing the parameter values ​​of the CNN model constituting the workforce augmentation model, changing the parameter values ​​of the layer constituting the LSTM, and changing the number of classification categories constituting the Dense Layer, and by inputting biosignals of new users into the modified workforce augmentation model as new data to perform fine-tuning and complete the learning, thereby enabling the model to be advanced into a personalized learning model tailored to each user.

[0017] In addition, the neuromorphic chip may include a 3nm CMOS-memristor neuron processor capable of performing ultra-high-speed neural computation by operating at an operating frequency of 2GHz and providing a spike response speed of 0.5ns, and a non-volatile memory having 512-step variable resistors, and may include a ReRAM synapse array capable of updating weights between neurons at an ultra-high speed of 0.8ns. Effects of the invention

[0018] The present invention can solve the limitations of conventional BIC systems.

[0019] Specifically, by improving the neural processing speed and precision of the BCI system, cognitive and physical abilities can be enhanced based on various signals received from the user's body.

[0020] In other words, by maximizing the user's body augmentation based on neural interfaces and neuromorphic computing, it becomes possible to precisely control human cognitive abilities and motor functions.

[0021] Through this, by combining ultra-precision neuromorphic augmentation technology with AI-based neural interfaces, it is possible to realize a next-generation Human Augmentation system that dramatically enhances human cognitive and physical capabilities. Furthermore, it is possible to break through the technological limitations of the neurotech industry and create innovative value in various fields such as artificial intelligence, healthcare, defense, and the space industry.

[0022] Key effects include augmented precision, ultra-low power / ultra-high speed processing, real-time adaptability, and the creation of economic value.

[0023] For example, the present invention utilizes neuromorphic-AI fusion technology to dramatically improve human cognitive and physical abilities compared to existing ones, overcomes the limitations of existing BCI (Brain-Computer Interface) and neural interface technologies, and realizes over 80% cognitive enhancement and over 95% physical naturalness, thereby strengthening memory, learning ability, sensory and motor functions beyond human limits.

[0024] In addition, conventional neuromorphic chips and BCI systems require the use of high-output GPUs for high-performance computation, and high-output GPUs cause high power consumption and processing delays, which limits the construction of real-time neural interfaces. However, the present invention implements ultra-low power (less than 1 pJ) and ultra-high-speed (within 50 μs) neural signal processing technology, thereby achieving more than 100 times improvement in computational performance and speed compared to existing systems.

[0025] In addition, conventional BCI and body augmentation technologies are unable to adapt in real time, requiring users to take tens of minutes or more of learning time to adapt to a new environment, but the present invention provides real-time adaptability within 5 seconds, enabling users to immediately utilize the neuro interface.

[0026] In addition, the present invention is expected to establish itself as a core technology platform for the Human Augmentation market after 2040, and its market value is also on the rise. Brief explanation of the drawing

[0027] FIG. 1 is a flowchart visually illustrating the process of human body augmentation being performed based on an artificial intelligence algorithm and a neural interface according to an embodiment of the present invention. FIG. 2 is a diagram showing the configuration of a system for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip according to an embodiment of the present invention. FIG. 3 is a diagram showing the configuration of a neuromorphic computing device according to one embodiment of the present invention. FIG. 4 is a diagram showing the configuration of a processor according to one embodiment of the present invention. FIG. 5 is an operation flowchart illustrating the process of driving a wearable device by generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating the process of constructing a manpower enhancement model according to one embodiment of the present invention. Specific details for implementing the invention

[0028] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0029] The following examples are detailed descriptions to aid in understanding the present invention and are not intended to limit the scope of the present invention. Accordingly, inventions within the same scope that perform the same function as the present invention will also fall within the scope of the present invention.

[0030] Before explaining the following, the term 'BCI (Brain-Computer Interface)' in this specification may refer to an interface device that connects the brain and a computer to enable direct interaction between them. Additionally, the terms 'neuromorphic' or 'neuromorphic chip' in this specification may correspond to a sub-module of a processor designed to solve the problems of conventional BCIs by fusing BCIs with artificial intelligence.

[0031] FIG. 1 is a flowchart visually illustrating the process of human body augmentation being performed based on an artificial intelligence algorithm and a neural interface according to an embodiment of the present invention.

[0032] FIG. 2 is a diagram showing the configuration of a system for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip according to an embodiment of the present invention.

[0033] Referring to FIG. 1, the BCI system (1) that can be performed through the present invention can be performed through the following four processes.

[0034] First, the system (1) can collect biosignals including brain signals of the user (200) and process the collected brain signals through a neuromorphic chip. Then, the system (1) can perform AI analysis on the processed data to calculate what action the user will perform and output an augmentation signal through the calculated result to control a wearable device (e.g., a prosthetic arm).

[0035] To implement the above process, the BCI system (1) may include a neuromorphic computing device (100), a plurality of sensors (210), and a wearable device (220).

[0036] A neuromorphic computing device (100) according to one embodiment of the present invention is driven by a neuromorphic chip to be described later, inputs biometric information received from a plurality of sensors (210) into a pre-set human resource augmentation model, and outputs information regarding the movement intention of a user (200) as an output value and provides it to a wearable device (220).

[0037] The neuromorphic chip of the present invention may include a 3nm CMOS-memristor neuron processor capable of performing ultra-high-speed neural computation by operating at an operating frequency of 2GHz and providing a spike response speed of 0.5ns, and a non-volatile memory having a 512-step variable resistor, and may include a ReRAM synapse array capable of updating weights between neurons at an ultra-high speed of 0.8ns, and a detailed description will be provided later through FIG. 4.

[0038] At this time, the bio-information may include at least one of brainwave signals (EEG), which are brain nerve information, electromyography signals (EMG) related to nerves, and photoplethysmography signals (fNIRS). However, as an additional embodiment, real-time video data or voice data capturing the user (200) may be additionally received and used as bio-information. Through this, the user's (200) movement intention can be calculated based on the user's (200) behavior, facial expression changes, or voice characteristics.

[0039] Additionally, the neuromorphic chip may be a sub-module included in the processor (130) of the neuromorphic computing device (100). In this case, the neuromorphic chip may be a chipset intended to be implemented through the present invention to overcome the low resolution and response speed of conventional chipsets.

[0040] Additionally, information regarding the user's (200) movement intention may be information regarding whether the user (200) has an intention to move a specific body part consciously or unconsciously. For example, changes in various biosignals given to the arm and fingers to perform the action of 'raising the right arm and extending all fingers' may correspond to information regarding the movement intention.

[0041] According to one embodiment of the present invention, the sensor (210) and the wearable device (220) may correspond to equipment that the user (200) wears, respectively.

[0042] First, the sensor (210) is intended to measure biosignals and may be equipment attached to the body of the user (200) to measure brainwave signals, electromyogram signals, and photocirculatory signals, respectively. For example, brainwave signals may be implemented as a brainwave measuring sensor worn on the head as shown in the drawing, and electromyogram signals may be implemented through a sensor attached to the skin over muscles such as arms, legs, and face to measure electrical signals of the muscles. In the case of photocirculatory signals, a near-infrared spectroscopic sensor is attached to the skin to measure changes in blood flow.

[0043] The wearable device (220) may include a prosthetic arm worn by a person or a robotic arm. That is, the prosthetic arm shown in the drawing may be included in the wearable device (220). However, as an optional embodiment, the wearable device (220) may be used to replace a manipulator for assembling products in an assembly plant or an input device for operating a computer.

[0044] Accordingly, the sensor (210) receives biometric data from the user (200) and provides it to the neuromorphic computing device (100), and the wearable device (220) drives the wearable device (220) according to the output value received from the neuromorphic computing device (100).

[0045] At this time, the output value is information about the movement intention that the neuromorphic chip has converted into information for driving the wearable device (220), or the information about the movement intention itself, so that the wearable device (220) can be driven by interpreting the information about the movement intention.

[0046] FIG. 3 is a diagram showing the configuration of a neuromorphic computing device (100) according to one embodiment of the present invention.

[0047] Referring to FIG. 3, the neuromorphic computing device (100) may include a communication module (110), memory (120), a processor (130), and a database (140).

[0048] To explain in detail, the communication module (110) provides a communication interface necessary to provide transmission and reception signals between the neuromorphic computing device (100) and a plurality of sensors (210) or wearable devices (220) in the form of packet data in conjunction with a communication network. Furthermore, the communication module (110) can perform the role of the neuromorphic computing device (100) receiving a data request from a plurality of sensors (210) or wearable devices (220) and transmitting data as a response thereto.

[0049] Here, the communication module (110) may be a device including hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0050] In a specific embodiment, the communication module (110) may have a Spike-Based Neural Communication Protocol. This can overcome the limitations of implementing a real-time interface, such as the large data overhead and limited neural signal transmission speed of conventional BCI systems and neuromorphic chips. Accordingly, the communication module (110) of the present invention can transmit neural data at an ultra-high speed of 60 Gbps by applying Opto-Electronic Interconnect-based spike signal communication technology.

[0051] That is, the communication module (110) performs 60Gbps optical-electronic hybrid communication, reducing data overhead by 95% compared to conventional wired signal transmission, and applies a spiking neural network (SNN) protocol to reproduce communication between neurons in a biomorphic manner, thereby minimizing delay and securing neural signal transmission efficiency of over 99%.

[0052] The memory (120) records a program for generating a force enhancement signal from a bio-signal using an AI algorithm and a neuromorphic chip. It also performs the function of temporarily or permanently storing data processed by the processor (130). Here, the memory (120) may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto.

[0053] The processor (130) is a type of central processing unit that controls the entire process of generating a force enhancement signal from a bio-signal using an AI algorithm and a neuromorphic chip. Each step performed by the processor (130) will be described later with reference to FIGS. 5 and 6.

[0054] Here, the processor (130) may include any type of device capable of processing data, such as a processor. Here, 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program, for example. Examples of such data processing devices embedded in hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0055] However, in order to achieve faster performance compared to conventional chipsets, specifications may include a 3nm CMOS-memristor-based neuron processor, a ReRAM synapse array, and a 3D TSV (Through-Silicon Via)-based neural network interface. An example description of specific specifications will be provided later through Fig. 4.

[0056] The database (140) can store biosignals for the user (200) and output values ​​for the biosignals. In this case, the data stored in the database (140) can be used as information for fine-tuning the workforce enhancement model.

[0057] Although not illustrated in FIG. 3, some of the data described above may be stored in a database (not illustrated) that is physically or conceptually separated from the database (140).

[0058] Meanwhile, the multiple sensors (210) described above can apply multimodal interface technology (EEG headset (256 channels), ECG patch, EMG sensor) that integrates and analyzes neural signals (EEG), electromyography signals (EMG), and photoplethysmography signals (fNIRS), and through the multimodal interface, the low spatial resolution and signal accuracy of conventional BCI systems can be overcome. That is, through the multimodal interface technology (EEG headset (256 channels), ECG patch, EMG sensor) applied to the multiple sensors (210), the coordination pattern of neurons and muscles can be precisely analyzed, and an ultra-precision neuromuscular interface can be realized.

[0059] Specifically, brainwave signal (EEG) analysis is performed using an EEG sensor, and brain signals can be collected with an ultra-high-speed temporal resolution of less than 1ms using a 256-channel high-density EEG sensor. At this time, the signal-to-noise ratio (SNR) of the brainwave signals can be improved by more than 20dB by applying a reinforcement learning-based noise removal algorithm.

[0060] The analysis of electromyogram (EMG) signals is performed through an EMG neural signal amplifier, and even minute muscle signals can be detected through a 128-channel electromyogram sensor, and neural activity can be measured with a resolution of 5 mm or less by applying amplification technology of a superconducting quantum interferometer (SQUID).

[0061] The analysis of photoplethysmosis (fNIRS) performed neural activity analysis based on near-infrared spectroscopy, 0.1mm 3 By analyzing changes in brain blood flow in real time with high resolution, metabolic activity of neurons can be precisely measured, and by applying this to an AI-based signal processing engine, analysis can be performed at a speed more than five times faster than existing methods.

[0062] Meanwhile, the wearable device (220) may include a modular augmentation platform applicable not only to wearables but also to implantable devices, and may support various cognitive and physical augmentation applications.

[0063] At this time, the wearable device (220) processes neural signals in real time through a neuron-chip interface via a 120Gbps high-speed neural data bus and can implement more than 90% natural movement based on real-time neural feedback through a smart prosthetic arm and exoskeleton robot integration interface. In addition, by including a cognitive enhancement module (AI-Hippocampus Engine) in the wearable device (220), memory and learning speed can be improved (more than 80%).

[0064] FIG. 4 is a diagram showing the configuration of a processor (130) according to one embodiment of the present invention.

[0065] Referring to FIG. 4, the processor (130) may include an AI engine (131) on which a workforce augmentation model is operated and a control unit (132) composed of a neuromorphic core.

[0066] As previously explained, the neuromorphic chip is a core processor (130) of a neuromorphic computing device (100) and may be a 3nm CMOS-memristor-based neuromorphic core for realizing ultra-high speed / ultra-low power neural signal processing. This is intended to overcome the limitations of low resolution (70% or less) and response speed (10ms or more) of existing neuromorphic chips (e.g., Intel Loihi, IBM TrueNorth), and the control unit (132) of the present invention can dramatically improve signal processing performance between neurons by applying a ReRAM synapse-based 3D TSV architecture.

[0067] Specifically, among the sub-modules constituting the control unit (132), the 3nm CMOS-memristor neuron processor provides a spike response speed of 0.5ns, which can provide a neural signal processing speed 20 times faster than that of a conventional neuromorphic chip. In addition, the 3nm CMOS-memristor neuron processor can perform ultra-high-speed neural computations through an operating frequency of 2GHz. For example, when a 3nm CMOS-memristor neuron circuit is applied and has a spike response speed of 0.5ns and an operating frequency of 2GHz, energy consumption was 0.38pJ±0.02pJ (experiment n=1000, based on average value).

[0068] Next, among the sub-modules constituting the control unit (132), the ReRAM synapse array can adjust weights between neurons through non-volatile memory (ReRAM) with 512-step variable resistors and can implement real-time neural network learning by supporting an ultra-high-speed weight update rate of 0.8ns. For example, if EEG / EMG neural signals are processed in real-time and the EEG / EMG input signal analysis time is 48.7μs±1.2μs, the computation speed is improved by about 100 times compared to conventional neuromorphic chips.

[0069] Next, among the sub-modules constituting the control unit (132), the 3D TSV (Through-Silicon Via) based neural network interface has 3 million neurons and 10 improved compared to conventional 2D chip designs. 13 It can provide computational performance of operations per second (10 teraops). Through this, the parallel processing speed of neural signals can be maximized to enable real-time neural interpretation and pattern analysis. For example, if it is a ReRAM (Resistive RAM) synapse array with 512-level variable resistance characteristics and a weight update speed of 0.8ns, it has a learning speed that is more than 10 times faster than conventional SRAM-based neural networks.

[0070] Through this, the neuromorphic chip of the present invention achieved an average signal processing speed of 48.7μs ± 1.2μs in an experiment using 1,000 CMOS-memristor neuron circuits, and reduced power consumption to 1 / 106 of that of a conventional GPU-based BCI system (400W).

[0071] Meanwhile, the AI ​​engine (131) can be implemented by combining a CNN-LSTM-based AI algorithm with Deep Reinforcement Learning. The AI ​​engine (131) can provide an AI augmentation engine that learns connectivity between neurons in real time and optimizes cognitive and physical abilities. That is, the AI ​​engine (131) can process biosignals in real time and learn and predict connectivity between neurons by utilizing CNN-LSTM-based neural network analysis and reinforcement learning. In addition, data security can be maximized by applying ultra-high-speed neural signal communication (60Gbps) and Quantum Random Number Generation (QRNG) encryption.

[0072] At this time, the AI ​​engine (131) may include a CNN-LSTM integrated neural network analysis block, a 64-bit ALU (Arithmetic Logic Unit) signal processing unit, and a QRNG (Quantum Random Number Generator) based neural signal security engine.

[0073] First, the CNN-LSTM integrated neural network analysis block can analyze spatial patterns with a Convolutional Neural Network (CNN) and learn temporal neural signal changes with a Long Short-Term Memory (LSTM). Through this, the CNN-LSTM integrated neural network analysis block can predict connectivity between neurons with an accuracy of over 90% (neural network analysis accuracy 98.7%±0.5% (experiment n=1000)).

[0074] Next, the 64-bit ALU (Arithmetic Logic Unit) signal processing unit is 10 10 It may be a high-speed signal processing engine with computational performance of cycles per second (10 TFLOPS). It supports ultra-low power (0.5W) computation, which can save the driving power of the wearable device (220).

[0075] Next, the QRNG (Quantum Random Number Generator)-based neural signal security engine applies 256-bit quantum random number (QRNG) generation technology to perfectly protect BCI data, and security can be improved by more than 1,000 times compared to existing AES-128 encryption. For example, when performing QRNG-based neural data protection, the probability of data leakage is 10 times lower than conventional methods. -6 It was possible to see that it decreased below this level.

[0076] In other words, neuromorphic chips can be summarized as performing the following four representative functions.

[0077] First, the neuromorphic chip can perform inference processing by inputting biosignals into a real-time CNN-LSTM model. Biosignals coming from multiple sensors (210) are converted into spike signals, and operations (Conv, LSTM, Dense) of the CNN-LSTM model are performed in the neuromorphic structure, and in particular, weight updates and time pattern prediction can be performed through a ReRAM-based synapse array.

[0078] Secondly, the neuromorphic chip can perform high-speed inference of a Transfer Learning applied model. In addition, parameters of a pre-trained model can be stored on the chip (weight mapping to ReRAM synapses), and only some LSTM / Dense parameters can be updated in real time as needed (personalized fine-tuning). Accordingly, by “inserting” a user-customized Transfer Learning model onto the neuromorphic chip to perform operations, rapid application is possible even if the user (200) or the environment changes.

[0079] Thirdly, the neuromorphic chip can perform spiking-based data communication and distributed processing. The neuromorphic chip processes input signals such as brainwaves / electromyograms in the form of a Spiking Neural Network (SNN), and by performing internal neuromorphic chip computations as well as inter-chip communication in the form of spiking, it is possible to implement a real-time feedback loop using 60Gbps optical-electronic communication and SNN protocols, and transmit output values ​​to a wearable device (220) without latency.

[0080] Fourth, the neuromorphic chip can perform AI computation, neural analysis, and control integration platforms in an integrated manner. For example, AI computation can perform CNN-LSTM models, neural analysis can analyze biosignal patterns in real time, and the control integration platform can perform control signals for controlling wearable devices (220), such as exoskeletons or prosthetic arms, on a single chip.

[0081] The neuromorphic chip implemented through the technology of the present invention has performance differences as shown in [Table 1] below compared to conventional technologies such as DARPA, Synchron, and Loihi.

[0082] division Existing technology The present invention Improvement rate Signal response rate 10ms or more 0.5ns 20,000-fold improvement Cognitive enhancement rate 20% 80% or more More than 4 times Naturalness of the body 50% 95% About 2 times Power consumption tens of milliwatts 1 pJ or less 10 6배 감소

[0083] As shown in [Table 1] above, the neuromorphic chip implemented through the present invention enables rapid response to the user's (200) body based on a signal response speed 20,000 times faster than conventional chipsets, a cognitive enhancement rate 4 times faster, and physical naturalness 2 times faster, and enables operation for a long time with a single installation through low power consumption.

[0084] FIG. 5 is an operation flowchart illustrating the process of driving a wearable device by generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip according to an embodiment of the present invention.

[0085] Referring to FIG. 5, the neuromorphic computing device (100) can receive biosignals from a plurality of sensors (210) connected to a plurality of parts of the user's (200) body (S110).

[0086] At this time, the biosignal may include at least one of the brainwave signal, electromyogram signal, and photoplethysmogram signal as previously described, and the brainwave signal or electromyogram signal may be collected by sampling at 250 to 1000 Hz.

[0087] At this time, the neuromorphic computing device (100) can label the movement intention of the user (200) for each collected bio-signal. For example, when the user (200) raises their left arm, the bio-signal measured before and after raising the left arm is labeled as 'left arm raising signal,' or when the user extends their left foot forward to walk, the bio-signal measured is labeled as 'left foot extending signal to walk,' and so on. The neuromorphic computing device (100) labels the action and bio-signal measured when the user (200) performs an action.

[0088] Next, the neuromorphic computing device (100) can preprocess biosignals (S120).

[0089] Specifically, the neuromorphic computing device (100) can apply noise removal, normalization, and time-axis-based synchronization of the biosignal.

[0090] First, the neuromorphic computing device (100) can perform synchronization so that the time axes of the brainwave signal, electromyogram signal, and photoplethysmogram signal are all matched identically. This is because the brainwave signal, electromyogram signal, and photoplethysmogram signal are collected through different sensors (210), so differences may occur in their respective measurement time axes, and thus synchronization is performed so that each biosignal has the same time axis.

[0091] Next, the neuromorphic computing device (100) removes noise with a 1-50Hz bandpass filter and then removes defects with ICA / FFT, and in the case of an electromyogram signal, can remove noise included in the biosignal using a QRS detector and baseline drift technique.

[0092] Next, the neuromorphic computing device (100) can perform preprocessing by normalizing biosignals according to time series using a z-score technique, and then matching the time series length values ​​of different biosignals using a padding or slicing technique.

[0093] Next, the neuromorphic computing device (100) can perform learning using the biosignal as training data to infer the type of movement intention and the probability value of the movement intention when the biosignal is input, by using a human force augmentation model including a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory) (S130).

[0094] At this time, the process of the neuromorphic computing device (100) generating a manpower augmentation model based on biosignals will be described in detail later through FIG. 6.

[0095] Next, the neuromorphic computing device (100) can input the human biosignals collected in real time into a trained human augmentation model to produce information about movement intention as an output value (S140).

[0096] In an optional embodiment, the neuromorphic computing device (100) can convert the calculated output value into a control command for driving each wearable device (220).

[0097] Finally, the neuromorphic computing device (100) transmits the output value to a wearable device (220) worn by a person (S150), and can apply force to the person's movement so that the wearable device (220) can perform the movement that the person intends to perform (S160).

[0098] If, prior to step (S150), the neuromorphic computing device (100) does not perform conversion for the control command, as an optional embodiment, the neuromorphic computing device (100) may provide the output value to a specific wearable device (220) after determining which wearable device (220) to transmit the output value to by comparing it with the labeling of the dataset.

[0099] Additionally, in step (S160), the wearable device (220) can receive a movement intention (i.e., an output value), generate a control command corresponding to it, and operate. Accordingly, a device applied to the user's (200) body, such as a prosthetic arm or a robotic arm, is driven in accordance with the control command, thereby enabling the user (200) to apply more force to the originally intended movement, thereby enabling the realization of manpower augmentation.

[0100] FIG. 6 is a flowchart illustrating the process of constructing a manpower enhancement model according to one embodiment of the present invention.

[0101] Referring to FIG. 6, the neuromorphic computing device (100) can input biosignals into a CNN model to produce spatial feature vectors between biosignals (S210).

[0102] In this case, the input value entered into the CNN model may be a vector value of the biosignal. In this case, the vector value may consist of the number of data samples from people participating in the biosignal collection, the sampling value on the time axis (unit is Hz), and the channel number of the biosignal collection sensor. For example, if the input value is [32, 250, 8], the number of data samples is 32, meaning biosignal samples collected from 32 people, a sampling value of 250 Hz, and 8 channels of brainwave signals (or other signals).

[0103] Meanwhile, the spatial feature vector can be calculated by learning the spatial correlation defined by each input data. Accordingly, the spatial feature vector may correspond to a value calculated as data representing the correlation between each biosignal measuring sensor (210) or between channels within each biosignal measuring sensor (210).

[0104] Next, the neuromorphic computing device (100) can input a spatial feature vector into an LSTM model to learn a time pattern and produce a temporal feature vector of the biosignal (S220).

[0105] At this time, the LSTM model can perform time series analysis with one or more LSTM layers. Accordingly, the LSTM model can learn time patterns for predicting movement intention, and based on this, the LSTM model outputs a temporal feature vector.

[0106] Next, the neuromorphic computing device (100) inputs the temporal feature vector into a linear transformation model that forms a Dense Layer and can calculate a score for each body classification that is pre-set (S230).

[0107] Specifically, when movements regarding each body part, including hand movements, foot movements, and head movements, are defined as respective classification categories, the neuromorphic computing device (100) can output how many points the data output by the LSTM model has in which classification category.

[0108] Next, the neuromorphic computing device (100) can input scores by body classification into Softmax and generate a human augmentation model that infers the type of human movement intention and the probability value of the movement intention as output values ​​(S240).

[0109] For example, the neuromorphic computing device (100) can calculate the percentage of the probability that the biosignal measured from the user (200) is intended to move the hand or the percentage of the probability that it is intended to move the foot, and select the movement with the highest probability as the type of movement intention.

[0110] Next, the neuromorphic computing device (100) can evaluate the manpower augmentation model (S250).

[0111] At this time, the neuromorphic computing device (100) can evaluate the manpower augmentation model by applying at least one of the Accuracy, F1 Score, and Confusion Matrix techniques.

[0112] At this time, the neuromorphic computing device (100) can evaluate the accuracy of the human force enhancement model by comparing the output value produced when input data is formed from biosignals actually measured from a person and input into the human force enhancement model with the person's actual movement.

[0113] If the accuracy of the neuromorphic computing device (100) is lower than a threshold, it can repeat steps (S210) to (S240) to perform additional learning.

[0114] Next, the neuromorphic computing device (100) can advance the workforce augmentation model (S260). At this time, step (S260) can be performed after the previously described step (S160).

[0115] The neuromorphic computing device (100) can change the workforce augmentation model by fixing the parameter values ​​of the CNN model that constitutes the workforce augmentation model, changing the parameter values ​​of the layer that constitutes the LSTM, and then changing the number of classification categories that constitute the Dense Layer.

[0116] Accordingly, the neuromorphic computing device (100) inputs biosignals of a new user (200) into a modified workforce augmentation model as new data to perform fine-tuning, and by completing the learning, it can advance into a customized learning model personalized for each user (200). For example, even if A and B have the same movement intention of raising their left arm, their brainwave signal patterns may differ. However, if the learning model of step (S260) is transferred to a learning model, it can be customized to fit the new user (200), and fine-tuning can be performed with only a small amount of data.

[0117] One embodiment of the present invention may also be implemented in the form of a recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include all computer storage media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0118] Although the method and system of the present invention have been described in relation to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0119] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0120] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0121] 1: BCI System 100: Neuromorphic computing device 110: Communication module 120: Memory 130: Processor 140: Database 131: AI Engine 132: Control unit 200: User 210: Sensor 220: Wearable devices

Claims

Claim 1 A method for generating a force augmentation signal from biosignals using an AI algorithm and a neuromorphic chip, comprising: (a) receiving biosignals from a plurality of sensors connected to a plurality of parts of a human body; (b) preprocessing the biosignals; (c) performing training using the biosignals as training data to infer the type of movement intention and the probability value of the movement intention when the biosignals are input, using a force augmentation model including a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory); (d) inputting the human biosignals collected in real time into the trained force augmentation model to produce information regarding the movement intention as an output value; A method for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, comprising the step of (e) transmitting the output value to a wearable device worn by the person to add force to the person's movement so that the wearable device can perform the movement the person intends to move, wherein the wearable device includes a prosthetic arm or a robotic arm worn by the person. Claim 2 A method for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, wherein, in claim 1, the biosignal comprises at least one of a brainwave signal, an electromyogram signal, and a photoplethysmogram signal, and the brainwave signal or electromyogram signal is collected by sampling at 250 to 1000 Hz. Claim 3 A method for generating an attraction enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, wherein the preprocessing process applies noise removal, normalization, and time-axis-based synchronization to the biosignal, the synchronization is performed so that the time axes of the brainwave signal, electromyogram signal, and photoplethysmogram signal are all matched identically, the noise removal is performed by removing noise with a 1-50Hz bandpass filter in the case of the brainwave signal and then removing defects using ICA / FFT, and by removing noise using a QRS detector and baseline drift technique in the case of the electromyogram signal, and the normalization is performed by normalizing the biosignals according to the time series using a z-score technique and then matching the time-series length values ​​of different biosignals using padding or slicing techniques. Claim 4 A method for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, wherein the step (c) comprises: (c-1) inputting the biosignal into the CNN model to calculate a spatial feature vector between the biosignals; (c-2) inputting the spatial feature vector into the LSTM model to learn a time pattern and calculate a temporal feature vector of the biosignal; (c-3) inputting the temporal feature vector into a linear transformation model that constitutes a Dense Layer and calculating a preset score for each body classification; and (c-4) inputting the score for each body classification into Softmax and generating a force enhancement model that infers the type of movement intention of the person and the probability value of the movement intention as the output value. Claim 5 A method for generating a power enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, wherein, in claim 4, the input value input to the CNN model is a vector value of the biosignal, and the vector value is composed of the number of data samples of people participating in the biosignal collection, a sampling value in the time axis (unit is Hz), and a channel number of the biosignal collection sensor. Claim 6 In claim 5, the above step (c-3) is defined as a classification category for movements of each body part including hand movements, foot movements, and head movements, and the linear transformation model outputs how many points the data output by the LSTM model has in which classification category; and after the above step (c-4), the method further includes the step of evaluating the human force enhancement model by applying at least one of the Accuracy, F1 Score, and Confusion Matrix techniques, and evaluating the human force enhancement model by comparing the output value produced when input data is constructed from a biosignal actually measured from a person and input into the human force enhancement model with the actual movement of the person. Claim 7 A method for generating a human force enhancement signal from a bio-signal using an AI algorithm and a neuromorphic chip according to claim 6, wherein, after step (d), the human force enhancement model is modified by fixing the parameter values ​​of the CNN model constituting the human force enhancement model, changing the parameter values ​​of the layer constituting the LSTM, and varying the number of classification categories constituting the Dense Layer, and the learning is completed by inputting bio-signals for a new user into the modified human force enhancement model as new data and performing fine-tuning, thereby enhancing it into a customized learning model personalized for each user, and the neuromorphic chip includes a 3nm CMOS-memristor neuron processor capable of providing a spike response speed of 0.5ns and operating at an operating frequency of 2GHz to perform ultra-high-speed neural computation, and a non-volatile memory having a 512-step variable resistor, and a ReRAM synapse array capable of updating weights between neurons at an ultra-high speed of 0.8ns. Claim 8 A device for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip comprises: a memory storing a program for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip; and a processor that executes the program stored in the memory to generate a force enhancement signal from a biosignal using the AI ​​algorithm and the neuromorphic chip. The processor receives biosignals from a plurality of sensors connected to a plurality of parts of a human body, preprocesses the biosignals, and performs learning using the biosignals as training data to infer the type of movement intention and the probability value of the movement intention when the biosignals are input, using a force enhancement model including a CNN (Convolutional Neural Network) and an LSTM (Long Short-Term Memory). The processor inputs the biosignals collected in real time into the trained force enhancement model to calculate information regarding the movement intention as an output value, and transmits the output value to a wearable device worn by the human, thereby applying force to the human's movement so that the wearable device can perform the movement intended by the human. A device for generating a force enhancement signal from a biosignal using an AI algorithm and a neuromorphic chip, wherein the wearable device includes a prosthetic arm or a robotic arm worn by a person.

Images