Method, device and program for reconstructing handmovement trajectories based on biometric signal

Abstract

A method for restoring the trajectory of a hand movement based on a biosignal according to one embodiment may include: a step of collecting electroencephalogram data and electromyogram data related to a user's hand movement in a first data collection unit; a step of collecting acceleration data and angular velocity data related to the user's hand movement in a second data collection unit; a step of performing preprocessing on each of the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data in a first preprocessing unit; a step of extracting proprioceptive data based on the preprocessed electromyogram data, acceleration data, and angular velocity data in a proprioceptive data extraction unit; and a step of training the trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data in a data learning unit.

Description

Technology Field

[0001] The present invention relates to a method, apparatus, and program for restoring the trajectory of hand movements based on biosignals. More specifically, the invention relates to a method, apparatus, and program that enables various interface interactions, such as communication and drawing, by providing a trajectory or text of hand movements or characters intended by a user based on biosignals when there is a hand movement, such as handwriting. Background Technology

[0002] Biosignals include electroencephalography (EEG) and electromyography (EMG).

[0003] Electroencephalography (EEG) is a non-invasive technology that measures the electrical activity of the brain. For example, it captures brain waves through electrodes attached to the skull and displays them as a graph of voltage changes over time. EEG is widely used to diagnose various brain-related diseases and for research, playing a significant role in neuroscience, psychology, and medical studies. Fundamentally, EEG measures signals originating from various regions of the brain. These signals are primarily attributed to the electrical activity of neurons and can be analyzed because their characteristics vary by region, time, and frequency. EEG is a critical biosignal, particularly because it carries fundamental information about hand movements and incorporates the user's inherent intentions. Compared to other representative biosignals such as fMRI, EEG offers faster temporal resolution, allowing for the detection of brain activity at minute time scales; however, it suffers from a high signal-to-noise ratio due to various obstructions between the electrodes, such as the scalp, skull, and hair. Additionally, its relatively low spatial resolution often makes it difficult to pinpoint exactly where in the brain a signal originated. Consequently, the quality of technology utilizing EEG alone is influenced by appropriate noise reduction and feature extraction techniques.

[0004] Electromyography (EMG) is a technology that measures and analyzes the electrical activity of muscles. This technology captures electromyography through electrodes attached to the muscles, enabling the identification of fine muscle movements and intentions. Used to monitor muscle contraction and relaxation, this technology is widely applied in fields such as medicine, rehabilitation, and sports science. EMG signals are based on the electrical activity of muscle fibers. These signals originate from the action potential generated in motor nerve fibers when muscles operate, allowing for the quantitative measurement of muscle strength, endurance, and fatigue. The temporal and spatial resolution of EMG is generally higher than that of electroencephalography (EEG). The signal-to-noise ratio of EMG varies depending on the quality of contact between the muscle and the electrode, external noise, and signal processing methods; therefore, noise removal and appropriate feature extraction may be required.

[0005] Proprioception refers to the sense of perceiving the body's position and movement, and it is obtained through proprioceptors distributed in muscles, tendons, joints, and skin. Proprioception informs us how our body parts (e.g., hand joints) are positioned in space and how they are moving.

[0006] Deep learning is a field of machine learning that focuses on data modeling using artificial neural networks and can solve difficult problems through the complex interaction of multiple layers of neural networks. It demonstrates high performance in unstructured data such as biological signals or in high dimensions and is utilized in various application fields such as image recognition, natural language processing, and speech recognition.

[0007] Artificial intelligence models in the field of deep learning can have the structure of encoders and decoders. An encoder can form a 'latent representation' by non-linearly extracting complex features from input data. This latent representation is a concise expression of the intrinsic properties or structure of the data, transforming the original data into a simpler and easier-to-process form. A decoder can utilize this latent representation to generate or predict desired results. Decoders can perform tasks such as classification, regression, image reconstruction, text translation, and speech recognition. The term 'latent space' refers to an abstract space where all possible latent representations exist, in which various characteristics of the data are mathematically expressed. In various embodiments of this document, a latent representation may also be referred to as a 'latent vector' or 'embedding'.

[0008] Taking image reconstruction as an example, an image encoder analyzes a given image and converts important information into a compressed form, namely a latent representation. For instance, when an encoder processes an image of a cat, it recognizes characteristics such as the cat's size, position, breed, facial expression, and contours, and compresses them into a concise data form. This latent representation contains the core characteristics of the cat's image. An image decoder can utilize latent representations to reconstruct a new image similar to the original, generate different poses or expressions of the cat, or classify the image. The latent space holds the 'core elements' of the data, and small changes within the latent space can result in significant changes to the image's characteristics. For instance, moving in a specific direction within the latent space can alter the cat's facial expression, while moving in a different direction can result in changes to the color or background. As such, latent representations help effectively process complex data like images and enable various applications, such as creative image generation, efficient image compression, and sophisticated image editing.

[0009] The encoder is automatically trained using latent representations to minimize the difference (loss value) between the output of the decoder and the actual data. Through this process, the latent space is automatically optimized according to the model's structure and objective function, allowing it to be constructed into a more accurate and efficient latent space. The latent space can be formed through automatic learning or artificially adjusted based on human data modeling results. For example, latent representations can be adjusted to emphasize or ignore specific data characteristics, or specific patterns or characteristics can be reinforced through a separate data regeneration model. The adjustment of latent representations within the latent space can be achieved through various technical methods. For instance, mechanisms such as 'concatenation,' 'sum,' or 'attention' can be used to combine or reinforce different latent representations.

[0010] The encoder / decoder structure is a concept generally applicable to most artificial intelligence models, providing a foundation that helps models effectively process complex data and enabling the derivation of more diverse and creative results through the flexible manipulation of the latent space.

[0011] A Brain-Computer Interface (BCI) is a technology capable of directly transmitting a user's intentions or thoughts to a computer system. By connecting the brain to a computer, it recognizes user intent and enables the performance of various control tasks through imagination alone. Active research on BCI technology is taking place in diverse fields, particularly in medicine, gaming, and virtual reality. It is especially utilized to provide functions such as message delivery, environmental control, and speech synthesis to individuals with limitations in movement or communication.

[0012] In existing BCI-based user interface control methods, command input using brainwaves was common. For example, by utilizing methods such as Steady-State Visual Evoked Potentials (SSVEP), brainwaves responding to visual stimuli of specific frequencies were analyzed, enabling cursor movement, character selection, and the execution of simple commands. This method, which allows for the simplified execution of complex control commands, is also used as an assistive tool for patients who have difficulty communicating.

[0013] Camera-based tracking, motion capture using various sensors, and touch screens are being utilized for tracking human movement, such as handwriting or hand tracking. Camera-based tracking captures and analyzes human movement in real time using external or built-in cameras. Motion capture measures human movement using sensors. Prior art literature

[0014] Republic of Korea Published Patent Application No. 10-2021-0059079 Republic of Korea Published Patent Application No. 10-2020-0110998 Republic of Korea Published Patent Application No. 10-2023-0086936 Republic of Korea Published Patent Application No. 10-2013-0141904 The problem to be solved

[0015] Existing user interface devices that track hand movements or handwriting are equipped with cameras, sensors, and / or touch equipment; however, they face challenges such as spatial constraints arising from installing these external devices at a certain distance from the user, sensor blind spots, and limitations in capturing fine movements. Furthermore, the reliance on external stimuli in existing BCI-based communication devices often makes it difficult to execute complex commands, as it requires a high level of concentration and training. Tracking movement solely using electroencephalograms (EEGs) among biosignals is also difficult to achieve in terms of diverse user interactions, as the high signal-to-noise ratio of EEGs limits the decoding of only simple controls. Consequently, there is a growing need for methods and devices that can overcome these limitations and be used more flexibly and conveniently by the user in various environments and situations.

[0016] The present invention utilizes only the user's biosignals for movement tracking, thereby eliminating spatial constraints and blind spots caused by the use of a camera. Furthermore, by applying an electroencephalogram (EEG) containing the user's intrinsic intention, it is possible to output hand movements, such as handwriting intended (or imagined) by the user, even without actually moving the hand. Additionally, the present invention provides a user-interface device that can be controlled in various ways purely by the user by utilizing a deep learning model trained by synthesizing various biosignals. In other words, by decoding the user's intrinsic biosignals without external stimuli, the present invention ensures user convenience while simultaneously enabling the restoration of complex hand movements, such as handwriting trajectories, rather than simple classification.

[0017] Meanwhile, the technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0018] While electroencephalography (EEG) provides high temporal resolution, its low signal-to-noise ratio makes it difficult to track direct hand movements. To complement EEG, proprioceptive data is utilized to infer not only more accurate movement paths but also imagined hand movements. This allows for a more accurate identification of the user's actual or intended movements, thereby making user-interface interactions more sophisticated. Proprioceptive data can be obtained by measuring and modeling the acceleration, angular velocity, and electromyography (EMG) of the hand joints. In this invention, such modeling can be performed by a deep learning regenerative-based model. Acceleration and angular velocity sensors monitor joint movements in real time, enabling precise tracking of the hand's position and movement path. Since EMG measures muscle contraction and relaxation through electrical activity, it can detect even minute hand movements. In particular, proprioceptive information can serve as an important intermediate step in inferring a clear movement trajectory from noisy electroencephalography (EEG).

[0019] The present invention is designed to train a model based on actual hand movement data, wherein the encoder extracts proprioceptive latent information from the EEG, and the decoder generates a movement path based on this proprioceptive information. This method enables robust decoding even for imagined hand movements. It has been scientifically proven that brain activity patterns in imagined movements appear similar to those in actual movements. Since the brain processes imagined movements identically to actual movements to activate similar neural pathways, information such as proprioception can be embedded in the electroencephalogram (EEG) even in these imagined movements. Consequently, a composite model based on proprioceptive information and the EEG can operate effectively while maintaining high accuracy and reliability under various environmental factors. This approach is advantageous for processing noisy EEG signals and predicting imagined movement paths. In conclusion, the utilization of such proprioceptive information serves as a key background technology for the present invention and can contribute to enhancing the accuracy and efficiency of user-interface interactions based on various biosignals.

[0020] A method for restoring the trajectory of a hand movement based on a biosignal according to one embodiment may include: a step of collecting electroencephalogram data and electromyogram data related to a user's hand movement in a first data collection unit; a step of collecting acceleration data and angular velocity data related to the user's hand movement in a second data collection unit; a step of performing preprocessing on each of the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data in a first preprocessing unit; a step of extracting proprioceptive data based on the preprocessed electromyogram data, acceleration data, and angular velocity data in a proprioceptive data extraction unit; and a step of training the trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data in a data learning unit.

[0021] The step of learning the trajectory data of the user's hand movement may include: a step of extracting a proprioceptive latent expression used for data regeneration of electromyography data, angular velocity data, and acceleration data from the extracted proprioceptive data; a step of extracting a first free latent expression used for trajectory tracking and a first co-latent expression used for proprioceptive tracking from the preprocessed electroencephalography data; and a step of determining a predicted trajectory based on the extracted proprioceptive latent expression, the first free latent expression, and the first co-latent expression. The data learning unit may determine the predicted trajectory by using a loss function that integrates weights assigned to each of the following: a regeneration loss value representing the similarity between the data regenerated by the data regeneration and the actual measurement data; a co-latent expression loss value representing the similarity between the first co-latent expression and the proprioceptive latent expression; and a trajectory loss value representing the similarity between the predicted trajectory and the actual trajectory. Each of the weights may vary over time.

[0022] The preprocessing of the collected electroencephalogram (EEG) data and electromyogram (EMG) data includes filtering by a band-block filter, filtering by a band-pass filter, independent component analysis to remove unnecessary signals, and filtering by a low-frequency band-pass filter, and the preprocessing of the angular velocity data and acceleration data may include filtering by a correction filter.

[0023] The above method may further include: a step of collecting electroencephalogram (EEG) data of the subject related to the imagination of the subject's hand movements in a third data collection unit after the learning of the data learning unit is performed; a step of performing preprocessing on the collected electroencephalogram (EEG) data of the subject in a second preprocessing unit; and a step of inferring trajectory data of the subject's hand movements through the artificial neural network based on the preprocessed electroencephalogram (EEG) data of the subject in a data inference unit. The step of inferring trajectory data of the subject's hand movements may further include: a step of extracting a second free latent expression used for trajectory tracking and a second co-latent expression used for proprioceptive tracking based on the preprocessed electroencephalogram (EEG) data input through the artificial neural network in an electroencephalogram encoder; and a step of inferring the trajectory of the subject's hand movements based on the second free latent expression and the second co-latent expression in a trajectory decoder. In the trajectory data processing unit, control can be made to transmit or display the trajectory data of the inferred target's hand movement. Effects of the invention

[0024] The present invention can provide a device and method capable of user-interface interaction by identifying the intrinsic intention of a target user even in the presence of minute movements or no movement.

[0025] This invention does not merely classify the movements of a target user, but rather restores the user's intended hand movements from biosignals. By inferring the user's intended handwriting / hand movements without spatial constraints or sensor blind spots, it enables the control of complex user interfaces such as writing, drawing, and cursor movement. This allows individuals with physical disabilities who have difficulty using their arms, or those in situations where using their hands is challenging—such as rush-hour subway rides, driving, or injuries—to conveniently interact with mobile phones or other mobile electronic devices solely through imagination to perform tasks like writing, drawing, and manipulation.

[0026] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other unmentioned technical effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0027] FIG. 1 is a configuration diagram of a device for restoring the trajectory of a hand movement according to one embodiment. FIG. 2 is a flowchart of the operation performed by a device that restores the trajectory of a hand movement according to one embodiment. FIG. 3 shows a configuration diagram of a process for learning and inferring user movements using biosignals according to one embodiment. Figure 4 shows the process of a data preprocessing module according to one embodiment. FIG. 5 illustrates the process of a learning module according to one embodiment. FIG. 6 illustrates the process of an inference module according to one embodiment. Specific details for implementing the invention

[0028] Detailed information regarding the purpose, technical configuration, and resulting effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. An embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0029] The embodiments disclosed herein should not be interpreted or used to limit the scope of the invention. It is obvious to those skilled in the art that the description including the embodiments herein has various applications. Accordingly, any embodiments described in the detailed description of the invention are illustrative for better explaining the invention and are not intended to limit the scope of the invention to the embodiments.

[0030] The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the detailed description. Additionally, while one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0031] Furthermore, the expression that it includes certain components is an “open-ended” expression that merely refers to the existence of such components and should not be understood as excluding additional components.

[0032] Furthermore, when it is stated that one component is “connected” or “joined” to another component, it should be understood that while it may be directly connected or joined to that other component, there may also be other components present in between.

[0033] Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0034] In various embodiments of this document, the term "user" may be used interchangeably with "subject." In various embodiments of this document, the term "target" may be used interchangeably with "target user" or "target subject." In various embodiments of this document, the user and the target may be distinct or at least partially overlap.

[0035] In the various embodiments of this document, hand movements are exemplified, but are not limited thereto, and the present invention may also be applied to movements of other body parts of a user or object.

[0036] FIG. 1 is a configuration diagram of a device (100) for restoring the trajectory of a hand movement according to one embodiment (hereinafter referred to as 'device (100)').

[0037] Referring to FIG. 1, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0038] The memory (110) can store data obtained from an external device or data generated by itself. The memory (110) can store instructions that can perform operations of the processor (120). For example, the memory (110) can store collected data, preprocessed data, extracted data, artificial neural networks, etc.

[0039] The processor (120) is a computational device that controls the overall operation. The processor (120) can execute instructions stored in memory (110). The operation of the device (100) according to the embodiment of the present document can be understood as an operation performed by the processor (120).

[0040] The input / output interface (130) may include a hardware interface or a software interface for inputting or outputting information.

[0041] The communication interface (140) enables the transmission and reception of information through a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0042] The device (100) can be implemented in various forms of devices capable of performing calculations through a processor (120) and transmitting and receiving information through a network. For example, it can be implemented in the form of a server, a computer device, a portable communication device, a smartphone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0043] The components of the device (100) may be divided in a manner different from that of FIG. 1. For example, the device (100) may be defined as including at least some of a first data collection unit, a second data collection unit, a first preprocessing unit, a proprioceptive data extraction unit, a data learning unit, a third data collection unit, a second preprocessing unit, a data inference unit, and a trajectory data processing unit.

[0044] FIG. 2 is a flowchart of an operation performed by a device (100) according to one embodiment. The operation of the device (100) according to the embodiment of FIG. 2 can be understood as an operation performed by a processor (120).

[0045] Each step disclosed in FIG. 2 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and any one step may be included in another step. The order of each operation disclosed in FIG. 2 is arranged only for ease of understanding and is not limited to a chronological order, and the order may be changed and operated differently according to the designer's choice.

[0046] In step 205, the first data collection unit can collect electroencephalogram data and electromyogram data related to the user's hand movements. The collected data can be stored in a predetermined database.

[0047] In step 210, acceleration data and angular velocity data related to the user's hand movements can be collected in the second data collection unit. The collected data can be stored in a predetermined database. Since it is difficult to use a large amount of data when using biosignals that vary significantly from person to person, auxiliary data such as acceleration data and angular velocity data can be utilized to improve consistency in deep learning even with a small amount of data.

[0048] In step 215, the first preprocessing unit may perform preprocessing on each of the collected electroencephalogram (EEG) data, electromyogram (EMG) data, acceleration data, and angular velocity data. The preprocessing of the collected EEG data and EMG data may include filtering by a band-stop filter, filtering by a band-pass filter, independent component analysis to remove unnecessary signals, and filtering by a low-frequency band-pass filter. The preprocessing of the angular velocity data and acceleration data may include filtering by a correction filter.

[0049] In step 220, the proprioceptive data extraction unit can extract proprioceptive data based on the preprocessed electromyography data, acceleration data, and angular velocity data.

[0050] In step 225, in the data learning unit, the trajectory data of the user's hand movement can be trained through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data. Training the trajectory data of the user's hand movement may include extracting a proprioceptive latent expression used for data regeneration of electromyogram data, angular velocity data, and acceleration data from the extracted proprioceptive data, extracting a first free latent expression used for trajectory tracking and a first co-latent expression used for proprioceptive tracking from the preprocessed electroencephalogram data, and determining a predicted trajectory based on the extracted proprioceptive latent expression, the first free latent expression, and the first co-latent expression.

[0051] The data learning unit above can determine the predicted trajectory by using a loss function that integrates weights assigned to each of the following: a regeneration loss value representing the similarity between the data regenerated by the data regeneration and the actual measurement data; a joint latent expression loss value representing the similarity between the first joint latent expression and the proprioceptive latent expression; and a trajectory loss value representing the similarity between the predicted trajectory and the actual trajectory. Each of the weights can be configured to vary over time.

[0052] In step 230, the third data collection unit may collect electroencephalogram data of the subject related to the subject's hand movement imagination. The subject may be separate from the user, or at least partially overlap.

[0053] In step 235, the second preprocessing unit may perform preprocessing on the collected electroencephalogram (EEG) data of the subject. The preprocessing on the collected electroencephalogram (EEG) data of the subject may include filtering by a band-block filter, filtering by a band-pass filter, independent component analysis to remove unnecessary signals, and filtering by a low-frequency band-pass filter.

[0054] In step 240, in the data inference unit, trajectory data of the target's hand movement can be inferred through the artificial neural network based on the preprocessed electroencephalogram data of the target. Inferring trajectory data of the target's hand movement may include, in the electroencephalogram encoder, extracting a second free latent expression used for trajectory tracking and a second co-latent expression used for proprioceptive tracking based on the preprocessed electroencephalogram data input through the artificial neural network, and in the trajectory decoder, inferring the trajectory of the target's hand movement based on the second free latent expression and the second co-latent expression.

[0055] In step 245, the trajectory data processing unit may control the transmission or display of trajectory data of the inferred target's hand movement. A device that receives the transmitted trajectory data may store or display the trajectory data.

[0056] The process of human movement involves the following simplified series of steps: (1) data collection from sensory organs, (2) analysis of sensory data and establishment of a movement plan in the brain, (3) transmission of action commands to muscle nerves, and (4) performance of the actual action through muscle contraction and relaxation. Since the brainwaves generated during this movement process are noisy and difficult to decode, a deep learning model must be able to efficiently simulate the above process in order to reconstruct fine movements, such as high-dimensional handwriting, based on brainwaves. To this end, the present invention collects 'proprioception' data, which is directly related to hand movements and is self-evidently processed in the brain, and induces the model to learn the latent expressions of this proprioception to improve efficiency. Proprioception includes joint positions, muscle tension, speed and direction of movement, etc., and based on this information, the brain establishes and executes an accurate movement plan. When a deep learning model simulates such human movement processes, utilizing proprioception data can reduce the complexity of brainwaves and build a model robust against noise. Specifically, proprioceptive data is used as auxiliary information to aid in the interpretation of brainwave signals, thereby enabling the model to serve as a link between noisy brainwaves and the precise trajectories of hand movements. This approach can significantly improve the accuracy of brainwave-based motion decoding.

[0057] According to various embodiments of the present invention, data such as the user's electroencephalogram, electromyogram, joint-specific angular velocity, and acceleration are collected, and an encoder / decoder model is trained to incorporate latent expressions that consider proprioceptive information, so that in actual use, operation is possible using only the electroencephalogram, and the corresponding hand movement can be restored simply by imagining the hand movement.

[0058] According to various embodiments of the present invention, an electroencephalogram encoder may correspond to a neural encoder and may include a combination of deep learning layers such as a fully connected layer, a convolutional neural network layer, a long short-term memory, a recurrent neural network, and a transformer, and may non-linearly extract features from a biological signal to create a useful embedding.

[0059] According to various embodiments of the present invention, the trajectory decoder can infer a trajectory from the embeddings of a neural encoder, and can infer an actual trajectory on a useful composite embedding composed of a free latent representation and an proprioceptive latent representation generated from a neural encoder, by complexly including deep learning layers such as a fully connected layer, a convolutional neural network layer, a long short-term memory, a recurrent neural network, and a transformer.

[0060] FIG. 3 shows a configuration diagram of a process for learning and inferring user movements using biosignals according to one embodiment.

[0061] The biosignal measurement and transmission module (310) collects biosignals (electroencephalogram and electromyogram) during the user's intended hand movements (including both actual and imagined movements). The data preprocessing module (320) performs noise removal on the collected biosignals through filtering such as independent component analysis, band cut-off filters, and / or band filters. The neural processing unit (330) infers the user's intended hand movement trajectory from the preprocessed biosignals, and this can be output to a display on the user interface (340). The neural processing unit (330) may be equipped with a learning module to maintain continuous performance customized to the user and an inference module to infer the user's intention in real time. In various embodiments of the present invention, the neural processing unit (330) performs learning considering proprioception by using electromyogram, acceleration, and / or angular velocity signals as auxiliary information along with electroencephalogram during learning, and may use only electroencephalogram among the biosignals during inference.

[0062] In various embodiments of the present invention, the process of restoring the trajectory of hand movements can be divided into two modes: a learning mode and an inference mode. The learning mode is a process of training a deep neural network through a learning module based on the user's arm movements according to a predetermined learning scenario and the bio-signals therefrom, and updating the inference module by fine-tuning the deep neural network through transfer learning to maintain consistent performance in the inference mode. The inference mode, which may also be called an operation mode, is a process of continuously outputting the user's intentional arm movements through the inference module using the deep neural network model obtained through the learning mode.

[0063] The biosignal measurement and transmission module (310) can be divided into a biosignal collection module and a transmission module. The biosignal collection module is a module that performs biosignal collection by recording the voltage difference of an electrode sensor. The biosignal collection module is composed of a device using non-invasive electrodes, for example, attached to the scalp and arm to collect signals (electromyography and electroencephalography) at a sampling frequency (e.g., 1000 Hz). In various embodiments of the present invention, acceleration signals and angular velocity signals based on an IMU (Inertial Measurement Unit) sensor attached to a part of the user's or subject's body (e.g., a joint) can be additionally collected. The (voltage) signals collected by the biosignal collection module can be input to a data preprocessing module through the transmission module.

[0064] Figure 4 shows the process of a data preprocessing module according to one embodiment.

[0065] The data preprocessing module (320) is a module that performs noise removal by preprocessing the biosignal (EMG / EEG) transmitted from the biosignal collection module. In the example of FIG. 4, the preprocessing of the EMG and EEG largely involves three frequency filtering steps and one dimensionality reduction step. Specifically, first, noise signals corresponding to a predetermined frequency (e.g., 60Hz) coming from the transmission line of the surrounding environment and its harmonics (e.g., 120Hz, 180Hz, etc.) are removed through a band-stop filter. Then, white noise and other high-frequency noise are removed through a band-pass filter (1-45Hz). Independent component analysis is performed with the noise removed primarily through the band-pass filter to remove noise signals generated by other movements, such as eye blinking. Then, a biosignal frequency band optimized for hand movements is extracted through a band-pass filter (1-8Hz).

[0066] Although not shown in Fig. 4, electromyogram and electroencephalogram signals can be further optimized for learning or inference by undergoing additional preprocessing such as baseline correction, windowing, and normalization.

[0067] The acceleration and angular velocity signals output from the IMU sensor can be used as proprioceptive data by passing through correction filters, such as Kalman filters or complementary filters, and then integrating them with preprocessed electromyography.

[0068] The neural processing unit (330) includes a learning module and an inference module, and may additionally be configured with a database and memory for an intermediate process to update the inference module using the learning results and output the inference results and device feedback. In the learning module, the deep neural network described later is fine-tuned using pre-processed biosignals and trajectory data. The deep neural network, after fine-tuning is completed, is stored in the database and transmitted to the inference module via memory. The inference module is always running, and the learning module can be turned on / off as needed by the user. While the learning module is turned on, the user performs hand movements according to a predetermined scenario, and the inference model can also be updated in real time.

[0069] FIG. 5 illustrates the process of a learning module according to one embodiment.

[0070] The learning module may consist of a proprioceptive regeneration unit (a regeneration-based model for obtaining latent representations of proprioception) and a main trajectory inference unit (a model that predicts trajectories from EEG by considering proprioceptive information). The proprioceptive regeneration unit is a regeneration-based model designed to regenerate the user's proprioceptive information and aims to create optimal latent representations for proprioceptive data. The main trajectory inference unit aims to infer (predict) hand movement trajectories from the EEG by considering proprioceptive information. The main trajectory inference unit may consist of an EEG encoder and a trajectory decoder. The EEG encoder can be trained to extract features by considering both latent representations of proprioception and latent representations containing features important for trajectory inference from the EEG. Some of the latent representations generated by the EEG encoder are induced to consider latent representations of proprioception, while the remainder are left as free latent representations. The trajectory decoder can infer the actual trajectory from a latent representation for proprioception and a free latent representation for trajectory inference.

[0071] The proprioceptive reconstruction unit inputs electromyography, angular velocity, and acceleration data into a model to compress and decompress the data, thereby creating a proprioceptive latent representation that generates data most closely matching the original data. The loss value measuring how similar the reconstructed data (sensor signal) is to the measured data (actual sensor signal) is called the 'reconstruction loss'. Reconstruction loss value The lower this value, the more effectively the latent expressions within the proprioceptive regeneration unit are considered to extract meaningful features from actual data, and the derived latent expressions enable the EEG encoder to learn with sophisticated and meaningful information. In other words, the regeneration loss value can induce the proprioceptive regeneration model to extract latent expressions of proprioception useful for trajectory generation. The criteria for evaluating the regeneration loss value may include, for example, at least one of the Mean Squared Error (MSE), Mean Absolute Percentage Error (MAP), Cosine Similarity, and Dynamic Time Warping.

[0072] The main trajectory inference unit consists of an EEG encoder and a trajectory decoder. The EEG encoder performs a conversion from EEG to a latent representation, and the trajectory decoder predicts the trajectory from the latent representation. The latent representation, which is the output of the EEG encoder, is divided into a free latent representation and a joint latent representation that is induced to be similar to the proprioceptive latent representation. The joint latent representation calculates the similarity with the proprioceptive latent representation extracted by the proprioceptive regeneration unit to obtain the 'joint latent loss'. It is defined as follows. That is, the neural encoder can be induced to extract the optimal latent representation of the proprioceptive from the EEG through the co-latent representation loss value. The criterion for evaluating the co-latent representation loss value may use at least one of, for example, Mean Squared Error (MSE), Mean Absolute Percentage Error (MAP), Cosine Similarity, and Dynamic Time Warping. By setting the model's cost function in a way that reduces this co-latent loss value, the EEG encoder is induced to consider the latent representation of the proprioceptive.

[0073] In this invention, a latent space alignment technique is employed to adjust the latent space. This technique primarily uses the similarity between the source latent representation and the target latent representation as a loss function to induce the encoder to generate a target latent representation similar to the source latent representation. These techniques play a crucial role in minimizing model loss by effectively adjusting the latent representation. In this process, the model emphasizes only the meaningful characteristics of the data, enabling the AI ​​model to generate more precise and customized outputs. This increases the utility of the model in various application fields.

[0074] In other words, the present invention follows a basic encoder-decoder structure that takes an electroencephalogram as input and a hand movement trajectory as output, but utilizes a Proprioceptive and Free Latent Space Optimization (PFLSO) method that derives part of the encoder's output latent representation as a free latent representation and part as a latent representation considering proprioception, thereby enabling stable trajectory inference even from complex EEG signals generated during hand movement tasks such as handwriting. That is to say, the PFLSO method helps the model effectively reflect proprioceptive information from the EEG signal to predict the optimal movement trajectory.

[0075] In the main trajectory inference unit, the latent vector, which is the output of the EEG encoder, enters the trajectory decoder as input, and the trajectory is finally output. The similarity between the predicted trajectory and the actual trajectory is calculated to obtain the 'trajectory loss'. It is defined as follows. By using the trajectory loss value, the neural encoder's free latent representation can be guided to extract the optimal latent representation from the EEG. The criterion for evaluating the trajectory loss value may use at least one of, for example, Mean Squared Error (MSE), Mean Absolute Percentage Error (MAP), Cosine Similarity, or Dynamic Time Warping. By determining the model cost function in a way that reduces this trajectory loss value, the entire module can be guided to properly consider the trajectory information.

[0076] The proprioceptive regeneration unit and the main trajectory inference unit have a combined weighted sum of three loss values—namely, the regeneration loss value, the joint latent loss value, and the trajectory loss value—as shown in Equation 1, and the entire module can be trained simultaneously using an appropriate optimization algorithm with this combined loss value. This is called an end-to-end learning method.

[0077]

[0078] The optimal values ​​for each of the weights of the three loss values ​​can be determined, for example, by a grid-search method. Additionally, each component of the module can be trained differentially over time by refining the training stages through flexibly changing weights according to the level of training. For example, in the early stages of training, the weight of the regeneration loss value can be set high and the others low to intensively train the proprioceptive regeneration unit, and from the middle of training onwards, the weight of the joint latent loss value can be set high to intensively train the main trajectory inference unit.

[0079] FIG. 6 illustrates the process of an inference module according to one embodiment.

[0080] The inference module is a module that infers a trajectory by taking EEG as input, based on a trained neural encoder and trajectory decoder. The trained main trajectory inference unit may include an EEG encoder optimized through the aforementioned PFLSO method; this EEG encoder receives only EEG signals as input and can consistently infer (predict) hand movement trajectories considering proprioceptive information. The trained EEG encoder receives noisy EEG as input and extracts (infers) the optimal co-latent representation (proprioceptive latent space) which has relatively less noise and is more similar to the trajectory data, as well as the optimal free latent representation (free latent space) for trajectory generation.

[0081] In the inference module, the joint latent representation is induced to contain information similar to the proprioceptive latent representation learned in the proprioceptive regeneration unit. This joint latent representation fully reflects the user's proprioceptive information, ensuring consistent performance in both actual and imagined movements. The free latent representation is the part left for the model to freely learn additional information necessary for trajectory prediction, helping the trajectory decoder to optimally predict the trajectory of the hand movement.

[0082] In Fig. 6, all latent vectors produced as outputs of the EEG encoder are input into the trajectory decoder. Based on these latent vectors, the trajectory decoder can predict the trajectory of the hand, such as X-coordinates, Y-coordinates, pressure (pen pressure), and speed (force). In this process, the trajectory decoder can derive optimal prediction results for the trajectory by reflecting proprioceptive information through co-latent representations and utilizing additional information necessary for trajectory prediction through free latent representations.

[0083] Due to consistent latent expression learning using proprioceptive data during the learning process, trajectory prediction can be performed consistently not only for actual movements but also for movements imagined by the user. If the PFLSO method is not used and latent expressions rely solely on the free latent space, and trajectories are inferred from EEG using only a simple encoder-decoder structure without deriving latent expressions through a separate regenerative model, the model will rely only on noisy EEG signals, resulting in reduced robustness. If the model is trained without considering proprioceptive information, it becomes difficult to maintain consistent performance in various environments. This leads to unstable results in both actual hand movements and imagined hand movements, which can ultimately significantly degrade usability and utility. According to various embodiments of the present invention, trajectories can be inferred consistently in both movements imagined by the user and movements actually performed through a composite model of EEG and proprioception.

[0084] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more items unless the relevant context clearly indicates otherwise.

[0085] In this document, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include all possible combinations of items listed together in the corresponding phrase. Terms such as “1,” “2,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that the component may be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0086] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of a component or part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0087] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) that can be read by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0088] Additionally, the processor of the embodiments of this document may call at least one instruction among one or more instructions stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to at least one called instruction. Such one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0089] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0090] Methods according to the various embodiments disclosed in this document may be provided as part of a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as a manufacturer's server, an application store's server, or the server's memory.

[0091] According to various embodiments, each component of the described components (e.g., module, program, user, target, etc.) may include a singular or multiple entities. According to various embodiments, one or more of the aforementioned components or operations may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as they were performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically; one or more of the operations may be executed in a different order; may be omitted; or one or more other operations may be added.

Claims

Claim 1 A method for reconstructing the trajectory of hand movement based on biosignals comprises: a step of collecting electroencephalogram (EEG) data and electromyogram (EMG) data related to a user's hand movement in a first data collection unit; a step of collecting acceleration data and angular velocity data related to the user's hand movement in a second data collection unit; a step of performing preprocessing on each of the collected EEG data, EMG data, acceleration data, and angular velocity data in a first preprocessing execution unit; a step of extracting proprioceptive data based on the preprocessed EMG data, acceleration data, and angular velocity data in a proprioceptive data extraction unit; and a step of training the trajectory data of the user's hand movement through an artificial neural network based on the preprocessed EEG data and the extracted proprioceptive data in a data learning unit; wherein the step of training the trajectory data of the user's hand movement comprises: a step of extracting a proprioceptive latent representation used for data regeneration of the EMG data, angular velocity data, and acceleration data from the extracted proprioceptive data; The method comprises: a step of extracting a first free latent expression used for trajectory tracking and a first co-latent expression used for proprioceptive tracking from the preprocessed electroencephalogram data; and a step of determining a predicted trajectory based on the extracted proprioceptive latent expression, the first free latent expression, and the first co-latent expression; wherein, after the learning of the data learning unit is performed, a third data collection unit collects electroencephalogram data of the subject related to the imagination of the subject's hand movements; a second preprocessing unit performs preprocessing on the collected electroencephalogram data of the subject; and a data inference unit infers trajectory data of the subject's hand movements through the artificial neural network based on the preprocessed electroencephalogram data of the subject.A method for restoring the trajectory of a hand movement, comprising: a step of inferring trajectory data of the hand movement of the subject, wherein, in an electroencephalogram encoder, a second free latent expression used for trajectory tracking and a second co-latent expression used for proprioceptive tracking are extracted based on the preprocessed electroencephalogram data input through the artificial neural network; and a step of inferring the trajectory of the hand movement of the subject based on the second free latent expression and the second co-latent expression in a trajectory decoder. Claim 2 delete Claim 3 A method for restoring the trajectory of a hand movement according to claim 1, wherein the data learning unit determines the predicted trajectory by using a loss function that integrates weighted values ​​for each of the following: a regeneration loss value representing the similarity between the data regenerated by the data regeneration and the actual measurement data; a joint latent expression loss value representing the similarity between the first joint latent expression and the proprioceptive latent expression; and a trajectory loss value representing the similarity between the predicted trajectory and the actual trajectory. Claim 4 A method for restoring the trajectory of a hand movement according to claim 3, wherein each weight can vary over time. Claim 5 A method for restoring the trajectory of a hand movement according to claim 1, wherein the preprocessing of the collected electroencephalogram data and electromyogram data includes filtering by a band-block filter, filtering by a band-pass filter, independent component analysis to remove unnecessary signals, and filtering by a low-frequency band-pass filter, and the preprocessing of the angular velocity data and acceleration data includes filtering by a correction filter. Claim 6 delete Claim 7 delete Claim 8 A method for restoring the trajectory of a hand movement according to claim 1, further comprising the step of controlling the trajectory data processing unit to transmit or display the trajectory data of the inferred hand movement of the target. Claim 9 A device for restoring the trajectory of hand movement based on biosignals comprises: a first data collection unit for collecting electroencephalogram (EEG) data and electromyogram (EMG) data related to a user's hand movement; a second data collection unit for collecting acceleration data and angular velocity data related to the user's hand movement; a first preprocessing unit for performing preprocessing on each of the collected EEG data, EMG data, acceleration data, and angular velocity data; a proprioceptive data extraction unit for extracting proprioceptive data based on the preprocessed EMG data, acceleration data, and angular velocity data; and a data learning unit for learning the trajectory data of the user's hand movement through an artificial neural network based on the preprocessed EEG data and the extracted proprioceptive data; wherein the operation of learning the trajectory data of the user's hand movement comprises the step of extracting a proprioceptive latent expression used for data regeneration of the electromyogram data, angular velocity data, and acceleration data from the extracted proprioceptive data; The method comprises: a step of extracting a first free latent expression used for trajectory tracking and a first common latent expression used for proprioceptive tracking from the preprocessed electroencephalogram data; and a step of determining a predicted trajectory based on the extracted proprioceptive latent expression, the first free latent expression, and the first common latent expression; wherein, after the data learning unit has performed the learning, a third data collection unit collects electroencephalogram data of the subject related to the imagination of the subject's hand movements, a second preprocessing unit performs preprocessing on the collected electroencephalogram data of the subject, and a data inference unit infers trajectory data of the subject's hand movements through the artificial neural network based on the preprocessed electroencephalogram data of the subject, and the operation of inferring trajectory data of the subject's hand movements includes, in an electroencephalogram encoder, a step of extracting a second free latent expression used for trajectory tracking and a second common latent expression used for proprioceptive tracking based on the preprocessed electroencephalogram data input through the artificial neural network;A device for restoring the trajectory of a hand movement, comprising: a step of inferring the trajectory of the hand movement of the target based on the second free potential representation and the second joint potential representation in the trajectory decoder. Claim 10 In a program stored in a recording medium for restoring the trajectory of hand movements based on biosignals, the program executes on a computer the following operations: collecting electroencephalogram (EEG) data and electromyogram (EMG) data related to a user's hand movements; collecting acceleration data and angular velocity data related to the user's hand movements; performing preprocessing on each of the collected EEG data, EMG data, acceleration data, and angular velocity data; extracting proprioceptive data based on the preprocessed EMG data, acceleration data, and angular velocity data; and training the trajectory data of the user's hand movements through an artificial neural network based on the preprocessed EEG data and the extracted proprioceptive data; wherein the operation of training the trajectory data of the user's hand movements comprises: extracting a proprioceptive latent representation used for data regeneration of EMG data, angular velocity data, and acceleration data from the extracted proprioceptive data; The method comprises: an operation of extracting a first free latent expression used for trajectory tracking and a first co-latent expression used for proprioceptive tracking from the preprocessed electroencephalogram data; and an operation of determining a predicted trajectory based on the extracted proprioceptive latent expression, the first free latent expression, and the first co-latent expression; wherein, after learning is performed by the data learning unit, the third data collection unit collects electroencephalogram data of the subject related to the imagination of the subject's hand movements; the second preprocessing unit performs preprocessing on the collected electroencephalogram data of the subject; and the data inference unit infers trajectory data of the subject's hand movements through the artificial neural network based on the preprocessed electroencephalogram data of the subject.A program stored in a recording medium, further comprising: an operation for inferring trajectory data of the hand movement of the subject, wherein the operation for inferring trajectory data of the hand movement of the subject comprises: an operation for extracting a second free latent expression used for trajectory tracking and a second co-latent expression used for proprioceptive tracking based on the preprocessed electroencephalogram data input through the artificial neural network in an electroencephalogram encoder; and an operation for inferring the trajectory of the hand movement of the subject based on the second free latent expression and the second co-latent expression in a trajectory decoder.

Images