Method, apparatus and program for reconstructing trajectory of hand movement based on biological signal

Abstract

To restore a complicated hand motion by decoding a user's inherent biological signal without external stimulation.SOLUTION: Collecting, by a first data collector, electroencephalogram (EEG) data and electromyogram (EMG) data associated with a gesture of a hand of a user; collecting, by a second data collector, acceleration data and angular velocity data associated with the gesture of the hand of the user; and performing, by a first preprocessor, preprocessing on each of the collected EEG data, EMG data, acceleration data, and angular velocity data. The method may further include extracting proprioception data based on the pre-processed EMG data, acceleration data, and angular velocity data, and allowing a data learner to learn the hand movement trajectory data of the user through an artificial neural network based on the pre-processed EEG data and the extracted proprioception data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method, device, and program for restoring a hand movement trajectory based on a biometric signal, and more particularly to a method, device, and program for providing a user's intended hand movement or character as a trajectory or text based on a biometric signal when there is a hand movement such as handwriting, thereby enabling various interface interactions such as communication and drawing.

[0002] Meanwhile, this invention was supported by the following national research and development projects: Project unique number: 2710060570 Project number: RS-2024-00336673 Department name: Ministry of Culture, Sports and Tourism Issue Management (Specialized) Institution Name: Information and Communications Planning and Evaluation Institute Research project name: Core technology development for realistic content Research title: Brain-to-Speech: Development of artificial intelligence technology for two-way communication for people with language disabilities Project implementation organization: Korea University Industry-Academia Cooperation Group Research period: April 1, 2024 - December 31, 2026 Project unique number: 2710033819 Project number: 2021-0-02068-003 Department name: Ministry of Science, ICT and Communication Issue Management (Specialized) Institution Name: Information and Communications Planning and Evaluation Institute Research project name: Information, communications and broadcasting innovation human resource development Research topic: Artificial Intelligence Innovation Herbal Research and Development Project implementation organization: Korea University Industry-Academia Cooperation Group Research period: 2021.07.01~2025.12.31 Project unique number: 1711193973 Project number: 2019-0-00079-005 Department name: Ministry of Science, ICT and Communication Issue Management (Specialized) Institution Name: Information and Communications Planning and Evaluation Institute Research project name: Information, communications and broadcasting innovation human resource development Research topic: Artificial Intelligence Graduate School Support (Korea University) Project implementation organization: Korea University Industry-Academia Cooperation Group Research period: 2024.01.01~2026.12.31 Project unique number: 1711190390 Project number: 2022R1A2C3009750 Department name: Ministry of Science, ICT and Communication Name of the project management (specialized) organization: Korea Research Foundation Research project name: Individual basic research (Ministry of Science, Technology, Information and Communication) Research title: MetaSkin: Development of next-generation neurohaptic interface core technology that enables communication and environmental control in Metabus through skin contact Project implementation organization: Korea University Industry-Academia Cooperation Group Research period: March 1, 2022 - February 28, 2026 Issue identification code: 2710083012 Grant Number: RS-2025-02304828 Ministry name: Ministry of Science, ICT and Technology Business management (specialized) organization name: Information and Communications Planning and Evaluation Institute Research Project Name: AI Star Fellowship Program Research title: AI Star Fellowship Program (Korea University) Project implementing organization: Korea University Industry-Academia Cooperation Group Research period: April 1, 2025 to December 31, 2030 [Background technology]

[0003] Biosignals include electroencephalography (EEG) and electromyography (EMG).

[0004] Electroencephalography (EEG) is a non-invasive technique for measuring electrical activity in the brain. For example, it captures brain waves through electrodes attached to the skull and displays them as a graph of voltage changes over time. Electroencephalography (EEG) is widely used to diagnose and study various brain-related disorders, particularly in neuroscience, psychology, and medical research. Essentially, EEG measures signals generated in various regions of the brain. These signals, primarily resulting from the electrical activity of neurons, exhibit distinct regional, temporal, and frequency characteristics, allowing for analysis. EEG is an important biosignal, particularly because it contains fundamental information about hand movements and the user's underlying intentions. Compared to other representative biosignals, such as fMRI, EEG has short-term resolution, enabling it to capture brain activity over minute time periods. However, it suffers from a high signal-to-noise ratio due to various obstructions, such as the scalp, skull, and hair, that exist between the electrodes. Furthermore, its relatively low spatial resolution often makes it difficult to accurately identify the brain region from which the signal originated. Therefore, the quality of techniques that rely solely on electroencephalograms is sensitive to appropriate noise reduction and feature extraction techniques.

[0005] Electromyography (EMG) is a technique for measuring and analyzing the electrical activity of muscles. This technique captures electromyograms (EMGs) through electrodes attached to the muscles, enabling detailed understanding of muscle movements and intentions. It is widely used in medicine, rehabilitation, sports science, and other fields to monitor muscle contraction and relaxation. EMG signals are based on the electrical activity of muscle fibers. These signals result from action potentials generated by motor nerve fibers when muscles are activated, allowing for quantitative measurement of muscle strength, endurance, fatigue, and other factors. The temporal and spatial resolutions of EMG are generally higher than those of EEG. The signal-to-noise ratio of EMG varies depending on factors such as the contact quality between the muscle and electrodes, external noise, and signal processing methods, requiring noise reduction and appropriate feature extraction.

[0006] Proprioception is the sense of awareness of the body's position and movement, achieved through proprioceptors distributed in muscles, tendons, joints, and skin. Proprioception tells us how a part of the human body (e.g., the wrist) is positioned in space and how it is moving.

[0007] Deep learning is a branch of machine learning that focuses on data modeling using artificial neural networks and can solve difficult problems through the complex interactions of multiple layers of neural networks. It demonstrates high performance in high-dimensional and non-standard data such as biosignals, and is used in a variety of application fields such as image recognition, natural language processing, and speech recognition.

[0008] An artificial intelligence model in the field of deep learning may have an encoder and decoder structure. The encoder can nonlinearly extract complex characteristics from input data to form a "latent representation." This latent representation is a simplified representation of the essential attributes and structure of the data, converting the original data into a simpler, more easily processed form. The decoder can use this latent representation to generate or predict a desired result. Decoders can perform tasks such as classification, regression, image reconstruction, text translation, and speech recognition. The term "latent space" refers to an abstract space in which all possible latent representations exist, and in this space, various characteristics of the data are mathematically expressed. In various embodiments herein, the latent representation may also be referred to as a "latent vector" or "embedding."

[0009] Taking image reconstruction as an example, an image encoder analyzes a given image and converts key information into a compressed form, i.e., a latent representation. For example, when processing a cat image, the encoder recognizes the cat's characteristics, such as its size, position, breed, facial expression, and contours, and compresses them into a concise data format. This latent representation contains the cat's core characteristics. An image decoder can use the latent representation to reconstruct a new image similar to the original, generate other poses or facial expressions for the cat, or classify the image. The latent space contains the "key elements" of the data, and small changes in the latent space can result in significant changes in the image's characteristics. For example, moving in a particular direction from the latent space can change the cat's facial expression, while moving in another direction can result in a change in color or background. In this way, latent representations are useful for effectively processing complex data such as images, enabling a variety of applications, including creative image generation, efficient image compression, and sophisticated image editing.

[0010] The encoder automatically learns to minimize the difference (loss value) between the decoder's output and the actual data using the latent representation. Through this, the latent space is automatically optimized based on the model structure and objective function, resulting in a more accurate and efficient latent space. The latent space can be formed by automatic learning or manually adjusted based on human data modeling results. For example, latent representations can be adjusted to emphasize or ignore specific data characteristics, or specific patterns or characteristics can be enhanced through a separate data reconstruction model. Adjusting latent representations in the latent space can be performed using various techniques. For example, mechanisms such as "concatenation," "sum," or "attention" can be used to combine or enhance different latent representations.

[0011] The encoder-decoder structure is a concept commonly applied to many artificial intelligence models, providing a foundation for helping them process complex data effectively and derive more diverse and creative results through flexible manipulation of the latent space.

[0012] Brain-Computer Interface (BCI) is a technology that can directly transmit a user's intentions and thoughts to a computer system. It connects the brain with a computer, recognizes the user's intentions, and allows various control tasks to be performed simply through imagination. BCI technology is being actively researched in various fields, particularly in medicine, games, and virtual reality. It is particularly used to provide functions such as message delivery, environmental control, and voice synthesis to people with limited mobility or communication.

[0013] Existing BCI-based user interface control methods have mainly used brain waves to input commands. For example, methods such as Steady-State Visual Evoked Potentials (SSVEP) analyze the brain waves that a user generates in response to visual stimuli of a specific frequency, allowing for cursor movement, character selection, and simple command execution. This method, which can simplify and execute complex control commands, is also used as an assistive device for patients who have difficulty communicating.

[0014] For tracking human body movements, such as handwriting / hand tracking, camera-based tracking, motion capture using various sensors, touch screens, etc. Camera-based tracking uses an external or built-in camera to capture and analyze human body movements in real time, while motion capture uses sensors to measure human body movements. [Prior art documents] [Patent documents]

[0015] [Patent Document 1] Korean Patent Publication No. 10-2021-0059079 [Patent Document 2] Korean Patent Publication No. 10-2020-0110998 [Patent Document 3] Korean Patent Publication No. 10-2023-0086936 [Patent Document 4] Korean Patent Publication No. 10-2013-0141904 Summary of the Invention [Problem to be solved by the invention]

[0016] Existing user interface devices that track hand movements or handwriting are equipped with cameras, sensors, and / or touch devices, but they face challenges such as spatial constraints due to the need to install these external devices at a certain distance from the user, sensor blind spots, and limitations in capturing fine movements. Furthermore, existing BCI-based communication devices that use external stimuli often require high levels of concentration and training, making it difficult to execute complex commands. Tracking movements using only electroencephalograms (EEGs) among biosignals limits the decoding of simple commands due to the high signal-to-noise ratio of EEGs, making it difficult to anticipate diverse interactions with users. There is a growing need for methods and devices that overcome these limitations, enhance user convenience, and enable more flexible use in a variety of environments and situations.

[0017] The present invention uses only the user's biometric signals for motion tracking, eliminating the spatial constraints and blind spots inherent in using a camera, and applies electroencephalograms (EEGs) that contain the user's inherent intentions, allowing the user to output hand movements, such as handwriting, intended (imagined) without actually moving their hands. The present invention also provides a user interface device that can be controlled in a variety of ways purely through the user's initiative by utilizing a deep learning model that collectively learns various types of biometric information. In other words, the present invention provides user convenience by decoding the user's inherent biometric signals without external stimuli, while also enabling the restoration of complex hand movements, such as handwriting trajectories, that cannot be simply classified.

[0018] Meanwhile, the technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0019] While electroencephalography (EEG) provides high resolution in a short time, its low signal-to-noise ratio makes it difficult to directly track hand movements. Proprioceptive data can be used to complement EEG, enabling more accurate inference of movement paths and even imagined hand movements. This allows for a more accurate understanding of the user's actual or intended movements, leading to more sophisticated user interface interactions. Proprioceptive data can be obtained by measuring and then modeling wrist acceleration, angular velocity, and electromyography (EMG). In the present invention, this modeling is performed using a deep learning reconstruction-based model. Accelerometers and angular velocity sensors monitor joint movements in real time, allowing for precise tracking of the hand's position and movement path. EMG measures muscle contraction and relaxation through muscle electrical activity, allowing for the detection of even the most subtle hand movements. In particular, proprioceptive information can be an important intermediate step in inferring a clean movement trajectory from noisy electroencephalography (EEG).

[0020] The present invention trains a model based on actual hand movement data, with an encoder extracting proprioceptive latents from EEG and a decoder generating a movement path based on this proprioceptive information. This method also enables robust decoding of imagined hand movements. It has been scientifically proven that imagined movements produce similar brain activity patterns to real movements. Because the brain processes imagined movements in the same way as real movements and activates similar neural pathways, proprioceptive information is inherent in the electroencephalogram (EEG) for such imagined movements. This combined model based on proprioceptive information and EEG can operate effectively and maintain high accuracy and reliability even under diverse environmental conditions. This approach is advantageous for processing noisy EEG signals and predicting imagined movement paths. Consequently, utilizing such proprioceptive information, an important background technology for the present invention, can contribute to improving the accuracy and efficiency of user interface interactions based on various biological signals.

[0021] According to one embodiment, a method for restoring a hand movement trajectory based on a biosignal may include: collecting electroencephalogram data and electromyogram data related to a user's hand movement in a first data collection unit; collecting acceleration data and angular velocity data related to the user's hand movement in a second data collection unit; preprocessing the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data in a first preprocessing unit; extracting proprioceptive data based on the preprocessed electroencephalogram data, acceleration data, and angular velocity data in a proprioceptive data extraction unit; and learning, through an artificial neural network, trajectory data of the user's hand movement based on the preprocessed electroencephalogram data and the extracted proprioceptive data in a data learning unit.

[0022] The step of training the trajectory data of the user's hand movement may include the steps of: extracting, from the extracted proprioceptive data, a proprioceptive latent representation used for data reconstruction of electromyogram data, angular velocity data, and acceleration data; extracting, from the preprocessed electroencephalogram data, a first free latent representation used for trajectory tracking and a first joint latent representation used for proprioceptive tracking; and determining a predicted trajectory based on the extracted proprioceptive latent representation, first free latent representation, and first joint latent representation. The data training unit may determine the predicted trajectory by using, as a loss function, a weighted integration of a reconstruction loss value indicating a similarity between data reconstructed by the data reconstruction and actual measurement data, a joint latent representation loss value indicating a similarity between the first joint latent representation and the proprioceptive latent representation, and a trajectory loss value indicating a similarity between the predicted trajectory and an actual trajectory. Each of the weighted values ​​may vary over time.

[0023] The preprocessing of the collected electroencephalogram data and electromyogram data may include filtering with a bandstop filter, filtering with a bandpass filter, analysis of independence to remove unwanted signals, and filtering with a low-frequency bandpass filter, and the preprocessing of the angular velocity data and acceleration data may include filtering with a correction filter.

[0024] The method may further include, after the data learning unit has completed learning, collecting electroencephalogram data of the subject related to the imagining of the subject's hand movement in a third data collection unit; preprocessing the collected electroencephalogram data of the subject in a second preprocessing unit; and inferring trajectory data of the subject's hand movement through the artificial neural network based on the preprocessed electroencephalogram data of the subject in a data inference unit. The inferring trajectory data of the subject's hand movement may further include extracting, in an electroencephalogram encoder, a second free latent representation used for trajectory tracking and a second joint latent representation used for proprioception tracking based on the preprocessed electroencephalogram data input through the artificial neural network; and inferring, in a trajectory decoder, a trajectory of the subject's hand movement based on the second free latent representation and the second joint latent representation. A trajectory data processing unit may be controlled to transmit or display the inferred trajectory data of the subject's hand movement. [Effects of the Invention]

[0025] The present invention can provide an apparatus and method for user interface interaction that can grasp the underlying intention of a target user even when the target user makes subtle movements or is motionless.

[0026] The present invention does not simply classify the target user's movements, but rather restores the user's intended hand movements from bio-signals, enabling the control of complex user interfaces such as handwriting, drawing, cursor movement, etc. by inferring the user's intended handwriting / hand movements without spatial constraints or sensor blind spots. This allows physically disabled people who have difficulty using their arms, or people in situations where it is difficult to use their hands, such as on the subway during rush hour, driving, or injured, to easily interact with mobile phones and other mobile electronic devices, writing, drawing, operating, etc., using only their imagination.

[0027] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other technical effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a configuration diagram of an apparatus for restoring a trajectory of hand movement according to an embodiment. [Figure 2] FIG. 1 is a flowchart of operations performed by an apparatus for reconstructing a trajectory of hand movement according to an embodiment. [Figure 3] FIG. 1 illustrates a block diagram of a process for learning and inferring user movements utilizing bio-signals according to one embodiment. [Figure 4] 1 illustrates a process of a data pre-processing module according to one embodiment. [Figure 5] 1 illustrates a process for a learning module according to one embodiment. [Figure 6] 1 illustrates the process of an inference module according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] The purpose, technical configuration, and effects of the present invention will be more clearly understood from the following detailed description based on the accompanying drawings. With reference to the accompanying drawings, the present invention will be described in detail.

[0030] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. It is obvious to those skilled in the art that the description including the embodiments of the present specification has various applications. Therefore, any embodiments described in the detailed description of the present invention are merely examples for better explaining the present invention, and are not intended to limit the scope of the present invention to the embodiments.

[0031] The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the detailed description. Also, although one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0032] Furthermore, a phrase that includes a certain element simply refers to the presence of that element as an "open" phrase, and should not be understood to exclude additional elements.

[0033] Furthermore, when a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but there may also be other components in between.

[0034] Various embodiments of the present invention will now be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the present invention to the particular embodiments, but rather to encompass various modifications, equivalents, and / or alternatives to the embodiments of the present invention.

[0035] In various embodiments of this document, user is used in conjunction with subject. In various embodiments of this document, subject is used in conjunction with target user or target subject. In various embodiments of this document, user and subject can be distinct or at least partially overlapping.

[0036] Although various embodiments of this document illustrate hand movements, the present invention is not limited to this and may also be applied to movements of other body parts of a user or subject.

[0037] FIG. 1 is a diagram illustrating the configuration of a device 100 for reconstructing a trajectory of hand movement (hereinafter referred to as "device 100") according to an embodiment.

[0038] Referring to FIG. 1, an apparatus 100 according to one embodiment may include a memory 110, a processor 120, an input / output interface 130, and a communication interface 140, respectively.

[0039] The memory 110 may store data obtained from an external device or data generated by the memory 110. The memory 110 may store instructions that cause the processor 120 to perform operations. For example, the memory 110 may store collected data, pre-processed data, extracted data, an artificial neural network, etc.

[0040] The processor 120 is a computing device that controls the overall operation. The processor 120 can execute instructions stored in the memory 110. The operations of the device 100 according to the embodiments of this document can be understood as operations performed by the processor 120.

[0041] The input / output interface 130 may include a hardware or software interface for inputting or outputting information.

[0042] The communication interface 140 allows for sending and receiving information over a communication network, and therefore may include a wireless communication module or a wired communication module.

[0043] The device 100 may be implemented in various forms, such as a server, a computer, a portable communication device, a smartphone, a portable multimedia device, a notebook, a tablet PC, etc., that perform calculations using the processor 120 and transmit and receive information over a network, but is not limited to these examples.

[0044] The components of the device 100 may be divided in a different manner than in Fig. 1. For example, the device 100 may be defined as including at least some of a first data collection unit, a second data collection unit, a first pre-processing unit, a proprioception data extraction unit, a data learning unit, a third data collection unit, a second pre-processing unit, a data inference unit, and a trajectory data processing unit.

[0045] 2 is a flow chart diagram of operations performed by the device 100 according to one embodiment. The operations of the device 100 according to the embodiment of FIG.

[0046] 2 are merely preferred embodiments for achieving the object of the present invention, and it goes without saying that some steps may be added or deleted as necessary, or one step may be included in another step. The procedures of each operation disclosed in FIG. 2 are merely arranged for ease of understanding, and such procedures are not limited to chronological procedures, and the procedures may be changed and operated differently at the designer's discretion.

[0047] In operation 205, the first data collection unit may collect electroencephalogram data and electromyogram data related to the user's hand movements, and the collected data may be stored in a predetermined database.

[0048] In operation 210, a second data collector may collect acceleration data and angular velocity data related to the user's hand movement. The collected data may be stored in a predetermined database. When using biosignals with high individual deviations, it is difficult to use a large amount of data. Therefore, by utilizing auxiliary data such as acceleration data and angular velocity data, consistency can be improved through deep learning even with a small amount of data.

[0049] In operation 215, a first pre-processing unit may perform pre-processing on each of the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data. The pre-processing on the collected electroencephalogram data and electromyogram data may include filtering using a bandstop filter, filtering using a bandpass filter, independent component analysis for removing unwanted signals, and filtering using a low-frequency bandpass filter. The pre-processing on the angular velocity data and acceleration data may include filtering using a correction filter.

[0050] In operation 220, a proprioceptive data extracting unit may extract proprioceptive data based on the pre-processed electromyogram data, acceleration data, and angular velocity data.

[0051] In operation 225, the data learning unit may learn trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data. Learning the trajectory data of the user's hand movement may include extracting a proprioceptive latent representation used for data reconstruction of electromyogram data, angular velocity data, and acceleration data from the extracted proprioceptive data, extracting a first free latent representation used for trajectory tracking and a first joint latent representation used for proprioception tracking from the preprocessed electroencephalogram data, and determining a predicted trajectory based on the extracted proprioceptive latent representation, first free latent representation, and first joint latent representation.

[0052] The data learning unit may determine the predicted trajectory by using, as a loss function, a weighted combination of a reconstruction loss value indicating a similarity between the data reconstructed by the data reconstruction and actual measurement data, a joint latent representation loss value indicating a similarity between a first joint latent representation and a proprioceptive latent representation, and a trajectory loss value indicating a similarity between the predicted trajectory and an actual trajectory. Each of the weighted values ​​may be configured to vary over time.

[0053] In step 230, a third data collection unit may collect electroencephalogram data of the subject related to the subject's imagined hand movements, the subject being separate from the user or at least partially overlapping with the user.

[0054] In operation 235, a second pre-processing unit may perform pre-processing on the collected electroencephalogram data of the subject. The pre-processing on the collected electroencephalogram data of the subject may include filtering using a band-stop filter, filtering using a band-pass filter, independent component analysis for removing unnecessary signals, and filtering using a low-frequency band-pass filter.

[0055] In operation 240, a data inference unit may infer trajectory data of the subject's hand movement through the artificial neural network based on the preprocessed electroencephalogram data of the subject. Inferring the trajectory data of the subject's hand movement may include extracting, in an electroencephalogram encoder, a second free latent representation used for trajectory tracking and a second joint latent representation used for proprioception tracking based on the preprocessed electroencephalogram data input through the artificial neural network, and inferring, in a trajectory decoder, a trajectory of the subject's hand movement based on the second free latent representation and the second joint latent representation.

[0056] In step 245, the trajectory data processing unit can be controlled to transmit or display the trajectory data of the inferred target hand movement. A device receiving the transmitted trajectory data can store or display the corresponding trajectory data.

[0057] Human movement processes involve a simplified series of steps: (1) data collection from sensory organs, (2) analysis of sensory data and establishment of a motor plan in the brain, (3) transmission of action commands to the muscles and nerves, and (4) actual execution of the movement through muscle contraction and relaxation. Because the EEG generated during this movement process is noisy and difficult to decode, a deep learning model must be able to efficiently simulate this process in order to reconstruct high-dimensional, detailed movements like handwriting based on EEG data. To achieve this, the present invention collects proprioception data, which is directly related to hand movements and trivially processed by the brain, and guides the model to learn the latent representation of this proprioception, thereby improving efficiency. Proprioception includes joint position, muscle tension, and movement speed and direction. Based on this information, the brain establishes and executes an accurate motor plan. When a deep learning model simulates this human movement process, utilizing proprioception data can reduce the complexity of EEG and build a model that is robust to noise. Specifically, proprioceptive data are used as auxiliary information to aid in the interpretation of EEG signals, and the model then serves as a link between noise-tolerant EEG signals and precise hand movement trajectories. This approach can significantly improve the accuracy of EEG-based movement decoding.

[0058] According to various embodiments of the present invention, data such as a user's electroencephalogram (EEG), electromyogram (EMG), angular velocity and acceleration for each joint, etc. are collected, and an encoder / decoder model is trained to have a latent representation that takes into account proprioceptive information. In actual use, the device can operate using only the EEG, and the hand movement can be restored simply by imagining the hand movement.

[0059] According to various embodiments of the present invention, the electroencephalogram encoder may be a neural encoder, which may include a combination of deep learning layers such as a fully connected layer, a convoluted neural network layer, a long-term short-term memory, a recurrent neural network, and a transformer, and may extract features nonlinearly from biosignals to create useful embeddings.

[0060] According to various embodiments of the present invention, the trajectory decoder can infer the embeddings of the neural encoder from the trajectory, and can infer the actual trajectory on a useful composite embedding consisting of the free latent representations and proprioceptive latent representations created from the neural encoder, including a combination of deep learning layers such as a fully connected layer, a convoluted neural network layer, a long short-term memory, a recurrent neural network, and a transformer.

[0061] FIG. 3 shows a block diagram of a process for learning and inferring user movements using bio-signals according to one embodiment.

[0062] The biosignal measurement and transmission module 310 collects biosignals (electroencephalogram and electromyogram) during the user's intentional hand movements (including both actual and imagined movements). The data preprocessing module 320 performs noise reduction on the collected biosignals through filtering, such as independent component analysis, bandstop filtering, and / or bandpass filtering. The neural processing unit 330 infers the user's intentional hand movement trajectory from the preprocessed biosignals, which can be output to a display from the user interface 340. The neural processing unit 330 may be equipped with both a learning module for maintaining continuous performance with user customization and an inference module for inferring the user's intention in real time. In various embodiments of the present invention, the neural processing unit 330 performs proprioception-based learning using electroencephalogram (EEG) signals, electromyogram signals, acceleration signals, and / or angular velocity signals as auxiliary information during learning, and may use only the EEG signals during inference.

[0063] In various embodiments of the present invention, the hand movement trajectory restoration process is divided into two modes: a learning mode and an inference mode. The learning mode is a process of learning a deep neural network through a learning module based on the user's arm movements and corresponding biosignals according to a predetermined learning scenario, and updating the inference module by fine-tuning the deep neural network through transfer learning to maintain stable performance of the inference mode. The inference mode, which can also be called an operational mode, is a process of continuously outputting the user's intended arm movements through the inference module using the deep neural network model obtained through the learning mode.

[0064] The biosignal measurement and transmission module 310 is divided into a biosignal collection module and a transmission module. The biosignal collection module collects biosignals by recording voltage differences between electrode sensors. The biosignal collection module is configured with a device using non-invasive electrodes and can be attached to the scalp and arm to collect signals (electromyograms and electroencephalograms) at a sampling frequency (e.g., 1000 Hz). In various embodiments of the present invention, acceleration signals and angular velocity signals can be additionally collected based on an inertial measurement unit (IMU) sensor attached to a part of the user's or subject's body (e.g., a joint). The (voltage) signals collected by the biosignal collection module can be input to the data pre-processing module via the transmission module.

[0065] FIG. 4 illustrates the process of the data pre-processing module according to one embodiment.

[0066] The data preprocessing module 320 preprocesses and removes noise from the biosignals (electromyogram / electroencephalogram) transmitted from the biosignal collection module. In the example of FIG. 4, the preprocessing of the electromyogram and electroencephalogram involves three rounds of frequency filtering and one round of dimensionality reduction. Specifically, a band-stop filter is first used to remove noise signals corresponding to a predetermined frequency (e.g., 60 Hz) and its harmonics (e.g., 120 Hz, 180 Hz, etc.) from the transmission lines in the surrounding environment. Then, a band-pass filter (1-45 Hz) is used to remove white noise and other high-frequency noise. After the initial noise removal by the band-pass filter, independent component analysis is performed to remove noise signals generated by other movements, such as blinking. Then, a band-pass filter (1-8 Hz) is used to extract a biosignal frequency band optimized for hand movements.

[0067] Although not shown in Figure 4, the EMG and EEG signals can be further optimized for learning or inference through additional preprocessing such as baseline correction, windowing, and normalization.

[0068] The acceleration and angular velocity signals output from the IMU sensor are passed through a correction filter such as a Kalman filter or a complementary filter, and then integrated with the preprocessed electromyogram to be used as proprioceptive data.

[0069] The neural processing unit 330 includes a learning module and an inference module, and may further include a database and memory for intermediate processes of updating the inference module using the learning results and outputting the inference results and device feedback. The learning module fine-tunes a deep neural network (described below) using preprocessed biosignals and trajectory data. The fine-tuned deep neural network is stored in a database and transmitted to the inference module via memory. The inference module is always running, and the learning module can be turned on or off as needed by the user. While the learning module is on, the user can perform hand movements according to a predetermined scenario, and the inference model can be updated in real time.

[0070] FIG. 5 illustrates the process of the learning module according to one embodiment.

[0071] The learning module can consist of a proprioception reconstruction unit (a reconstruction-based model for obtaining proprioception latent representations) and a main trajectory inference unit (a model for predicting trajectories from EEG (electroencephalogram) taking proprioception information into account). The proprioception reconstruction unit is a reconstruction-based model designed to reconstruct the user's proprioception information and aims to create an optimal latent representation for proprioception data. The main trajectory inference unit aims to infer (predict) hand movement trajectories from EEG taking proprioception information into account. The main trajectory inference unit can consist of an EEG encoder and a trajectory decoder. The EEG encoder can be trained to extract features by considering both the proprioception latent representation from the EEG and a latent representation containing important features for trajectory inference. The latent representation generated by the EEG encoder is partially induced to take into account the proprioception latent representation, and the rest is left as a free latent representation. The trajectory decoder can infer the actual trajectory from the proprioception latent representation and the free latent representation for trajectory inference.

[0072] The proprioceptive reconstruction unit inputs electromyogram, angular velocity, and acceleration data into a model to compress the data and then decompress it to create a proprioceptive latent representation that can generate data that most closely matches the original data. The loss value that measures how similar the reconstructed data (sensor signal) is to the measured data (actual sensor signal) is called the "reconstruction loss value." Reconstruction loss value The lower the loss, the more effectively the proprioceptive reconstruction unit extracts meaningful features from the measured data. The induced latent representation guides the EEG encoder to learn with sophisticated and meaningful information. In other words, the reconstruction loss can guide the proprioceptive reconstruction model to extract a latent representation of proprioception that is useful for trajectory generation. The reconstruction loss can be evaluated using at least one of the following criteria: mean squared error (MSE), mean absolute percentage error (MAP), cosine similarity, and dynamic time warping.

[0073] The main trajectory inference unit consists of an EEG encoder and a trajectory decoder. The EEG encoder converts EEG data into a latent representation, and the trajectory decoder predicts the trajectory from the latent representation. The latent representation output by the EEG encoder is divided into a free latent representation and a joint latent representation derived similarly to the proprioceptive latent representation. The joint latent representation is calculated by calculating the similarity between the proprioceptive latent representation extracted from the proprioceptive reconstruction unit and a joint latent representation loss. JPEG2026012663000003.jpg6150. That is, the joint latent representation loss value can be used to guide the neural encoder to extract the optimal latent representation of proprioception from EEG. The criterion for evaluating the joint latent representation loss value can be at least one of the following: mean squared error (MSE), mean absolute percentage error (MAP), cosine similarity, and dynamic time warping. The cost function of the model is set in a way to reduce the joint latent loss value, and the EEG encoder is guided to consider the latent representation of proprioception.

[0074] In this invention, to adjust the latent space, we mainly use a latent space alignment technique, which uses the similarity between the source latent representation and the target latent representation as a loss function to guide the encoder to create a target latent representation similar to the source latent representation. This technique plays an important role in effectively adjusting the latent representation and minimizing the loss of the model. In this process, the model emphasizes only the significant characteristics of the data, allowing the artificial intelligence model to generate more precise and customized outputs. This increases the usefulness of the model in various application fields.

[0075] That is, while the present invention follows a basic encoder-decoder architecture that takes electroencephalograms as input and hand movement trajectories as output, it uses a proprioceptive and free latent space optimization (PFLSO) method that guides the encoder's output latent representations partially into free latent representations and partially into latent representations that take proprioceptive information into account, thereby enabling stable trajectory inference even for complex EEG signals generated in hand movement tasks such as handwriting. In other words, the PFLSO method helps the model effectively incorporate proprioceptive information from EEG signals to predict optimal movement trajectories.

[0076] In the main trajectory inference unit, the latent vector output by the EEG encoder is fed into the trajectory decoder, which finally outputs a trajectory. The similarity between the predicted trajectory and the actual trajectory is calculated as a "trajectory loss." The trajectory loss value is defined as JPEG2026012663000004.jpg6150. The trajectory loss value can be used to guide the neural encoder's free latent representation to extract the optimal latent representation from EEG. The trajectory loss value can be evaluated using at least one of the following criteria: mean squared error (MSE), mean absolute percentage error (MAP), cosine similarity, and dynamic time warping. A model cost function can be formulated in a way that reduces the trajectory loss value, guiding the entire module to effectively take trajectory information into account.

[0077] The proprioception reconstruction unit and the main trajectory inference unit have an integrated loss value, which is a composite weighted sum of three loss values, namely, the reconstruction loss value, the joint latent loss value, and the trajectory loss value, as shown in Equation 1. Using this integrated loss value, the entire module can be trained simultaneously by an appropriate optimization algorithm. This is called the end-to-end training method.

[0078]

number

[0079] The optimal weights of the three loss values ​​can be determined, for example, by a grid-search method. Also, the weights can be flexibly changed according to the learning level to refine the learning stage, allowing each component of the module to be differentially learned over time. For example, in the early stages of learning, the weights of the reconstruction loss value can be set high and the remaining weights can be set low to intensively learn the proprioception reconstruction unit, and from the middle stages of learning, the weights of the joint latent loss value can be set high to intensively learn the main trajectory inference unit.

[0080] FIG. 6 illustrates the process of the inference module according to one embodiment.

[0081] The inference module receives EEG input and infers a trajectory based on a trained neural encoder and trajectory decoder. The trained main trajectory inference unit may include an EEG encoder optimized through the PFLSO method. This EEG encoder receives only EEG signal input and can consistently infer (predict) a hand movement trajectory taking proprioceptive information into account. The trained EEG encoder receives noisy EEG input and extracts (infers) an optimal joint latent representation (proprioceptive latent space) that is relatively less noisy and has more similarities to the trajectory data, and an optimal free latent representation (free latent space) for trajectory generation.

[0082] In the inference module, a joint latent representation is induced to contain information similar to the proprioceptive latent representation learned in the proprioceptive reconstruction unit. This joint latent representation fully reflects the user's proprioceptive information and ensures consistent performance for both actual and imagined movements. The free latent representation is the remaining part of the model to freely learn additional information necessary for trajectory prediction, helping the trajectory decoder to optimally predict the hand movement trajectory.

[0083] In Figure 6, all latent vectors output from the EEG encoder are input to the trajectory decoder. Based on these latent vectors, the trajectory decoder can predict the trajectory of the hand, including the X and Y coordinates, pressure (pen pressure), and speed (force). In this process, the trajectory decoder reflects proprioceptive information through the joint latent representation and utilizes additional information required for trajectory prediction through the free latent representation, thereby deriving the optimal prediction result for the trajectory.

[0084] By learning a consistent latent representation using proprioceptive data during the learning process, trajectory prediction can be performed consistently for both actual and imagined movements. If the latent representation were to rely solely on free latent space without using the PFLSO method, and a trajectory were inferred from EEG using only a simple encoder-decoder structure, without deriving a latent representation through a separate reconstruction model, the model would rely solely on noisy EEG signals and become less robust. If the model is trained without considering proprioceptive information, it would be difficult to maintain consistent performance in various environments. This could result in unstable results for both actual and imagined hand movements, ultimately significantly reducing usability and usefulness. According to various embodiments of the present invention, a combined model of EEG and proprioception can be used to consistently infer trajectories for both imagined and actual movements.

[0085] The various embodiments and terms used herein should be understood not to limit the technical features described herein to a specific embodiment, but to include various modifications, equivalents, or alternatives of the embodiment. In connection with the description of the drawings, similar reference numerals are used for similar or related components. The singular form of a noun corresponding to an item may include one or more items unless the relevant context clearly dictates otherwise.

[0086] In this document, each of the terms "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" includes all possible combinations of the items listed with that term. Terms such as "first," "second," "primary," or "secondary" are used merely to distinguish one component from another and do not limit the component in any other aspect (e.g., importance or procedure). When referring to one (e.g., first) component as "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively," it means that the component is connected to the other component directly (e.g., by wire), wirelessly, or through a third component.

[0087] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrated component or the smallest unit or portion of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0088] Various embodiments of this document may be embodied as software (e.g., a program) including one or more instructions stored on a recording medium (e.g., memory) readable by a device (e.g., an electronic device). The recording medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0089] Furthermore, the processor of the embodiments of this document can call and execute at least one instruction from one or more instructions stored on a recording medium. This allows the device to be operated to perform at least one function according to the called at least one instruction. Such one or more instructions may include code generated by a compiler or code executed by an interpreter. The processor may be a general-purpose processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), and / or the like.

[0090] A machine-readable recording medium may be provided in the form of a non-transitory recording medium. Here, "non-transitory" simply means that the recording medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between data being stored semi-permanently on the recording medium and data being stored temporarily.

[0091] Methods according to various embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded between sellers and buyers as a commodity. The computer program product may be distributed in the form of a machine-readable recording medium (e.g., a compact disc read-only memory (CD-ROM)) or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., the Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated on a machine-readable recording medium such as a manufacturer's server, an application store server, or the memory of a server.

[0092] According to various embodiments, each of the described components (e.g., modules, programs, users, objects, etc.) may include one or more entities. According to various embodiments, one or more of the components or operations described above may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components may be integrated into a single component. In such cases, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component in the multiple components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.

Claims

1. 1. A method for reconstructing a hand movement trajectory based on a biometric signal, comprising: collecting electroencephalogram data and electromyogram data related to the user's hand movements in a first data collection unit; collecting acceleration data and angular velocity data associated with the user's hand movement at a second data collection unit; a first pre-processing unit performing pre-processing on the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data; extracting proprioceptive data based on the pre-processed electromyogram data, acceleration data, and angular velocity data in a proprioceptive data extracting unit; a step of causing a data learning unit to learn trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data; A method for recovering the trajectory of hand movements, including

2. The step of learning trajectory data of the user's hand movement includes: extracting from the extracted proprioceptive data a proprioceptive latent representation used for data reconstruction of electromyographic data, angular velocity data, and acceleration data; extracting a first free latent representation used for trajectory tracking and a first joint latent representation used for proprioception tracking from the preprocessed electroencephalogram data; determining a predicted trajectory based on the extracted proprioceptive latent representation, the first free latent representation, and the first joint latent representation; The method for reconstructing a hand movement trajectory according to claim 1 , comprising:

3. The data learning unit a reconstruction loss value indicating the degree of similarity between the data reconstructed by the data reconstruction and the actual measurement data; a joint latent representation loss value indicating the similarity between the first joint latent representation and the proprioceptive latent representation; a trajectory loss value indicating the similarity between the predicted trajectory and an actual trajectory; The method for reconstructing a trajectory of hand movement according to claim 2 , wherein the predicted trajectory is determined by using a loss function obtained by assigning a weight to each of the weights and integrating the weights.

4. The method for reconstructing a trajectory of hand movement according to claim 3 , wherein each of the weighted values ​​can be varied over time.

5. The pre-processing of the collected electroencephalogram data and electromyogram data includes filtering with a band-stop filter, filtering with a band-pass filter, independent component analysis for removing unnecessary signals, and filtering with a low-frequency band-pass filter; The method for reconstructing a trajectory of hand movement according to claim 1 , wherein pre-processing of the angular velocity data and acceleration data includes filtering with a correction filter.

6. After the data learning unit has completed learning, collecting electroencephalogram data of the subject related to the subject's imagination of hand movements in a third data collection unit; a second pre-processing unit performing pre-processing on the collected electroencephalogram data of the subject; a data inference unit inferring hand movement trajectory data of the subject through the artificial neural network based on the preprocessed electroencephalogram data of the subject; The method for reconstructing a hand movement trajectory according to claim 1 , further comprising:

7. The step of inferring trajectory data of the subject's hand movement includes: extracting, in an electroencephalogram encoder, a second free latent representation used for trajectory tracking and a second joint latent representation used for proprioception tracking based on the preprocessed electroencephalogram data input through the artificial neural network; inferring, in a trajectory decoder, a trajectory of the subject's hand movement based on the second free latent representation and the second joint latent representation; The method for reconstructing a hand movement trajectory according to claim 6, comprising:

8. The method for restoring a hand movement trajectory according to claim 6, further comprising the step of controlling a trajectory data processing unit to transmit or display the inferred target hand movement trajectory data.

9. 1. An apparatus for reconstructing a trajectory of hand movement based on a biometric signal, a first data collection unit that collects electroencephalogram data and electromyogram data related to the user's hand movements; a second data collection unit that collects acceleration data and angular velocity data related to the user's hand movement; a first preprocessing unit that performs preprocessing on each of the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data; a proprioceptive data extractor that extracts proprioceptive data based on the preprocessed electromyogram data, acceleration data, and angular velocity data; a data learning unit that learns trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data; A device for recovering a trajectory of hand movement, comprising:

10. A program stored on a recording medium for restoring a trajectory of a hand movement based on a biosignal, the program comprising: collecting electroencephalogram and electromyogram data associated with the user's hand movements; collecting acceleration and angular velocity data associated with the user's hand movements; An operation of performing pre-processing on each of the collected electroencephalogram data, electromyogram data, acceleration data, and angular velocity data; extracting proprioceptive data based on the preprocessed electromyogram data, acceleration data, and angular velocity data; an operation of learning trajectory data of the user's hand movement through an artificial neural network based on the preprocessed electroencephalogram data and the extracted proprioceptive data; A program stored on a recording medium that causes a computer to execute the above.

Images