System and implementation method for testing and training brain functions

Abstract

Disclosed are a system and method for testing and training brain functions for planning and executing actions. The system includes an electroencephalogram sensor device wearable on the head of a trainee, a processor configured to receive and analyze electroencephalogram signals acquired from the trainee in response to visual stimuli presented to the trainee, and a memory that, when executed by the processor, (i) gives an instruction to make the trainee imagine performing the action, (ii) measures electroencephalogram signals with the electroencephalogram sensor device, (iii) calculates at least one characteristic selected from a concentration index, a motor control index, and an arousal index, and (iv) stores an instruction for repeatedly providing a feedback pattern to the trainee. In some embodiments, a display is provided as a visual stimulus presented to the trainee for the trainee to imagine performing an action and respond.

Description

Technical Field

[0001] The present invention relates to a system and method for neurofeedback training for enhancing the nerves of a trainee, and more particularly for improving the performance of the trainee.

Background Art

[0002] Training of motor ability for planning and executing movements is performed by visualization.

[0003] Brain stimulation is used as training for improving the physical movements of a subject. It is known that physical ability is improved by passing an electric current (externally) through a specific brain region. This has been shown, for example, in athletes. Brain stimulation shows almost the same results as brain training naturally achieved by training during its use. The main drawback of the device for brain stimulation is the duration of its effect. It has been reported that the effect of training lasts only about 20 minutes after the device is removed.

[0004] As commercially available neurointervention technologies, devices for neurofeedback of mental or cognitive states are known. These devices are easy-to-use EEG sets with a sophisticated design, and users can train specific brain activations without the help of specialized hardware or an operator. These products target globally widespread and common brain processes such as relaxation, stress reduction, and concentration. Using auditory or visual feedback, users can learn how to switch to a relaxed or concentrated mental state. These devices are used limited to general processes as described above as the recording process of their headset.

[0005] Currently, such commercially available EEG sets have a low signal-to-noise ratio and a limited variety of detectable brain activities. Due to the rapid progress of technology, it is thought that such a situation will also change in a few years.

[0006] US2019 / 0247662 discloses a method for facilitating a skill learning process or improving the performance of a task, the method comprising: determining an electroencephalogram pattern reflecting the neural activity of an expert subject engaged in each technique or task; processing the electroencephalogram pattern determined by at least one automated processor; and subjecting the brain of each subject of each skill or task to brain synchronization by one or more stimuli selected from the group consisting of sensory excitation, peripheral excitation, transcranial excitation, and deep brain stimulation, depending on the processed temporal pattern extracted from the electroencephalogram reflecting the neural cell activity of the expert.

[0007] <Sports> In professional sports, competition is always a constant. Almost all sports are fundamentally based on competition. In a competitive environment, sports clubs and professional athletes strive to improve their performance. Success in professional sports, in addition to the pure goal of "being number one", often becomes an important factor for players and clubs to increase their funds (KPMG study https: / / assets.kpmg / content / dam / kpmg / br / pdf / 2019 / 06 / the-european-elite-2019a.pdf). For this reason, these clubs are constantly in need of new technologies that can give athletes an edge over their competitors. The sports tech market is continuously growing (MarketandMarket Research; www.marketsandmarkets.com / Market-Reports / sports-technology-market-104958738.html), and one of the most promising areas within it is the use of newly developed brain-computer interfaces (BCIs) (Ourcrowd: Top 10 Technology Trends for 2020 and Beyond - at Summit; https: / / summit.ourcrowd.com / top-10-tech-trends-for-2020-and-beyond-at-2020-ourcrowdsummit / ).

[0008] In traditional training, basically, one tries to train the brain by repeating, that is, repeating until perfection. "In any sport, skilled athletes accumulate game practice until their bodies remember the main skills, sub-skills, and their movements" (G. Landrum, "Empowerment: The Competitive Edge in Sports, Business & Life", 2005). An important process in practice is, in addition to the actual growth of muscles through training, to form a stable and efficient exercise plan. This means that the neural circuits in the brain are reconstructed, which in turn enhances the proficiency of movements later. However, traditional training, while having helped professional athletes perform at their best, is not easy to achieve today. Large-scale physical movements mainly have drawbacks such as being time-consuming, costly, causing injuries and fatigue, and reducing motivation.

[0009] Thanks to recent state-of-the-art technologies and scientific discoveries, it has become possible to directly read brain activity "online". With the ability to read the brain in real time and analyze its movements, it is expected that new ways to train the brain can be developed, overcoming the above-mentioned drawbacks while achieving the same results as physical training.

[0010] In addition to continuously exploring new training methods, an essential change in the training strategies of sports clubs is the use of new and robust data science tools. These clubs understand the importance of investing in athletes' performance and use large amounts of data from various sources (such as speed, strength, team dynamics, etc.) to extract the weaknesses to be trained while predicting future performance. In many industries, including the sports industry, the recognition that "data is the new gold" has widely penetrated, and experts are recently in a frenzy of "gold rush". Information about the brain has been difficult to obtain and collect until recently.

[0011] As an excellent example of how brain data can contribute to sports, consider how well an athlete can maintain concentration during a game. "Concentrating on the important things is the key to effective performance. During play, especially in long games such as soccer, baseball, and football, it is difficult to maintain concentration throughout. Those who can maintain concentration the longest tend to make the fewest errors and gain an advantage." The conventional methods for measuring an athlete's ability to concentrate on their task have been to either observe the play or indirectly measure it from the performance of a specific task (unrelated to actual sports movements) for examining concentration. Neuroscientific research has shown that an individual's level of attention strongly correlates with specific brain patterns that can be measured from an external device (SokJooTan et al., A Brief Review of the Application of Neuroergonomics in Skilled Cognition During Expert Sports Performance, Front. Hum. Neurosci., 2019; https: / / www.frontiersin.org / articles / 10.3389 / fnhum.2019.00278 / full).

[0012] Commercially, there are mainly two ways to directly intervene in brain activity to improve performance. One is brain stimulation. By passing an electric current (from the outside) through specific brain regions, an athlete's performance can be improved. Brain stimulation shows results almost equivalent to those achieved naturally by training during its use. The main drawback of devices for brain stimulation is the duration of their effects. It has been reported that the effects of training last only about 20 minutes after the device is removed. This means that in most competitive sports, the effects of the intervention weaken before the necessary situation arrives. Also, these actions may be considered a form of doping, which is a problem for competitive sports.

[0013] <Surgeon's Tests and Training> In the United States, approximately 15 million surgeries are performed annually in operating rooms (Weiss & Elixhauser, 2006). Surgeries on inpatients are considered "hot spots" where medical errors are likely to occur, with a mortality rate ranging from 0.4% to 0.8% and the incidence rate of major complications reaching 3% to 17% (Haynes et al., 2009). According to research, about half of surgical complications are avoidable (Gawande, Thomas, Zinner, & Brennan, 1999; Kable, Gibberd, & Spigelman, 2002), and it has been confirmed that the number of adverse events significantly decreases in teams with high functionality (Mazzocco et al., 2009). The introduction of new technologies may improve patient safety, but at the same time, new and significant demands will be imposed on surgeons' capabilities and workloads. In the United States, more than 1 million laparoscopic surgeries are performed annually, and surgeons are performing surgeries with indirect and narrow visual access and minimal tactile feedback. In such a situation, new skills beyond the traditional master-apprentice relationship and accompanying new training methods are required (Van Hove, Tuijthof, Verdaasdonk, Stassen, & Dankelman, 2010). In fact, as the medical pattern shifts towards prevention and quality improvement, aspects of the operating room that were not previously considered are being focused on more clearly, and the performance of surgeons and trainees is being scrutinized (Kao & Thomas, 2008; Kohn, Corrigan, & Donaldson, 2000; Pavlidis et al., 2012; Risucci, Geiss, Gellman, Pinard, & Rosser, 2001). Therefore, the need to provide systems and methods for decoding and measuring brain patterns during the training of surgeries and dental treatments has been long desired.

[0014] <Aviation Testing and Training> According to Boeing, 80% of aviation accidents are caused by human error.

[0015] According to NASA, pilot error is cited as the main cause of 78.6% of general aviation fatal accidents and 75.5% of all general aviation accidents that occurred in the United States in 2004.

[0016] Pilot errors are classified as follows. (i) General judgment errors due to various causes such as lack of training, fatigue, and inattention. (ii) Weather-related errors such as not fully understanding the impact of weather on an aircraft in flight. (iii) Machine-related errors such as being unable to handle internal failures of an aircraft.

[0017] In scheduled air transportation, more than about half of the accidents whose causes are known are said to be due to pilot error.

[0018] The recent suspension of air travel has caused an increase in pilot errors (https: / / www.latimes.com / business / story / 2021-01-29 / airline-pilots-flight-errors-pandemic), highlighting the need for additional and effective training systems. The systems and methods of the present invention can be easily applied for the purpose of improving pilot performance and evaluation. "There is a growing interest in the introduction of tools for monitoring cognitive performance in work in natural environments and in daily life. In a new research field known as neuroergonomics, the use of wearable and portable brain monitoring sensors such as functional near-infrared spectroscopy (fNIRS) is advancing to investigate cortical activity in various behaviors of humans outside the laboratory" (Front. Hum. Neurosci., 2018).

[0019] The need for means, systems, and methods for training pilots to prevent pilot error is very high.

[0020] That is, systems and methods for decoding and measuring brain patterns during training such as sports activities, surgeon training, aviation testing, and training have been longed for for many years.

Summary of the Invention

[0021] One object of the present invention is to disclose a system for testing and training brain functions for planning and executing movements. The system includes: (a) an electroencephalogram sensor device wearable on the head of a trainee; (b) a processor configured to receive and analyze electroencephalogram signals obtained from the trainee in response to visual stimuli presented to the trainee; and (c) a memory that, when executed by the processor, (i) gives an instruction to cause the trainee to imagine performing an action, (ii) measures electroencephalogram signals with the electroencephalogram sensor device, (iii) calculates at least one characteristic selected from the following (1) to (3): (1) a concentration index, (2) a movement control index, (3) an arousal index, (iv) provides a feedback pattern to the trainee based on at least one of concentration, movement control, arousal, and movement readiness, and (v) stores instructions for repeating steps (c) to (e) as necessary.

[0022] Another object of the present invention is to disclose the above-described system adapted for planning and executing movements related to sports, surgery, aviation, or other activities.

[0023] In a system according to a further object of the present invention, a display configured to present visual stimuli to the trainee is provided.

[0024] In a system according to a further object of the present invention, the trainee is instructed to imagine performing an action in response to the presentation of visual stimuli.

[0025] In a system according to a further object of the present invention, the presentation of visual stimuli, the measurement of electroencephalogram signals by the electroencephalogram sensor device, and the provision of the feedback pattern are performed continuously.

[0026] In the system according to a further object of the present invention, the memory comprises an instruction for calculating a concentration index as a rate of change of the electroencephalogram signals of the vertex electrode and the frontal electrode at the alpha frequency, beta frequency, and theta frequency with respect to the electroencephalogram signals of the vertex electrode and the frontal electrode measured at rest.

[0027] In the system according to a further object of the present invention, the memory comprises an instruction for calculating a motor control index as a rate of change of the electroencephalogram signal in the sensorimotor area electrode obtained from the trainee in response to a visual stimulus with respect to the electroencephalogram signal of the sensorimotor area electrode measured at rest.

[0028] In the system according to a further object of the present invention, the memory comprises an instruction for calculating an arousal index as a rate of change of the electroencephalogram signal in the vertex electrode obtained with the eyes open of the trainee at the alpha frequency with respect to the electroencephalogram signal in the vertex electrode obtained with the eyes closed.

[0029] In the system according to a further object of the present invention, the memory comprises an instruction for analyzing at least one of the concentration index, motor control index, and arousal index of the trainee or a group consisting of trainees and presenting progress data of the training in a time series.

[0030] In the system according to a further object of the present invention, the feedback pattern is selected from the group consisting of a static avatar, a dynamic avatar, a text message, a sound pattern, a tactile pattern, and any combination thereof.

[0031] In the system according to a further object of the present invention, the feedback pattern relates to a visual environment related to an operation.

[0032] In the system according to a further object of the present invention, the visual environment is selected from the group consisting of a soccer stadium, a baseball field, a basketball court, a rugby stadium, an athletic stadium, and any combination thereof.

[0033] A system according to a further object of the present invention includes a memory containing an instruction to calculate a comprehensive index of readiness for sports, surgery, and flight control as a composite index obtained by normalizing at least two indexes selected from the group consisting of a concentration index, a motor control index, and an arousal index by their sum.

[0034] A further object of the present invention is to disclose a method for testing and training the brain function of a trainee to plan and execute movements in the system according to a further object of the present invention. This method includes: (a) providing the system according to claim 1 for testing and training the brain function of planning and executing movements; (b) instructing the trainee to imagine performing an action; (c) measuring an electroencephalogram signal in an electroencephalogram electrode device; (d) calculating a concentration index, a motor control index, and an arousal index; (e) providing the trainee with a feedback pattern characterizing at least one of concentration, motor control, arousal, and movement readiness; and (f) repeating steps (c) to (e) as necessary.

[0035] A method according to a further object of the present invention includes providing a display configured to display visual stimuli to a trainee.

[0036] In the method according to a further object of the present invention, the display of visual stimuli, the measurement of electroencephalogram signals in the electroencephalogram sensor device, and the provision of the feedback pattern are performed continuously.

[0037] In the method according to a further object of the present invention, the step of calculating the concentration index includes calculating the rate of change of electroencephalogram signals at the alpha frequency, beta frequency, and theta frequency of the parietal and frontal electrodes obtained from the trainee in response to visual stimuli with respect to the electroencephalogram signals at the alpha frequency, beta frequency, and theta frequency of the parietal and frontal electrodes measured at rest.

[0038] The method according to a further object of the present invention, the step of calculating the motor control index includes calculating the motor control index as the rate of change of the brain wave signal at the sensorimotor area electrode at the Mu frequency obtained from the trainee in response to visual stimulation, with respect to the brain wave signal of the sensorimotor area electrode measured at rest.

[0039] The method according to a further object of the present invention, the step of calculating the arousal index includes calculating the arousal index as the rate of change of the brain wave signal at the vertex electrode obtained in the open-eye state of the trainee at the alpha frequency, with respect to the brain wave signal at the vertex electrode obtained in the closed-eye state.

[0040] The method according to a further object of the present invention includes analyzing at least one of the concentration index, the motor control index, and the arousal index of the trainee or a group consisting of trainees, and presenting the training progress data in time series.

[0041] The method according to a further object of the present invention includes calculating a comprehensive index of the readiness for sports, surgery, and flight control as a composite index obtained by normalizing at least two indexes selected from the group consisting of the concentration index, the motor control index, and the arousal index by their sum.

Brief Description of the Drawings

[0042]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 4F

Figure 4G

Figure 4H

Figure 4I

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0043] The following describes the best mode contemplated by the inventors for practicing the present invention so that those skilled in the art can practice the method for using the present invention. However, the overall principle of the present invention is particularly defined to provide a system and method for testing and training the brain functions of a trainee who plans and executes physical movements, and various modifications obvious to those skilled in the art can be made. In this specification, the trainee includes athletes, women, surgeons and surgical trainees, pilots and student pilots, or those engaged in improving physical movements or undergoing tests.

[0044] <Testing and Training in Sports> The system and method of the present invention can provide coaches and athletes with relevant information (for example, how long a player can maintain high concentration, their periodic changes on a daily / weekly / monthly basis, etc.) for assembling an optimal training protocol. In addition, information regarding the brain states of various characteristics of athletes (for example, concentration, motor cortex volume) is useful for coaches to judge the states of athletes and team players during training, formulate strategies for competitions and games, and scout new team members.

[0045] Today, sports technology mainly targets the performance of athletes through physical strength, flexibility, and other physical training. Until recently, the brain, which is the main organ controlling movement, was at least not directly trainable. Therefore, there were limitations in improving athletes through normal training. It is important to understand that the brain controls each movement element from planning to execution. The brain formulates a movement plan for which muscles to contract, in what order, and with what strength (except for spinal reflexes) before almost all movements. This movement plan determines how close the actual movement will be to the athlete's ultimate goal.

[0046] For the user, the present invention has the following effects: (1) performance improvement, (2) assistance in recovery from injury, and (3) a unique approach to new individual key performance indicators (KPIs).

[0047] Through short training sessions once or several times a week, the user first learns how to strengthen specific brain patterns that control their movements. In this case, the readiness of the brain network can respond more quickly as needed, improving the speed and accuracy of performance. The main effect of neurofeedback training appears over time. That is, through the continuous adaptation of the neural network, the neural network is being reconstructed. As described above, these neural changes are similar to the natural changes associated with conventional training, but are characterized by being carried out at a higher intensity and having no risk of injury or fatigue.

[0048] At the end of each training session, various performance measurements are totaled and presented to the user. That is, the user's strengths and weaknesses, the changes in the user's performance over time, and the fields or goal settings preferred by the user are presented.

[0049] Players can also observe their performance in different sessions, confirm their performance in the game, and compare them with their performance in the field.

[0050] The present invention is different from the prior art in that it provides long-term effects based on data. The performance of the brain is enhanced not by the use of external stimuli, but by learning / training. Importantly, while the effects of external stimuli are limited to a very short time, any type of learning / training here has an effect that lasts for a long time. The feedback provided by the system of the present invention relates to the current brain state of the tested trainee. Such feedback allows the trainee to learn how to control the activation levels of these specific brain regions at will. These changes are stored in the brain in the same way as any learning process.

[0051] The system of the present invention is configured to record electroencephalogram signals related to the activation of specific brain regions related to movement, such as the primary motor cortex. For this reason, high-quality EEG hardware is used to ensure the collection of optimal data. By collecting such unique data from the user and analyzing it online and offline, a very individualized training environment for improving the user's performance is created.

[0052] Overall, the system of the present invention functions as a brain-computer interface (BCI) for sports and training, and includes (1) a brain signal recording device, (2) real-time signal analysis software, and (3) a user front end (a game-form training environment). The data thereof is described in "The Brain at Work" (Parasuraman and Rizzo, 2008).

[0053] Figure 1 shows a schematic diagram of a system for testing and training brain functions for planning and executing sports activities. Reference numeral 10 indicates the trainee. The system 100 includes a memory unit 50 that stores instructions for the processing unit 40. The processing unit is connected to an electroencephalogram (EEG) sensor device 20 that can be worn on the head of the trainee and can detect EEG signals generated on the surface of the trainee's head. The trainee 10 is instructed to imagine performing a predetermined sports activity 15 in response to visual stimuli shown on the display 30. The display of visual stimuli to the trainee 10 is performed simultaneously with the measurement of EEG signals in the EEG sensor device 20. According to the instructions stored in the memory unit 50, the processing unit calculates concentration indexes, motor control indexes, and arousal indexes (details will be described later). Next, a feedback message characterizing at least one of the above-mentioned concentration, motor control, arousal, and sports readiness is given to the trainee 10. The present invention includes feedback in the form of static avatars, dynamic avatars, text messages, sound patterns, and tactile patterns.

[0054] Referring to FIG. 2a, an exemplary EEG cap is made of stretchable synthetic fibers of various sizes. The cap holds sensors directly above the associated brain regions. The standard EEG cap is known as the "10-20 system". The "International 10-20 system" is recognized as a method for representing the positions of scalp electrodes. With the standard test system, the research results (clinical or research) of subjects can be reliably aggregated, reproduced, and effectively analyzed and compared using scientific methods. This system is based on the relationship between the positions of the electrodes and the regions below them, particularly the cerebral cortex.

[0055] Referring to FIG. 2b, in the EEG sensor device, each EEG sensor records the electric field below it. Neurons transmit information to each other through changes in electric charge. As a result, a difference occurs in the surrounding electric field. By analyzing this change, the functions of the brain can be read. The EEG sensor device includes sensors that can be worn as follows. (1) Frontal region electrodes located on the midline of the frontal lobe, (2) The vertex region electrode located on the midline of the parietal lobe (3 - 5) The sensorimotor cortex electrode attached to the motor cortex

[0056] Also, for signal collection, a ground sensor and a reference sensor are used.

[0057] It is desirable to maximize the signal - to - noise ratio by cleaning the lower surface of the sensor before measurement and electrically contacting the skin surface through a conductive gel as needed.

[0058] Referring to Figure 2c, the sensor is connected to an amplifier that amplifies the magnitude of the electrical signal. The amplifier is attached to the cap using a cord. Thereby, the amplified electroencephalogram signal is wirelessly transmitted to a computer, smartphone, or tablet connected by a USB connector. Most commercially available electroencephalogram signal amplifiers can be used in the present invention

[0059] EEG data is streamed to a processing unit and structured by software according to a preset sensor montage. Referring to Figure 2d, a noise - removal function is applied in the next step. For example, removing the electrostatic interference of the electrical network (notch filter). With this software, the user can check the connection level of each sensor and add gel if the impedance is too high or detect malfunction of the sensor.

[0060] EEG data is streamed using the LSL protocol (LabStreamingLayer) and acquired by data - analysis code.

[0061] Figure 3 shows a method 200 for testing and training the brain functions for planning and executing sports activities. In method 200, first, a system 100 (Figure 1) for testing and training the brain functions for planning and executing the above-described sports activities is prepared (step 210). After instructing the trainee to imagine performing a predetermined sports activity in response to the display of a visual stimulus (step 220), the above-described visual stimulus is displayed to the trainee to be tested while measuring the brain wave signal by the electroencephalogram sensor device (step 230). The obtained brain wave signal is processed to calculate a concentration index, a motor control index, and an arousal index (step 240). Based on the calculated indices, a feedback message is output that characterizes at least one of concentration, motor control, arousal, and sports readiness (step 250). Steps 220 to 250 are repeated as necessary.

[0062] EEG data is received via an LSL socket. Data analysis includes filtering. The raw EEG data is analyzed by shifting each cycle for each time window (e.g., a 50% shift of 500 samples or 1,000 samples). The purpose of the analysis is to extract the characteristics of the brain functions that contribute to the success of motor control.

[0063] A person's concentration and activity level in the motor cortex can be inferred from the variation in the power of a specific frequency band. These are detected by at least five scalp electrodes.

[0064] The calculation of the "concentration" (or brain involvement) index is performed according to the following algorithm. The raw data is first filtered by an IIR filter at the half-frequency of the frequency range [alpha: 8 - 11 Hz, beta: 16 - 22 Hz, theta 4 - 7 Hz] for the data from the sensors attached to the vertex (alpha) and the frontal lobe (beta and theta). The power of each frequency band (alpha, beta, and theta) is calculated from the corresponding filtered data using the "band power" method. The concentration index for each cycle is the ratio of the power of beta, theta, and alpha.

[0065] Threshold - Accuracy: At the start of each session, the system determines a baseline as a characteristic of each trainee. The trainee is instructed to sit still for several minutes (default is 2 minutes) before the indicated simulation with eyes open to create an open - eye baseline. The system collects metrics during the baseline collection and uses them to set a threshold for the trainee. In this way, a customized boundary value is set for each user. When the value is exceeded, the system determines that the user has a high level of concentration and can give positive feedback to the user.

[0066] According to an embodiment of the present invention, the above - mentioned threshold can be set as the sum of a lower limit value and the product of the difficulty level and the difference between the upper limit value and the lower limit value.

[0067] The upper limit value is the average of the metrics. The lower limit value is the value obtained by subtracting two standard deviations from the average of the metrics, and the difficulty level is a value set by the user to adjust the challenge level according to the user's ability.

[0068] Training using in - game feedback: During the game, metrics are continuously calculated and compared with the trainee's threshold baseline via the backend module. If the current value of the concentration metric exceeds the set threshold, the module sends a message for providing positive user feedback to the simulation game. As an example, in the case of soccer, the circle indicating the target of the kick shrinks, and by the user noticing it, the user can make a more accurate shot. In the current trial, if the user exceeds the threshold multiple times (for example, in soccer, at least 5 times within 8 seconds), intermittent feedback is given (in soccer, the ball is kicked accurately towards the target).

[0069] The algorithm for calculating the movement control index is as follows. First, the raw data is filtered by an IIR filter at the half-frequency of the frequency range [Mu: 12 Hz to 15 Hz] of the somatosensory area sensors such as C3, Cz, and C4 on the motor cortex. The power in the Mu frequency band and the data filtered from these channels are calculated by the band power function. The indices from the somatosensory area sensors such as C3 and C4 are then used to evaluate the current Mu desynchronization, which is the movement preceding the movement.

[0070] Threshold - kick force: All Mu power indices calculated during the acquisition of the baseline data with eyes open from the somatosensory areas such as the C3 and C4 electrodes are used to evaluate the trainee pattern of the Mu rhythm of a specific user.

[0071] For example, the average Mu power at positions C3 and C4 (above the left motor cortex controlling the right hand and foot or vice versa) is used as the motor brain activity threshold.

[0072] Training based on feedback during the game: During the game, the current Mu power is constantly compared with the average value of the data collected at the baseline with eyes open. Specifically, the feedback is defined as positive when the sum of the instantaneous Mu power and the product of the difficulty level and the STD of MU at the baseline is smaller than the average MU at the reference time.

[0073] "Mu power" represents the instantaneous Mu value. The difficulty is a value set by the user to adjust the challenge level according to the user's ability. The STD of Mu at the baseline is the standard deviation of all Mu indices collected at the baseline with eyes open. If adding a part of the standard deviation to the current Mu power results in a value lower than the average Mu at the baseline, positive feedback is given because of the activation of the motor cortex.

[0074] In soccer training, a circular bar is gradually filled and changes color from red to green (through yellow and orange). The filling of the bar indicates to the user how to perform neural actions to better activate these areas. If the user is able to "fill" this bar in one attempt, the power of the kick is strong enough to score a point. Similar criteria are applied to training and scouting in basketball, hockey, golf, American football and other sports games that require quick reactions and shooting accuracy. Measurement of physical vigor and ability to exercise attention is also important in the training process of various sports such as long jump, high jump, javelin, hammer throw and discus.

[0075] Only if both feedbacks (motor control and concentration) are simultaneously and sufficiently controlled by the user can the avatar (the on-screen soccer player) score a goal.

[0076] The algorithm for calculating the wakefulness (sleep) index is as follows: Alpha band power at the central parietal sensor (e.g., Pz) is collected at an eyes-closed baseline (a few minutes (default 2 minutes), immediately following the eyes-open baseline) and compared to the eyes-open baseline of alpha power. Alpha power at this location is known to be related to the user's level of wakefulness. High alpha power is typically associated with fatigue.

[0077] Threshold - Sleep Detection: Calculate and compare moving averages of baseline data for both eyes open and eyes closed. First, average over one time window, then increase the interval until the moving average for eyes closed is at least two standard deviations higher than the alpha value for eyes open. The moving average for the smallest time interval that meets this condition is set as the threshold for "sleep detection."

[0078] Feedback: During the game, the current alpha power in the central vertex sensor is constantly compared to a threshold. When the threshold is exceeded, a warning is given (in the case of soccer, the countdown numbers turn red). If multiple sleep indices are detected during a trial, that trial is disqualified (in soccer, the player drops the ball with their foot or faints on the grass. Then, an audio message saying "Wake up!" plays).

[0079] As mentioned above, soccer is the first example, but the same applies to other sports and training environments.

[0080] By using advanced ML tools such as clustering algorithms, SVM classifiers, and artificial neural networks that utilize data from early adopters in this field, a powerful pattern detection mechanism can be created that provides users with highly user-specific, EEG-noise-robust, and rich data-driven insights.

[0081] The last element of the system of the present invention is a training environment for the user. The interface displays the user's brain activity in real time. This is essential in the neurofeedback learning process and closes the loop starting from the brain data acquired using EEG. The inventors designed the product so that athletes can train important brain functions for improving performance. So far, the basic intracranial processes monitored by the system of the present invention, namely, the images of movements at a high level of concentration, have been described. These intracranial processes are common to different types of sports. However, in current research, it has been shown that the efficiency of the feedback environment for this type of neurotraining is greatly improved by an appropriate learning environment. Therefore, it is necessary to construct each virtual environment for each type of sport. The purpose is to enable people to train in a familiar environment and, ultimately, to efficiently reflect the learned content in "real-world" sports competitions and the like.

[0082] The interface segment of the present invention is in a high-context computer game format environment. For example, the first environment of the present invention is a soccer trainer, and the first task is a free kick towards the goal.

[0083] Figures 4a - 4i show an example of a change in visual stimuli that improves test metrics. Specifically, the trainee is instructed to sit still and concentrate on the monitor and imagine kicking a ball towards a target set on the screen. In each trial, the system gives the player 8 seconds. The neural activity of the actual "kick" and the imagined "kick" is very similar. When this system detects strong activation in a specific band in a specific brain region related to this movement, the index of kick power increases (as shown in the figure below). Due to the correlation between this neural activation and the symbols shown on the screen, the user can intuitively control the activities of the neural network as described above, which is most important for controlling their own leg during a kick.

[0084] Through the accumulation of practice, the user learns how to control these brain regions, but at the same time, it also causes changes in their "connections". The brain is an organ that is constantly changing. The connections of neurons change, and the neural network is created or strengthened by being constantly activated. Such changes are the components of learning and lead to improvement through training and repetition.

[0085] When the player imagines the action of kicking the ball, the trainee receives continuous feedback regarding this activity. The change in the circular bar (8 is still empty, 0 is full power) and color indicate high activation, and the number in the center of the circle indicates a countdown to inform the user of the timing of the kick. Only when the user reaches sufficient power can the avatar score a goal.

[0086] Activation of the network through repetitive training using this brain trainer can avoid many problems in conventional training while causing activities equivalent to actual field activities.

[0087] Figures 5a to 5d show other embodiments of the present invention. In this game, the trainee can control the accuracy of the kick. The accuracy of the kick is controlled by the player's level of involvement and concentration. When the system recognizes the user's concentration, the circle surrounding the target is gradually locked onto the target. The ball is shot towards the goal frame only when the white circle is locked onto the target.

[0088] In this way, the user obtains continuous feedback on two mental processes that could not be felt before. The second form of feedback is called "intermittent feedback". When both inputs can be controlled to a sufficient level, the kick to be kicked after 8 seconds is successful and scores. If only one condition is met, it is a failure. During this trial (one kick), if the player cannot reach a sufficient level of concentration, the ball cannot be kicked accurately. If the user cannot obtain sufficient power through the motor imagery trial, the kick becomes weak and is stopped by the goalkeeper.

[0089] By training over time, the majority of users learn how to adjust these neural processes at will, and important changes occur in the neural network related to the player's performance.

[0090] When the player reaches sufficient power and accuracy, the avatar scores a goal. Scoring a goal is inherently very rewarding for a soccer player. In the brain, there is a general "rule" that neural activity leading to reward results in stronger connections and an increase in resources related to this activity. Such changes increase the speed and accuracy of the player's movements, giving a competitive advantage.

[0091] At the end of the session, the user can observe their performance and check for changes in their ability to concentrate during the game and activate the motor areas of the brain.

[0092] Through the interface, the coach can assemble training sessions, for example, the number of repetitions, position, leg to use, etc. Then, the player and the coach can use a specially designed interface presented at the end of the session to observe the performance achieved by the player in the most recent session, compare it with past sessions, and perform other important analyses. Here, statistical values based on the results of the training are shown. This enables the user, coach, or supervisor to grasp the overall picture of the player's abilities and progress using the brain data of the trainee.

[0093] The object of the present invention is to create a basis for developing a series of neuro - interface applications for various sports, rehabilitation methods, and training, whereby the user can improve their motor ability. These products are designed taking into account the needs of each target group of users. The procedures of the present invention are applicable to basketball, American football, hockey, racing, golf, tennis, other sports, and the rehabilitation process.

[0094] <Surgeon's Tests and Training> Surgeons use sophisticated instruments for long periods of time and often communicate with nurses and anesthesiologists under time pressure, operating complex monitor interfaces. Surgeons have technical skills acquired through long training. They also utilize non-technical skills (Yule et al., 2008). These include situation awareness (collection and understanding of information, prediction of future states) and task management (responding to changes). Planned action, for example, is deciding whether to change a laparoscopic operation to an open surgery. When the main tasks (such as suturing) are accompanied by difficulties different from normal, there is a possibility that the detection of important alarms may not work well (Frederic Dehais et al., 2014), or appropriate planning may be impaired. Even almost automated mental processes, such as camera angle correction (Klein, Riley, Warm, & Matthews, 2005) and the mismatch between the optical axis of the endoscope and the placement of the instrument on the monitor (Patil, Hanna, & Cuschieri, 2004), may potentially deprive resources from the overall function of the surgeon. Changes in mental workload due to training or new instrument design have a wide impact not only on efficiency but also on patient treatment outcomes. Kinematic and physiological measurements are useful for monitoring the workload of surgeons. When measuring the workload of surgeons, a hybrid or multimodal approach is more desirable than a single method. This is because these methods can provide more information to clarify the operator's function from multiple perspectives. Different measurement methods have different strengths and weaknesses and can complement each other's drawbacks. Furthermore, as hardware miniaturization progresses and sensor design improves, the cost and effort of introducing additional modalities are decreasing (Gramann et al., 2011). Yurko et al. (2010) used NASA-TLX to analyze the performance of novice trainees in laparoscopic surgery and explain the degree of transfer of skills acquired in the simulator to the operating room (OR).Such systems and methods can provide relevant information to surgeons, their trainers, and educators for assembling optimal training protocols (e.g., how long a surgeon can maintain a high level of concentration and how it changes periodically over days / weeks / months, etc.). Additionally, information regarding various characteristics of a surgeon's brain state (such as concentration, motor cortex ability, etc.) can also be useful for surgical instructors and surgical directors to judge the state of trainees and test the suitability of candidates for this highly demanding profession that requires great manual dexterity.

[0095] The systems and methods of the present invention can, by appropriate application, provide surgeons and dentists with relevant information for assembling optimal training protocols (e.g., how long a surgeon can maintain a high level of concentration and the periodic changes in the high-concentration state on a daily / weekly / monthly basis, etc.). Information on the brain state regarding various characteristics of a surgeon (such as concentration, motor cortex ability, etc.) is useful for surgical educators, examiners, and supervisors in determining the conditions, suitability as an expert, and evaluation of trainee surgeons, operating room (OR) staff, and surgical teams.

[0096] Similar to the case of sports, before almost all types of movements (excluding spinal reflexes) performed by surgeons and dentists, the mind makes a motor plan (which muscles to contract, in what order, and with what strength). This motor plan determines how useful the actual movement is for the doctor's ultimate goal.

[0097] The use of the solution of the present invention has the following effects on surgical PI doctors: (1) improvement in performance; (2) a unique approach to new individual KPIs (Key Performance Indicators).

[0098] KPIs for surgeons can include clearly defined performance indicators that are used to observe, analyze, optimize, and transform a surgeon's processes in order to equally enhance the satisfaction of both patients and healthcare providers. These indicators are commonly used by healthcare facilities to compare their performance with other facilities and identify areas for improvement. For example, the error rate in the operating room measures the number of mistakes made by the surgeon during patient treatment. The error rate can be expressed as (number of treatment errors / total number of treatments) * 100.

[0099] Embodiments of the present invention disclose a system for testing and training brain functions that plan and execute actions. The system includes (a) an electroencephalogram sensor device wearable on the head of a trainee, (b) a processor configured to receive and analyze electroencephalogram signals acquired from the trainee in response to visual stimuli presented to the trainee, and (c) a memory that, when executed by the processor, (i) gives an instruction to cause the trainee to imagine performing an action, (ii) measures electroencephalogram signals with the electroencephalogram sensor device, (iii) calculates at least one characteristic selected from the following (1) to (3): (1) a concentration index, (2) a movement control index, (3) an arousal index, (iv) provides a feedback pattern to the trainee based on at least one of concentration, movement control, arousal, and action readiness, and (v) repeats steps (c) to (e) as necessary, and stores instructions therefor.

[0100] By implementing a system and method for testing and training the brain functions that plan and execute hand movements of the present disclosure, the construct validity can be demonstrated. The system of the present invention can be used to examine and train psychomotor ability, visual-spatial ability, and perceptual ability, and can be used to associate with objective tests such as those in which the predictive performance on surgical performance has already been shown for such basic abilities. The functional involvement of psychomotor ability in the adaptation, fixation, and development of skills in endoscopic surgery has been demonstrated (Gallagher AG, McClure N, McGuigan J, et al.). Ergonomic analysis of the fulcrum effect in the acquisition of endoscopic skills. Endoscopy 1998;30:617-20. A comprehensive index for the entire surgery (endoscopic sinus surgery) was developed and culminated in ES3 developed by Lockheed Martin to teach the essentials of ESS operation to otolaryngology residents (Rudman DT, Stredney D, Sessanna D et al. Training simulator for functional endoscopic sinus surgery, Laryngoscope 1998;108:1643-7, Edmond CV, Heskamp D, Sluis D, et al. Training simulator for ENT endoscopic surgery, In: Morgan KS, eds. The integration of medicine and virtual reality, Amsterdam: IOS Press, 1997:518-28.25 Wiet GJ, Yagel R, Stredney D, et al.) A three-dimensional approach to virtual simulation of functional endoscopic sinus surgery (Stud Health Technol Inform 1997;39:167-79).

[0101] ES3 mainly consists of the following four components. A computer manufactured by Silicon Graphics that functions as a simulation host platform, a haptic system controller PC that performs the necessary high-speed control of the physical instrument handle related to the virtual surgical instrument, a virtual voice recognition instructor PC that responds to voice commands for controlling the simulator, a physical replica of the endoscope, a mechanically connected surgical hand tool handle, and an electromechanical platform that houses a mannequin of the external head anatomy.

[0102] Reduce errors and receive feedback on performance in the following areas through laparoscopic surgery and surgical simulation corresponding to the functional aspect. 1. As a simulation, grasp the tissue, transfer it from one gripper to the other, pass the intestine through hand-to-hand transfer, remove the tool from the surgical field, accurately reinsert it, cauterize three subtargets, and maintain the object in the target box while cauterizing three consecutive subtargets. 2. Visual spatial ability: Evaluated using card rotation, cube comparison, and map planning tests from the factor reference cognitive test kit. These tests assess the subject's understanding of various spatial arrangements of objects. 3. Perceptual evaluation: Measured by a test called PicSOr (pictorial surface orientation). Each item is an image of the tip of a rotating arrow touching the surface of a cube or sphere drawn on a computer monitor. The subject moves the arrow (using the cursor keys) until the axis of the arrow is perpendicular to the surface of the object at the point where the arrow touches the surface. This relatively purely tests the subject's ability to restore the pictorial clues identifying the direction of the structure in the (virtual) image space and compare the implied orientation. The most important indicators of performance are the correlation between the theoretically correct arrow orientation and the setting chosen by the subject, and the slope of the fitted regression line.

[0103] The results and predictions of the system and method of the present invention can be evaluated for internal validity and consistency, and using the ES3 described herein, and can be linked to other reference measurements of the cognitive and psychomotor abilities of the surgeon in training. The types of errors can be quantitatively defined based on the following metrics. An incorrect operation, something outside the tolerance of the tissue or instrument. The instrument operation was performed correctly, but the order is off or it is inappropriate for the surgical site. An inefficient application or use of force, an inefficient operation or sequence of operations. Inappropriate technical variability in performance. Inappropriate "downtime" or "poor progress" indicating confusion or disorientation. By appropriately combining with a standard system such as ES3, the present system and method can provide accurate context-based analysis and feedback to surgical students regarding error recognition and correction, in addition to objective comparison.

[0104] The database is the basic unit for integrating the project. The metrics component identifies quantifiable measurements, which become fields in the database. The system of the present invention can acquire measurements during training and transmit them to a database in an automated and standardized format, including web-based ones.

[0105] The data may show the overall evaluation results regarding the technical skills of the surgeon in training. By evaluating these statistics together with the stored measurements of other cognitive and interpersonal skills, primary metrics in the overall competency assessment can also be generated. The present system and method can support the training and evaluation of surgeons and OR teams by providing data in a recursive and iterative feedback cycle. Furthermore, it can assist in population-based analysis based on demographic data, training, and performance for many simulation procedures and groups of surgeons, and define the parameters of competency, skills, and training for submission to appropriate surgical councils, societies, state, and federal agencies for the purposes of certification, regulation, and policy-making.

[0106] Once a week or through several short training sessions, trainee doctors or qualified surgeons first learn how to enhance specific brain patterns that control their movements. In this case, the readiness of the brain network can respond more quickly as needed, improving the speed and accuracy of performance. The main effects of neurofeedback training appear over time. That is, through the continuous adaptation of the neural network, the neural network is being reconstructed. As mentioned above, these neural changes are similar to the natural changes associated with conventional training, but are characterized by being carried out at a higher intensity and without the risk of injury or fatigue.

[0107] At the end of each training session, various performance measurements are totaled and presented to the user. That is, the user's strengths and weaknesses, the changes in the user's performance over time, as well as the fields or goal settings preferred by the user are presented.

[0108] Players can also observe their performance in different sessions, confirm their performance in the game, and compare them with their performance in the field.

[0109] The present invention is different from the prior art in that it provides long-term effects based on data. The performance of the brain is enhanced through learning / training rather than by the use of external stimuli. It should be emphasized that while the effects of external stimuli are limited to a very short time, any type of learning / training here has a long-lasting effect. The feedback provided by the system of the present invention relates to the current state of the brain of the tested trainee. Through such feedback, the trainee can learn how to control the activation level of these specific brain regions at will. These changes are stored in the brain in the same way as any learning process.

[0110] The system of the present invention is configured to record electroencephalogram signals related to the activation of specific brain regions related to movements such as the primary motor area. For this reason, high-quality EEG hardware is used to ensure the collection of optimal data. By collecting such unique data from users and analyzing it online and offline, a highly individualized training environment is created to improve the performance of users.

[0111] Overall, the system of the present invention functions as a brain-computer interface (BCI) for surgical training and includes (1) an electroencephalogram signal recording device, (2) real-time signal analysis software, and (3) a user front-end (a training environment in the form of an operating room).

[0112] FIG. 1 shows a schematic diagram of a system for testing and training brain functions for planning and performing surgeries and operations. In FIG. 1, reference numeral 10 indicates a trainee who is to be tested. The system 100 includes a memory unit 50 that stores instructions for the processing unit 40. The processing unit is connected to an electroencephalogram sensor device 20 that can be worn on the head of the trainee and can detect electroencephalogram signals generated on the surface of the trainee's head. The trainee 10 is instructed to imagine performing a predetermined surgery 15 in response to a visual stimulus shown on the display 30. The display of the visual stimulus to the trainee 10 is performed simultaneously with the measurement of the electroencephalogram signals in the electroencephalogram sensor device 20. According to the instructions stored in the memory unit 50, the processing unit calculates a concentration index, a motor control index, and an arousal index (details will be described later). Next, a feedback message characterizing at least one of the above-mentioned concentration, motor control, arousal, and sports readiness is given to the trainee 10. The present invention includes feedback in the form of static avatars, dynamic avatars, text messages, sound patterns, and tactile patterns.

[0113] Referring to FIG. 2a, an exemplary electroencephalogram (EEG) cap is made of stretchable synthetic fibers of various sizes. The cap holds sensors directly above the associated brain regions. The standard EEG cap is known as the "10-20 system". The "International 10-20 system" is recognized as a method for representing the positions of scalp electrodes. With this standard test system, the research results (clinical or research) of subjects can be reliably aggregated, reproduced, and effectively analyzed and compared using scientific methods. This system is based on the relationship between the positions of the electrodes and the regions beneath them, particularly the cerebral cortex.

[0114] Referring to FIG. 2b, in an EEG sensor device, each EEG sensor records the electric field beneath it. Neurons communicate with each other through changes in electric charge. As a result, a difference occurs in the surrounding electric field. By analyzing this change, the functions of the brain can be read. The EEG sensor device is equipped with sensors that can be worn as follows. (1) Frontal region electrodes located on the midline of the frontal lobe, (2) Parietal region electrodes located on the midline of the parietal lobe, (3 - 5) Somatosensory cortex electrodes attached to the motor cortex.

[0115] Also, for signal collection, a ground sensor and a reference sensor are used. The lower surface of the sensor is preferably cleaned before measurement and electrically contacted with the skin surface through a conductive gel as needed to maximize the signal-to-noise ratio.

[0116] Referring to FIG. 2c, the sensor is connected to an amplifier that amplifies the magnitude of the electrical signal. The amplifier is attached to the cap using a cord. Thereby, the amplified EEG signal is wirelessly transmitted to a computer, smartphone, or tablet connected by a USB connector. Most commercially available EEG signal amplifiers can be used in the present invention.

[0117] EEG data is streamed to the processing unit and structured by software according to a preset sensor montage. Referring to FIG. 2d, a noise removal function is applied in the next step. For example, electrostatic interference in the electrical network is removed (notch filter). This software allows the user to check the connection level of each sensor and add gel if the impedance is too high or detect malfunction of the sensor.

[0118] EEG data is streamed using the LSL protocol (LabStreamingLayer) and acquired by the data analysis code.

[0119] FIG. 3 shows a method 200 for testing and training brain functions for planning and performing surgery. In method 200, first, a system 100 (FIG. 1) for testing and training brain functions for planning and performing the above-described surgery is prepared (step 210). After instructing the trainer to imagine performing a predetermined surgery in response to the display of a visual stimulus (step 220), while displaying the above-described visual stimulus to the trainee to be tested, an electroencephalogram signal is measured by the electroencephalogram sensor device (step 230). The obtained electroencephalogram signal is processed to calculate a concentration index, a motor control index, and an arousal index (step 240). Based on the calculated indices, a feedback message characterizing at least one of concentration, motor control, arousal, and surgical readiness is output (step 250). Steps 220 to 250 are repeated as necessary.

[0120] EEG data is received via an LSL socket. Data analysis includes filtering. The raw EEG data is analyzed by shifting (e.g., 50% shift with 500 samples or 1,000 samples) every cycle for each time window. The purpose of the analysis is to extract the characteristics of brain functions contributing to the success of motor control.

[0121] The concentration and activity levels of a person in the motor cortex can be inferred from the fluctuations in the power of specific frequency bands. In this system, these are detected by at least five scalp electrodes.

[0122] The calculation of the "concentration" (or brain involvement) index is performed according to the following algorithm. The raw data is first filtered by an IIR filter at the half-power frequencies of the frequency ranges [alpha: 8 - 11 Hz, beta: 16 - 22 Hz, theta: 4 - 7 Hz] for the data from sensors attached to the vertex (alpha) and frontal (beta and theta) regions. The power of each frequency band (alpha, beta, and theta) is calculated from the corresponding filtered data using the "band power" method. The concentration index for each cycle is the ratio of the powers of beta, theta, and alpha.

[0123] Threshold - accuracy: At the start of each session, the system determines a baseline as a characteristic of each trainee. The trainee sits still with eyes open for several minutes (by default, 2 minutes) before the instructed surgical activity or OR simulation to create an open - eye baseline. The system collects the concentration index for the baseline and uses it to set a threshold for the trainee. In this way, a customized boundary value is set for each user. When the value is exceeded, the system can determine that the user has a high concentration and provide positive feedback to the user.

[0124] According to an embodiment of the present invention, the above - mentioned threshold can be set as the sum of a lower limit value and the product of the difficulty level and the difference between the upper limit value and the lower limit value.

[0125] The upper limit value is the average of the index. The lower limit value is the value obtained by subtracting two standard deviations from the average of the index, and the difficulty level is a value set by the user to adjust the challenge level according to the user's ability.

[0126] Training Using Surgical or Procedural Feedback: During a surgery or procedure, metrics are continuously calculated and compared via a backend module to a trainee's threshold baseline. If the current value of a concentration metric exceeds a set threshold, the module sends a message to provide positive user feedback for a simulation game. For example, in a suturing, incision, or removal procedure, a circle indicating the target of the suturing, incision, or removal procedure shrinks, and as the user notices this, the surgery becomes more accurate. In the current trial, if the user exceeds the threshold multiple times (e.g., in surgery, performs a predetermined number of sutures within a predetermined number of seconds), intermittent feedback (good suturing procedure) is given.

[0127] The algorithm for calculating the motion control metric is as follows. First, raw data is filtered by an IIR filter at the half-frequency of the frequency range [Mu: 12 Hz - 15 Hz] of motor-sensory area sensors such as C3, Cz, C4, etc. on the motor cortex. The power of the Mu frequency band and the data filtered from these channels are calculated by a band-power function. Metrics from motor-sensory area sensors such as C3 and C4 are then used to evaluate the current Mu desynchronization, which is the movement preceding the movement.

[0128] Threshold - Force Applied to a Manual Surgical Instrument: All Mu power metrics calculated during the acquisition of baseline data with eyes open from motor-sensory areas such as C3 and C4 electrodes are used to evaluate the trainee pattern of the Mu rhythm of a specific user.

[0129] For example, the average Mu power at positions C3 and C4 (above the left motor cortex controlling the right limb or vice versa) is used as the motor brain activity threshold.

[0130] Training based on feedback of actions or procedures: During an action or procedure, the current Mu power is constantly compared with the average value of the data collected at the baseline with eyes open. Specifically, the feedback is defined as positive when the sum of the instantaneous Mu power and the product of the difficulty level and the STD of MU at the baseline is smaller than the average MU at the reference time.

[0131] "Mu power" represents the instantaneous Mu value. The difficulty level is a value set by the user to adjust the challenge level according to the user's ability. The STD of Mu at the baseline is the standard deviation of all Mu metrics collected at the baseline with eyes open. If the current Mu power plus a part of the standard deviation is lower than the average Mu at the baseline, positive feedback is given because of the activation of the motor cortex.

[0132] In surgical training, a circular bar is gradually filled and the color changes from red to green (passing through yellow and orange). As the bar is filled, the user is shown how to perform the nerve actions that better activate these areas. If the user can "fill" this bar in one trial, the force applied to the surgical manual instrument is strong enough to perform a given hand movement. Similar criteria apply to the selection and training in any surgery that requires quick and sure accuracy. Also, the ability to exert physical vitality and attention is important in the training process of general surgery, plastic surgery, cardiac surgery, thoracic surgery, neurosurgery, ophthalmic surgery, dental surgery, veterinary surgery, and other types of surgery.

[0133] The avatar (the surgeon on the screen) performs the procedure only when both types of feedback (motor control and concentration) are controlled simultaneously and sufficiently by the user.

[0134] The algorithm for calculating the wakefulness (sleep) index is as follows: Alpha band power at the central parietal sensor (e.g., Pz) is collected at an eyes-closed baseline (a few minutes (default 2 minutes), immediately following the eyes-open baseline) and compared to the eyes-open baseline of alpha power. Alpha power at this location is known to be related to the user's level of wakefulness. High alpha power is typically associated with fatigue.

[0135] Threshold - Sleep Detection: Calculate and compare moving averages of baseline data for both eyes open and eyes closed. First, average over one time window, then increase the interval until the moving average for eyes closed is at least two standard deviations higher than the alpha value for eyes open. The moving average for the smallest time interval that meets this condition is set as the threshold for "sleep detection."

[0136] Feedback: During surgery, the current alpha power at the central parietal sensor is constantly compared to a threshold. If the threshold is exceeded, a warning is given (in surgery, the countdown numbers may turn red). If multiple sleep indices are detected during a trial, the trial is disqualified (in surgery, if the surgeon makes a mistake and the steps on the screen are incomplete, a warning is given and an audio message is played saying "Wake up!").

[0137] By using advanced ML tools such as clustering algorithms, SVM classifiers, and artificial neural networks, using data from early adopters in the field, we can create powerful pattern detection mechanisms that are highly user-specific, robust to EEG noise, and provide rich data-driven insights to users.

[0138] The last element of the system of the present invention is a training environment for the user. The interface displays the user's brain activity in real time. This is essential in the neurofeedback learning process and closes the loop starting from the brain data acquired using EEG. The inventors have designed the product so that medical students and surgeons can train important brain functions to improve their performance. So far, the basic intracerebral processes monitored by the system of the present invention, namely, the imagery of movement at a high level of concentration, have been described. These intracerebral processes are common in different types of surgeries. However, in current research, it has been shown that the efficiency of the feedback environment for this type of neurotraining is greatly improved by an appropriate learning environment. Therefore, for each type of surgery, a respective virtual environment is constructed. The purpose is to enable people to train in a familiar environment and, ultimately, to efficiently reflect the learned content in surgeries in the "real world" operating room.

[0139] The interface segment of the present invention is a surgical environment in the form of a computer game designed for high context. For example, the first environment of the present invention is an intestinal surgery that removes growths in the intestine and sutures the intestine safely and effectively.

[0140] Figures 4a to 4i show an example of the change in visual stimuli that improves the test index. Specifically, the surgeon is instructed to sit still and concentrate on the monitor and imagine removing the target tumor on the screen. In each trial, the system gives the surgeon a predetermined number of seconds to perform the action. The neural activities of the actual resection and the imagined resection are very similar. When the system detects strong activation in a specific band in a specific brain region related to this action, the index of the force applied to the surgical hand tool increases (as shown in the figure below). Due to the correlation between this neural activation and the symbols shown on the screen, the user can intuitively control the activities of the neural network as described above, which is most important for controlling the hand during the operation.

[0141] Through repeated practice, users learn how to control these brain regions, while at the same time causing changes in their "connections". The brain is an organ that is constantly changing. Neuronal connections change, and neuronal networks are created or strengthened by being constantly activated. Such changes are the components of learning and lead to improvement through training and repetition.

[0142] For example, when a player imagines the action of excising the area between healthy tissue and tumor tissue, the trainee receives continuous feedback regarding this activity. The change in the circular bar (8 is still empty, 0 is full power) and color indicate high activation, and the number in the center of the circle indicates a countdown that informs the user of the start timing of the excision as an irreversible act. Only when the user reaches sufficient power can the avatar excise the tissue.

[0143] Activation of the network through repeated training using this brain trainer can avoid many problems in conventional practice and can cause activities equivalent to actual field activities.

[0144] Figure 6 shows another embodiment of the present invention regarding the operation of a surgical device in laparoscopic surgery. In this game, the trainee can control the accuracy of the laser beam. The accuracy is controlled by the player's level of involvement and level of concentration. When the system recognizes the user's concentration, the circle surrounding the target is gradually locked onto the target. Only when the white circle is locked onto the target does the laser intensity increase to the surgical level required to remove the tissue.

[0145] In this way, the user obtains continuous feedback regarding two mental processes that could not be sensed hitherto. The second form of feedback is what is called "intermittent feedback". As a result of the effort, if both types of input can be controlled at a sufficient level, then at the end of a predetermined allotted time, the manual operation pressure applied to the surgical instrument will be appropriate, and the intended result of the work using the instrument will be achieved. If only one of the conditions is met, the surgery will fail. During the trial, if the surgeon cannot reach a sufficient level of concentration, the accuracy of the surgery will decrease. If the user cannot exert sufficient force through the trial of motor imagery, for example, the tissue will not be excised correctly.

[0146] By training over time, the majority of users learn how to adjust these neural processes voluntarily, and significant changes occur in the neural networks related to the player's performance.

[0147] When the surgeon reaches sufficient power and accuracy, the avatar performs the surgery and the trainee is given a warning. Successfully performing an important surgery is very rewarding for the surgeon. In the brain, there is a general "rule" that neural activity leading to reward results in stronger connections and an increase in the resources related to this activity. These changes improve the speed and accuracy of the surgeon when performing the operation, leading to an improvement in subjective confidence and objective performance.

[0148] At the end of the session, the user can observe their performance and confirm the changes in the ability to concentrate during the progress of the operation and activate the motor areas of the brain.

[0149] Through the interface, the surgical trainer can assemble training sessions such as the number of repetitions, position, type of complication, and simulated emergencies. Then, the trainee and the trainer or supervisor can use a specially designed interface presented at the end of the session to observe the performance achieved by the surgeon in the most recent session, compare it with past sessions, or perform other important analyses. Here, statistical values based on the training results are presented. This enables the trainee surgeon and the supervisor, trainer, or supervisor to grasp the overall picture of the surgeon's capabilities and progress using the brain data obtained from the training.

[0150] <Aviation Testing and Training> It has been proposed to improve the basic neurocognitive processes of pilots during flight and to enhance safety and efficiency in human-machine collaboration. This can be achieved by (i) improving human performance and linking it to improved "on-the-job" functions, (ii) using it to assist in the design of complex systems, or (iii) dynamically adjusting the user interface and task parameters during use.

[0151] Pilots cope with uncertain environments and engage in complex interactions with the flight deck (Causse et al., 2013; Cakir et al., 2016; Reynal et al., 2016). For example, several studies have reported that pilots' working memory (WM) is particularly used for processing flight paths, monitoring flight parameters, and maintaining awareness of new situations (Causse et al., 2011a, b). WM is also an important factor when following air traffic control (ATC) instructions (Morrow et al., 1993). This activity actually requires memorizing flight parameters (e.g., heading, altitude, speed) to follow the appropriate flight path. However, it is well known that human working memory has fundamental limitations (Baddeley, 1992; Miller, 1994) and is easily overwhelmed when task demands become excessive (Durantin et al., 2014a). Human factors research has reported that various environmental stress factors can negatively affect pilots' performance in following ATC clearances (Billings and Cheaney, 1981; Taylor et al., 1994, 2005; Scerbo et al., 2003; Risser et al., 2006; Rome et al., 2012; Dehais et al., 2017). Therefore, introducing monitoring technologies that infer cognitive limits in the cockpit could be a promising approach to enhancing flight safety (Roy et al., 2017; Verdiere et al., 2018).

[0152] In fact, the development of brain-computer interface (BCI) technology has shown an interesting prospect of continuously monitoring and utilizing the neural mechanisms underlying brain dynamics and cognition. Among the three categories of BCI (active, reactive, passive) (Zander and Kothe, 2011; Vecchiato et al., 2016), the first two types aim to convert brain activities into messages or commands to spontaneously control remote devices (such as a mouse cursor). Passive BCI (pBCI) has received particular attention in neuroergonomics applications (Cutrell and Tan, 2008; Frey et al., 2017; Gramann et al., 2017). These enable the derivation of various mental states by leveraging the interpretation of unlabeled brain activities during tasks (Blankertz et al., 2010; Roy et al., 2013; Van Erp et al., 2015; Zander et al., 2017). These mental state inference systems have provided unique insights into the development of human-system interaction for overcoming cognitive limitations (Zander and Kothe, 2011; Brouwer et al., 2013). Although some pBCIs have been successfully implemented in driving (Dijksterhuis et al., 2013) and flight simulators (Gateau et al., 2015; Arico et al., 2016; Cakir et al., 2016; Callan et al., 2016; Verdiere et al., 2018), few attempts have been made to test these systems in more realistic settings. On the other hand, very few studies have attempted to test these adaptive systems in realistic settings (Callan et al., 2015).

[0153] Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have recently gained popularity with the miniaturization, portability, and wirelessness of sensors (Ayaz et al., 2013; Strait et al., 2014; Naseer and Hong, 2015; Schudlo and Chau, 2015).

[0154] The present invention is different from the prior art in the field of pilot training in that it brings long-term effects to the pilot based on data. The performance of the brain is enhanced by learning / training rather than by the use of external stimuli. It should be emphasized that while the effect of external stimuli is limited to a very short time, any type of learning / training here has an effect that lasts for a long time. The feedback provided by the system of the present invention relates to the current brain state of the tested trainee. Such feedback enables the trainee to learn how to control the activation level of these specific brain regions at will. These changes are stored in the brain in the same way as any learning process.

[0155] The system of the present invention is configured to record electroencephalogram signals related to the activation of specific brain regions related to movement, such as the primary motor cortex. For this purpose, high-quality EEG hardware is used to ensure the collection of optimal data. By collecting such unique data from the user and analyzing it online and offline, a very individualized training environment is created to improve the user's performance.

[0156] Overall, the system of the present invention functions as a brain-computer interface (BCI) for pilot training and comprises (1) a brain signal recording device, (2) real-time signal analysis software, and (3) a user front end (a training environment in the form of a cockpit).

[0157] Figure 1 shows a schematic diagram of a system for testing and training the brain functions that plan and execute the operation and control of a pilot. Reference numeral 10 indicates a trainee. The system 100 includes a memory unit 50 that stores instructions for the processing unit 40. The processing unit is connected to an electroencephalogram sensor device 20 that can be worn on the head of the trainee and can detect electroencephalogram signals generated on the surface of the trainee's head. The trainee 10 is instructed to imagine performing a predetermined flight control operation 15 in response to a visual stimulus displayed on the display 30. The display of the visual stimulus to the trainee 10 is performed simultaneously with the measurement of the electroencephalogram signal in the electroencephalogram sensor device 20. According to the instructions stored in the memory unit 50, the processing unit calculates an attention index, a motor control index, and an arousal index (details will be described later). Next, a feedback message characterizing at least one of the above-mentioned attention, motor control, arousal, and flight readiness is given to the trainee 10. The present invention includes feedback in the form of a static avatar, a dynamic avatar, a text message, a sound pattern, and a tactile pattern.

[0158] Referring to FIG. 2a, an exemplary electroencephalogram cap is made of stretchable synthetic fibers of various sizes. The cap holds sensors directly above the associated brain regions. The standard EEG cap is known as the "10-20 system". The "International 10-20 system" is recognized as a method for representing the positions of scalp electrodes. With this standard test system, the research results (clinical or research) of subjects can be reliably aggregated, reproduced, and effectively analyzed and compared using scientific methods. This system is based on the relationship between the positions of the electrodes and the regions below them, particularly the cerebral cortex.

[0159] Referring to FIG. 2b, in the electroencephalogram sensor device, each EEG sensor records the electric field below it. Neurons transmit information to each other by changes in electric charge. As a result, a difference occurs in the surrounding electric field. By analyzing this change, the functions of the brain can be read. The electroencephalogram sensor device includes sensors that can be worn as follows. (1) Frontal region electrodes located on the midline of the frontal lobe, (2) The vertex region electrode located on the midline of the parietal lobe, (3 - 5) The sensorimotor cortex electrode attached to the motor cortex.

[0160] Also, for signal collection, a ground sensor and a reference sensor are used.

[0161] It is desirable to maximize the signal - to - noise ratio by cleaning the lower surface of the sensor before measurement and electrically contacting the skin surface through a conductive gel as needed.

[0162] Referring to Figure 2c, the sensor is connected to an amplifier that amplifies the magnitude of the electrical signal. The amplifier is attached to the cap using a cord. Thereby, the amplified electroencephalogram signal is wirelessly transmitted to a computer, smartphone, or tablet connected by a USB connector. Most commercially available electroencephalogram signal amplifiers can be used in the present invention.

[0163] EEG data is streamed to the processing unit and structured by software according to a preset sensor montage. Referring to Figure 2d, a noise removal function is applied in the next step. For example, to remove the electrostatic interference of the electrical network (notch filter). With this software, the user can check the connection level of each sensor and add gel if the impedance is too high or detect malfunction of the sensor.

[0164] EEG data is streamed using the LSL protocol (LabStreamingLayer) and acquired by data analysis code.

[0165] In FIG. 3, a method 200 for testing and training brain functions that plan and execute flight control operations is shown. In method 200, first, a system 100 (FIG. 1) for testing and training the brain functions that plan and execute the above-described piloting actions is prepared (step 210). After causing a trainer to imagine performing a predetermined pilot action in response to the display of a visual stimulus (step 220), the above-described visual stimulus is displayed to a trainee to be tested while measuring an electroencephalogram signal in an electroencephalogram sensor device (step 230). The obtained electroencephalogram signal is processed to calculate a concentration index, a motor control index, and an arousal index (step 240). Based on the calculated indices, a feedback message is output that characterizes at least one of concentration, motor control, arousal, and flight readiness (step 250). Steps 220 to 250 are repeated as necessary.

[0166] EEG data is received via an LSL socket. Data analysis includes filtering. The raw EEG data is analyzed by shifting (e.g., 50% shift with 500 samples or 1,000 samples) each cycle for each time window. The purpose of the analysis is to extract the characteristics of brain functions that contribute to the success of motor control of the cockpit and pilot control.

[0167] A person's concentration and activity level in the motor cortex can be inferred from the variation in power in a specific frequency band. In this system, these are detected by at least five scalp electrodes.

[0168] The calculation of the "concentration" (or brain involvement) index is performed according to the following algorithm. The raw data is first filtered by an IIR filter at the half-power frequencies of the frequency ranges [alpha: 8 - 11 Hz, beta: 16 - 22 Hz, theta: 4 - 7 Hz] for the data from the sensors attached to the vertex (alpha) and the frontal lobe (beta and theta). The power of each frequency band (alpha, beta, and theta) is calculated from the corresponding filtered data using the "band power" method. The concentration index for each cycle is the ratio of the powers of beta, theta, and alpha.

[0169] Threshold - Accuracy: At the start of each session, the system determines a baseline as a characteristic of each trainee. The pilot trainee sits still for several minutes (2 minutes by default) before the instructed flight maneuvers or simulations with eyes open to create an open-eye baseline. The system collects the concentration index in the baseline and uses it to set a threshold for the trainee, which determines a customized boundary value for each user. When the value is exceeded, the system determines that the user has a high concentration and can give positive feedback to the user.

[0170] According to an embodiment of the present invention, the above-mentioned threshold can be set as the sum of the lower limit value and the product of the difficulty level and the difference between the upper limit value and the lower limit value.

[0171] The upper limit value is the average of the index. The lower limit value is the value obtained by subtracting two standard deviations from the average of the index, and the difficulty level is a value set by the user to adjust the challenge level according to the user's ability.

[0172] During training flights using aircraft control feedback, metrics are continuously calculated and compared via a backend module to a pilot's threshold baseline during training. If the current value of the concentration metric exceeds the set threshold, the module sends a message to the flight simulation to provide positive user feedback. As an example, during takeoff, landing, or in-flight procedures, a circle indicating the target shrinks, and by the user noticing this, they can recognize the execution of takeoff, landing, or in-flight procedures and perform the operation and aircraft handling procedures more accurately. In the current trial, if the user exceeds the threshold multiple times (e.g., when performing a landing approach to the runway within a predetermined number of seconds), intermittent feedback is provided (an appropriate and safe approach by adjusting the throttle and flaps appropriately).

[0173] The algorithm for calculating the motion control metric is as follows. First, the raw data is filtered by an IIR filter at the half-frequency of the frequency range [Mu: 12 Hz - 15 Hz] of the somatosensory area sensors such as C3, Cz, C4, etc. on the motor cortex. The power in the Mu frequency band and the data filtered from these channels are calculated by the band power function. The metrics from the somatosensory area sensors such as C3 and C4 are then used to evaluate the current Mu desynchronization, which is the movement preceding the movement.

[0174] Threshold - Force applied to the flight manual joystick: All Mu power metrics calculated during the acquisition of baseline data with eyes open from the somatosensory areas such as C3 and C4 electrodes are used to evaluate the trainee pattern of the Mu rhythm of a specific user.

[0175] For example, the average Mu power at positions C3 and C4 (above the left motor cortex controlling the right limb or vice versa) is used as the motor brain activity threshold.

[0176] Training based on feedback of actions or procedures: During an action or procedure, the current Mu power is constantly compared with the average value of the data collected at the open-eye baseline. Specifically, the feedback is defined as positive when the sum of the instantaneous Mu power and the product of the difficulty level and the STD of MU at the baseline is smaller than the average MU at the reference time.

[0177] "Mu power" represents the instantaneous Mu value. The difficulty level is a value set by the user to adjust the challenge level according to the user's ability. The STD of Mu at the baseline is the standard deviation of all Mu metrics collected at the open-eye baseline. If the current Mu power plus a part of the standard deviation is lower than the average Mu at the baseline, positive feedback is given because of the activation of the motor cortex.

[0178] In the pilot training, the circular bar is gradually filled and the color changes from red to green (through yellow and orange). As the bar is filled, the user is shown how to perform the nerve actions that better activate these areas. If the user can "fill" this bar in one trial, the force required to control the joystick or other manual device is strong enough to perform a given flight action. Similar criteria apply to the selection and training in any commercial flight that requires quick and reliable accuracy. Also, the ability to exert physical vitality and attention is important in the training process for various types of aircraft, light aircraft, propeller aircraft, jet aircraft, cargo aircraft, passenger aircraft, gliders, and helicopters.

[0179] The avatar (the pilot on the screen) performs the procedure only when both types of feedback (motor control and concentration) are controlled simultaneously and sufficiently by the user.

[0180] The algorithm for calculating the wakefulness (sleep) index is as follows: Alpha band power at the central parietal sensor (e.g., Pz) is collected at an eyes-closed baseline (a few minutes (default 2 minutes), immediately following the eyes-open baseline) and compared to the eyes-open baseline of alpha power. Alpha power at this location is known to be related to the user's level of wakefulness. High alpha power is typically associated with fatigue.

[0181] Threshold - Sleep Detection: Calculate and compare moving averages of baseline data for both eyes open and eyes closed. First, average over one time window, then increase the interval until the moving average for eyes closed is at least two standard deviations higher than the alpha value for eyes open. The moving average for the smallest time interval that meets this condition is set as the threshold for "sleep detection."

[0182] Feedback: During surgery, the current alpha power at the central parietal sensor or other predefined location is constantly compared to a threshold. If, during landing, the pilot makes an error, such as overshooting, it is determined that the on-screen procedures have not been followed and a warning is given, along with an audio message saying, "Wake up! You're in danger!"

[0183] By using advanced ML tools such as clustering algorithms, SVM classifiers, and artificial neural networks, using data from early adopters in the field, we can create powerful pattern detection mechanisms that are highly user-specific, robust to EEG noise, and provide rich data-driven insights to users.

[0184] The last element of the system of the present invention is a training environment for the user. The interface displays the user's brain activity in real time. This is essential in the neurofeedback learning process and closes the loop starting from the brain data obtained using EEG. The inventors designed the product so that the pilot can train important brain functions for improving their performance. So far, the basic intracerebral processes monitored by the system of the present invention, that is, the images of movements at a high level of concentration, have been described. These intracerebral processes are common in different types of flights under different conditions in different aircraft. However, in current research, it has been shown that the efficiency of the feedback environment for this type of neurotraining is greatly improved by an appropriate learning environment. Therefore, for each type of flight, a respective virtual environment is constructed. The goal is to enable the pilot to train in a familiar environment, thereby enabling the pilot to efficiently reflect the learned content in the flight and control of an "actual world" aircraft.

[0185] The interface segment of the present invention is a high-context computer game-form environment. For example, the first environment of the present invention is an intestinal surgery that excises the growth in the intestine and sutures the intestine safely and effectively.

[0186] Figures 4a to 4i show an example of a change in visual stimuli that improves test metrics. Specifically, the pilot is instructed to sit still and concentrate on the monitor and imagine landing on the screen. In each trial, the system gives the pilot a predetermined number of seconds. The neural activity of the actual flight operation and the neural activity of the imagined flight operation are very similar. When this system detects strong activation in a specific band in a specific brain region related to this operation, the index of the force applied to the joystick or other manual flight control unit increases (the image below). Due to the correlation between this neural activation and the symbols shown on the screen, the user can intuitively control the activities of the above-mentioned neural networks that are most important for controlling the arm and hand operating the joystick.

[0187] Through the accumulation of practice, the user learns how to control these brain regions, but at the same time, it also causes changes in their "connections". The brain is an organ that is constantly changing. The connections of neurons change, and neural networks are created or strengthened by being constantly activated. Such changes are the components of learning and lead to improvement through training and repetition.

[0188] When the player pilot imagines, for example, the movement of taking off in strong crosswinds, the pilot receives continuous feedback regarding this movement. The change in the circular bar (8 is still empty, 0 is the full power state) and the color indicate high activation, and the number in the center of the circle indicates a countdown that tells the user the start timing of "flap down" as an irreversible action. The avatar can only take off when the user reaches sufficient power.

[0189] The activation of the network through repeated training using this brain trainer can cause activities equivalent to actual field activities while avoiding many problems in conventional practice.

[0190] FIG. 7 is a diagram showing another embodiment of the present invention, and shows a case where a safe course is maintained during the occurrence of turbulent flow or engine failure. In this game, the pilot can control how much power to distribute to the remaining engines and how to supplement the lift of the wings. The accuracy is controlled by the player's level of involvement and concentration. When the system recognizes the user's concentration, the circle surrounding the target is gradually locked onto the target. Only when the white circle is locked onto the target does the remaining engine output reach the required level.

[0191] In this way, the user obtains continuous feedback on two mental processes that could not be felt before. The second form of feedback is what is called "intermittent feedback". As a result of the effort, if both types of input can be controlled at a sufficient level, at the end of the predetermined allotted time, the manual operation pressure applied to the surgical instrument will be appropriate and the intended result will be achieved by the flight control operation. If only one of the conditions is met, the pilot fails. If the pilot cannot reach a sufficient level of concentration during this trial, the accuracy of the flight control operation will be low. If the pilot cannot exert sufficient force by the trial of motor imagery, the flight will become unstable and there is a possibility of continuous failure.

[0192] By training over time, most users learn how to adjust these neural processes at will, and important changes occur in the neural network related to the pilot's performance.

[0193] When the pilot reaches sufficient power and accuracy, the avatar takes over flight control and the trainee is warned. Successfully performing important flight operations is highly rewarding for the pilot. In the brain, there is a general "rule" that neural activity leading to reward results in stronger connections and an increase in resources related to this activity. These changes improve the pilot's speed and accuracy when performing actions, leading to increased subjective confidence and objective performance.

[0194] At the end of the session, the pilot can observe their performance and confirm changes in their ability to concentrate during the operation and activate the brain's motor areas.

[0195] Through the interface, the pilot's trainer can assemble training sessions with parameters such as the number of repetitions, position, complexity, and simulated emergencies. After that, the pilot and trainer during training can use a specially designed interface presented at the end of the session to observe the performance achieved by the pilot in the most recent session, compare it with past sessions, and perform other important analyses. Here, statistical values based on the training results are shown. This enables the pilot and trainer during training to use the brain data obtained from the training to grasp the overall picture of the pilot's abilities and progress.

[0196] Accordingly, the present invention provides a system and method for testing and training brain functions for planning and executing actions. In particular, the invention of the present disclosure is applicable to all types of human actions, such as sports, surgery, aviation, post-traumatic and post-illness rehabilitation.

Claims

1. A system for testing and training the brain functions that plan and execute actions, (a) an electroencephalogram sensor device that can be attached to the head of the person being trained, (b) A processor configured to receive and analyze electroencephalogram signals obtained from the trainee in response to a visual stimulus displayed to the trainee, (c) memory, which, when executed by the processor, (i) Instruct the trainee to imagine performing the aforementioned action, (ii) The electroencephalogram sensor device measures the electroencephalogram signal, (iii) Calculate at least one characteristic selected from (1) to (3) below, (1) Concentration index, (2) Motor control indicators, (3) Arousal level index, (iv) Based on at least one of the concentration level, motor control level, arousal level, and readiness level, provide the trainee with a feedback pattern that shows a real-time display of brain activity, including changes in visual stimuli as the test indicator improves. (v) Provide intermittent feedback in a virtual environment showing the results of the trainee's efforts, (vi) Repeat steps (ii) to (v) as needed. The memory for storing instructions for, system.

2. The system includes a display configured to show visual stimuli to the trainee, and the memory includes instructions for presenting visual messages and / or playing audio messages in response to the display of the visual stimuli, instructing the trainee to imagine performing the action. The system according to claim 1.

3. The trainee is instructed to imagine performing the action in response to the display of the visual stimulus. The system according to claim 2.

4. The display of the visual stimulus, the measurement of the electroencephalogram signal in the electroencephalogram sensor device, and the provision of the feedback pattern are performed continuously. The system according to claim 1.

5. The memory includes instructions for calculating the concentration index as the rate of change of the electroencephalogram signals of the parietal and frontal electrodes at alpha, beta, and theta frequencies, obtained from the electroencephalogram signals of the parietal and frontal electrodes measured at rest. The system according to claim 1.

6. The memory includes instructions for calculating the motor control index as the rate of change of the electroencephalogram (EEG) signal at The system according to claim 1.

7. The memory includes instructions for calculating the arousal index as the rate of change of the electroencephalogram signal at the parietal electrode at alpha frequency, obtained from the trainee in an open-eye state, compared to the electroencephalogram signal at the parietal electrode in an open-eye state. The system according to claim 1.

8. The memory includes instructions for analyzing at least one of the concentration index, motor control index, and arousal index of the trainee or a group of trainees, and for presenting training progress data in chronological order. The system according to claim 1.

9. The feedback pattern is selected from the group consisting of static avatars, dynamic avatars, text messages, sound patterns, haptic patterns, and any combination thereof. The system according to claim 1.

10. The aforementioned feedback pattern relates to the visual environment associated with the aforementioned operation. The system according to claim 1.

11. The memory includes instructions for inputting the electroencephalogram signal into a machine learning model and obtaining a detection pattern robust to EEG noise from the machine learning model, wherein at least one of the concentration index, the motor control index, and the arousal index is calculated according to the detection pattern robust to EEG noise. The system according to claim 1.

12. The memory includes instructions for monitoring the connection level of each sensor in the electroencephalogram sensor device and for generating a display for adding gel. The system according to claim 1.

13. The aforementioned memory is A baseline threshold is calculated based on at least one of the concentration index, motor control index, and arousal index calculated during the baseline acquisition time interval. Equipped with instructions for, The feedback pattern is generated according to at least one of the concentration index, the motor control index, and the arousal index calculated during iterations against the baseline threshold. The system according to claim 1.

14. The memory includes instructions for dynamically adapting the feedback pattern according to at least one of the calculated concentration index, motor control index, and arousal index, compared to a set difficulty level indicating the task level relative to the trainee's abilities. The system according to claim 1.

Images