Task-matched tactile feedback exercise device and method

Abstract

The application provides a kind of task matching visual and tactile feedback exercise training device and method, applied to biological signal processing technical field.The device includes: acquisition equipment;Electric stimulation device is used to exert first electric stimulation to target object;Second electric stimulation is exerted to target object according to second electric stimulation instruction;Processor is used to obtain training coupling matrix according to training eeg feature vector in training eeg signal and training electromyographic signal in training electromyographic signal;The training coupling matrix is updated using the update weight and the update coupling matrix corresponding to the historical training, to obtain the update coupling matrix;Based on particle swarm fitness function and update coupling matrix, training eeg feature vector and training electromyographic feature vector are extracted and fused to obtain target brain muscle eeg fusion feature;The target brain muscle eeg fusion feature is classified and identified using vector machine, and the second electric stimulation instruction is applied to the electric stimulation device according to the training result.

Description

Technical Field

[0001] This invention relates to the field of biosignal processing technology, and more specifically to a task-matched visual-tactile feedback motion training device and method. Background Technology

[0002] Brain-computer interface (BCI)-based auxiliary training technologies can fully leverage the trainee's initiative to help establish a close interactive connection between the brain's motor intention information and external motor assistance training equipment, thereby achieving a synchronous activation effect between the cortex and muscles. This has become one of the key development directions currently being focused on.

[0003] In the process of realizing the above-mentioned inventive concept, research has found that: existing technologies can only rely on traditional motor imagery paradigms and single sensory feedback pathways, resulting in low initiative and poor fusion perception of the training subjects, making it difficult to achieve coupling and coordination between electromyography and electroencephalography under self-regulation. At the same time, the lack of multi-sensory feedback and multimodal interactive training paradigms based on real natural task orientation during the training process leads to the technical problem of relatively general training effect. Summary of the Invention

[0004] In view of the above problems, the present invention provides a task-matched visual-tactile feedback motion training device and method.

[0005] According to a first aspect of the present invention, a task-matched visual-tactile feedback motor training device is provided, comprising: a data acquisition device disposed on the head and hand of a target object, for acquiring electroencephalogram (EEG) signals and electromyographic (EMG) signals of the target object; an electrical stimulation device attached to the hand of the target object, for applying a first electrical stimulation to the target object before motor training to prompt for motor training; and applying a second electrical stimulation to the target object according to a second electrical stimulation command to prompt for the training result of the motor training, wherein the intensity of the first electrical stimulation and the second electrical stimulation are different; and a processor electrically connected to the data acquisition device and the electrical stimulation device, for: responding to the target object performing motor training, using the EEG signals received from the data acquisition device as training EEG signals, and transferring the data from the data acquisition device... The received electromyographic (EMG) signals are used as training EMG signals. A training coupling matrix is ​​constructed based on the training EEG feature vectors and training EMG feature vectors in the training EEG signals. The training coupling matrix is ​​updated using the update weights and update coupling matrix corresponding to historical training, resulting in an update coupling matrix corresponding to motion training. Based on the particle swarm fitness function and the update coupling matrix corresponding to motion training, feature extraction and fusion processing are performed on the training EEG feature vectors and training EMG feature vectors to obtain the target EEG-EMG fusion feature. The target EEG-EMG fusion feature is then classified and identified using a vector machine to obtain the training result. Based on the training result, a second electrical stimulation command is applied to the electrical stimulation device.

[0006] A second aspect of the present invention provides a task-matched visual-tactile feedback method for motor training, comprising: responding to a target object for motor training, using an electroencephalogram (EEG) signal received from a data acquisition device as a training EEG signal, and using an electromyography (EMG) signal received from the data acquisition device as a training EMG signal; constructing a training coupling matrix based on the training EEG feature vector and the training EMG feature vector in the training EEG signal and the training EMG feature vector in the training EMG signal; updating the training coupling matrix using update weights and the update coupling matrix corresponding to historical training to obtain an update coupling matrix corresponding to the motor training; performing feature extraction and fusion processing on the training EEG feature vector and the training EMG feature vector based on a particle swarm fitness function and the update coupling matrix corresponding to the motor training to obtain a target EEG-EMG fusion feature; performing feature classification and recognition on the target EEG-EMG fusion feature using a vector machine to obtain a training result; and applying a second electrical stimulation command to an electrical stimulation device based on the training result.

[0007] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0008] A fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0009] A fifth aspect of the invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0010] According to the task-matched visual-tactile feedback motor training device and method of the present invention, in response to the target object training according to the motor imagination task, under the prompting of the first electrical stimulation guided by the prompt, the device collects training EEG signals and training EMG signals in multiple dimensions and multiple channels, thereby inducing the target object to train a new paradigm of motor intention based on natural tasks designed for everyday life scenarios. This lays a solid and fundamental foundation for subsequent processing related to the decoding of motor intentions by neuromuscular coupling and for controlling visual-tactile stimulation and task guidance.

[0011] Then, based on the trained EEG and EMG feature vectors, a training coupling matrix corresponding to the current training is constructed. Combined with the updated weights and updated coupling matrices corresponding to historical training, the training coupling matrix for the current training is updated, constructing an updated coupling matrix. This updated coupling matrix, combined with the multi-layer particle swarm architecture constructed for EEG and EMG in this invention, allows for feature extraction and fusion of the two feature vectors (EEG + EMG) to obtain the target EEG-EMG fusion features. This achieves the updating of the coupling matrix containing the initial EEG and EMG features in the current round, considering the influence of the previous round's training results (second electrical stimulation) on the current round and the influence of the EEG and EMG features from the previous round's historical training. This fully considers the impact of the second electrical stimulation on the target object's next task, enhances the correlation between multiple training rounds, and constructs an accurate neuro-muscle coupling decoding motion intent. Furthermore, by combining a targeted, multi-layered particle swarm architecture, the accuracy and reliability of fusion analysis are improved. This enables the neuromuscular system to move from low-level analysis and mining to high-level fusion perception, fully realizing the coupling and synergy between electromyography (EMG) and electroencephalography (EEG), and improving the training effect of assisted training.

[0012] Furthermore, by combining a second electrical stimulus controlled according to the training results, two different modes of electrical stimulation are applied. While providing visual cues, the target object is also prompted by dual tactile cues through the hand cortex. This establishes a two-way perceptual feedback pathway and a multimodal interactive training paradigm, thereby improving the target object's initiative, sensory feedback, training enthusiasm, practical application process, and auxiliary training effect. Attached Figure Description

[0013] The above-mentioned contents, other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the present invention with reference to the accompanying drawings, which will be described in conjunction with the drawings.

[0014] Figure 1 A schematic diagram of a task-matching visual-tactile feedback motion training device according to an embodiment of the present invention is shown.

[0015] Figure 2 A schematic diagram of a task-matching visual-tactile feedback motion training device according to another embodiment of the present invention is shown.

[0016] Figure 3 A schematic diagram of the workflow for obtaining the first updated coupling matrix using a processor according to an embodiment of the present invention is shown.

[0017] Figure 4 A schematic diagram of the workflow for obtaining target brain electromyography fusion features using a processor according to an embodiment of the present invention is shown.

[0018] Figure 5 A schematic diagram illustrating the workflow of generating training results using a processor according to an embodiment of the present invention is shown.

[0019] Figure 6 A schematic diagram illustrating the workflow of a task-matching visual-tactile feedback motion training device according to an embodiment of the present invention is shown.

[0020] Figure 7 A schematic diagram illustrating the workflow of a task-matching visual-tactile feedback motion training device according to another embodiment of the present invention is shown.

[0021] Figure 8 A flowchart of a task-matching visual-tactile feedback motion training method according to an embodiment of the present invention is shown.

[0022] Figure 9 A block diagram of a task-matching visual-tactile feedback motion training method according to an embodiment of the present invention is shown. Detailed Implementation

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0028] Brain-computer interface (BCI)-based motor rehabilitation training technology can fully leverage the subjective initiative of the target individual while establishing a close interactive connection between the brain's motor intention information and external motor assistive training equipment, thus possessing significant scientific and clinical practical value.

[0029] Building upon this foundation, existing technologies apply functional electrical stimulation (FES) driven by motor imagery (MI) brain-computer interfaces to assist in motor rehabilitation training. This achieves a synchronous activation effect between the cortex and muscles, strengthening the activity of the brain's sensorimotor cortex to aid in training that promotes repair and remodeling. However, these existing technologies still struggle to provide effective training for target subjects. Research during the development process revealed that existing technologies rely solely on traditional motor imagery paradigms and single sensory feedback pathways, resulting in low subject initiative, poor fusion perception, and difficulty in achieving self-regulated coupling and synergy between electromyography (EMG) and electroencephalography (EEG). Furthermore, the lack of multi-sensory feedback and multimodal interactive training paradigms based on realistic, natural tasks leads to relatively limited training effectiveness.

[0030] In view of this, embodiments of the present invention provide a task-matched visual-tactile feedback motor training device, comprising: a data acquisition device disposed on the head and hand of a target object, for acquiring electroencephalogram (EEG) signals and electromyographic (EMG) signals of the target object; an electrical stimulation device attached to the hand of the target object, for applying a first electrical stimulation to the target object before motor training to prompt for motor training; and applying a second electrical stimulation to the target object according to a second electrical stimulation command to prompt for the training results of the motor training, wherein the intensities of the first and second electrical stimulations are different; and a processor electrically connected to the data acquisition device and the electrical stimulation device, for: responding to the target object performing motor training, using the EEG signals received from the data acquisition device as training EEG signals, and transferring the data from the data acquisition device... The received electromyographic (EMG) signals are used as training EMG signals. A training coupling matrix is ​​constructed based on the training EEG feature vectors and training EMG feature vectors in the training EEG signals. The training coupling matrix is ​​updated using the update weights and update coupling matrix corresponding to historical training, resulting in an update coupling matrix corresponding to motion training. Based on the particle swarm fitness function and the update coupling matrix corresponding to motion training, feature extraction and fusion processing are performed on the training EEG feature vectors and training EMG feature vectors to obtain the target EEG-EMG fusion feature. The target EEG-EMG fusion feature is then classified and identified using a vector machine to obtain the training result. Based on the training result, a second electrical stimulation command is applied to the electrical stimulation device.

[0031] Figure 1 A schematic diagram of a task-matching visual-tactile feedback motion training device according to an embodiment of the present invention is shown.

[0032] like Figure 1As shown, the task-matched visual-tactile feedback motor training device of the present invention may include a data acquisition device, an electrical stimulation device, and a processor. This task-matched visual-tactile feedback motor training device can be used to assist in the training of a target subject, thereby helping the target subject to perform assisted exercises for related motor functions.

[0033] Specifically, the acquisition device can be set on the head and hands of the target object to acquire the target object's electroencephalogram (EEG) and electromyogram (EMG) signals.

[0034] The acquisition devices can include electroencephalogram (EEG) acquisition devices and electromyography (EMG) acquisition devices. EEG acquisition devices can be placed on the target subject's head to assist in acquiring the target subject's electroencephalogram (EEG) signals, while EMG acquisition devices can be placed on the target subject's hand to assist in acquiring the target subject's electromyography (EMG) signals.

[0035] An electrical stimulation device can be attached to the hand of a target subject to apply a first electrical stimulus to the target subject before exercise training to prompt the exercise training; and to apply a second electrical stimulus to the target subject according to a second electrical stimulation instruction to prompt the target subject for the training results of the exercise training.

[0036] The intensity of the first and second electrical stimuli differs.

[0037] The first electrical stimulation can be a relatively gentle triangular wave (soft vibration), while the second electrical stimulation can be a more intense rectangular wave (mild tingling). At the start of motor imagery training, the first electrical stimulation is applied to the target subject's hand (left, right, or both median nerves) to prompt the subject to engage in motor imagery training and facilitate the acquisition of electromyographic signals during the training process. After obtaining the target subject's training result for the current motor imagery training, depending on the success or failure of the result, a second electrical stimulation of varying intensity is applied to the target subject's hand (left, right, or both median nerves) to indicate the current training outcome.

[0038] Before training begins, the target subject's sensory threshold (PT) can be measured, defined as the minimum current intensity required for the subject to stably perceive a stimulus across multiple stimulations. The currents for both the first and second electrical stimuli can be set above the sensory threshold but significantly below the discomfort threshold to avoid inducing pain. The first electrical stimulus uses a biphasic symmetrical triangular wave with a frequency of 100 Hz, a pulse width of 250 µs, and a current amplitude of 1.2 PT–1.86 PT to induce a gentle, vibratory tactile sensation. The stimulation duration can be 2 seconds. After the first electrical stimulation, the target subject can be asked to begin imagining the actions described in the task description; this imagination process can last for 5 seconds.

[0039] After obtaining the training results, a second electrical stimulation can be applied to the corresponding median nerve in the left, right, or both hands using patch electrodes connected to the electrical stimulator. The second electrical stimulation can use a biphasic rectangular wave with a frequency of 40 Hz, a pulse width of 150 µs, and a current amplitude of 1.3 PT-1.7 PT to produce a slight tingling sensation without causing pain. The duration can be 3 seconds. The end of the second electrical stimulation signifies the completion of the current set of motor imagery training.

[0040] The processor, electrically connected to the acquisition and electrical stimulation devices, is used to perform motion training in response to the target object. It uses EEG signals received from the acquisition device as training EEG signals and EMG signals received from the acquisition device as training EMG signals. A training coupling matrix is ​​constructed based on the training EEG and EMG feature vectors in the training EEG and EMG signals. The training coupling matrix is ​​updated using update weights and the update coupling matrix corresponding to historical training, resulting in an update coupling matrix corresponding to motion training. Based on the particle swarm fitness function and the update coupling matrix corresponding to motion training, feature extraction and fusion processing are performed on the training EEG and EMG feature vectors to obtain the target brain-motor fusion feature. A vector machine is used to classify and identify the target brain-motor fusion feature to obtain the training result. Based on the training result, a second electrical stimulation command is applied to the electrical stimulation device.

[0041] A complete motor training session for the target subject can include a 20-second preparation period and a task period of less than 10 minutes. During the preparation period, the target subject needs to remain resting to maintain a baseline level of brain activation. The task period can include multiple training sets, each consisting of a 7-second task state, a 5-second rest state, and a 3-second feedback state. During the task state, the target subject needs to visualize a designated limb and a designated movement for a daily living task based on synchronized visual-tactile cues (task content). During the rest state, the target subject needs to stop visualizing and remain resting. During the feedback state, the target subject needs to receive synchronized visual cues and tactile stimulation based on the analyzed training results.

[0042] Tactile cues involve providing tactile feedback to the target object through a second electrical stimulus, while visual cues involve directly displaying the target object various information such as the current training results, the progress of related tasks, the total number of training sessions completed, the total number of successful training sessions, and the number of failed training sessions.

[0043] The training coupling matrix can be characterized as a 14*2 multidimensional coupling strength vector matrix containing the training EEG feature vectors and training EMG feature vectors. The training EEG feature vectors include features corresponding to seven leads in the alpha band: FC3 (Frontal Central 3, left frontal midline), C3 (Central 3, left central), CP3 (Central Parietal 3, left parietal midline), Cz (Central), CP4 (Central Parietal 4, right parietal midline), C4 (Central 4, right central), and FC4 (Frontal Central 4, right frontal midline); and features corresponding to seven leads in the beta band: FC3, C3, CP3, Cz, CP4, C4, and FC4.

[0044] The update weights corresponding to historical training can be the weight values ​​determined based on the training results of historical training (the previous training task), and the update coupling matrix corresponding to historical training can be the update coupling matrix of historical training (the previous training task).

[0045] By combining the historical training from the previous round, the update coupling matrix for each round is updated, thereby obtaining a constraint matrix for feature extraction and fusion that can reflect the influence of the training results (second electrical stimulation) from the previous round on the current round of training, the influence of the EEG and EMG features from the previous round of training, and the influence of the EEG and EMG features from the current round of training.

[0046] After obtaining the target brain-muscle electrofusion features, a vector machine capable of classifying and recognizing these features is used to process them, thereby obtaining the training results. If the training is successful, a gentler second electrical stimulus (or no second electrical stimulus) can be applied to the target subject using an electrical stimulation device. If the training fails, a stronger second electrical stimulus can be applied to the target subject using the same device.

[0047] It should be noted that the training results and the entire training process presented here are only for medical staff to assist the target subjects in relevant training, and are not intended as any direct diagnostic or rehabilitation results.

[0048] According to an embodiment of the present invention, in response to the target object training according to a motor imagery task, under the prompting of a first electrical stimulation guided by a cue, training EEG signals and training EMG signals are collected in multiple dimensions and multiple channels. This leads to a new paradigm of inducing the target object to train motor intentions based on natural tasks designed for everyday life scenarios. This lays a solid and fundamental foundation for subsequent processing related to decoding motor intentions through neuromuscular coupling, as well as for controlling visual-tactile stimulation and task guidance.

[0049] Then, based on the trained EEG and EMG feature vectors, a training coupling matrix corresponding to the current training is constructed. Combined with the updated weights and updated coupling matrices corresponding to historical training, the training coupling matrix for the current training is updated, constructing an updated coupling matrix. This updated coupling matrix, combined with the multi-layer particle swarm architecture constructed for EEG and EMG in this invention, allows for feature extraction and fusion of the two feature vectors (EEG + EMG) to obtain the target EEG-EMG fusion features. This achieves the updating of the coupling matrix containing the initial EEG and EMG features in the current round, considering the influence of the previous round's training results (second electrical stimulation) on the current round and the influence of the EEG and EMG features from the previous round's historical training. This fully considers the impact of the second electrical stimulation on the target object's next task, enhances the correlation between multiple training rounds, and constructs an accurate neuro-muscle coupling decoding motion intent. Furthermore, by combining a targeted, multi-layered particle swarm architecture, the accuracy and reliability of fusion analysis are improved. This enables the neuromuscular system to move from low-level analysis and mining to high-level fusion perception, fully realizing the coupling and synergy between electromyography (EMG) and electroencephalography (EEG), and improving the training effect of assisted training.

[0050] Furthermore, by combining a second electrical stimulus controlled according to the training results, two different modes of electrical stimulation are applied. While providing visual cues, the target object is also prompted by dual tactile cues through the hand cortex. This establishes a two-way perceptual feedback pathway and a multimodal interactive training paradigm, thereby improving the target object's initiative, sensory feedback, training enthusiasm, practical application process, and auxiliary training effect.

[0051] It should be noted that the training results obtained by the present invention are only an intermediate result, and the exercise training carried out by the present invention is only an intermediate process. Diagnostic results or health status cannot be directly derived from the training results obtained by the method of the present invention or from the training process carried out.

[0052] Figure 2 A schematic diagram of a task-matching visual-tactile feedback motion training device according to another embodiment of the present invention is shown.

[0053] like Figure 2 As shown, in Figure 1Based on the device shown, the task-matching visual-tactile feedback motion training device may further include a display screen for displaying motion imagery training tasks to the target object.

[0054] Specifically, the display screen can be electrically connected to the processor to display the exercise tasks and completion status corresponding to the exercise training to the target object.

[0055] Motor completion can be characterized as the progress of completion related to the motor imagery training task in that round, for example, the target object completes 60% of the task.

[0056] The display can present visual cues related to the motor imagery training task to the target object. The task instruction for these visual cues can be to move an object from its starting position, over obstacles, to its destination using the left, right, or both hands. The height of the obstacles indicates the task difficulty, which can be categorized into low, medium, and high difficulty levels. The black rectangular bar on the display represents the obstacle height for the current training set. Higher obstacles require greater effort from the user's motor imagery to induce stronger event-related synchronous neural responses; otherwise, the object movement task may not be completed. After each round of the task, the display shows whether the task was successful. If successful, a black object may appear at the destination in the center of the screen; otherwise, no object may be displayed at the destination.

[0057] According to an embodiment of the present invention, by employing a display and an electrical stimulator to provide simultaneous visual and tactile cues to the target object for a motor imagery task, multidimensional cues and assistance are provided for the target object's motor imagery training task from multiple visual and tactile perspectives. This facilitates the establishment of a two-way perceptual feedback pathway and a multimodal interactive training paradigm, thereby improving the target object's initiative, sensory feedback, training enthusiasm, practical application process, and auxiliary training effect.

[0058] According to an embodiment of the present invention, regarding the construction of the training coupling matrix, the processor is further configured to: in response to motion training of the target object, extract synchronous energy features from the acquired training EEG signals based on event-related spectrum perturbation to obtain a training EEG feature vector.

[0059] When the target object is undergoing motor imagery training, the target object in the task state is subjected to precise real-time acquisition of EEG signals (including the use of differential amplification and isolation design with reduced noise interference to acquire EEG signals, the sampling frequency of EEG signals can be set to 1000Hz, and 24-bit A / D (analog / digital) conversion is used to ensure the accuracy of the acquired signals) to obtain the initial EEG signal.

[0060] The initial EEG signal is then standardized and preprocessed to obtain the training EEG signal. This standardization preprocessing may include downsampling the initial EEG signal (to 200 Hz), followed by filtering the initial EEG signal using a zero-phase digital bandpass filter and a notch filter to obtain the training EEG signal. The zero-phase digital bandpass filter has a filtering range of 0.1–100 Hz, and the notch filter has a filtering range of 50 Hz.

[0061] Based on event-related spectrum perturbation, synchronous energy feature extraction is performed on the preprocessed training EEG signal to obtain a 14-dimensional training EEG feature vector, which includes features corresponding to the seven leads FC3, C3, CP3, Cz, CP4, C4 and FC4 in the alpha band, and features corresponding to the seven leads FC3, C3, CP3, Cz, CP4, C4 and FC4 in the beta band.

[0062] According to an embodiment of the present invention, the integrated electromyographic value of the acquired training electromyographic signal is calculated based on a predetermined electromyographic channel to obtain the training electromyographic feature vector.

[0063] Simultaneously with acquiring the target subject's electroencephalogram (EEG) signals, the target subject's electromyographic (EMG) signals are also precisely acquired in real time (including using differential amplification and isolation designs to reduce noise interference for EMG signal acquisition; the sampling frequency of the EMG signals can be set to 2000 Hz, using 24-bit A / D conversion to ensure the accuracy of the acquired signals), resulting in initial EMG signals. These initial EMG signals are then subjected to standardized preprocessing to obtain training EMG signals. Standardized preprocessing may include first correcting the baseline drift of the initial EMG signals using a moving average method to ensure the stability of the acquired signals. Then, the initial EEG signals are filtered using a zero-phase digital bandpass filter and a notch filter to obtain the training EMG signals. The zero-phase digital bandpass filter has a filtering range of 10-500 Hz, and the notch filter has a filtering range of 50 Hz.

[0064] Then, the relevant sub-electromyography values ​​are calculated on the training electromyography signals, thereby obtaining a 2D training electromyography feature vector that includes features corresponding to the left hand and features corresponding to the right hand.

[0065] Furthermore, the above methods can be used to collect and process electromyographic (EMG) and electroencephalogram (EEG) signals from a single task, facilitating subsequent feature recognition and analysis. Simultaneously, EMG and EEG signals corresponding to multiple tasks can also be collected and processed, allowing for unified feature recognition and analysis across multiple tasks, yielding multiple results corresponding to each task, which can then be displayed to the target audience.

[0066] The process of acquiring and processing electromyographic (EMG) and electroencephalogram (EEG) signals corresponding to multiple tasks may include: during the initial acquisition of initial EEG and initial EMG signals, based on the unified control of the brain-motor synchronization timestamp, digital trigger event marking processing is performed on the initial EEG and initial EMG signals to facilitate subsequent synchronization analysis and feature extraction (i.e., truncation based on synchronization event codes), with an event time difference accuracy of <1ms between brain-motor signals.

[0067] Then, based on the synchronization event code, the initial EEG and initial EMG signals are precisely extracted to obtain the initial EEG and initial EMG signals corresponding to each training task, facilitating temporal feature alignment between the EEG and EMG signals. The initial EEG and initial EMG signals corresponding to each training task are then subjected to standardization preprocessing operations as described above, thereby obtaining the training EEG feature vector and training EMG feature vector corresponding to each training task.

[0068] According to an embodiment of the present invention, a training coupling matrix is ​​constructed based on the training EEG feature vector and the training EMG feature vector.

[0069] The Pearson correlation coefficient between the training EEG feature vector and the training EMG feature vector corresponding to the trial of the motor imagery training task is calculated to extract the temporal features and physiological features of the two feature vectors. The absolute value of the calculated Pearson correlation coefficient is then used as the coupling strength to construct a 14*2 training coupling matrix corresponding to the trial of the motor imagery training task.

[0070] According to embodiments of the present invention, by extracting features from the preprocessed training EEG and EMG signals, relatively accurate and high-quality EEG and EMG feature vectors related to motor control can be obtained, thereby constructing a training coupling matrix corresponding to the test trial. This achieves the simultaneous acquisition of basic feature vectors corresponding to the test trial for subsequent feature fusion and recognition, and the construction of relevant basic parameters for subsequent feature fusion. This allows for further updating and iteration of the relevant basic parameters to obtain the relevant coupling parameters for subsequent feature fusion, thereby accurately identifying the neuromuscular coupling decoding of motor intentions and improving the training effect of assisted training.

[0071] According to an embodiment of the present invention, for obtaining the updated coupling matrix, the processor can also be used to: in the case of the nth motion training in N motion training of the target object, determine the (n-1)th updated weight corresponding to the (n-1)th motion training based on the training result of the (n-1)th motion training as historical training and the second electrical stimulation instruction.

[0072] The updated weight is one of the first weight corresponding to the training result representing successful training and the second weight corresponding to the training result representing failed training, where n>1 and N≥2.

[0073] The target subject needs to undergo N motor imagery training sessions. When the target subject undergoes the nth motor imagery training session, it is necessary to first determine the update weight of the previous session based on the training results of the previous session. This is to adapt to the dynamic changes in the neuromuscular coordination pattern of the target subject during actual training. The training coupling matrix of this round of training is progressively optimized using the relevant task execution results and feedback stimuli of the previous training session. This allows for adaptive updating and correction of the brain-muscle electrocoupling with a feedback regulation mechanism, thereby taking into account the influence of the second electrical stimulation on this round of training in subsequent feature extraction and fusion.

[0074] Specifically, the (n-1)th update weight corresponding to the (n-1)th exercise training is determined as shown in formula (1).

[0075] (1);

[0076] in, This can be represented as the (n-1)th updated weight corresponding to the (n-1)th training exercise. This can be represented as the first updated weights when the training result is successful. This can be represented as the second updated weights when the training result is a training failure. This can be represented as the training result of the (n-1)th exercise training session. This can be represented as the training result of the (n-1)th exercise training session being considered a successful training session. It can be characterized as the training result of the (n-1)th exercise training being a training failure. At the same time, the constraint condition of the (n-1)th update weight corresponding to the (n-1)th exercise training is as shown in formula (2), and the constraint condition between the first update weight and the second update weight is as shown in formula (3).

[0077] (2);

[0078] (3);

[0079] According to an embodiment of the present invention, the training coupling matrix of the nth exercise is updated using the (n-1)th update weight and the (n-1)th update coupling matrix corresponding to the (n-1)th exercise training, so as to obtain the nth update coupling matrix corresponding to the nth exercise training.

[0080] Given the (n-1)th update weight, the training coupling matrix of the nth exercise is updated using the (n-1)th update weight and the (n-1)th update coupling matrix in an exponentially weighted moving average manner. The update process can be shown in formula (4).

[0081] (4);

[0082] in, This can be represented as the nth update coupling matrix corresponding to the nth training iteration. This can be represented as the (n-1)th update coupling matrix corresponding to the (n-1)th exercise training. It can be represented as the training coupling matrix of the nth exercise training, which can be shown in formula (5).

[0083] (5);

[0084] in, It can be represented as the absolute value of the Pearson correlation coefficient obtained by calculating the training EEG feature vector and training EMG feature vector corresponding to the nth exercise training, i can be represented as the index of the EEG feature in the training EEG feature vector, and j can be represented as the index of the EMG feature in the training EMG feature vector.

[0085] According to an embodiment of the present invention, for the nth exercise training, an (n-1)th update weight is determined based on the training results of the (n-1)th exercise training. Then, based on the dynamic changes of the neuro-muscle synergy mode in continuous training, and based on the feedback regulation idea of ​​adaptive update correction for progressive optimization of the training coupling matrix of the next round according to the associated training task execution results and feedback stimuli, the training coupling matrix of the nth exercise training is updated using the (n-1)th update weight and the (n-1)th update coupling matrix, resulting in the nth update coupling matrix corresponding to the nth exercise training. This achieves the integration of the influence and feedback of the second electrical stimulation from historical training into the coupling matrix of the current training round through the above update mechanism. This allows the current coupling matrix to be appropriately corrected along the deviation direction between mental influence (historical training) and neuromuscular synergy (imagined execution of the current training task), thereby progressively and adaptively optimizing the brain-muscle functional coupling relationship, enhancing the correlation between multiple training rounds, and constructing an accurate neuro-muscle coupling decoding motion intention for auxiliary training.

[0086] Simultaneously, by obtaining the accurate nth update coupling matrix, it is possible to use the nth update coupling matrix as a dynamic physiological prior constraint under the bidirectional perception feedback path and multimodal interactive training paradigm, and introduce it into the subsequent feature fusion and particle swarm optimization process to guide the search direction of multimodal feature weights, so that the feature optimization process can simultaneously satisfy classification performance constraints and auxiliary physiological rationality constraints.

[0087] Figure 3 A schematic diagram of the workflow for obtaining the first updated coupling matrix using a processor according to an embodiment of the present invention is shown.

[0088] like Figure 3 As shown, for the first update coupling matrix corresponding to the first exercise in N exercise training sessions, the processor can also be used to: extract synchronous energy features from multiple acquired adaptive training EEG signals based on event-related spectrum perturbation before the target object performs its first exercise training, thereby obtaining multiple adaptive EEG feature vectors.

[0089] Among them, the multiple adaptation training EEG signals are EEG signals collected from the target subject after performing multiple sets of exercise adaptation training before exercise training.

[0090] Before the target subject begins formal motor imagery training, multiple rounds of adaptive motor imagery training can be conducted. This adaptive training is similar to the formal training, presenting a similar training task to the target subject via a monitor to assist them in entering a training state.

[0091] During the process of multiple screening and adaptation motor imagery training for the target subjects, multiple EEG signals were collected from the target subjects to obtain the adaptation training EEG signals corresponding to the task state of each screening and adaptation motor imagery training.

[0092] Then, after the relevant standardization preprocessing and other operations as described above, based on the event-related spectrum perturbation, synchronous energy feature extraction is also performed on multiple adaptive training EEG signals to obtain multiple adaptive EEG feature vectors.

[0093] According to an embodiment of the present invention, based on a predetermined electromyography (EMG) channel, the integrated EMG values ​​of multiple acquired adaptive training EMG signals are calculated to obtain multiple adaptive EMG feature vectors.

[0094] Among them, the multiple adaptation training electromyographic signals are electromyographic signals collected when the target subject undergoes multiple sets of exercise adaptation training before exercise training.

[0095] During the process of multiple screening adaptation motor imagery training for the target subjects, multiple electromyographic (EMG) signals are also collected from the target subjects to obtain the adaptation training EMG signals corresponding to the task state of each screening adaptation motor imagery training.

[0096] Then, after the relevant standardization preprocessing operations as described above, based on the predetermined electromyography channels, the integrated electromyography values ​​of multiple adaptive training electromyography signals are calculated to obtain multiple adaptive electromyography feature vectors.

[0097] According to an embodiment of the present invention, a target adaptive EEG feature vector and a target adaptive EMG feature vector are determined from multiple adaptive EEG feature vectors and multiple adaptive EMG feature vectors according to a predetermined feature selection rule.

[0098] The predetermined feature selection rule can be characterized as selecting from multiple adaptive EEG feature vectors and multiple adaptive EMG feature vectors to obtain both high-quality partially adapted EEG feature vectors and adaptive EMG feature vectors that correspond to successful training and high-quality partially adapted EEG feature vectors and adaptive EMG feature vectors that correspond to failed training.

[0099] By filtering multiple adaptive EEG feature vectors and multiple adaptive EMG feature vectors based on predetermined feature filtering rules, high-quality target adaptive EEG feature vectors and multiple target adaptive EMG feature vectors with comprehensive information elements can be obtained. The time series of the i-th target adaptive EEG feature vector can be as shown in formula (6), and the time series of the j-th target adaptive EMG feature vector can be as shown in formula (7).

[0100] (6);

[0101] Among them, X i This can be characterized as the time series of EEG features of the i-th target adaptation, x i1 This can be represented as the i-th target-adapted EEG feature in the first target-adapted EEG feature vector obtained through screening, x i2 This can be represented as the i-th target-adapted EEG feature in the second target-adapted EEG feature vector obtained through screening, x iK This can be characterized as the i-th target-adapted EEG feature in the K-th target-adapted EEG feature vector obtained through screening.

[0102] (7);

[0103] Among them, Y j This can be characterized as the time series of electromyographic features of the j-th target adaptation, y j1This can be represented as the j-th target-adapted electromyographic feature obtained from the first target-adapted electromyographic feature vector, y j2 This can be represented as the selection of the j-th target-adapted electromyographic feature in the second target-adapted electromyographic feature vector, y jK This can be characterized as selecting the j-th target-adaptive electromyographic feature from the K-th target-adaptive electromyographic feature vector.

[0104] According to an embodiment of the present invention, an initial coupling matrix is ​​constructed based on the target-adapted EEG feature vector and the target-adapted EMG feature vector, and the initial coupling matrix is ​​used as the first updated coupling matrix corresponding to the first exercise training.

[0105] The Pearson correlation coefficient between the EEG feature time series and the EMG feature time series of each target adaptation is calculated, and the absolute value of the calculated Pearson correlation coefficient is used as the coupling strength, thereby constructing an initial coupling matrix of 14*2 corresponding to the multiple screening adaptation motor imagery training. The initial coupling matrix can be as shown in formula (8).

[0106] (8);

[0107] in, It can be represented as the initial coupling matrix. It can be represented as the absolute value of the Pearson correlation coefficient calculated based on the time series of EEG characteristics and the time series of EMG characteristics adapted to each target.

[0108] Once the initial coupling matrix is ​​obtained, it can be used as the first updated coupling matrix corresponding to the first motion training, so as to constrain subsequent feature fusion.

[0109] According to an embodiment of the present invention, before the target subject undergoes the first formal assisted motor training, multiple screening and adaptive motor imagery training sessions can be performed to ensure that the target subject's brain-muscle electrophysiological activity is in a relatively stable and suitable state. Then, EEG and EMG signals are collected from the target subject during these multiple screening and adaptive motor imagery training sessions, and target-adapted EEG feature vectors and target-adapted EMG feature vectors are extracted based on predetermined feature selection rules. An initial coupling matrix is ​​constructed based on the Pearson coefficient between the target-adapted EEG and target-adapted EMG feature vectors, thereby determining the updated coupling matrix for the first motor imagery training in the subsequent formal training. This achieves the goal of considering both the effect of the target subject's motor imagery training and determining the first update constraint through the aforementioned adaptive training, improving the accuracy and reliability of subsequent fusion analysis.

[0110] Figure 4A schematic diagram of the workflow for obtaining target brain electromyography fusion features using a processor according to an embodiment of the present invention is shown.

[0111] like Figure 4 As shown, the processor can also be used to: update the initial particle swarm velocity function according to the nth update coupling matrix, and obtain the underlying particle swarm velocity function corresponding to the nth motion training.

[0112] The underlying particle swarm velocity function can include the underlying particle swarm EEG velocity function and the underlying particle swarm EMG velocity function. The underlying particle swarm velocity function can be characterized as the velocity step used to iteratively update the EEG weights or EMG weights.

[0113] By replacing the update coupling matrix in the initial particle swarm velocity function with the current nth update coupling matrix, the nth update coupling matrix can be introduced as a physiological guide term into the update iteration of EEG weights and EMG weights.

[0114] According to an embodiment of the present invention, based on the fitness function of the underlying subgroup, the EEG weights and EMG weights corresponding to the nth exercise training are iteratively updated using the underlying particle swarm velocity function corresponding to the nth exercise training, to obtain the target EEG weights and target EMG weights.

[0115] The fitness function of the lower-level subgroup can include the lower-level subgroup EEG fitness function and the lower-level subgroup EMG fitness function. By fusing features from the training EEG feature vector and the training EMG feature vector individually, and by fusing features from the whole, a hierarchical particle swarm architecture is adopted. The lower-level particle swarm architecture performs individual fusion extraction for the two feature vectors, while the top-level particle swarm architecture performs comprehensive fusion extraction for the two feature vectors, thereby improving the accuracy and reliability of the fusion analysis.

[0116] By updating the EEG weights or EMG weights based on the velocity step value obtained from the bottom particle swarm velocity function, the iteratively updated EEG weights and EMG weights can be obtained. Then, the fitness function of the bottom subgroup is used to verify the fitness between the updated EEG weights and the training EEG feature vector, as well as the EMG weights and the training EMG feature vector. By combining the EEG threshold and the EMG threshold, the target EEG weights and target EMG weights are gradually converged and determined. The training EEG feature vector can be as shown in formula (9), the training EMG feature vector can be as shown in formula (10), the EEG weights can be as shown in formula (11), and the EMG weights can be as shown in formula (12).

[0117] (9);

[0118] Among them, F eeg This can be represented as a training EEG feature vector. This can be characterized as the training EEG characteristics corresponding to FC3 in the alpha band. This can be characterized as the training EEG characteristics corresponding to C3 in the alpha band. It can be characterized as the training EEG characteristics corresponding to FC4 in the beta band.

[0119] (10);

[0120] Among them, F emg This can be represented as a training electromyography feature vector. This can be characterized by the electromyographic features of the training muscle corresponding to the left hand. It can be characterized as the training electromyographic features corresponding to the right hand.

[0121] (11);

[0122] Among them, W e This can be characterized as EEG weighting. This can be characterized as the EEG weights of the training EEG features corresponding to FC3 in the alpha band. It can be characterized as the EEG weights of the training EEG features corresponding to FC4 in the beta band.

[0123] (12);

[0124] Among them, W m This can be characterized as electromyographic weighting. This can be characterized as the electromyographic weights of the training electromyographic features corresponding to the left hand. It can be characterized as the electromyographic weights of the training electromyographic features corresponding to the right hand.

[0125] According to an embodiment of the present invention, the target EEG fusion feature corresponding to the nth exercise training is obtained based on the target EEG weights and the training EEG feature vector corresponding to the nth exercise training, and the target EMG fusion feature corresponding to the nth exercise training is obtained based on the target EMG weights and the training EMG feature vector corresponding to the nth exercise training.

[0126] Once the target EEG weights and target EMG weights are determined, the EEG weights corresponding to each EEG feature are multiplied by each element to obtain the target EEG fusion feature. Similarly, the EMG weights corresponding to each EMG feature are multiplied by each element to obtain the target EMG fusion feature.

[0127] According to an embodiment of the present invention, based on the fitness function of the top subgroup, the target EEG fusion feature and the target EMG fusion feature are fused using the modality fusion coefficient to obtain the target EEG fusion feature.

[0128] The fitness function of the top-level subgroup can be characterized as a condition used to constrain feature fusion. By using modal fusion coefficients to fuse target EEG fusion features and target EMG fusion features, and then using the fitness function of the top-level subgroup to verify the fitness of the modal fusion coefficients, combined with the fusion threshold, the target modal fusion coefficients and target EEG / EMG fusion features are gradually converged and determined.

[0129] According to an embodiment of the present invention, the underlying particle swarm velocity function is updated using the obtained nth update coupling matrix to obtain the underlying particle swarm velocity function for the current motion imagery training. Then, the underlying particle swarm architecture in the hierarchical particle swarm architecture is used to separately fuse and extract the two feature vectors, while the top-level particle swarm architecture performs a comprehensive fusion and extraction of the two feature vectors to obtain the target brain-muscle electromyography (EMG) fusion features. This achieves enhanced correlation between multiple training rounds while considering the influence of the second electrical stimulation, constructs accurate neuromuscular coupling decoding of motion intent, improves the accuracy and reliability of fusion analysis, and significantly enhances the coupling and synergy between EMG and EEG, as well as the training effect of assisted training, from low-level analytical mining to high-level fusion perception of neuromuscular interactions.

[0130] According to an embodiment of the present invention, taking the iterative update of EEG weights as an example, the processor can also be used to: in the case of M rounds of iterative update of EEG weights, based on the underlying particle swarm velocity function corresponding to the nth exercise training, and according to the (m-1)th update velocity of the (m-1)th iterative update, the training EEG feature vector and the velocity coefficient, determine the m-th update velocity of the m-th iterative update.

[0131] The update speed of the m-th iteration can be shown in formula (13).

[0132] (13);

[0133] in, This can be characterized as the update rate corresponding to the i-th training EEG feature in the m-th iteration update. It can be characterized as inertial weight (used to keep the particle moving in the direction and trend of the previous iteration, preventing discontinuous and abrupt changes in the EEG weight coefficient, and the value is usually dynamically decreasing from 0.4 to 0.9). It can be represented as the update rate corresponding to the i-th training EEG feature in the (m-1)-th iteration update, c1 can be represented as the individual learning factor (used to adjust the intensity of the particle's learning from its own historical best), and r1 can be represented as the individual random factor. It can be characterized as the historical EEG weight corresponding to the historical best update speed of the i-th training EEG feature (i.e., the EEG weight corresponding to the historical maximum fitness EEG resolution value of the i-th training EEG feature, used to characterize the best weight combination corresponding to the i-th training EEG feature searched by the current particle). G can be represented as the (m-1)th EEG weight corresponding to the ith training EEG feature in the (m-1)th iteration update, c2 can be represented as the social learning factor (used to adjust the intensity of particle learning towards the optimal group), r2 can be represented as the social random factor, and G e It can be characterized as the historical EEG weight corresponding to the historical best update speed among all training EEG features (i.e., the EEG weight corresponding to the historical maximum fitness EEG resolution value among all training EEG features, which can be the highest weight combination among all EEG signals). It can be characterized as the coupling guidance strength coefficient (used to modulate the strength of physiological prior influence), r3 can be characterized as the electromyographic random factor, G t It can be characterized as the historical electromyographic weights corresponding to the historical best update speed of all training electromyographic features (i.e., the electromyographic weights corresponding to the historical maximum fitness electromyographic resolution of all training electromyographic features).

[0134] In the above formula (13), the first term can be the inertial term, which is used to maintain the continuity and stability of particle motion; the second term can be the individual cognition term, which is used to guide the particle to approach its own historical optimal solution; the third term can be the social learning term, which is used to guide the particle to approach the group optimal solution; the fourth term can be the coupling guidance term; the value range of c1 and c2 can be 1.5-2.0; r3 can be a random number in the interval [0,1]; m>1; M≥2.

[0135] According to an embodiment of the present invention, the EEG weights updated in the (m-1)th iteration are processed using the m-th update rate to obtain the m-th EEG weight.

[0136] Based on the current m-th update speed, the m-1th EEG weight updated in the m-1th iteration is updated in the m-th round to calculate the m-th EEG weight.

[0137] According to an embodiment of the present invention, based on the fitness function of the underlying subgroup, the m-th fitness EEG resolution value is obtained according to the m-th EEG weight and the training EEG feature vector.

[0138] The fitness function of the bottom subgroup EEG can be represented by formula (14). By substituting the m-th EEG weight and the training EEG feature vector into the fitness function of the bottom subgroup, the fitness EEG resolution value of the m-th subgroup can be obtained.

[0139] (14);

[0140] Among them, J e (W e This can be represented as the EEG resolution value of the m-th fitness level, Acc. eeg This can be represented by the calculation of EEG classification accuracy. It can be characterized as element-wise multiplication.

[0141] According to an embodiment of the present invention, if the EEG resolution value of the mth fitness is greater than the EEG threshold, the EEG fusion feature obtained based on the mth EEG weight and the trained EEG feature vector is confirmed as the target EEG fusion feature. If the EEG resolution value of the mth fitness is less than or equal to the EEG threshold, the above iterative update operation on the EEG weight is repeated until the target EEG fusion feature is obtained.

[0142] If the fitness EEG resolution value of the mth fitness is less than or equal to the EEG threshold, construct the (m+1)th update rate, thereby iteratively updating the mth EEG weight to obtain the (m+1)th EEG weight, and repeat the relevant verification until the fitness EEG resolution value obtained last time is greater than the EEG threshold.

[0143] In addition to setting EEG thresholds to constrain the target of iterative updates, the number of iterations can also be set. After reaching the predetermined number of iterations, the EEG weights obtained from the last iteration are determined as the target weights, and the EEG fusion features are determined as the target EEG fusion features.

[0144] According to embodiments of the present invention, the bottom-level particle swarm architecture in a hierarchical particle swarm architecture is used to iteratively extract and fuse two feature vectors separately. Combined with the constraints of the fitness function of the bottom-level subgroup, the target EEG fusion feature is obtained. This enhances the correlation between multiple training rounds while considering the influence of the second electrical stimulation, constructs an accurate neuromuscular coupling decoding of motor intent, and improves the accuracy and reliability of fusion analysis. This allows for detailed and accurate low-level analysis and mining of neuromuscular signals, thereby improving the training effect of assisted training.

[0145] In the process of iteratively updating the electromyography weights and obtaining the target electromyography fusion features, the above process of iteratively updating the electroencephalogram weights and obtaining the target electroencephalogram fusion features and related processing formulas can be referred to (the elements in the formulas will have adaptive changes depending on the different objects being processed). Among them, the underlying subgroup electromyography fitness function in the underlying subgroup fitness function can be as shown in formula (15).

[0146] (15);

[0147] Among them, J m (W m ) can be characterized as the electromyographic resolution value of the m-th fitness, Acc emg This can be represented as the calculation of electromyography classification accuracy.

[0148] According to an embodiment of the present invention, for obtaining the target brain-myoelectric fusion feature, the processor can also be used to: in the case of performing P rounds of fusion of the target brain-myoelectric fusion feature and the target electromyography fusion feature, obtain the p-th brain-myoelectric fusion feature based on the target brain-myoelectric fusion feature, the target electromyography fusion feature and the p-th modal fusion coefficient.

[0149] The obtained brain-muscle electrofusion characteristics can be shown as in formula (16), where P≥p≥1.

[0150] (16);

[0151] Among them, F fused(p) This can be characterized as the p-th brain-muscle electrofusion feature. This can be represented as the fusion coefficient of the p-th EEG modality. This can be characterized as the fusion coefficient of the p-th electromyographic mode. This can be characterized as a target EEG fusion feature. It can be characterized as the target electromyography fusion feature, wherein the constraint conditions satisfied between the EEG modal fusion coefficient and the electromyography modal fusion coefficient are shown in formula (17).

[0152] (17);

[0153] During the p-th round of fusion in the P-round fusion process, the p-th EEG fusion feature can be obtained by combining the target EEG fusion feature, the target EMG fusion feature, and the fusion coefficient of the p-th modality in this round.

[0154] According to an embodiment of the present invention, based on the fitness function of the top subgroup, the p-th fitness fusion resolution value is obtained according to the p-th electroencephalogram (EEG) fusion feature and fusion coefficient.

[0155] By incorporating the p-th electroencephalogram (EEG) fusion feature and fusion coefficient into the fitness function of the top subgroup, the calculated EEG fusion feature and fitness fusion resolution value in the iteration rounds are constrained and verified, thereby gradually converging to determine the target EEG fusion feature and target fusion coefficient. The fitness fusion resolution value can be shown in formula (18).

[0156] (18);

[0157] in, This can be represented as the fusion resolution value of the p-th fitness. This can be represented as a classification accuracy weighting coefficient. It can be represented as the p-th electroencephalogram (EEG) classification fusion resolution value (refer to the calculation method of the above formula (14) or formula (15)), S align This can be characterized as a physiological coupling baseline coefficient. It can be characterized as the alignment similarity between the brain-muscle electrofusion features corresponding to the p-th EEG modality fusion coefficient and the p-th EMG modality fusion coefficient and the neuromuscular physiological coupling pattern.

[0158] Among them, the first term in the above formula (18) can be used to characterize the classification accuracy of the fusion features, and the second term can be used to characterize the weighted coupling alignment degree, which is used to guide the fusion result to conform to the physiological coupling pattern of the neuromuscular system.

[0159] According to an embodiment of the present invention, if the p-th fitness fusion resolution value is greater than the fusion threshold, the p-th electroencephalogram (EEG) fusion feature is confirmed as the target EEG fusion feature. If the p-th fitness fusion resolution value is less than or equal to the fusion threshold, the above-described operation of fusing the fusion features is repeated until the target EEG fusion feature is obtained.

[0160] If the fitness fusion resolution value of the p-th fitness is less than or equal to the fusion threshold, the fusion coefficients of the p-th EEG modality and the p-th EMG modality are iteratively updated, and the relevant verification is repeated until the fitness fusion resolution value obtained last time is greater than the fusion threshold.

[0161] In addition to setting a fusion threshold to constrain the target of iterative updates, the number of iterative updates can also be set. After reaching the predetermined number of iterative updates, the EEG modal fusion coefficient and EMG modal fusion coefficient obtained in the last iterative update are determined as the target modal fusion coefficient, and the EEG-EMG fusion feature is determined as the target EEG-EMG fusion feature.

[0162] According to an embodiment of the present invention, the top-level particle swarm architecture in a hierarchical particle swarm architecture is used to fuse two feature vectors as a whole. Combined with the constraints of the fitness function of the top-level subgroup, the brain-muscle electrofusion features are iteratively updated until the target brain-muscle electrofusion features are obtained. This achieves high-level fusion perception of neuromuscular nerves by fully utilizing the specifically constructed multi-layer particle swarm architecture while considering the influence of the second electrical stimulation. Under the condition that the obtained target brain-muscle electrofusion features conform to the physiological coupling characteristics, specific behavioral training features are analyzed and highlighted, fully realizing the coupling and synergy between electromyography and electroencephalography. This allows the subsequent use of a vector machine to accurately identify the current auxiliary training results of the target brain-muscle electrofusion features, thereby improving the training effect of auxiliary training.

[0163] Figure 5 A schematic diagram illustrating the workflow of generating training results using a processor according to an embodiment of the present invention is shown.

[0164] like Figure 5 As shown, based on the training results, the processor can also be used to: obtain the electromyographic similarity value based on the similarity function, according to the target electromyographic fusion features and the ideal electromyographic fusion features corresponding to the exercise training.

[0165] During the process of multiple screening and adaptation motor imagery training for the target subjects, from the multiple target adaptation EEG feature vectors and multiple target adaptation EMG feature vectors obtained from the screening, some target adaptation EEG feature vectors and some target adaptation EMG feature vectors corresponding to the target subjects' task success (tasks of various difficulty levels) are further selected.

[0166] Then, the selected partial target-adapted EEG feature vectors and partial target-adapted EMG feature vectors are subjected to the feature extraction and fusion operations described above to obtain the target EEG / EMG fusion features corresponding to the successful completion of tasks at each difficulty level, and these are used as the ideal EEG / EMG fusion features.

[0167] Furthermore, if the target brain-motor fusion features corresponding to the successful completion of tasks at various difficulty levels cannot be obtained during multiple screening and adaptation motor imagery training processes, the target brain-motor fusion features corresponding to the first successfully completed task at each difficulty level in the formal motor imagery training, identified by the vector machine, can be taken as the ideal brain-motor fusion features.

[0168] The tasks are interconnected based on their difficulty. Task patterns include difficulty level (obstacle height: low, medium, high) and instruction type (left hand, right hand, both hands). For example, for a given instruction type, if the task is successful at the current difficulty level, the difficulty of the same instruction is increased in the next training session until three consecutive training sessions with successful high-difficulty tasks are completed, at which point the instruction type is changed. If the task is unsuccessful at the current difficulty level, the difficulty of the same instruction is decreased in the next training session, and if three consecutive training sessions with unsuccessful low-difficulty tasks are completed, the instruction type is changed to avoid fatigue of the ipsilateral hemisphere. Throughout a complete training session, the ratio of instruction types is maintained at 1:1:1 to ensure that both hemispheres receive a sufficient frequency of assisted training.

[0169] According to an embodiment of the present invention, a sports training decision score is obtained based on a predetermined recognition rule, according to the electromyographic similarity value and the particle swarm optimization fusion coefficient.

[0170] The predetermined identification rule can be characterized as a predetermined rule for calculating the sports training decision score, which can be shown in formula (19).

[0171] (19);

[0172] Where Score can be represented as the exercise training decision score, k can be represented as the total number of fusion features in the target brain-motor fusion features, q can be represented as the sequence number of a certain fusion feature in the target brain-motor fusion features, and w q This can be represented as the target (EEG or EMG) weight corresponding to the q-th fusion feature in the target EEG / EMG fusion features, f rt,q This can be characterized as the q-th fusion feature in the target brain-muscle electrofusion features, f ideal,q It can be represented as the qth fusion feature in the ideal brain-motor fusion feature, and sim can be represented as a similarity function. The similarity function can be calculated as shown in formula (20).

[0173] (20);

[0174] Where, sim(f) rt,q , f ideal,q This can be characterized as calculating the similarity between the q-th fusion feature in the target EMG fusion features and the q-th fusion feature in the ideal EMG fusion features, where range q It can be characterized as the calibration range corresponding to the q-th fusion feature in the target brain electromyography fusion features, and max can be characterized as taking the maximum value.

[0175] According to an embodiment of the present invention, the degree of exercise completion corresponding to the exercise training is determined based on the exercise training decision score and a predetermined decision threshold, and training results are generated.

[0176] After calculating the exercise training decision score, the training result is determined based on the predetermined decision threshold. The relationship between the exercise training decision score and the predetermined decision threshold can be shown in formulas (21) to (25).

[0177] 0.8≤Score≤1.0(21)

[0178] 0.6≤Score<0.8(22)

[0179] 0.4≤Score<0.6(23)

[0180] 0.2≤Score<0.4(24)

[0181] Score < 0.2 (25);

[0182] Among them, formula (21) can be characterized as judging the training result as having high task execution quality and output completion of 100%; formula (22) can be characterized as judging the training result as having output completion of 80%; formula (23) can be characterized as judging the training result as having output completion of 40%; formula (24) can be characterized as judging the training result as having output completion of 20%; formula (25) can be characterized as judging the training result as having failed task execution and not outputting positive completion feedback.

[0183] According to an embodiment of the present invention, by pre-acquiring ideal electroencephalogram (EEG) fusion characteristics and comparing the target EEG fusion characteristics obtained in the current round of motor imagery training with the ideal EEG fusion characteristics, the current task completion level is determined. This allows for visual cues to be displayed to the target object on a monitor, and the generation of a corresponding second electrical stimulation command to apply the second electrical stimulation to the target object. This achieves the application of two different modes of electrical stimulation, simultaneously providing visual cues and dual tactile cues through the hand cortex, establishing a bidirectional perceptual feedback pathway and a multimodal interactive training paradigm. This enhances the target object's initiative, sensory feedback, training motivation, practical application, and auxiliary training effects.

[0184] Furthermore, this invention can also determine whether the second electrical stimulation has a moderating effect on the task-state stability of motor imagery tasks. This is achieved by introducing a task-state stability assessment index based on weighted EEG characteristics to evaluate the effect of the second electrical stimulation. Specifically, the task-state stability assessment index can characterize whether, after introducing the second electrical stimulation as outcome-confirming tactile feedback, the activation of the sensorimotor cortex related to motor intention in subsequent motor imagery tasks exhibits a more stable and consistent expression across multiple tasks.

[0185] For example, within the task-state time window of each motor imagery task, sensorimotor-related training EEG feature vectors are extracted. Based on the obtained target EEG weights, the training EEG features in each dimension of the training EEG feature vector are weighted and summed to obtain the task-state stability assessment value. By calculating the task-state stability assessment value in multiple consecutive motor imagery tasks and analyzing its fluctuations across trials, a task-state stability assessment index is constructed to reflect the stability of cortical activation in the motor imagery task state under the influence of the second electrical stimulation.

[0186] Furthermore, when a second electrical stimulus is introduced, if the fluctuation range of the task-state stability assessment value decreases and the stability index increases in the subsequent motor imagery task, it indicates that the second electrical stimulus helps enhance the stability of cortical activation expression during the motor imagery task. Similarly, if the fluctuation range increases and the stability index decreases, it indicates that the second electrical stimulus does not help enhance the stability of cortical activation expression during the motor imagery task for the target subject, and therefore the application of the second electrical stimulus to the target subject can be suspended.

[0187] Figure 6 A schematic diagram illustrating the workflow of a task-matching visual-tactile feedback motion training device according to an embodiment of the present invention is shown.

[0188] like Figure 6 As shown on the left, a complete assisted training session for the target subject can include multiple sets of motor imagery training and a preparatory period (not multiple screening and adaptation motor imagery training). Each motor imagery training session can include a task state, a rest state, and a feedback state of varying durations. During the task state, the target subject is prompted with the specific task, task difficulty, and the hand to be trained via a display. Simultaneously, a tactile cue (first electrical stimulation) is sent to the target subject's corresponding hand to indicate the start of training. The training EEG and EMG signals collected during the task state are then processed. During the feedback state, the training results for the current round and the difficulty of the next round are presented to the target subject via a display. Simultaneously, based on the training results of the current round, the intensity and waveform of the second electrical stimulation are adaptively selected and applied to the target subject's corresponding hand to provide tactile feedback on the training results.

[0189] Figure 7 A schematic diagram illustrating the workflow of a task-matching visual-tactile feedback motion training device according to another embodiment of the present invention is shown.

[0190] like Figure 7 As shown, during the target object's motor imagery training (task state), training EEG and EMG signals are collected in real time. A hierarchical particle swarm optimization architecture is used to fuse the training EEG and EMG feature vectors from the training EEG and EMG signals to obtain the target brain-EMG fusion features. Then, a vector machine is used to classify and identify these target brain-EMG fusion features. Based on the training results, the intensity and waveform (pattern) of the second electrical stimulus applied to the target object are controlled in real time. Simultaneously, visual cues are provided to the target object via a display, and the next round of motor imagery training task is updated.

[0191] Figure 8 A flowchart of a task-matching visual-tactile feedback motion training method according to an embodiment of the present invention is shown.

[0192] like Figure 8 As shown, the embodiments of the task-matching visual-tactile feedback motion training method include operations S810 to S840.

[0193] In operation S810, in response to the target object performing motion training, the EEG signal received from the acquisition device is used as the training EEG signal, and the EMG signal received from the acquisition device is used as the training EMG signal. Based on the training EEG feature vector in the training EEG signal and the training EMG feature vector in the training EMG signal, a training coupling matrix is ​​constructed.

[0194] In operation S820, the training coupling matrix is ​​updated using the update weights and update coupling matrix corresponding to the historical training, thus obtaining the update coupling matrix corresponding to the motion training.

[0195] In operation S830, based on the particle swarm fitness function and the update coupling matrix corresponding to the exercise training, feature extraction and fusion processing are performed on the training EEG feature vector and the training EMG feature vector to obtain the target EEG fusion feature.

[0196] In the operation of S840, a vector machine is used to classify and identify the target brain electromyography fusion features to obtain training results, and based on the training results, a second electrical stimulation command is applied to the electrical stimulation device.

[0197] The effects achieved by the above method are the same as those achieved by the above device, and will not be described again here.

[0198] Figure 9A block diagram of a task-matching visual-tactile feedback motion training method according to an embodiment of the present invention is shown.

[0199] like Figure 9 As shown, an electronic device according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0200] RAM 903 stores various programs and data required for the operation of the electronic device. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0201] According to embodiments of the present invention, the electronic device may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0202] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the task-matching visual-tactile feedback motion training method according to embodiments of the present invention.

[0203] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0204] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the method for assisting in the detection of silent atrial fibrillation provided in the embodiments of the present invention.

[0205] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0206] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0207] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0208] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A task-matching visual-tactile feedback motion training device, characterized in that, include: The acquisition device is installed on the head and hands of the target object to acquire the electroencephalogram (EEG) and electromyogram (EMG) signals of the target object. An electrical stimulation device is attached to the hand of the target object to apply a first electrical stimulation to the target object before exercise training to prompt exercise training; and to apply a second electrical stimulation to the target object according to a second electrical stimulation instruction to prompt the target object's exercise training results, wherein the intensity of the first electrical stimulation and the second electrical stimulation are different. The processor, electrically connected to the acquisition device and the electrical stimulation device, is used for: In response to the target object undergoing motion training, the electroencephalogram (EEG) signal received from the acquisition device is used as the training EEG signal, and the electromyography (EMG) signal received from the acquisition device is used as the training EMG signal. A training coupling matrix is ​​constructed based on the training EEG feature vector in the training EEG signal and the training EMG feature vector in the training EMG signal. The training coupling matrix is ​​updated using the update weights and update coupling matrix corresponding to the historical training to obtain the update coupling matrix corresponding to the motion training. Based on the particle swarm fitness function and the update coupling matrix corresponding to the exercise training, feature extraction and fusion processing is performed on the training EEG feature vector and the training EMG feature vector to obtain the target EEG fusion feature. The target brain-muscle electrofusion features are classified and identified using a vector machine to obtain training results, and the second electrical stimulation command is applied to the electrical stimulation device based on the training results.

2. The apparatus according to claim 1, characterized in that, The processor is also used for: In response to the target object undergoing motion training, based on event-related spectrum perturbation, synchronous energy feature extraction is performed on the collected training EEG signals to obtain a training EEG feature vector; Based on the predetermined electromyography channel, the collected training electromyography signals are integrated to calculate the electromyography value, and the training electromyography feature vector is obtained. The training coupling matrix is ​​constructed based on the training EEG feature vector and the training EMG feature vector.

3. The apparatus according to claim 1, characterized in that, The processor is also used for: In the case of the nth exercise training in N exercise training sessions for the target object, based on the training result of the (n-1)th exercise training as historical training and the second electrical stimulation command, the (n-1)th update weight corresponding to the (n-1)th exercise training is determined, wherein the update weight is one of a first weight corresponding to the training result representing training success and a second weight corresponding to the training result representing training failure, where n>1 and N≥2; Using the (n-1)th update weight and the (n-1)th update coupling matrix corresponding to the (n-1)th exercise training, the training coupling matrix of the nth exercise training is updated to obtain the nth update coupling matrix corresponding to the nth exercise training.

4. The apparatus according to claim 3, characterized in that, The processor is also used for: Based on the nth update coupling matrix, the initial particle swarm velocity function is updated to obtain the underlying particle swarm velocity function corresponding to the nth motion training. Based on the fitness function of the underlying subgroup, the EEG weights and EMG weights corresponding to the nth exercise training are iteratively updated using the velocity function of the underlying particle swarm corresponding to the nth exercise training, so as to obtain the target EEG weights and target EMG weights. Based on the target EEG weights and the training EEG feature vector corresponding to the nth exercise training, the target EEG fusion feature corresponding to the nth exercise training is obtained. Based on the target EMG weights and the training EMG feature vector corresponding to the nth exercise training, the target EMG fusion feature corresponding to the nth exercise training is obtained. Based on the fitness function of the top subgroup, the target EEG fusion feature and the target EMG fusion feature are fused using the modality fusion coefficient to obtain the target EEG fusion feature.

5. The apparatus according to claim 4, characterized in that, The processor is also used for: With M rounds of iterative updates to the EEG weights Based on the underlying particle swarm velocity function corresponding to the nth exercise training, the m-1th update velocity of the m-1th iteration update is determined according to the m-1th update velocity of the m-1th iteration update, the training EEG feature vector and velocity coefficient, where m>1 and M≥2. Using the m-th update rate, the EEG weights updated in the (m-1)-th iteration are processed to obtain the m-th EEG weight; Based on the fitness function of the underlying subgroup, the fitness EEG resolution value is obtained according to the m-th EEG weight and the training EEG feature vector. If the m-th fitness EEG resolution value is greater than the EEG threshold, the EEG fusion feature obtained based on the m-th EEG weight and the training EEG feature vector is confirmed as the target EEG fusion feature. If the m-th fitness EEG resolution value is less than or equal to the EEG threshold, the above iterative update operation on the EEG weight is repeated until the target EEG fusion feature is obtained.

6. The apparatus according to claim 4, characterized in that, The processor is also used for: In the case of P-round fusion of the target EEG fusion features and the target EMG fusion features Based on the target EEG fusion feature, the target EMG fusion feature, and the p-th modality fusion coefficient, the p-th EEG fusion feature is obtained, where P ≥ p ≥ 1; Based on the fitness function of the top subgroup, the fitness fusion resolution value of the p-th brain-myoelectric fusion feature and fusion coefficient is obtained. If the p-th fitness fusion resolution value is greater than the fusion threshold, the p-th electroencephalogram (EEG) fusion feature is confirmed as the target EEG fusion feature. If the p-th fitness fusion resolution value is less than or equal to the fusion threshold, the above-mentioned operation of fusing the fusion features is repeated until the target EEG fusion feature is obtained.

7. The apparatus according to claim 3, characterized in that, The processor is also used for: Before the target object undergoes its first exercise training, based on event-related spectrum perturbation, multiple adaptive training EEG signals are subjected to synchronous energy feature extraction to obtain multiple adaptive EEG feature vectors. The multiple adaptive training EEG signals are EEG signals collected when the target object undergoes multiple sets of exercise adaptation training before exercise training. Based on a predetermined electromyography (EMG) channel, the integrated EMG values ​​of multiple acquired adaptive training EMG signals are calculated to obtain multiple adaptive EMG feature vectors. The multiple adaptive training EMG signals are EMG signals acquired by the target object when it undergoes multiple sets of adaptive exercise training before exercise training. According to predetermined feature selection rules, target adaptive EEG feature vector and target adaptive EMG feature vector are determined from the plurality of adaptive EEG feature vectors and the plurality of adaptive EMG feature vectors; An initial coupling matrix is ​​constructed based on the target-adapted EEG feature vector and the target-adapted EMG feature vector, and the initial coupling matrix is ​​used as the first updated coupling matrix corresponding to the first exercise training.

8. The apparatus according to claim 1, characterized in that, The processor is also used for: Based on the similarity function, the electroencephalogram (EEG) similarity value is obtained according to the target EEG fusion features and the ideal EEG fusion features corresponding to the exercise training. Based on the predetermined recognition rules, the exercise training decision score is obtained according to the electromyography similarity value and the particle swarm optimization fusion coefficient. Based on the exercise training decision score and the predetermined decision threshold, the exercise completion rate corresponding to the exercise training is determined, and the training result is generated.

9. The apparatus according to claim 1, characterized in that, The device further includes: The display screen, electrically connected to the processor, is used to display the exercise task and exercise completion rate corresponding to the exercise training to the target object.

10. A task-matched visual-tactile feedback motion training method, applied to a task-matched visual-tactile feedback motion training device as described in any one of claims 1-9, characterized in that, The method includes: In response to the target object undergoing motion training, the electroencephalogram (EEG) signal received from the acquisition device is used as the training EEG signal, and the electromyography (EMG) signal received from the acquisition device is used as the training EMG signal. A training coupling matrix is ​​constructed based on the training EEG feature vector in the training EEG signal and the training EMG feature vector in the training EMG signal. The training coupling matrix is ​​updated using the update weights and update coupling matrix corresponding to the historical training to obtain the update coupling matrix corresponding to the motion training. Based on the particle swarm fitness function and the update coupling matrix corresponding to the exercise training, feature extraction and fusion processing is performed on the training EEG feature vector and the training EMG feature vector to obtain the target EEG fusion feature. The target brain-muscle electrofusion features are classified and identified using a vector machine to obtain training results, and the second electrical stimulation command is applied to the electrical stimulation device based on the training results.

Images