Motor imagery device manipulation method, apparatus and system

Abstract

The motion imagination device control method, device and system, the method comprises the following steps: inputting the brain electrical signal of a wearer collected by a motion imagination device into a basic feature extraction model to obtain corresponding feature information; inputting the feature information into at least one classification and recognition model corresponding to a motion action to obtain the freedom degree discrimination result of each action; and generating the activity instruction of the motion imagination device based on all the freedom degree discrimination results. The present application separates the model, fully utilizes the existing data to realize the basic motion imagination recognition function, does not need to use a random initial model and specially train the user, divides the neural network model on the device into two parts of basic feature extraction and classification and discrimination, and can make the model more robust.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically, it designs a method, device and system for controlling motion visualization devices. Background Technology

[0002] Human physiological data, such as electroencephalograms (EEGs) and electromyograms (EMGs), serve as special input signals, enabling more convenient control of external devices and monitoring of human condition. With the development of machine learning and artificial intelligence technologies, increasingly sophisticated human-machine interfaces are being integrated into society and widely applied in medical rehabilitation and daily life.

[0003] BCI systems based on motor imagery can control external devices, using robotic arms or similar devices to replace trunk movements. This not only has applications in the medical field, providing a new control method for people with disabilities or movement disorders, helping them achieve self-care and even complete rehabilitation, but also plays a role in daily work assistance and entertainment. There are various methods for collecting brain activity information in BCI systems. Signals are categorized as "invasive" and "non-invasive" based on the access method, and in principle, both can provide input to the BCI system. EEG is a typical non-invasive brain-computer interface technology, and the electrode cap is easy to wear. Due to its good temporal resolution, ease of use, portability, and relatively low price, EEG technology is widely used to decode motor intentions and control the movement of external devices such as robotic arms or exoskeletons. However, EEG currently has many shortcomings. EEG data has strong temporal attributes and user specificity. Furthermore, due to differences in health status, usage habits, and hardware devices, EEG data exhibits randomness in constantly changing real-world environments. Using neural network models in specific scenarios may be ineffective, leading to several limitations. Summary of the Invention

[0004] The present invention provides a method, apparatus and system for controlling a motion visualization device, in order to solve at least one problem existing in the prior art.

[0005] According to one aspect of the present invention, a method for controlling a motion visualization device is provided, comprising:

[0006] The electroencephalogram (EEG) signals collected by the wearer from the motion visualization device are input into a basic feature extraction model to obtain the corresponding feature information;

[0007] The feature information is input into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0008] Activity commands for the motion visualization device are generated based on the results of all degrees of freedom to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0009] In a preferred embodiment, the number of classification and recognition models is multiple, and the motion visualization device control method further includes:

[0010] Each classification and recognition model is stored in a separate user device;

[0011] The step of inputting the feature information into at least one classification and recognition model includes:

[0012] The feature information is sent to all user devices so that the user devices can output the degree of freedom discrimination result corresponding to each classification and recognition model.

[0013] In a preferred embodiment, it further includes:

[0014] The EEG signal is filtered.

[0015] The filtered EEG signal is aligned to divide it into multiple data segments of fixed time length; the EEG signal is then input into a basic feature extraction model to obtain corresponding feature information, including:

[0016] Each data segment is input into the basic feature extraction model to obtain the feature segment corresponding to each data segment.

[0017] In a preferred embodiment, the motion visualization device is generated based on the results of all degrees of freedom determination, including:

[0018] If a classification recognition model outputs a degree of freedom determination result of yes, then the motion action execution instruction corresponding to the classification recognition model is determined.

[0019] In a preferred embodiment, it further includes:

[0020] Obtain information about the expected action input by the user;

[0021] If the expected action information is compared with the activity command, and they are inconsistent, the expected action information and the corresponding EEG signal are used as training data to train and update each classification recognition model.

[0022] In a preferred embodiment, it further includes:

[0023] Store the updated training data generated within one period.

[0024] After one period of time, the feature extraction model is trained using all the updated training data from that period of time, and the currently stored feature extraction model is replaced locally with the trained feature extraction model.

[0025] According to another aspect of the present invention, a motion visualization device control device is provided, comprising:

[0026] The first input module inputs the EEG signals collected by the motion imagery device from the wearer into a basic feature extraction model to obtain the corresponding feature information;

[0027] The first input module inputs the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0028] The control module generates activity commands for the motion visualization device based on the results of all degrees of freedom discrimination, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0029] In a preferred embodiment, the number of classification and recognition models is multiple, and the motion visualization device control device further includes:

[0030] The classification and recognition model distributed storage module stores each classification and recognition model to a separate user device.

[0031] The second input module is specifically used to send the feature information to all user devices so that the user devices can output the degree of freedom discrimination result corresponding to each classification and recognition model.

[0032] In a preferred embodiment, the preprocessing module includes:

[0033] The filtering operation unit performs filtering operations on the electroencephalogram (EEG) signal.

[0034] An alignment unit performs an alignment operation on the filtered EEG signal, splitting the EEG signal into multiple data segments of fixed time length; the step of inputting the EEG signal into a basic feature extraction model to obtain corresponding feature information includes:

[0035] The fragment input unit inputs each data fragment into the basic feature extraction model to obtain the feature fragment corresponding to each data fragment.

[0036] In a preferred embodiment, the control module is specifically used to determine the motion action execution command corresponding to the classification and recognition model if the output degree of freedom judgment result of a classification and recognition model is yes.

[0037] In a preferred embodiment, it further includes:

[0038] The expected action information acquisition module acquires the expected action information input by the user.

[0039] The classification and recognition model update module compares the expected action information with the activity command. If they are inconsistent, the expected action information and the corresponding EEG signal are used as update training data to train and update each classification and recognition model.

[0040] In a preferred embodiment, it further includes:

[0041] Update the training data storage module to store the updated training data generated within one cycle.

[0042] The offline update module, after one cycle, uses all the updated training data from that cycle to train the feature extraction model, and replaces the currently stored feature extraction model locally with the trained feature extraction model.

[0043] According to another aspect of the present invention, a motion visualization device control system includes:

[0044] Motion imagery device that can collect the wearer's electroencephalogram (EEG) signals; and

[0045] A motion visualization device control unit, comprising:

[0046] The first input module inputs the EEG signal into a basic feature extraction model to obtain the corresponding feature information;

[0047] The first input module inputs the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0048] The control module generates activity commands for the motion visualization device based on the results of all degrees of freedom discrimination, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0049] According to another aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the above-described motion visualization device control method.

[0050] According to another aspect of the present invention, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described motion visualization device control method.

[0051] The method, device, and system for controlling a motor imagery device proposed in this invention separates the models. First, the wearer's EEG signals are input into a basic feature extraction model to obtain feature information. Then, the feature information is input into a classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to achieve basic motor imagery recognition functions. There is no need to use a random initial model and train it specifically for the user. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram illustrating the temporal variation of the EEG signal in an embodiment of the present invention is shown.

[0054] Figure 2 A schematic diagram of the structure of a motion visualization device control system is shown in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a motion visualization device control method according to an embodiment of the present invention is shown;

[0056] Figure 4 A schematic diagram of the structure of a motion visualization device control device according to an embodiment of the present invention is shown;

[0057] Figure 5 A schematic diagram of a computer device structure applicable to an embodiment of the present invention is shown. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Definitions of some terms in this area:

[0060] BCI is an abbreviation for Brain Computer Interface. It is a direct connection pathway established between the human or animal brain (or a culture of brain cells) and an external device.

[0061] Neural network: A mathematical model that uses a structure similar to the synaptic connections of the brain to process information; in the following text, "model" refers to the neural network model.

[0062] Parameters: These refer to the weight parameters of neurons in a neural network, representing the strength of the connections between units and determining the degree to which the input affects the output.

[0063] EEG: an abbreviation for electroencephalogram, is a graphic obtained by amplifying and recording the spontaneous bioelectric potentials of the cerebral cortex from the scalp. It is a non-invasive technique that records the spontaneous, rhythmic electrical activity of brain cell groups through electrodes placed on the scalp to record brain signals.

[0064] Ground truth: refers to the classification accuracy of the training set in supervised learning, used to prove or disprove a hypothesis. The data that are correctly labeled are called ground truth.

[0065] Online learning, also known as online machine learning, is a type of machine learning algorithm characterized by its ability to continuously receive data and dynamically update the model in real time, making it suitable for processing large-scale and streaming data.

[0066] Currently, BCI systems based on motor imagery can control external devices, using robotic arms or similar devices to replace trunk movements. This not only has applications in the medical field, providing a new control method for people with disabilities or movement disorders, helping them achieve self-care and even complete rehabilitation, but also plays a role in daily work assistance and entertainment. There are various methods for collecting brain activity information in BCI systems, with signals categorized as "invasive" and "non-invasive" based on the access method; in principle, both can provide input to the BCI system. EEG is a typical non-invasive brain-computer interface technology, with electrode caps that are easy to wear. Due to its good temporal resolution, ease of use, portability, and relatively low cost, EEG technology is widely used in decoding motor intentions and controlling the movement of external devices such as robotic arms or exoskeletons.

[0067] EEG signals have the following characteristics: 1) The signal is non-stationary, formed by the synchronous summation of postsynaptic potentials of a large number of neurons in the cortex, and is the result of the joint activity of many neurons, containing a large amount of noise. 2) The EEG signals corresponding to brain activity at the same subjective level vary greatly among different users. 3) The quality and distribution of EEG signals vary greatly among different users and at different times. In fact, the generation of EEG data is a constantly evolving stochastic process.

[0068] like Figure 1 The diagram illustrates the temporal variation of EEG signals. In machine learning, it's generally assumed that training and real-world data must reside in the same feature space and have the same data distribution. However, EEG data possesses strong temporal attributes and user specificity. Furthermore, due to differences in health status, usage habits, and hardware devices, EEG data exhibits randomness in constantly changing real-world environments. A general neural network model may perform poorly in specific scenarios. In fact, one of the biggest challenges preventing the adoption of motion imagery decoding outside the laboratory is the limited lifespan of decoding models. Calibration training is typically required to collect data and retrain the model. In real-world daily use, trained models must be able to maintain their performance over extended periods.

[0069] Training a unique model from scratch for each user is too costly and impractical. Furthermore, the performance of such a specific model degrades with changing user states and environments. A feasible solution is to continuously collect user data and use machine learning to adjust the model to the current user, allowing it to adaptively update and adapt to user characteristics over time, making the model more robust and accurate in user experience. However, traditional machine learning methods, also known as batch learning or offline learning, have three characteristics: ① They require storing large amounts of data and transferring it multiple times during training, which is unfriendly to hardware usage and training efficiency; ② Training and inference processes need to be run separately, making the model unusable in real-time until the network model configuration is updated; ③ The training dataset is assumed to be independent and identically distributed, resulting in poor adaptability to constantly changing dynamic environments. On the other hand, motion perception involves long-term, continuous use in daily life. Because it requires long learning times and is inefficient, retraining all data every time new data is added makes large-scale offline training impractical and unable to effectively adapt to data shifts.

[0070] In a specific embodiment, the inventors discovered that, to address the mutual adaptation problem in the human-computer interface (BCI) training process, a method of collecting data and using machine learning to adapt to user characteristics has been proposed. Users actively adjust their brain activity by observing the feedback from the BCI system to adapt to its operation. After a certain number of experiments, a classifier is trained based on the feedback results. This classifier is then used to identify and provide feedback on the experimental data until training is complete. This method emphasizes focusing on the user's own data rather than being limited to static datasets in BCIs. However, it requires users to participate in training multiple times to generate an initial model and repeatedly train their motor imagery abilities, adding unnecessary workload. Furthermore, this method, used for system training, cannot adaptively update the model online during continuous daily use, requiring additional offline training costs.

[0071] Furthermore, one of the biggest challenges preventing the application of motion visualization technology outside the laboratory is the limited lifespan of the decoding model, which must be able to maintain its performance for extended periods in real-world daily use.

[0072] Based on this, and considering the strong temporal and personal characteristics of EEG data mentioned in the background, as well as the unsuitability of a single, general model for everyday motion imagery, this invention proposes an adaptive learning motion imagery system that combines continuous inference with fine-tuning and updating the model. The streaming computing model of online learning is well-suited for utilizing continuously arriving massive amounts of data, enabling timely responses to changes in data distribution, and effectively addressing the shortcomings of traditional machine learning in solving motion imagery problems.

[0073] The present invention will now be described in detail with reference to the accompanying drawings.

[0074] This invention first provides a specific embodiment of a motion visualization device control system, see [link to specific embodiment]. Figure 2 As shown, it includes a control device for a motor imagery device and a motor imagery device. The motor imagery device can collect the wearer's electroencephalogram (EEG) signals; the control device for the motor imagery device can specifically be used to: input the EEG signals into a basic feature extraction model to obtain corresponding feature information; input the feature information into at least one classification and recognition model corresponding to a motor action to obtain the degree of freedom discrimination result for each action; and generate activity instructions for the motor imagery device based on all degree of freedom discrimination results to control the motor imagery device; wherein, the basic feature extraction model and the classification and recognition model are obtained through offline training based on user information data.

[0075] In specific implementation, the control device is divided into units according to the functions of the control device in one embodiment. The motion visualization device control device includes a computer main body, a parameter configuration unit, an update unit, an identification unit, a storage unit, a signal processing unit, and an interaction unit. Each unit can be an independent processing device. For example, the motion visualization device control device of the present invention can be a device main body with multi-device communication connection composed of a server cluster. The various units in the above device are described one by one below.

[0076] ① Data acquisition unit: includes hardware such as EEG helmet (i.e., motor imagery device), which converts brain activity into digital signals that can be processed by the system.

[0077] ② Signal Processing Unit: This unit includes a preprocessing module and a basic feature extraction model. The preprocessing module performs filtering, alignment, and segmentation on the collected data, converting the physiological data into a standard data format for use as input to the neural network. The basic feature extraction model is commonly implemented using CNN, LSTM, or a combination of both. It extracts EEG signals as feature representations as intermediate values, reducing data dimensionality and complexity, and serves as input to subsequent models to facilitate classification by the recognition unit.

[0078] ③ Recognition Unit: This unit contains multiple classification models. It receives signal features extracted by the processing unit and sends them to each model for inference. Each model corresponds to a binary classification task with one degree of freedom in motion visualization. This separation of multiple degrees of freedom improves the robustness of the entire recognition unit and the flexibility of the device. Single-degree-of-freedom recognition and classification can be a simple binary classification task with fewer parameters and lower complexity, such as simple feature matrix calculations, improving the efficiency of online learning and reducing the cost of loss calculation and parameter updates.

[0079] ④ Parameter Configuration Unit: Accepts pre-trained general-purpose models as input from external data to support the system in performing basic inference and classification functions, and collects more reliable data and labels for online learning. The overall model can also be configured from external sources at any time.

[0080] ⑤ Storage unit: Stores a small dataset - its content consists of a limited number of EEG signal data samples within a certain period or the most recent time, and labels obtained from inference and feedback loss calculations, in the form of data pairs (x,y) for subsequent offline learning and training.

[0081] ⑥ Update Unit: Includes three components - Model Parameter Configuration Module: (1) Accepts model parameters from the initial parameter configuration unit and configures the neural network model parameters in the system's signal processing unit and recognition unit. (2) Feature Extraction Update Module: Performs batch learning using stored labeled offline datasets at certain intervals (e.g., X hours), and combines learning with the corresponding classification model. The frequency is low and the accuracy is high, updating the parameters of the basic feature extraction model. (3) Classification Model Update Module: Performs online learning after obtaining the ground truth of the fragment data, and updates the parameters of all recognition units or a single classification model according to the feedback type.

[0082] ⑦ Interactive Unit: Enables human-computer interaction with the user, including (1) the realization of motion imagination functions, such as virtual reality data communication, hardware auxiliary device actions, etc. (2) Recording user feedback information on the overall functional realization or specific degree-of-freedom components, which serves as ground truth for completing online learning.

[0083] The system's data transmission is divided into three parts: the exchange between external user datasets and local devices, the data and feedback generated during user use, and the storage of data and the learning and updating of the model.

[0084] Using existing user information datasets, a general neural network model with sufficient reasoning ability and recognition accuracy is trained offline, meaning it has a general accuracy rate for all users. This model can be divided into feature extraction and classification / discrimination parts, and can be deployed to users' local devices for daily use via a parameter configuration unit.

[0085] The device collects the user's EEG signals during use and preprocesses the signals to produce a standard dataset in a format suitable for input into a neural network for inference and training. The data is then classified and identified by the model, and the interactive unit enables motion visualization, including invoking the hardware robotic arm. Users can evaluate the motion visualization results, which are recorded by the device.

[0086] For the current data segment, if the feedback result is incorrect, the correct result is easily obtained because the recognition task is a binary classification task. The data segment and its corresponding true result label are used as data pairs, and the classification recognition model's parameters are updated using an online learning algorithm. Labeled data from every X hours is saved to the device's storage unit. When the device is not in use, or periodically using all recently stored data, the feature extraction model's parameters are updated through offline batch training. The parameter iteration of both models allows the model to better fit the current user's data distribution, resulting in higher accuracy as a personalized model.

[0087] Users provide positive or negative feedback on the overall or individual degrees of freedom of the motion visualization result. If no feedback is provided, the inference result is assumed to be labeled as correct.

[0088] As can be seen, the motor imagery device control system provided in this aspect, by separating the model, first inputs the wearer's EEG signal into the basic feature extraction model to obtain feature information, and then inputs the feature information into the classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to realize basic motor imagery recognition functions. There is no need to use a random initial model and train it specifically for the user. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust.

[0089] Figure 3 An embodiment of the present invention illustrates a method for controlling a motion visualization device, comprising:

[0090] S100: Input the EEG signals of the wearer collected by the motion imagery device into a basic feature extraction model to obtain the corresponding feature information;

[0091] S200: Input the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0092] S300: Generate activity instructions for the motion visualization device based on the discrimination results of all degrees of freedom, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0093] The proposed method for controlling a motor imagery device involves separating the model. First, the wearer's EEG signals are input into a basic feature extraction model to obtain feature information. Then, the feature information is input into a classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to achieve basic motor imagery recognition functions. There is no need to use a random initial model and train it specifically for the user. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust.

[0094] In this invention, an existing user information dataset is used to train a general neural network model W with sufficient reasoning ability and recognition accuracy through offline learning. The model can be divided into a basic feature extraction model W. e And classification and discrimination model W c .

[0095] In a preferred embodiment, to avoid the processing burden on the device, the classification and recognition model corresponding to each motion action is imported into a separate computer device, and the number of the classification and recognition models is multiple. The motion visualization device control method further includes:

[0096] Each classification and recognition model is stored in a separate user device; the step of inputting the feature information into at least one classification and recognition model includes:

[0097] The feature information is sent to all user devices so that the user devices can output the degree of freedom discrimination result corresponding to each classification and recognition model.

[0098] In this embodiment, W is distributed and deployed to the user's local device, with different degrees of freedom corresponding to W. c1 ...W cn Here, n represents the number of degrees of freedom. Multiple classification and recognition models can be deployed on multiple computing devices, thereby reducing the burden on a single computer and improving the overall computing speed.

[0099] In some embodiments, the method further includes: preprocessing the electroencephalogram (EEG) signals.

[0100] In this invention, preprocessing can standardize EEG signals. Of course, it is understood that preprocessing is not necessary. For example, if the acquired EEG signals are relatively standardized or the accuracy requirement is not high, preprocessing is not required.

[0101] In this embodiment of the invention, the preprocessing operation can be performed in the following manner:

[0102] The EEG signal is filtered.

[0103] The filtered EEG signal is aligned to split it into multiple data segments of fixed time length.

[0104] In this embodiment, the step of inputting the EEG signal into a basic feature extraction model to obtain corresponding feature information includes: inputting each data segment into the basic feature extraction model to obtain feature segments corresponding to each data segment.

[0105] Since each EEG signal can contain multiple motor actions, meaning each data segment corresponds to at least one motor action, it is not necessary to traverse all data during classification. When determining the specific degrees of freedom, the degrees of freedom of that action information will only be generated if the EEG information contains the action information corresponding to the classification model. Therefore, for the entire EEG signal, even if other action information that does not correspond to the classification model is input into the classification model, the degrees of freedom of that action information cannot be output. By segmenting the data, the amount of data processing can be greatly reduced.

[0106] For example, suppose a movement has degrees of freedom of 1 and -1, where 1 indicates that the EEG signal contains the movement and -1 indicates that the EEG signal does not contain the movement. In other words, when the entire EEG signal is input into the classification model corresponding to the movement, only the output of the EEG signal corresponding to the movement corresponds to 1, and other movements are input into the specific classification model and output -1. However, the entire process requires inputting the complete EEG signal into each classification model in order to identify all movements.

[0107] After data slicing, the EEG signal is divided into multiple sub-signals. At this point, the data processing only requires extracting features from each sub-signal and then inputting the features into the corresponding classification and discrimination model, thus avoiding excessive input to the model at one time.

[0108] Furthermore, in some embodiments, the motion visualization device's activity instructions are generated based on the results of all degrees of freedom determination, including:

[0109] If a classification recognition model outputs a degree of freedom determination result of yes, then the motion action execution instruction corresponding to the classification recognition model is determined.

[0110] For example, the original data D is input into the basic feature extraction model W. e In this process, the feature representation F of the corresponding data segment is obtained. The feature representation F is then input into the classification and discrimination model W. ci In this process, the discrimination result T for the corresponding degree of freedom is obtained. i In a binary classification task, the result T∈{-1,+1} indicates whether that degree of freedom is active. Based on the discrimination results T1……T n The equipment will then perform the corresponding actions.

[0111] In other embodiments, the present invention further includes: acquiring expected action information input by the user; comparing the expected action information with the activity instruction; if they are inconsistent, using the expected action information and the corresponding EEG signal as update training data to train and update each classification recognition model.

[0112] In this embodiment, the user can default to the response meeting expectations, i.e., the judgment result is correct, or manually provide correct feedback, indicating that the model output result {D, F, T} is the real triplet dataset, and store the data pair {D, T}.

[0113] Users can also indicate that the response is not as expected, i.e., the judgment result is incorrect, suggesting that there is a problem with the model's output result {D, F, T}. Using the data pair {F, -T} as the true label data, the classification model W is then evaluated. ci Using an online machine learning algorithm, the model parameters are updated to obtain W. ci Store the data pair {D, -T}. Continue this process for multiple iterations from s3 to s9, updating W multiple times. c1 ...W cn This improves the accuracy of model discrimination.

[0114] Furthermore, some embodiments of the present invention also include:

[0115] Store the updated training data generated within one period.

[0116] After one period of time, the feature extraction model is trained using all the updated training data from that period of time, and the currently stored feature extraction model is replaced locally with the trained feature extraction model.

[0117] In this embodiment, after collecting data and updating it online for one cycle, such as a fixed X hours, or when the device is in an unused state, offline batch machine learning is performed on the local model W using all the data stored during this cycle, pairing {D, T} or {D, -T}, updating only the feature extraction model to W. e ′.

[0118] The updated W e ′ and Wc This forms a new personalized model W', which enhances the adaptability to user data distribution.

[0119] It is understandable that EEG data has strong temporal and personal characteristics, and that a general single neural network model is not suitable for use in continuous daily motion visualization. This invention proposes a motion visualization method that can adaptively learn changes in user characteristics. It achieves high-efficiency and low-cost online learning while in use, enabling the model to reflect the migration and changes in data distribution in a timely manner without retraining the overall model or storing large amounts of data offline.

[0120] Furthermore, by pre-training a general neural network model using a limited dataset of relevant user groups, basic motion image recognition functions can be achieved by making full use of existing data, without the need to use a random initial model and specifically train it on users.

[0121] Simultaneously dividing the neural network model on the device into two parts—basic feature extraction and classification—makes the model more robust. Each degree of freedom is a simple binary classification task, allowing users to more accurately report model label errors. Applying online learning parameter updates only to the classification model improves training efficiency, reduces training costs, and facilitates low-power implementation of the hardware device.

[0122] Furthermore, storing only labeled data within a certain period reduces the requirements for storage space and data exchange, enabling batch offline learning at a lower frequency. This allows the feature extraction capability to generalize across different degrees of freedom, improving accuracy and better balancing the conflict between accuracy and training cost.

[0123] The inventive concept of this solution can not only be applied to adaptive learning of motor imagery, but also extended to the application of EEG signals or other physiological signals. It is very suitable for processing massive amounts of continuously arriving data, can respond promptly to changes in data distribution, and can be adjusted according to actual needs, thus exhibiting high flexibility.

[0124] Based on the same inventive concept, such as Figure 4 As shown, another embodiment of the present invention provides a motion visualization device control device, comprising:

[0125] The first input module 10 inputs the electroencephalogram (EEG) signals of the wearer collected by the motion visualization device into a basic feature extraction model to obtain the corresponding feature information;

[0126] The first input module 20 inputs the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0127] The control module 30 generates activity commands for the motion visualization device based on the discrimination results of all degrees of freedom, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0128] The motion imagery device control device proposed in this invention separates the model. First, the wearer's EEG signal is input into the basic feature extraction model to obtain feature information. Then, the feature information is input into the classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to realize basic motion imagery recognition functions. There is no need to use a random initial model and train it specifically for the user. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust.

[0129] Specifically, in this invention, an existing user information dataset is used to train a general neural network model W with sufficient reasoning ability and recognition accuracy through offline learning. The model can be divided into a basic feature extraction model W. e And classification and discrimination model W c .

[0130] In a preferred embodiment, the number of classification and recognition models is multiple, and the motion visualization device control device further includes:

[0131] The classification and recognition model distributed storage module stores each classification and recognition model to a user device; the second input module is specifically used to send the feature information to all user devices so that the user devices output the degree of freedom discrimination result corresponding to each classification and recognition model.

[0132] In this embodiment, W is distributed and deployed to the user's local device, with different degrees of freedom corresponding to W. c1 ...W cn Here, n represents the number of degrees of freedom. Multiple classification and recognition models can be deployed on multiple computing devices, thereby reducing the burden on a single computer and improving the overall computing speed.

[0133] In a preferred embodiment, it further includes:

[0134] The preprocessing module preprocesses the electroencephalogram (EEG) signals.

[0135] In this invention, preprocessing can standardize EEG signals. Of course, it is understood that preprocessing is not necessary. For example, if the acquired EEG signals are relatively standardized or the accuracy requirement is not high, preprocessing is not required.

[0136] In a preferred embodiment, the preprocessing module includes:

[0137] The filtering operation unit performs filtering operations on the electroencephalogram (EEG) signal.

[0138] An alignment unit performs an alignment operation on the filtered EEG signal, splitting the EEG signal into multiple data segments of fixed time length; the step of inputting the EEG signal into a basic feature extraction model to obtain corresponding feature information includes:

[0139] The fragment input unit inputs each data fragment into the basic feature extraction model to obtain the feature fragment corresponding to each data fragment.

[0140] In this embodiment, the step of inputting the EEG signal into a basic feature extraction model to obtain corresponding feature information includes: inputting each data segment into the basic feature extraction model to obtain feature segments corresponding to each data segment.

[0141] Since each EEG signal can contain multiple motor actions, meaning each data segment corresponds to at least one motor action, it is not necessary to traverse all data during classification. When determining the specific degrees of freedom, the degrees of freedom of that action information will only be generated if the EEG information contains the action information corresponding to the classification model. Therefore, for the entire EEG signal, even if other action information that does not correspond to the classification model is input into the classification model, the degrees of freedom of that action information cannot be output. By segmenting the data, the amount of data processing can be greatly reduced.

[0142] For example, suppose a movement has degrees of freedom of 1 and -1, where 1 indicates that the EEG signal contains the movement and -1 indicates that the EEG signal does not contain the movement. In other words, when the entire EEG signal is input into the classification model corresponding to the movement, only the output of the EEG signal corresponding to the movement corresponds to 1, and other movements are input into the specific classification model and output -1. However, the entire process requires inputting the complete EEG signal into each classification model in order to identify all movements.

[0143] After data slicing, the EEG signal is divided into multiple sub-signals. At this point, the data processing only requires extracting features from each sub-signal and then inputting the features into the corresponding classification and discrimination model, thus avoiding excessive input to the model at one time.

[0144] In a preferred embodiment, the control module is specifically used to determine the motion action execution command corresponding to the classification and recognition model if the output degree of freedom judgment result of a classification and recognition model is yes.

[0145] For example, the original data D is input into the basic feature extraction model W. e In this process, the feature representation F of the corresponding data segment is obtained. The feature representation F is then input into the classification and discrimination model W.ci In this process, the discrimination result T for the corresponding degree of freedom is obtained. i In a binary classification task, the result T∈{-1,+1} indicates whether that degree of freedom is active. Based on the discrimination results T1……T n The equipment will then perform the corresponding actions.

[0146] In a preferred embodiment, it further includes:

[0147] The expected action information acquisition module acquires the expected action information input by the user.

[0148] The classification and recognition model update module compares the expected action information with the activity command. If they are inconsistent, the expected action information and the corresponding EEG signal are used as update training data to train and update each classification and recognition model.

[0149] In this embodiment, the user can default to the response meeting expectations, i.e., the judgment result is correct, or manually provide correct feedback, indicating that the model output result {D, F, T} is the real triplet dataset, and store the data pair {D, T}.

[0150] Users can also indicate that the response is not as expected, i.e., the judgment result is incorrect, suggesting that there is a problem with the model's output result {D, F, T}. Using the data pair {F, -T} as the true label data, the classification model W is then evaluated. ci Using an online machine learning algorithm, the model parameters are updated to obtain W. ci Store the data pair {D, -T}. Continue this process for multiple iterations from s3 to s9, updating W multiple times. c1 ...W cn This improves the accuracy of model discrimination.

[0151] In a preferred embodiment, it further includes:

[0152] Update the training data storage module to store the updated training data generated within one cycle.

[0153] The offline update module, after one cycle, uses all the updated training data from that cycle to train the feature extraction model, and replaces the currently stored feature extraction model locally with the trained feature extraction model.

[0154] In this embodiment, after collecting data and updating it online for one cycle, such as a fixed X hours, or when the device is in an unused state, offline batch machine learning is performed on the local model W using all the data stored during this cycle, pairing {D, T} or {D, -T}, updating only the feature extraction model to W. e ′.

[0155] The updated W e ′ and Wc This forms a new personalized model W', which enhances the adaptability to user data distribution.

[0156] It is understandable that EEG data has strong temporal and personal characteristics, and that a general single neural network model is not suitable for use in continuous daily motion visualization. This invention proposes a motion visualization method that can adaptively learn changes in user characteristics. It achieves high-efficiency and low-cost online learning while in use, enabling the model to reflect the migration and changes in data distribution in a timely manner without retraining the overall model or storing large amounts of data offline.

[0157] Furthermore, by pre-training a general neural network model using a limited dataset of relevant user groups, basic motion image recognition functions can be achieved by making full use of existing data, without the need to use a random initial model and specifically train it on users.

[0158] Simultaneously dividing the neural network model on the device into two parts—basic feature extraction and classification—makes the model more robust. Each degree of freedom is a simple binary classification task, allowing users to more accurately report model label errors. Applying online learning parameter updates only to the classification model improves training efficiency, reduces training costs, and facilitates low-power implementation of the hardware device.

[0159] Furthermore, storing only labeled data within a certain period reduces the requirements for storage space and data exchange, enabling batch offline learning at a lower frequency. This allows the feature extraction capability to generalize across different degrees of freedom, improving accuracy and better balancing the conflict between accuracy and training cost.

[0160] The inventive concept of this solution can not only be applied to adaptive learning of motor imagery, but also extended to the application of EEG signals or other physiological signals. It is very suitable for processing massive amounts of continuously arriving data, can respond promptly to changes in data distribution, and can be adjusted according to actual needs, thus exhibiting high flexibility.

[0161] From a hardware perspective, the present invention provides an embodiment of an electronic device for implementing all or part of the motion visualization device control method, wherein the electronic device specifically includes the following:

[0162] The device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to enable information transmission between devices; the electronic device may be a desktop computer, tablet computer, or mobile terminal, etc., but this embodiment is not limited to these.

[0163] Figure 5 This is a schematic block diagram illustrating the system configuration of an electronic device 9600 according to an embodiment of the present invention. Figure 5 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 5 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0164] In one embodiment, the motion visualization device control functions can be integrated into the central processing unit 9100. For example, the central processing unit 9100 can be configured to perform the following controls:

[0165] S100: Input the EEG signals of the wearer collected by the motion imagery device into a basic feature extraction model to obtain the corresponding feature information;

[0166] S200: Input the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action;

[0167] S300: Generate activity instructions for the motion visualization device based on the discrimination results of all degrees of freedom, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data.

[0168] As can be seen from the above description, the electronic device provided by the embodiments of the present invention, by separating the model, first inputs the wearer's EEG signal into the basic feature extraction model to obtain feature information, and then inputs the feature information into the classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to realize the basic motor imagery recognition function. There is no need to use a random initial model and train the user specifically. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust.

[0169] In another embodiment, the motion visualization device control device can be configured separately from the central processing unit 9100. For example, the motion visualization device control device can be configured as a chip connected to the central processing unit 9100, and the motion visualization device control function can be realized through the control of the central processing unit.

[0170] like Figure 5 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 5All components shown; in addition, the electronic device 9600 may also include Figure 5 For components not shown, please refer to existing technologies.

[0171] like Figure 5 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0172] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0173] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0174] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0175] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0176] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0177] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0178] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all steps of the motion visualization device control method in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the motion visualization device control method in the above embodiments.

[0179] As can be seen from the above description, the computer-readable storage medium provided in the embodiments of the present invention, by separating the model, first inputs the wearer's EEG signal into the basic feature extraction model to obtain feature information, and then inputs the feature information into the classification and recognition model. Both the basic feature extraction model and the classification and recognition model are trained locally offline, making full use of existing data to realize the basic motor imagery recognition function. There is no need to use a random initial model and specifically train the user. Dividing the neural network model on the device into two parts, basic feature extraction and classification and discrimination, can make the model more robust.

[0180] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for controlling a motion visualization device, characterized in that, include: The electroencephalogram (EEG) signals collected by the wearer from the motion visualization device are input into a basic feature extraction model to obtain the corresponding feature information; The feature information is input into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action; Based on the results of all degrees of freedom discrimination, activity commands for the motion visualization device are generated to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained through offline training based on user information data; Obtain information about the expected action input by the user; If the expected action information is not consistent with the activity command, the expected action information and the corresponding EEG signal are used as training data to train and update each classification recognition model. Store the updated training data generated within one period. After one period of time, the feature extraction model is trained using all the updated training data from that period of time, and the currently stored feature extraction model is replaced locally with the trained feature extraction model.

2. The method for controlling a motion visualization device according to claim 1, characterized in that, The number of classification and recognition models is multiple, and the motion imagination device control method further includes: storing each classification and recognition model to a user device.

3. The method for controlling a motion visualization device according to claim 1, characterized in that, Also includes: The EEG signal is filtered. The filtered EEG signal is aligned to split it into multiple data segments of fixed time length. The step of inputting the EEG signal into a basic feature extraction model to obtain corresponding feature information includes: Each data segment is input into the basic feature extraction model to obtain the feature segment corresponding to each data segment.

4. The method for controlling a motion visualization device according to claim 2, characterized in that, Based on the results of all degrees of freedom discrimination, activity instructions for the motion visualization device are generated, including: If a classification recognition model outputs a degree of freedom determination result of yes, then the motion action execution instruction corresponding to the classification recognition model is determined.

5. A control device for a motion visualization device, characterized in that, include: The first input module inputs the EEG signals collected by the motion imagery device from the wearer into a basic feature extraction model to obtain the corresponding feature information; The first input module inputs the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action; The control module generates activity commands for the motion visualization device based on the results of all degrees of freedom discrimination, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data; The expected action information acquisition module acquires the expected action information input by the user. The classification and recognition model update module compares the expected action information with the activity command. If they are inconsistent, the expected action information and the corresponding EEG signal are used as update training data to train and update each classification and recognition model. Update the training data storage module to store the updated training data generated within one cycle. The offline update module, after one cycle, uses all the updated training data from that cycle to train the feature extraction model, and replaces the currently stored feature extraction model locally with the trained feature extraction model.

6. A motion visualization device control system, characterized in that, include: Motion imagery device that can collect the wearer's electroencephalogram (EEG) signals; as well as A motion visualization device control unit, comprising: The first input module inputs the EEG signal into a basic feature extraction model to obtain the corresponding feature information; The first input module inputs the feature information into at least one classification and recognition model corresponding to a motion action to obtain the degree of freedom discrimination result for each action; The control module generates activity commands for the motion visualization device based on the results of all degrees of freedom discrimination, so as to control the motion visualization device; wherein, the basic feature extraction model and the classification and recognition model are obtained by offline training based on user information data; The expected action information acquisition module acquires the expected action information input by the user. The classification and recognition model update module compares the expected action information with the activity command. If they are inconsistent, the expected action information and the corresponding EEG signal are used as update training data to train and update each classification and recognition model. Update the training data storage module to store the updated training data generated within one cycle. The offline update module, after one cycle, uses all the updated training data from that cycle to train the feature extraction model, and replaces the currently stored feature extraction model locally with the trained feature extraction model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the motion visualization device control method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the motion visualization device control method according to any one of claims 1 to 4.

Images