Method for decoding motor intention based on multi-modal information

Abstract

The present application belongs to the technical field of physiological electrical signals, and particularly relates to a method for decoding motion intention based on multi-modal information, wherein when processing multi-modal digital signals, first, the classification results of action states such as fine actions or non-fine actions are obtained by combining video image signals and motion signals; if it is a fine action, the decoding results of motion intention are obtained by combining video image signals and electroencephalogram signals; if it is a non-fine action, the decoding results of motion intention are obtained by combining motion signals and electroencephalogram signals. Compared with the flat decoding results of only using one model, the features are more focused, each classification model only focuses on the core features required for the current classification, the model and features are better adapted, and the accuracy of recognition is improved.

Description

Technical Field

[0001] This invention belongs to the field of physiological electrical signal technology, specifically relating to a method for decoding motion intent based on multimodal information. Background Technology

[0002] Real-time decoding of user movement intentions via neural signals has become a core technology for achieving limb function reconstruction, peripheral device control, and neurorehabilitation. Since decoding movement intentions using a single modality is susceptible to external interference, leading to decreased accuracy, real-time movement intention decoding technologies often employ multimodal signal fusion to improve accuracy. For example, patent CN117390575A discloses a movement intention recognition and rehabilitation method based on brain-computer interfaces and multimodal interactive AI. This method combines EEG, EMG, motion signals, speech signals, and motion signals from normal and abnormal movements to train a deep learning model capable of movement intention recognition and rehabilitation, improving the accuracy of movement intention recognition and the effectiveness of rehabilitation. Another example is patent CN119026014A, which discloses a movement intention detection method, medical robot system, storage medium, and product. This method uses fusion to complement EEG and EMG signals, overcoming the problems of weak EEG signal strength and inaccurate EMG signal results due to user muscle fatigue, thus improving the accuracy of movement intention recognition. For example, patent CN120448748B discloses a configuration method and device for a motion intention decoding model based on EEG signal data. During the training process, by jointly analyzing the EEG signal data and the motion posture data obtained from video analysis, the model system can quickly capture the correlation between neural signals and motion trajectories, comprehensively capture motion intentions, and significantly improve the model's recognition accuracy for various complex motion patterns.

[0003] Clearly, compared with single-modal signals, multimodal fusion of EEG with other signals can effectively improve the accuracy and robustness of motion intention decoding; however, this also makes the classification model more complex and makes it difficult to simultaneously take into account the optimal combination of different training actions. Summary of the Invention

[0004] This invention provides a method for decoding motion intent based on multimodal information to solve the combinatorial optimization problem of classification models.

[0005] To address the aforementioned technical problems, this invention provides a method for decoding motion intention based on multimodal information, comprising: acquiring multimodal digital signals, including: video image signals, motion signals, and electroencephalogram (EEG) signals; combining the video image signals and motion signals to obtain a classification result of the action state, i.e., fine action or non-fine action; and acquiring a decoding result of the motion intention, i.e., if it is a fine action, then combining the video image signals and EEG signals to obtain the decoding result of the motion intention; if it is a non-fine action, then combining the motion signals and EEG signals to obtain the decoding result of the motion intention.

[0006] Furthermore, it also includes: constructing a multi-level classification model library; the multi-level classification model library includes: a first-level model library and a second-level model library; the first-level model library includes at least one first classification model; the second-level model library includes at least one second classification model and at least one third classification model; wherein the first classification model obtains the classification result of the action state by combining video image signals and motion signals; the second classification model obtains the classification result of motion information based on video image signals or motion signals; and the third classification model obtains the classification result of neural information based on electroencephalogram (EEG) signals.

[0007] Furthermore, the method of obtaining the classification result of the action state by combining video image signals and motion signals includes: motion information time synchronization, that is, obtaining the motion signal corresponding to each frame of video image based on the sampling rate of the motion sensor and the sampling rate of the video image signal; obtaining the first motion information feature based on the video image signal; obtaining the second motion information feature based on the motion signal; using the first motion information feature and the second motion information feature as the input values ​​of the first classification model, and using the classification result of the action state as the output value of the first classification model; training the first classification model based on the test data of both the video image signal and the motion signal; and using the first classification model to detect the real-time data of both the video image signal and the motion signal to obtain the classification result of the action state.

[0008] Furthermore, acquiring the first motion information feature based on the video image signal includes: acquiring information data of each frame of the video image signal; using optical tracking based on marker points or markers to correspond the identified parts to the positions of the video image sensor; finding the marked pixels according to the image processing method to calculate attitude information, position information, and velocity information; inputting the attitude information, position information, and velocity information into a first optimization model for optimization to obtain the first motion information feature; acquiring the second motion information feature based on the motion signal includes: acquiring information data of the motion sensor at each moment; acquiring attitude information by integration based on angular velocity data; acquiring velocity information based on attitude information and acceleration data; integrating the velocity information to obtain position information; inputting the attitude information, velocity information, and position information into a first optimization model for optimization to obtain the second motion information feature; the first optimization model includes at least one of the Kalman filter model and the particle filter model.

[0009] Furthermore, the secondary model library is divided into at least two secondary model groups based on the classification results of the action state. Each secondary model group includes a second classification model and a third classification model. For the second and third classification models within the same secondary model group, the number of categories in their classification results is the same.

[0010] Furthermore, the classification results of motion information obtained based on video image signals include: selecting a secondary model group 1 based on the action category of fine movements, including a second classification model 1 and a third classification model 1; obtaining third motion information features based on video image signals; using the third motion information features as the input value of the second classification model 1 and the action category of fine movements as the output value of the second classification model 1; training the second classification model 1 based on the test data of the video images corresponding to the fine movements; and using the second classification model 1 to detect real-time data of video image signals to obtain the classification probability of motion information.

[0011] Furthermore, the classification results of motion information obtained based on motion signals include: selecting a secondary model group according to the action category of non-fine movements, that is, the secondary model group includes the second classification model and the third classification model; obtaining a fourth motion information feature based on motion signals; using the fourth motion information feature as the input value of the second classification model and the action category of non-fine movements as the output value of the second classification model; training the second classification model based on the test data of motion signals corresponding to non-fine movements; and using the second classification model to detect real-time data of motion signals to obtain the classification probability of motion information.

[0012] Furthermore, the classification results of neural information obtained based on EEG signals include: segmenting the test data of EEG signals to obtain the neural information features and action category labels of each segment in the test data of EEG signals; segmenting the real-time data of EEG signals in the same way to obtain the neural information features of the real-time data segments; selecting the third classification model corresponding to the secondary model group according to the action category of fine movements or non-fine movements; inputting the neural information features of the real-time data segments into the corresponding third classification model to obtain the classification probability of the current data segment and the classification probability of the previous data segment; obtaining the number of transitions between action categories according to the action category labels of the test data segments; obtaining the transition probability matrix of action categories according to the number of transitions between each action category; fusing the classification probability of the current data segment, the classification probability of the previous data segment, and the transition probability matrix using a second optimization model to obtain the optimized classification probability of neural information; wherein the second optimization model is configured as a Hidden Markov Model.

[0013] Furthermore, the decoding result of the obtained motion intent is configured to fuse the classification probabilities of the corresponding motion information and the classification probabilities of the neural information according to the classification result of the action state, including: setting a recognition framework. ,in No. The target state refers to the type of action, whether it is a fine or non-fine movement. Represents the number of state categories; obtains the corresponding classification probability distribution based on the action category of fine-grained or non-fine-grained actions; sets the number of categories. Classification probability of each information source ,in No. Information source on state The classification probability, and satisfying Transform the classification probability into a probability assignment function, i.e. ,in No. Information source on state Support level; setting up two information sources and fusion results The conflict coefficient is ; C and C are subsets of the recognition framework. Indicates the degree of conflict between different information sources. This represents the probability allocation after fusion, i.e., the decoding result of the motion intent.

[0014] The present invention also provides a motion intent decoding system, comprising: a processor having a storage module thereon; the processor being adapted to run the steps of the method to obtain the decoding result of the motion intent; the storage module being adapted to store multimodal digital signals and a multi-level classification model library, so as to call the corresponding classification model when running the method.

[0015] This invention also provides an implantable rehabilitation training device, comprising: an electrode sensor implanted in the body for collecting intracranial electroencephalogram (EEG) signals; a video sensor located outside the body for collecting video image signals; a motion sensor located on a training peripheral for collecting motion signals; a lower-level computer comprising an in-body unit and an external unit, wherein the in-body unit receives the EEG signals and transmits them to the external unit; a higher-level computer equipped with the aforementioned motion intention decoding system to obtain the decoding results of motion intentions; and a training peripheral for executing a training program based on the decoding results of motion intentions.

[0016] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0019] The beneficial effects of this invention are that the method for decoding motion intent based on multimodal information, when processing multimodal digital signals, first combines video image signals and motion signals to obtain the classification result of the action state, such as fine action or non-fine action; if it is a fine action, then combines video image signals and EEG signals to obtain the decoding result of the motion intent; if it is a non-fine action, then combines motion signals and EEG signals to obtain the decoding result of the motion intent. By combining the characteristics of digital signals with the classification result of the action state, compared with the flat decoding result using only a single model, the features are more focused. Each classification model only focuses on the core features required for the current classification, adapting to better models and features, and improving the accuracy of recognition.

[0020] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. To make the foregoing objects, features, and advantages of the invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the hierarchical decoding process for motion intent.

[0023] Figure 2 This is a schematic diagram illustrating the role of a multi-level classification model library in decoding motion intent.

[0024] Figure 3 This is a schematic diagram of the principle of implantable rehabilitation training equipment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] During rehabilitation training, as the body gradually recovers, electromyographic (EMG) signals often gradually increase. This can lead to inconsistencies between training data and collected data, preventing the classification model from adapting in real time. Therefore, rehabilitation devices that rely on decoding EEG signals to execute training movements typically use pure EEG signals or a fusion of EEG signals with other non-EMG signals to decode motor intentions. See [link to relevant documentation]. Figure 1 At least one embodiment provides a method for decoding motion intention based on multimodal information, including: acquiring multimodal digital signals, including video image signals, motion signals, and electroencephalogram (EEG) signals; combining video image signals and motion signals to obtain a classification result of the action state, i.e., fine action or non-fine action; i.e., if it is a fine action, then combining video image signals and EEG signals to obtain a decoding result of the motion intention; if it is a non-fine action, then combining motion signals and EEG signals to obtain a decoding result of the motion intention.

[0027] Optionally, the video image signal is acquired using a video sensor, such as a wide-angle high-definition camera, which can cover the entire scene of the training action. The video sensor is a non-contact acquisition method that will not affect the subject and can be placed around the training scene.

[0028] Optionally, the motion signal is collected using motion sensors, such as angular velocity sensors and accelerometers, which mainly collect motion information of large joints and long bone segments. They are generally placed in the trunk or large limb segments to collect angular velocity data, acceleration data, etc. This can avoid interference with load-sensitive motion parts, and while being able to collect stable signals, it also allows users to quickly adapt to the presence of the sensors.

[0029] Optionally, the electrode sensor is implanted in the body for a long period of time and is suitable for collecting intracranial electroencephalogram (EEG) signals, such as cortical electrodes, deep brain electrodes, microfilament electrodes, subcutaneous electrodes, etc.

[0030] Optionally, the movement state during rehabilitation training can be divided into fine motor skills and non-fine motor skills (gross motor skills and resting state). Taking arm training as an example, gross motor skills are defined as movements involving large muscle groups (such as the arms, legs, and torso), usually related to body balance, positional movement, posture transitions, and force output; manifestations include moving the body and maintaining posture; movement categories include arm raising, arm lowering, arm extension, and arm retraction. Fine motor skills are defined as precise and coordinated movements involving small muscle groups (especially the fingers, wrists, toes, and also facial muscles), usually related to hand-eye coordination, object manipulation, and high dexterity; manifestations include precise grasping and fine manipulation; movement categories include palm extension and palm grasping.

[0031] In actual training, fine motor skills rely on tactile feedback and are sensitive to load. Wearing motion sensors may interfere with the movement, causing changes in movement patterns. Moreover, the motion signal changes are small during fine motor skills, making it difficult for motion sensors to distinguish them. Video signals, on the other hand, have high spatial resolution and can effectively identify fine motor skills. Large movements mainly involve large joints and long bone segments. The amplitude of large movement signals is significant, and motion sensors can detect them very stably. However, large movements are prone to occlusion between parts, affecting the recognition of movements in the video. Obviously, different movement states and movement categories within each movement state will lead to imbalanced samples. Different types of signal sensors have different advantages, and the features acquired from the digital signals are not entirely the same, resulting in different suitable classification models. If classification results and feature fusion are directly performed, it is easy to introduce a large number of interference factors into the classification results, making the classification model more complex and difficult to simultaneously consider the optimal combination of different training movement categories. Therefore, it is necessary to build a multi-level classification model library, which calls the corresponding classification model based on the type of digital signal and the movement state and movement category of the training movement.

[0032] Optionally, the first and third motion information features obtained from video image signals, and the second and fourth motion information features obtained from motion signals, can be set with core extraction features according to the classification results requirements or data sources. These features can be the same or different, and generally include, but are not limited to, features such as posture information, position information, and velocity information. When the first and second motion information features contain the same core extraction features, the classification results of the action state obtained by jointly using video image signals and motion signals can share one first classification model. In this case, the first-level classification (i.e., action state) of the training action is a binary classification, namely, fine action and non-fine action; in the second-level classification (i.e., action category), the action category of fine action is a binary classification, the action category of non-fine action (including large action and resting state) is a five-category classification, and the action category of large action is a four-category classification. Of course, the corresponding model library can set different classification models and the number of classification results according to the requirements of motion intent decoding, thereby realizing hierarchical recognition and using different architectures for different tasks, different signals, and different classification results. This solves the problem that it is difficult for a single model to simultaneously take into account the optimal architecture for different types. At the same time, it can also reduce the types of classification results for each model, making the features more focused. Each classification model only focuses on the core features required for the current classification, which can effectively improve accuracy and effectively avoid the problem of imbalanced samples when building the optimal architecture for different types.

[0033] Optional, see Figure 2 The multi-level classification model library includes: a primary model library and a secondary model library; the primary model library includes at least one first classification model; the secondary model library includes at least one second classification model and at least one third classification model; wherein the first classification model combines video image signals and motion signals to obtain the classification result of action state; the second classification model obtains the classification result of motion information based on video image signals or motion signals; and the third classification model obtains the classification result of neural information based on electroencephalogram (EEG) signals. Preferably, see Figure 3 The secondary model library is divided into at least two secondary model groups based on the classification results of the action state. Each secondary model group includes a second classification model and a third classification model. For the second classification model and the third classification model within the same secondary model group, the number of categories in their classification results is the same.

[0034] For example, if a patient's injury requires rehabilitation training for both fine and non-fine movements, and the decoding requirement is configured as fine and non-fine movements (including gross motor skills and resting state), then a multi-level classification model library is set up, including one binary primary model group, one binary secondary model group, and one five-category secondary model group. During decoding, the program instruction first calls the primary model group to determine whether the movement state is classified as fine or non-fine. If it is a fine movement, the program instruction calls the binary secondary model group one, and the decoding result is either hand extension or hand grasping. If it is a non-fine movement, the program instruction calls the five-category secondary model group two, and the decoding result is any one of arm raising, arm lowering, arm extension, arm retraction, or resting state. As another example, if the patient's injury is mainly concentrated in the area of ​​fine motor skills, and the decoding requirement is configured as fine movements, then a multi-level classification model library is set up, including one binary primary model group and one binary secondary model group one. During decoding, the program instructions prioritize calling the first-level model group to determine whether the action state is classified as a fine-grained or non-fine-grained action. If it is a fine-grained action, the program instructions call the second-level model group (group 1) for binary classification; otherwise, no decoding is performed. For example, if the patient's injury is mainly concentrated in areas requiring large movements, and the decoding requirement is configured as a large movement, then a multi-level classification model library is set up, including one first-level model group for binary classification and one second-level model group (group 3) for four-class classification. During decoding, the program instructions prioritize calling the first-level model group to determine whether the action state is classified as a fine-grained or non-fine-grained action. If it is a fine-grained action, no decoding is performed; otherwise, the program instructions call the second-level model group (group 3) for four-class classification. Of course, when decoding requirements differ, the number of classification results for the second and third classification models can be set according to the state and category of the trained action. Optionally, the classification model can be, for example, but not limited to, a support vector machine, logistic regression, or other training classifiers, and can be trained based on corresponding training data to obtain different types of classification models.

[0035] In some embodiments, obtaining the classification result of the action state by jointly using video image signals and motion signals includes: motion information time synchronization, i.e., obtaining the motion signal corresponding to each frame of video image based on the sampling rate of the motion sensor and the sampling rate of the video image signal; obtaining a first motion information feature based on the video image signal; obtaining a second motion information feature based on the motion signal; using the first motion information feature and the second motion information feature as input values ​​of a first classification model, and using the classification result of the action state as the output value of the first classification model, wherein the classification result of the action state is binary, i.e., fine action or non-fine action; training the first classification model based on test data of both video image signals and motion signals; and obtaining the classification result of the action state of real-time data of both video image signals and motion signals based on the first classification model. Of course, there can be multiple first classification models, each obtaining a corresponding classification probability based on the first motion information feature and the second motion information feature, and then fusing the two classification probabilities to obtain the classification result of the action state.

[0036] Optionally, obtaining the first motion information feature based on the video image signal includes: acquiring information data of each frame of video image signal; optical tracking based on marker points or markers to correspond the identified parts to the positions of the video image sensor; finding the marked pixels according to image processing methods (such as threshold segmentation, feature detection, etc.) to calculate attitude information, position information, and velocity information; and inputting the attitude information, position information, and velocity information into the first optimization model for optimization to obtain the first motion information feature.

[0037] Optionally, obtaining the second motion information feature based on the motion signal includes: acquiring information data from the motion sensor at each moment; obtaining attitude information by integrating the angular velocity data; obtaining velocity information by integrating the attitude information and acceleration data; obtaining position information by integrating the velocity information; and inputting the attitude information, velocity information, and position information into the first optimization model two for optimization to obtain the second motion information feature.

[0038] Optionally, the first optimization model (including the first optimization model one and the first optimization model two) includes at least one of the Kalman filter model and the particle filter model.

[0039] Specifically, firstly, based on the sampling rate of the motion sensor and the sampling rate of the video image, the time corresponding to the motion sensor for each frame of the video image is obtained. If the corresponding time does not exist, interpolation is used to obtain it. The motion information obtained from both methods is then input into a Kalman filter model, outputting optimized attitude, velocity, and position information. Next, based on clearly labeled test data containing both motion signals and video image signals, optimized motion information for each part is extracted. Based on the optimized motion information and labels, a classification probability model for motion information is trained, i.e., the first classification model. Based on the first classification model, the motion information of the online signal is detected in real time to obtain the classification probability of the trained action, i.e., fine or non-fine action.

[0040] In some embodiments, the classification result of motion information obtained based on video image signals includes: selecting a secondary model group, such as secondary model group one, based on the action category of fine movements, including a second classification model one and a third classification model one; obtaining third motion information features based on video image signals; using the third motion information features as input values ​​of the second classification model one and the action category of fine movements as output values ​​of the second classification model one; training the second classification model one based on test data of video images corresponding to fine movements; and using the second classification model one to detect real-time data of video image signals to obtain the classification probability of motion information.

[0041] In some embodiments, the classification result of motion information obtained based on motion signals includes: selecting a secondary model group, namely secondary model group two, according to the action category of non-fine movements, including a second classification model two and a third classification model two; obtaining a fourth motion information feature based on motion signals; using the fourth motion information feature as the input value of the second classification model two, and using the action category of non-fine movements as the output value of the second classification model two; training the second classification model two based on the motion signal test data corresponding to non-fine movements; and using the second classification model two to detect real-time data of motion signals to obtain the classification probability of motion information.

[0042] In some embodiments, the classification results of neural information obtained based on EEG signals include: dividing the test data of EEG signals to obtain the neural information features and action category labels of each segment in the test data of EEG signals; dividing the real-time data of EEG signals in the same way to obtain the neural information features of the real-time data segments; selecting the corresponding third classification model in the secondary model group according to the action category of fine movements or non-fine movements, such as selecting the third classification model one for fine movements, selecting the third classification model two for non-fine movements, and selecting the third classification model three for large movements; inputting the neural information features of the real-time data segments into the third classification model to obtain the classification probability of the current data segment and the classification probability of the previous data segment; obtaining the number of transitions between action categories according to the action category labels of the test data segments; obtaining the transition probability matrix of action categories according to the number of transitions between each action category; fusing the classification probability of the current data segment, the classification probability of the previous data segment, and the transition probability matrix using a second optimization model to obtain the optimized classification probability of neural information; wherein the second optimization model is configured as a Hidden Markov Model.

[0043] Optionally, the classification probability of neural information obtained based on EEG signals further includes: online updating of the transition probability matrix; the online updating of the transition probability matrix includes: acquiring data segments with high neural information classification probability values ​​and stable classification states; constructing a new transition probability matrix based on the number of transitions between data segments; weighted summing of the new transition probability matrix and the original transition probability matrix, and normalizing all transition probabilities for each action category. This enables online automatic optimization of the transition probability matrix, avoiding inaccurate optimization results caused by mismatch between training data and real-time data.

[0044] In some embodiments, Dempster-Shafer evidence theory (DST) is used to fuse the classification probabilities of multimodal information. See Figure 3When the first classification model classifies the action state as a fine movement, it calls the second-level model group one. This involves using the classification probabilities of the second classification model one (based on video image signals to obtain motion information) and the third classification model two (based on EEG signals to obtain neural information), fusing these probabilities to obtain the decoding result of the movement intention. When the first classification model classifies the action state as a non-fine movement, it calls the second-level model group two. This involves using the classification probabilities of the second classification model two (based on motion signals to obtain motion information) and the third classification model two (based on EEG signals to obtain neural information), fusing these probabilities to obtain the decoding result of the movement intention. When the first classification model classifies the action state as a large movement within a non-fine movement, it calls the second-level model group three. This involves using the classification probabilities of the second classification model three (based on motion signals to obtain motion information) and the third classification model three (based on EEG signals to obtain neural information), fusing these probabilities to obtain the decoding result of the movement intention. When the first classification model classifies the action state as meeting other requirements, the corresponding second-level model group N can be called. In this way, the continuity of motion information can be used to constrain the decoding results of EEG signals, thereby improving their accuracy. Simultaneously, employing a hierarchical recognition approach allows for the adaptation of better models and features, making the closed-loop feedback of the brain-computer interface more natural and continuous. The classification probability fusion process of the two is as follows.

[0045] (1) Set the system's recognition framework as follows ;in, No. Possible target states represent the category of the secondary classification result (i.e., fine action or non-fine action). Indicates the number of state categories;

[0046] (2) For classification results from different information sources, obtain the corresponding classification probability distributions. For example, for the first... Given information sources, the probability of their classification output is: ;in, No. Information source on state The classification probabilities satisfy: The above classification probabilities are converted into a probability assignment function: ;in, No. Information source on state Support level.

[0047] (3) For two information sources and The fusion result is calculated using Dempster's combination rule: The conflict coefficient is ; ; C and C are subsets of the recognition framework. Indicates the degree of conflict between different information sources. This represents the probability allocation after fusion.

[0048] Because EEG signals, motion signals, and video images have different sampling rates, the classification probability of motion information closest to the EEG signal's time is selected for subsequent fusion with the EEG signal's classification probability. Since the number of action categories in the classification results is the same across the same secondary model group, the classification probabilities of motion information and EEG signals can be weighted and integrated using a "trust assignment" method, and then fused through a conflict coefficient to obtain a more reliable probability, i.e., the decoding result of the motion intent. The classification probability adopts a maximum value retention method, using the category corresponding to the highest probability as the final classification result. A probability threshold is set; only when the maximum probability exceeds the threshold and the action category of the current segment is inconsistent with the action category of the previous segment is the action category of the classification result considered to need to be switched; otherwise, the action category of the current segment remains consistent with the classification result of the previous segment, thus obtaining the action category classification result.

[0049] Taking the probability fusion of fine motor skills as an example, we set categories. The third classification model 1 classifies the probability of EEG signals. The second classification model has the following classification probabilities for video image signals. The first value of the probability represents the probability of judging it as a grasping hand, and the second value represents the probability of judging it as an outstretched hand. Calculate the conflict coefficient. ; Calculate the probability of hand grasping after fusion. ; Calculate the probability of the palm extending after fusion. If the decoding result of the current motion intention is determined to be a hand grasping motion in fine motor skills, then the current motion intention is determined to be a hand grasping motion.

[0050] Taking the probability fusion of some large actions and resting states as an example, the category is set as follows: The classification probability of the third classification model N for EEG signals in a three-class classification system. The classification probability of the second classification model N for motion signals in the three-class classification system. The first probability value represents the probability of judging it as an arm extending forward, the second value is the probability of judging it as an arm retracting, and the third value is the probability of judging it as... The probability of conflict; calculating the conflict coefficient. ; Calculate the probability of the arm extending forward after fusion. ; Calculate the probability of arm contraction after fusion. ; Calculate the probability of the resting state after fusion. If the decoding result of the current movement intention is determined to be an arm extension during a large movement, then the current movement intention is determined to be ...

[0051] In some embodiments, a motion intent decoding system is also provided, comprising: a processor having a storage module thereon; the processor being adapted to run the steps of the method to obtain the decoding result of the motion intent; the storage module being adapted to store signal data and a multi-level classification model library to call the corresponding classification model when running the method; the method for decoding motion intent based on multimodal information may be a computer program executed by the processor and run through software.

[0052] In some embodiments, see Figure 3 Furthermore, an implantable rehabilitation training device is provided, comprising: an electrode sensor implanted in the body for collecting intracranial electroencephalogram (EEG) signals; a video sensor located outside the body for collecting video image signals; a motion sensor located on a training peripheral for collecting motion signals; a lower-level machine comprising an in-body machine and an external machine, wherein the in-body machine receives the EEG signals and transmits them to the external machine; a higher-level machine equipped with the aforementioned motion intention decoding system to obtain the decoding results of motion intentions; and a training peripheral for executing a training program based on the decoding results of motion intentions.

[0053] Specifically, the implantable rehabilitation training device includes structural components and hardware circuitry; the method operates via a host computer; the human-computer interaction system's interface is a display module; signal acquisition is provided by electrode sensors, motion sensors, video sensors, etc. Details are as follows.

[0054] (1) Lower-level machine (MCU / ARM / DSP hardware + embedded software), storing / forwarding digital signals. This includes an in-vivo unit and an external unit. The in-vivo unit is wired to the electrode sensors to transmit digital signals. The external unit communicates wirelessly with the in-vivo unit via coils and is wirelessly powered. An amplifier module (hardware analog circuit) amplifies the original analog EEG signal; an analog-to-digital converter module (hardware) converts the amplified analog EEG signal into a digital signal at a fixed sampling rate, i.e., test EEG signal and real-time EEG signal. The amplifier module and analog-to-digital converter module can be located on the in-vivo unit or integrated on the electrode sensors.

[0055] (2) Host computer, including PC hardware + PC software or APP hardware + APP software, receives digital signals such as video image signals, motion signals and EEG signals sent by the external machine. The host computer's processor contains a data processing module, a multi-level classification model library, etc., which run as software, process digital signals and display them in the display module in conjunction with the UI workflow.

[0056] Optionally, when the host computer includes a user terminal and a management terminal, the processor can be located on either the user terminal or the management terminal. The user terminal or management terminal can be understood as a cloud platform, computer, server, tablet, mobile phone, etc., with functions such as storage, data exchange, and analysis. The method for decoding motion intent based on multimodal information can run on computer software or APP software. The data processing module and multi-level classification model library on it can be understood as the process of inputting parameter options or starting program instructions through the software's operation interface and executing corresponding method steps through program instructions.

[0057] (3) Display module: Works with the human-computer interaction system to present the UI operation interface. Users can operate the UI to set parameters, select the classification results of the action state, display the decoding results of the motion intention, or preset the corresponding classification model according to the decoding requirements, etc.

[0058] (4) Training peripherals can be understood as flexible wearable glove robots, worn on the trainer's hand, and can control the execution of training actions based on the instructions formed by the decoding results of the motion intention.

[0059] In some embodiments, a computer device is also provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the method. The processor may be a central processing unit (CPU), an ASIC, or one or more integrated circuits configured to implement embodiments of the present invention. In specific implementations, if the memory and processor are implemented independently, they can be interconnected via a bus to communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a PCI bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. If the memory and processor are integrated on a single chip, they can communicate with each other through an internal interface.

[0060] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method. The storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0061] In some embodiments, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method. The computer program may include program code, which includes computer operation instructions and may be stored in a computer-readable storage medium. Based on this understanding, when the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be implemented in the form of a software product or sold or used as an independent product, the computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in the various embodiments of the present invention.

[0062] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0063] Based on the above-described preferred embodiments of the present invention, and through the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention. That is, the technical scope of the present invention is not limited to the contents of the specification.

Claims

1. A method for decoding motion intent based on multimodal information, characterized in that, include: Acquire multimodal digital signals, including video image signals, motion signals, and electroencephalogram (EEG) signals; The classification results of action state are obtained by combining video image signals and motion signals, including fine actions or non-fine actions; Obtain the decoding result of the motion intent, i.e. For fine motor skills, the decoding results of the motor intention are obtained by combining video image signals and EEG signals; If it is a non-fine movement, the decoding result of the movement intention is obtained by combining the motor signal and the EEG signal.

2. The method according to claim 1, characterized in that, Also includes: Build a multi-level classification model library; The multi-level classification model library includes: a primary model library and a secondary model library; the primary model library includes at least one first classification model; the secondary model library includes at least one second classification model and at least one third classification model; wherein... The first classification model combines video image signals and motion signals to obtain the classification results of action states; The second classification model obtains classification results based on motion information obtained from video image signals or motion signals; The third classification model is based on the classification results obtained from neural information acquired through electroencephalogram (EEG) signals.

3. The method according to claim 2, characterized in that, The classification results of action state obtained by combining video image signals and motion signals include: Motion information time synchronization, that is, obtaining the motion signal corresponding to each frame of video image based on the sampling rate of the motion sensor and the sampling rate of the video image signal; First motion information features are obtained based on video image signals; Second motion information features are obtained based on motion signals; The first motion information feature and the second motion information feature are used as the input values ​​of the first classification model, and the classification result of the action state is used as the output value of the first classification model. The first classification model is trained based on test data of both video image signals and motion signals; The first classification model is used to detect real-time data of both video image signals and motion signals to obtain the classification results of the action state.

4. The method according to claim 3, characterized in that, The features for obtaining the first motion information based on video image signals include: Obtain information data for each frame of video image signal: Optical tracking based on markers or markers to correlate identified parts with the positions of video image sensors; The marked pixels are located using image processing methods to calculate attitude, position, and velocity information. The attitude information, position information, and velocity information are input into the first optimization model for optimization to obtain the first motion information feature; Features for obtaining second motion information based on motion signals include: Acquire information data from motion sensors at every moment; Attitude information is obtained by integrating angular velocity data; Velocity information is obtained based on attitude information and acceleration data; Location information is obtained by integrating the velocity information; The attitude information, velocity information, and position information are input into the first optimization model for optimization to obtain the second motion information feature; The first optimization model includes at least one of the Kalman filter model and the particle filter model.

5. The method according to claim 2, characterized in that, The secondary model library is divided into at least two secondary model groups based on the classification results of action states. Each secondary model group includes a second classification model and a third classification model; and For the second and third classification models within the same secondary model group, the number of categories in their classification results is the same.

6. The method according to claim 5, characterized in that, The classification results based on motion information obtained from video image signals include: Based on the movement category of fine motor skills, a secondary model group is selected, namely secondary model group one, which includes the second classification model one and the third classification model one; Third motion information features are obtained from video image signals; The third motion information feature is used as the input value of the second classification model, and the motion category of fine movements is used as the output value of the second classification model. The second classification model is trained based on test data from video images corresponding to fine motor actions; The classification probability of motion information is obtained by using the real-time data of the video image signal detected by the second classification model.

7. The method according to claim 5, characterized in that, The classification results based on motion information obtained from motion signals include: A secondary model group is selected based on the movement category of non-fine movements. That is, secondary model group two includes the second classification model two and the third classification model two. Fourth motion information features are obtained based on motion signals; The fourth motion information feature is used as the input value of the second classification model, and the motion category of non-fine motion is used as the output value of the second classification model. The second classification model is trained based on test data of motion signals corresponding to non-fine movements; The classification probability of motion information is obtained by using real-time data of motion signals detected by the second classification model.

8. The method according to claim 6 or 7, characterized in that, The classification results based on neural information obtained from electroencephalogram (EEG) signals include: The test data of EEG signals were segmented to obtain the neural information characteristics of each segment of the EEG signal test data and their action category labels; Real-time EEG signals were divided in the same way to obtain the neural information features of real-time data segments; Select the corresponding third classification model from the secondary model group based on the action category of fine motor or non-fine motor; The neural information features of real-time data segments are input into the corresponding third classification model to obtain the classification probability of the current data segment and the classification probability of the previous data segment. Obtain the number of transitions between action categories based on the action category labels of the test data fragments; Based on the number of transitions between each action category, obtain the transition probability matrix for each action category; The classification probabilities of the current data segment, the previous data segment, and the transition probability matrix are fused using the second optimization model to obtain the optimized classification probabilities of the neural information; whereby... The second optimization model is configured as a hidden Markov model.

9. The method of claim 8, characterized in that, The decoding result for obtaining motion intent is configured to fuse the classification probabilities of corresponding motion information and neural information based on the classification result of the action state, including: Set up the recognition framework ,in No. The target state refers to the type of action, whether it is a fine or non-fine movement. Indicates the number of state categories; Obtain the corresponding classification probability distribution based on the action category of fine motor or non-fine motor; Set the first Classification probability of each information source ,in No. Information source on state The classification probability, and satisfying ; Transform the classification probability into a probability assignment function, i.e. ,in No. Information source on state Support level; Set up two information sources and fusion results The conflict coefficient is , C and C are subsets of the recognition framework. Indicates the degree of conflict between different information sources. This represents the probability allocation after fusion, i.e., the decoding result of the motion intent.

10. A motion intent decoding system, characterized in that, include: The processor has a storage module on it; The processor is adapted to perform the steps of the method as described in any one of claims 1-9 to obtain the decoding result of the motion intent; The storage module is adapted to store multimodal digital signals and a multi-level classification model library, so as to call the corresponding classification model when the method is run.

11. An implantable rehabilitation training device, characterized in that, include: Electrode sensors, implanted in the body, are suitable for collecting intracranial electroencephalogram (EEG) signals. Video sensors, located externally, are suitable for acquiring video image signals; Motion sensors, located on the training peripherals, are used to collect motion signals; The lower-level machine includes an in-body machine and an external machine. The in-body machine receives the EEG signals and transmits them to the external machine. A host computer is provided with the motion intent decoding system as described in claim 10 to obtain the decoding result of the motion intent; The training peripheral executes the training program based on the decoding results of the motion intent.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method as claimed in any one of claims 1-9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-9.

Images