Method and device for configuring motion intention decoding model based on electroencephalogram signal data

Abstract

The invention provides a method and a device for configuring a motion intention decoding model based on electroencephalogram signal data, which are used for providing a set of novel training mechanism of the motion intention decoding model, and performing conjoint analysis on the electroencephalogram signal data and motion posture data obtained by analyzing videos in the training process. A model system can quickly capture association between neural signals and motion tracks and comprehensively capture motion intentions, the recognition precision of the model for various complex motion modes is remarkably improved, meanwhile, computing resources needed by model processing are reduced, the whole decoding process is more efficient, individual adaptability and generalization are considered, and the method is suitable for popularization and application. And the real-time application requirement of intrusive brain-computer interface research can be well met.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a configuration method and device for a motion intention decoding model based on electroencephalogram (EEG) signal data. Background Art

[0002] The instructions generated by neural signals direct actual motor movements, but the complexity of neural signals and the complexity of the actual movement process limit the decoding process of neural signals. Invasive brain-computer interfaces (BCIs) implant electrodes into the cerebral cortex to capture electrophysiological signals generated by neuronal activity, namely electroencephalograms (EEGs), to analyze the state of brain function.

[0003] At present, the development of brain-computer interface technology using non-human primates, especially macaques, as research subjects is a key step in realizing the transformation of invasive brain-computer interfaces into clinical applications. However, there are still some technical challenges that need to be overcome in brain-computer interface control experiments on non-human primates, such as the analysis of movement processes and movement intention signals.

[0004] Most current research focuses on classifying EEG signals of brain-computer interfaces. First, the EEG signal features are extracted using time-frequency domain transformation methods, and then classified using machine learning methods. Although with the advancement of technology in recent years, deep learning technology has also been used in EEG signal classification and has achieved good results.

[0005] However, the inventors of this application found that the existing invasive brain-computer interface research, the technology of obtaining macaque EEG signals through implanted electrodes has gradually matured, but the motion intention decoding model configured by deep learning technology still has limitations in terms of processing performance in real-time analysis and accurate decoding. Summary of the Invention

[0006] The present application provides a configuration method and device for a motion intention decoding model based on EEG signal data, which is used to provide a novel training mechanism for the motion intention decoding model. During the training process, the EEG signal data and the motion posture data obtained by video analysis are jointly analyzed, so that the model system can quickly capture the relationship between neural signals and motion trajectories, and comprehensively capture motion intentions, significantly improving the model's recognition accuracy for various complex motion patterns. At the same time, it also reduces the computing resources required for model processing, making the entire decoding process more efficient, taking into account individual adaptability and generalization, and can well meet the real-time application needs of invasive brain-computer interface research.

[0007] In a first aspect, the present application provides a method for configuring a motion intention decoding model based on EEG signal data, the method comprising: Acquiring monitoring images of the sample moving object while the sample moving object is performing a designated moving task; Performing image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task and obtain corresponding sample motion posture data; When the sample moving subject performs a designated movement task, the EEG signal of the sample moving subject is collected through an invasive brain-computer interface system, and corresponding sample EEG signal data is obtained; The sample motion posture data and sample EEG signal data are used as training samples to train the motion intention decoding model, wherein the motion posture data is used as annotations, and the motion intention decoding model is a deep learning model. The motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

[0008] In a first aspect, the present application provides a configuration device for a motion intention decoding model based on EEG signal data, the device comprising: A first acquisition unit is configured to acquire a monitoring image of the sample moving object while the sample moving object is performing a designated moving task; The recognition unit is used to perform image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task and obtain corresponding sample motion posture data; The second acquisition unit is used to acquire EEG signals of the sample moving subject through the invasive brain-computer interface system while the sample moving subject is performing a specified movement task, and obtain corresponding sample EEG signal data; The training unit is used to use sample motion posture data and sample EEG signal data as training samples to train a motion intention decoding model, wherein the motion posture data is used as annotations, the motion intention decoding model is a deep learning model, and the motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

[0009] In a third aspect, the present application provides a processing device including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method provided in the first aspect of the present application is executed.

[0010] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the method provided in the first aspect of the present application.

[0011] From the above content, it can be concluded that this application has the following beneficial effects: Focusing on the goal of motion intention decoding in invasive brain-computer interface research, this application provides a novel training mechanism for motion intention decoding models. During the training process, the EEG signal data and the motion posture data obtained by video analysis are jointly analyzed, so that the model system can quickly capture the relationship between neural signals and motion trajectories, and comprehensively capture motion intentions, which significantly improves the model's recognition accuracy for various complex motion patterns. At the same time, it also reduces the computing resources required for model processing, making the entire decoding process more efficient, taking into account individual adaptability and generalization, and can well meet the real-time application needs of invasive brain-computer interface research. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of a method for configuring a motion intention decoding model based on EEG signal data in this application; Figure 2 A structural diagram of a configuration device for a motion intention decoding model based on EEG signal data in this application; Figure 3 This is a structural diagram of the processing equipment for this application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0015] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0016] The division of modules in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. Moreover, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed into multiple circuit modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application.

[0017] Before introducing the configuration method of the motion intention decoding model based on EEG signal data provided by this application, the background content involved in this application is first introduced.

[0018] The configuration method, device and computer-readable storage medium of the motion intention decoding model based on EEG signal data provided in this application can be applied to processing equipment, and provides a novel training mechanism for the motion intention decoding model. During the training process, the EEG signal data and the motion posture data obtained by video analysis are jointly analyzed, so that the model system can quickly capture the correlation between neural signals and motion trajectories, and comprehensively capture motion intentions, which significantly improves the model's recognition accuracy for various complex motion patterns. At the same time, it also reduces the computing resources required for model processing, making the entire decoding process more efficient, taking into account individual adaptability and generalization, and can well meet the real-time application needs of invasive brain-computer interface research.

[0019] The method for configuring a motion intention decoding model based on EEG signal data mentioned in this application can be performed by a configuration device for a motion intention decoding model based on EEG signal data, or by a server, physical host, or user equipment (UE) or other different types of processing devices that integrate the configuration device for a motion intention decoding model based on EEG signal data. The configuration device for the motion intention decoding model based on EEG signal data can be implemented in hardware or software, and the UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, desktop computer, or personal digital assistant (PDA). The processing device can be set up in a device cluster.

[0020] It should be noted that this application involves the collection of two types of data: monitoring images and EEG signals. The corresponding cameras (which can be further extended to related equipment / devices with cameras) and invasive brain-computer interface systems can be used in the form of external devices / external connections, or they can be included in the scope of processing equipment in the form of equipment clusters, or they can involve software and hardware modifications to integrate the original equipment functions into the processing equipment. This can be flexibly adjusted according to actual conditions.

[0021] Corresponding to the actual applications that the trained model may subsequently involve, the processing equipment may also be divided into the model processing mainly involved in the present application and the model applications that may subsequently be involved, to form a device cluster.

[0022] Similarly to the above, if the display of model processing results is also involved, the processing device also needs to be equipped with a corresponding display screen (including a touch screen), or it can be connected to other display screen devices or other devices with display screens to meet the needs of result display.

[0023] In the case of device clusters or devices involving multiple parts, the connection between the device parts can be either wired or wireless. Wireless connection is suitable for remote communication, such as web services and other communication methods, which corresponds to the diverse application requirements in actual situations.

[0024] It can be understood that the specific device type and device deployment form of the processing device that executes the configuration method of the motion intention decoding model based on EEG signal data of the present application, or is equipped with the corresponding application service of the configuration method of the motion intention decoding model based on EEG signal data of the present application, is relatively flexible and can be adaptively configured according to actual needs.

[0025] Next, we will introduce the configuration method of the motion intention decoding model based on EEG signal data provided by this application.

[0026] First, see Figure 1 , Figure 1 A flow chart of a configuration method of a motion intention decoding model based on EEG signal data provided by the present application is shown. The configuration method of a motion intention decoding model based on EEG signal data provided by the present application may specifically include the following steps S101 to S104: Step S101, collecting a monitoring image of the sample moving object while the sample moving object is performing a specified moving task; It can be understood that this application focuses on identifying / detecting the movement intention of moving objects or test objects based on EEG signals. To this end, image recognition is introduced to assist model training. As for the data source object of the training sample in the model training process, that is, the sample moving object, this application specifically requires it to perform the corresponding specified motion task, based on the specified related motion, to capture related monitoring images and EEG signals, and through further data processing, obtain the processed data that can be used for model training.

[0027] Among them, the monitoring image is usually a monitoring video, and the subsequent data processing is specifically carried out on the video frames obtained by parsing the monitoring video. Alternatively, the monitoring image can also be a monitoring picture in some cases, which is adjustable.

[0028] In addition, for the scenario where the sample motion object performs a specified motion task, the present application may also involve the processing of guiding the sample motion object to perform the specified motion task, such as through image display, voice prompts, vibration reminders, light reminders or even manual guidance, which is possible in actual situations.

[0029] If the sample moving object is a non-human object such as a macaque, it is easy to understand that its execution of a specified movement task usually involves a period of training, such as 2 months, to fully enable the sample moving object to better complete the specified movement task and provide high-quality monitoring images and EEG signal data.

[0030] Step S102: performing image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task, and obtaining corresponding sample motion posture data; It can be understood that for the surveillance images obtained previously, a pre-configured image recognition algorithm / model can be used to perform image recognition on the video frames therein to identify the specific motion postures involved in the sample motion objects in the image during the performance of specified motion tasks. The corresponding motion actions of these specific motion postures will also be reflected in the EEG signals. In this way, the sample motion posture data obtained through mature and high-precision image recognition can be used as a label of the corresponding sample EEG signal data or the predicted true value in the model training process, constituting high-quality training samples to promote the training process of the model for motion intention decoding based on EEG signal data.

[0031] Step S103, while the sample moving subject is performing the designated movement task, the EEG signal of the sample moving subject is collected through the invasive brain-computer interface system, and corresponding sample EEG signal data is obtained; It can be understood that in the process of the sample motion object performing the specified motion task, on the one hand, it involves the collection of monitoring images, and on the other hand, it also involves the collection of EEG signals. The specific EEG signal collection and processing needs to be carried out through the invasive brain-computer interface system.

[0032] Among them, considering how the invasive brain-computer interface system collects EEG signals to obtain corresponding sample EEG signal data, and considering that it belongs to the scope of existing technology, no detailed explanation will be given here.

[0033] In step S104, the sample motion posture data and the sample EEG signal data are used as training samples to train a motion intention decoding model, wherein the motion posture data is used as annotations, the motion intention decoding model is a deep learning model, and the motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

[0034] It can be understood that after obtaining the sample EEG signal data of the model input involved in the actual application of the corresponding model and the sample motion posture data that has a direct correlation with the motion posture involved in the sample EEG signal and is highly accurate, the two can be configured into the data format of training samples that can be used in the specific model training link. In this way, specific model training can be carried out based on the configured training samples to train the motion intention decoding model.

[0035] Among them, the motion intention decoding model is a deep learning model, which is built by artificial intelligence (AI) technology with powerful learning ability. It has better model performance in machine learning models. Therefore, on the basis of the configuration of training samples that this application scheme focuses on, it can conduct in-depth joint analysis of EEG signal data and motion posture data to capture the correspondence between EEG signals and motion postures, comprehensively capture the motion intention of the sample motion object, and carry out in-depth model training. On the one hand, it can significantly improve the model's recognition accuracy for various complex motion patterns, and on the other hand, it can effectively reduce the computing resources required for model processing, making the entire decoding process more efficient, taking into account the adaptability and generalization of different individual motion behaviors.

[0036] In this case, the specific model type (or model structure) used by the motion intention decoding model can obviously be adaptively configured according to actual needs. Existing solutions, such as long short-term memory networks (LSTM) and recurrent neural networks (RNN), can be used. Further optimization and improvement can also be made based on the existing solutions. Even novel self-developed solutions can be used to further enhance model performance under a high-quality model training mechanism.

[0037] The specific model training process usually includes: In each round of model training, a sample EEG signal data is input into the model to prompt the model to carry out the corresponding motion intention decoding processing and realize forward propagation. Then, based on the motion intention decoding processing results output by the model, the corresponding sample motion posture data is combined to calculate the loss function, and the model parameters are optimized through the loss function calculation results to realize reverse propagation. After a large number of rounds of model training, if the preset model training requirements such as training time, number of training times or prediction accuracy are met, the model training can be completed and a motion intention decoding model that can be put into practical use is obtained.

[0038] Among them, the specific model training architecture and the specific loss function used in the training process are easy to understand. They are similar to the model types introduced above. You can use the existing solution, or make further optimization and improvement on the basis of the existing solution, or even adopt a novel self-developed solution. This can be flexibly deployed according to actual conditions to meet the diverse application needs in actual applications.

[0039] Continue to the above Figure 1Each step of the illustrated embodiment and its possible implementation in practical applications are described in detail.

[0040] As an exemplary implementation, the above step S102 performs image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task and obtain the corresponding sample motion posture data, which may specifically include: 1) Use the Yolo-V8 algorithm to perform target detection on the surveillance image and obtain the corresponding hand information, where the hand information includes hand position and hand size; It can be understood that the computer vision method or target detection processing developed here is a preliminary image recognition processing for hand movement posture, so as to locate the hand from the video frame involved in the overall monitoring image, which may specifically involve the hand position and hand size.

[0041] In addition, it can be seen that this application specifically uses the yolo-v8 deep learning algorithm in the processing link here. The yolo-v8 algorithm has excellent target detection performance, so it can well meet the needs of obtaining hand information here.

[0042] 2) The hand information is further processed by the key point detection algorithm to obtain the corresponding hand key point information; After obtaining hand information including hand position and hand size, the hand information can be subjected to more precise positioning detection of the hand key points through the configured relevant key point detection algorithm, thereby obtaining more delicate and specific hand information related to the hand movement posture. For the sake of convenience, the deep-level hand information obtained here is recorded as hand key point information.

[0043] 3) Through the deep learning-based hand posture estimation network, the hand information and hand key point information are associated with the data to carry out hand posture estimation processing and obtain sample motion posture data.

[0044] After obtaining the preliminary hand information and the deep hand key point information, the two can be associated with each other to ensure that each detected target is correctly matched with its corresponding key point. This processing can be carried out in detail based on the relative position relationship, that is, matching is performed using the relative position relationship between the target's position information and the key points.

[0045] Next, the processing results of target detection and key point detection after completing data association can be used to accurately estimate the posture of the hand movement. Specifically, based on the deep learning-based posture estimation network, the hand posture key points are regressed to obtain accurate posture information of the hand movement, and then the hand movement trajectory is constructed to complete the configuration of the sample motion posture data.

[0046] Furthermore, for sample motion objects such as non-human primates, especially macaques, this application also designs specific task content for designated motion tasks.

[0047] Correspondingly, as an exemplary embodiment, the specified motion task involved in this application may specifically include the following motion process: 1) After the first signal light turns on, touch the first sensor for 500ms; The first signal light is usually arranged adjacent to the first sensor.

[0048] As an example, the first signal light can be a "Center light" next to the central sensor. The lighting of the first signal light serves as a signal for the start of a task. The lighting of the first signal light is used to prompt the sample moving object to touch the central sensor, place the right hand on the central sensor and keep it still for 500ms.

[0049] 2) 500ms after the first sensor is touched, the first signal light is turned off and one of the three second signal lights is randomly turned on; After the first sensor is continuously touched for 500ms, the system will turn off the first signal light. If the system is configured with one first signal light, three second signal lights and a sensor corresponding to each second signal light, a second signal light will be randomly selected to light up to remind the sample moving object to touch the sensor (that is, the target sensor) corresponding to the lit second signal light (that is, the target light).

[0050] 3) After the target light turns on, touch the target sensor corresponding to the target light among the three second sensors for 200ms.

[0051] It can be understood that in the next touch action of the target sensor, it needs to be maintained for 200ms.

[0052] Continuing with the previous example, if the previous training used the right hand for touch movements, then the right hand of the sample moving subject moves from the central sensor to the target sensor (target object) for grasping for 200ms.

[0053] During the training process, when the sample moving object completes the above-mentioned movement and completes a task, it can stop touching the target sensor, release its grip, and can be rewarded with rewards such as drinking water.

[0054] It can be understood that in the embodiment here, this application focuses on the hand movement posture of the sample moving object and stipulates the standard movement direction. An effective training grasping training process can help achieve stable grasping movements, standardize the deconstruction of hand movements, optimize the accuracy of the training and capture process, and facilitate better parameterization of the movement process, laying a good foundation for subsequent model training.

[0055] At the same time, we continue to focus on the motion intention decoding goals involved in hand motion postures. In the specific parameterization configuration work, this application also provides a specific adaptation solution to achieve higher quality hand posture parameterization effects.

[0056] Specifically, as an exemplary embodiment, the sample motion posture data may be data describing hand motion postures, and the different hand motion primitives involved include grasping motion, upward motion, downward motion, leftward motion, rightward motion, forward motion, backward motion, and stop motion, a total of 8 hand motion primitives; The sample motion posture data is specifically configured with parameterized encoding representing different hand motion primitives, which is decomposed into a time window of 200ms and is expressed as follows: M={m1, m2,..., mx}, Among them, M is the sample motion posture data, my is the time series data of the y-th time window, y=1,2,...x.

[0057] As an example, the motion primitive codes of the above eight hand motion primitives may be the corresponding binary 0-7. Of course, the specific coding mechanism may be flexibly configured according to the actual situation.

[0058] In this way, the hand movement process after time window decomposition is converted into a series of motion primitives through one-to-one mapping, and the movement process is parameterized and encoded according to the motion primitive coding.

[0059] In the embodiment here, it can be seen that the present application constructs a specific motion trajectory encoding and deconstruction technology. By capturing the hand motion trajectory of the sample motion object and extracting its key process, the motion trajectory is decomposed into multiple motion primitives and the motion primitives are parameterized. This can effectively improve the system's recognition accuracy of hand grasping movements and enhance the real-time and stability of decoding, which helps to enhance the system's adaptability to individual differences and different environmental conditions, and ensure that the system continues to work stably under dynamic changes.

[0060] In addition, it can be understood that in addition to focusing on the motion intention decoding target involved in the hand movement of the moving object, the present application may also involve the motion intention decoding target of other body parts other than the hand of the moving object, or, it may also involve the motion intention decoding target of multiple body parts or even the whole body. This can be adaptively adjusted according to actual needs. Therefore, for the sample moving object, it is necessary to configure an adapted specified motion task to obtain the corresponding training samples for model training. Among them, the above hand motion task operation ideas can be used for processing, or other types of improved operation ideas can be used for processing.

[0061] At the same time, for the collection and processing of sample EEG signal data, this application may also involve corresponding preprocessing operations to enhance data quality.

[0062] Accordingly, focusing on the preprocessing step, as an exemplary embodiment, step S103 collects EEG signals of the sample moving subject through the invasive brain-computer interface system during the sample moving subject's performance of the specified motion task, and obtains corresponding sample EEG signal data, which may specifically include: 1) While the sample motion subject is performing a designated motion task, the EEG signal of the sample motion subject is collected through an invasive brain-computer interface system, and corresponding first sample EEG signal data with a sampling frequency of 100 Hz for each signal channel is obtained; It can be seen that for the invasive brain-computer interface system, the EEG signal acquisition frequency is 1000Hz, which ensures that subtle changes in brain activity can be captured with high resolution. After the initial signal acquisition is completed, a series of data preprocessing steps can be continued to remove noise and unnecessary interference to ensure that the obtained neural signals can truly reflect the brain activity of the macaque during movement.

[0063] 2) processing the first sample EEG signal data into second sample EEG signal data with a sampling frequency of 200 Hz for each signal channel through downsampling; It can be understood that in the first EEG signal data initially collected, the sampling frequency of each signal channel is 1000 Hz, which means that 1000 data points are collected per second. Although a high sampling rate can provide higher signal accuracy, in many neural signal processing applications, a higher sampling frequency may cause a burden on data processing and storage. Therefore, downsampling processing can be continued to reduce the signal frequency to 200 Hz to significantly reduce the amount of data while ensuring that the key information in the data is retained. Especially in the decoding analysis of the brain's motor control area, a sampling frequency of 200 Hz is sufficient to capture the important neural activities of the brain when performing motor tasks.

[0064] 3) Using a fourth-order Butterworth filter with a passband of 0.5 Hz to 50 Hz, the second sample EEG signal data is bandpass filtered to obtain corresponding sample EEG signal data.

[0065] It can be understood that in order to remove low-frequency noise (such as electromyographic signals, power frequency interference, etc.) and high-frequency noise (such as electrode noise and environmental interference), the present application can continue to perform band-pass filtering on the downsampled second EEG signal data, and in the specific processing process, a fourth-order Butterworth filter can be used, and its passband is designed to be between 0.5Hz and 50Hz. This frequency band filter can effectively remove low-frequency interference below 0.5Hz (such as artifacts caused by eye movements or ECG signals) and high-frequency noise above 50Hz. The role of band-pass filtering is to retain low-frequency and medium-frequency neural signals related to motor control that can reflect the electrical activity of the cerebral cortex, especially the motor cortex.

[0066] In this process, the selection and parameter setting of the filter are crucial. For example, a passband of 0.5Hz to 50Hz can ensure that the filtered signal not only retains useful neural activity components but also avoids signal distortion or loss of key information.

[0067] In this way, through the preprocessing operation designed in the embodiment herein, high-quality and highly simplified sample EEG signal data is obtained, laying a good foundation for subsequent high-quality model training.

[0068] In addition, the training samples involve data on both EEG signals and movement postures. Each corresponding sample EEG signal data and sample movement posture data of a movement action at the same time is regarded as a training sample.

[0069] In this regard, the present application may also involve more detailed matching processing between sample EEG signal data and sample motion posture data in order to further ensure the matching between the two types of data, ensure that the time / timestamp remains synchronized, strictly correspond to the actual motion process, and ensure the high quality level of the training samples.

[0070] Correspondingly, as an exemplary embodiment, the method of the present application may further include: 1) The sample EEG signal data is divided into time windows of 200ms, and there are overlapping parts between adjacent time windows during the division process. The divided sample EEG signal data is represented as follows: E={e1,e2,...,eo}, Where E is the sample EEG signal data, ep is the time series data of the pth time window, p=1,2,...o; 2) Combine the sample motion posture data and sample EEG signal data in a time window of 200ms. The resulting training sample dataset is represented as follows: Q={(e1,m1), (e2,m2),..., (en,mn)}, The 0ms window size corresponds to the setting of the sample motion posture data previously decomposed into a 200ms window size. Q is the dataset used as training samples, n is the number of time series data pairs of sample motion posture data and sample EEG signal data, (eu, mu) is a paired neural signal segment and motion primitive label, and u = 1, 2, ..n.

[0071] During the specific processing, it is understandable that the sample EEG signal data and the sample motion posture data (or monitoring images) will be recorded with their respective timestamps, so that they can be matched based on the time point marked by the timestamp through the configured specific calibration method.

[0072] In this case, the time points of motion events / motion primitives can be used as calibration basis to match the sample EEG signal data at each time point with the sample motion posture data. This synchronous processing can ensure the accuracy and reliability of the data, which is helpful for subsequent neural signal decoding and motion behavior analysis.

[0073] As for the collected sample EEG signal data, this application divides it into time windows in order to associate it with the corresponding motion trajectory. Each 200ms time window contains a time series with a frequency of 200 Hz, that is, each time window contains 40 data points. The selection of this window size fully considers the time delay of the brain when performing motor tasks and the propagation characteristics of motor signals. The 200 ms window size can better capture the dynamic changes of nerves during the execution of movement, while avoiding signal distortion or information loss caused by too small a window.

[0074] Furthermore, to ensure timing consistency, time window division employs the same compensation method as the window size. Specifically, time window compensation refers to the overlap between each time window, meaning that there is some overlapping data between two adjacent time windows. The degree of overlap can be flexibly configured based on specific indicators such as data volume, number of timestamps, or duration span. This compensation method maximizes the retention of neural information within each time window, reduces signal loss, and improves the accuracy of neural signal classification and decoding.

[0075] In this way, after time window division, the sample EEG signal data E represented above is obtained, where each ep represents the brain activity performance of the sample motion subject in a specific time period when performing a certain motion behavior.

[0076] Correspondingly, the motion trajectory of the sample motion object composed of different motion primitives when performing a specified motion task, that is, the sample motion posture data, can be obtained. The motion primitive m can be corresponded one-to-one with the sample EEG signal data e in the time window, thereby establishing the relationship between neural signals and motion behavior.

[0077] Through the above-mentioned time window division and labeling process, a complete motion-neural signal dataset can be obtained. Each item in this dataset includes a neural signal segment e and its corresponding motion primitive label m, that is, the mapping relationship between a neural signal time series and a motion behavior, forming a dataset Q. Each data point in it contains a 200ms EEG signal sequence and its corresponding motion action label. This dataset will serve as the basis for subsequent EEG signal decoding and classification, and accurately predict the motion behavior of the moving object through deep learning methods.

[0078] In this way, after completing the configuration of high-quality training samples through the above processing, it is obvious that the specific model training process can be started to train the motion intention decoding model that uses EEG signal data as model input and is used to perform motion intention decoding processing on the motion intention of the corresponding motion object (or to predict the corresponding motion intention of the motion object).

[0079] As for the motion intention decoding model, this application has also specially designed a specific model structure solution to better adapt to the application scenario of this application, starting from the model itself, to further enhance the model's motion intention decoding classification / prediction performance.

[0080] Specifically, as an exemplary embodiment, the motion intention decoding model of the present application is based on convolutional neural networks (CNN) and recurrent neural networks (RNN), creating a two-dimensional CNN-GRU fusion neural network framework, which can specifically include the following configuration contents: 1) The model inputs one-dimensional time series signal data, which is transformed into a two-dimensional image signal after frequency domain transformation, and is input into the first half of the model. The transformation process includes converting the signal from the time domain to the frequency domain using the Discrete Fourier Transform (DFT), estimating the power spectral density using the square of the Fourier transform amplitude of the signal, and smoothing the estimated power spectral density using the Welch method. Discrete Fourier transform processing can help identify periodic or repetitive patterns in a signal. Smoothing the power spectral density using the Welch method can reduce the effects of noise, helping to more clearly extract the frequency characteristics of the signal.

[0081] 2) The first half of the model consists of three 2D convolutional layers and three max pooling layers. After each 2D convolutional layer, the data is then processed by a max pooling layer, and then input into the next 2D convolutional layer, until the third max pooling layer at the last position outputs to the second half of the model. Among them, the maximum pooling operation can help reduce the spatial dimension of features and reduce the amount of computation while retaining key information. The main function of the first half of the model is to extract the spatial features of the input signal, especially the frequency features.

[0082] 3) The second half of the model consists of two GRU layers connected in series. The GRU layer is used to extract temporal features. The output of the second GRU layer is processed in sequence by the flatten layer, the fully connected layer, the dropout layer, and the output layer to obtain the model output. Among them, the GRU layer is used to process the feature sequence extracted in the first half of the previous model and further extract temporal features. GRU controls the flow of information through a gating mechanism, which can avoid the gradient disappearance or gradient explosion problems in traditional RNNs.

[0083] The flatten layer, fully connected layer, dropout layer and output layer are responsible for mapping the features extracted from the GRU layer to the final output category space and mapping the complex features to the final motion intention classification results.

[0084] In this way, under the design of the above model structure, GRU is embedded into the overall model as a layer to better extract the features of the timing signal, and finally achieve accurate classification of the motion intention signal.

[0085] 4) The specifications of the three 2D convolutional layers are 998×26×32, 497×11×64, and 246×3×64, respectively. The specifications of the three maximum pooling layers are 499×13×32, 248×5×64, and 123×1×64, respectively. 5) The sizes of the flatten layer, fully connected layer, dropout layer, and output layer are 1×128, 1×64, 1×64, and 1×2, respectively.

[0086] It can be seen that this application further provides further improved configuration in terms of parameters for the above-mentioned model structure designed, regarding the different model structure parts involved, so as to better complete the configuration work of the motion intention decoding model, and provide a good model foundation for high-performance motion intention decoding classification / prediction tasks, with better application value and practical significance.

[0087] After completing the model training, it is obvious that subsequent model application links can be involved according to application requirements.

[0088] In this regard, the application method may further include: Obtaining target EEG signal data to be processed; Input the target EEG signal data into the movement intention decoding model; Obtain the processing results corresponding to the target EEG signal data output by the movement intention decoding model.

[0089] It is understandable that the target EEG signal data may involve the acquisition and processing of the sample EEG signal data mentioned above. Of course, in practical applications, it may also be directly the ready-made EEG signal data, which can be flexibly adjusted according to the actual situation.

[0090] After obtaining the processing results of the model output, data application processing such as local storage, remote storage, result forwarding, result display, output completion prompts, or further data analysis can be continued. Obviously, this link can be flexibly adjusted according to pre-configured and real-time configured data application strategies / rules. In this way, the network model configured in the present application scheme with high-performance decoding and classification performance can be used to promote high-quality invasive brain-computer interface research.

[0091] Finally, for the above program content, in general, there are: Focusing on the goal of motion intention decoding in invasive brain-computer interface research, this application provides a novel training mechanism for motion intention decoding models. During the training process, the EEG signal data and the motion posture data obtained by video analysis are jointly analyzed, so that the model system can quickly capture the relationship between neural signals and motion trajectories, and comprehensively capture motion intentions, which significantly improves the model's recognition accuracy for various complex motion patterns. At the same time, it also reduces the computing resources required for model processing, making the entire decoding process more efficient, taking into account individual adaptability and generalization, and can well meet the real-time application needs of invasive brain-computer interface research.

[0092] The above is an introduction to the configuration method of the motion intention decoding model based on EEG signal data provided by this application. In order to facilitate better implementation of the configuration method of the motion intention decoding model based on EEG signal data provided by this application, this application also provides a configuration device of the motion intention decoding model based on EEG signal data from the perspective of functional modules.

[0093] See Figure 2 , Figure 2 This is a schematic diagram of a configuration device for a motion intention decoding model based on EEG signal data in this application. In this application, the configuration device 200 for a motion intention decoding model based on EEG signal data may specifically include the following structure: The first acquisition unit 201 is configured to acquire a monitoring image of the sample moving object while the sample moving object is performing a designated moving task; The recognition unit 202 is used to perform image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task and obtain corresponding sample motion posture data; The second acquisition unit 203 is used to acquire EEG signals of the sample moving subject through the invasive brain-computer interface system while the sample moving subject is performing a designated movement task, and obtain corresponding sample EEG signal data; The training unit 204 is used to use the sample motion posture data and the sample EEG signal data as training samples to train the motion intention decoding model, wherein the motion posture data is used as a label, the motion intention decoding model is a deep learning model, and the motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

[0094] In an exemplary embodiment, the identification unit 202 is specifically configured to: The yolo-v8 algorithm is used to perform target detection on the surveillance image to obtain the corresponding hand information, where the hand information includes hand position and hand size; The hand information is further processed by the key point detection algorithm to obtain the corresponding hand key point information; Through the hand posture estimation network based on deep learning, the hand information and hand key point information are associated with the data to carry out hand posture estimation processing and obtain sample motion posture data.

[0095] In another exemplary embodiment, the designated motion task specifically includes the following motion process: After the first signal light is on, touch the first sensor for 500ms; 500ms after the first sensor is touched, the first signal light is turned off and one of the three second signal lights is randomly turned on; After the target light is on, touch the target sensor corresponding to the target light among the three second sensors for 200ms.

[0096] In another exemplary embodiment, the sample motion gesture data is specifically data describing a hand motion gesture, and the different hand motion primitives involved include grasping motion, upward motion, downward motion, leftward motion, rightward motion, forward motion, backward motion, and stop motion; The sample motion posture data is specifically configured with parameterized encoding representing different hand motion primitives, which is decomposed into a time window of 200ms and is expressed as follows: M={m1, m2,..., mx}, Among them, M is the sample motion posture data, my is the time series data of the y-th time window, y=1,2,...x.

[0097] In another exemplary embodiment, the second acquisition unit 203 is specifically configured to: While the sample motion subject is performing a designated motion task, an invasive brain-computer interface system is used to collect EEG signals of the sample motion subject, and corresponding first sample EEG signal data with a sampling frequency of 100 Hz for each signal channel is obtained; Processing the first sample EEG signal data into second sample EEG signal data with a sampling frequency of 200 Hz for each signal channel through downsampling processing; The second sample EEG signal data is bandpass filtered using a fourth-order Butterworth filter with a passband of 0.5 Hz to 50 Hz to obtain corresponding sample EEG signal data.

[0098] In another exemplary embodiment, the apparatus further includes a processing unit 205 configured to: The sample EEG signal data is divided into time windows of 200ms, and there are overlapping parts between adjacent time windows during the division process. The divided sample EEG signal data is represented as follows: E={e1,e2,...,eo}, Where E is the sample EEG signal data, ep is the time series data of the pth time window, p=1,2,...o; The sample motion posture data and sample EEG signal data are combined in a time window of 200ms. The resulting dataset, which serves as a training sample, is represented as follows: Q={(e1,m1), (e2,m2),..., (en,mn)}, Among them, Q is the dataset used as training samples, n is the number of time series data pairs of sample motion posture data and sample EEG signal data, (eu,mu) is a paired neural signal segment and motion primitive label, and u=1,2,..n.

[0099] In another exemplary embodiment, the motion intention decoding model includes the following configuration content: The model inputs one-dimensional time series signal data, which is transformed into a two-dimensional image signal after frequency domain transformation, and is input into the first half of the model. The transformation process includes: using discrete Fourier transform to convert the signal from the time domain to the frequency domain, estimating the power spectral density by the square of the Fourier transform amplitude of the signal, and smoothing the estimated power spectral density using the Welch method; The first half of the model consists of three 2D convolutional layers and three maximum pooling layers. After each 2D convolution layer, the maximum pooling layer performs maximum pooling, and then inputs the next 2D convolutional layer until the third maximum pooling layer at the last position outputs to the second half of the model. The second half of the model consists of two GRU layers connected in series. The GRU layer is used to extract temporal features. The output of the second GRU layer is processed in sequence by the flatten layer, the fully connected layer, the dropout layer, and the output layer to obtain the model output. The specifications of the three 2D convolutional layers are 998×26×32, 497×11×64, and 246×3×64, respectively. The specifications of the three maximum pooling layers are 499×13×32, 248×5×64, and 123×1×64, respectively. The specifications of the flatten layer, fully connected layer, dropout layer, and output layer are 1×128, 1×64, 1×64, and 1×2, respectively.

[0100] This application also provides a processing device from the perspective of hardware structure, see Figure 3 , Figure 3 The schematic diagram of the structure of the processing device of the present application is shown. Specifically, the processing device of the present application may include a processor 301, a memory 302 and an input / output device 303. The processor 301 is used to execute the computer program stored in the memory 302 to implement the following Figure 1 The steps of the configuration method of the motion intention decoding model based on EEG signal data in the corresponding embodiment; or, the processor 301 is used to execute the computer program stored in the memory 302 to implement the following Figure 2 The memory 302 is used to store the functions of each unit in the embodiment corresponding to the processor 301. Figure 1 A computer program required for the configuration method of the movement intention decoding model based on EEG signal data in the corresponding embodiment.

[0101] For example, the computer program may be divided into one or more modules / units, one or more of which are stored in the memory 302 and executed by the processor 301 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in a computer device.

[0102] The processing device may include, but is not limited to, a processor 301, a memory 302, and an input / output device 303. Those skilled in the art will appreciate that the illustrations are merely examples of processing devices and do not limit the processing device. The processing device may include more or fewer components than shown, or a combination of certain components, or different components. For example, the processing device may also include a network access device, a bus, etc., and the processor 301, the memory 302, the input / output device 303, etc. are connected via a bus.

[0103] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the processing device and connects various parts of the entire device using various interfaces and lines.

[0104] Memory 302 can be used to store computer programs and / or modules. Processor 301 implements various functions of the computer device by running or executing computer programs and / or modules stored in memory 302 and accessing data stored in memory 302. Memory 302 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the data storage area may store data generated based on the use of the processing device. Furthermore, memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0105] When the processor 301 is used to execute the computer program stored in the memory 302, it can specifically implement the following functions: Acquiring monitoring images of the sample moving object while the sample moving object is performing a designated moving task; Performing image recognition processing on the monitoring image to identify the motion posture of the sample moving object in the process of performing the specified motion task and obtain corresponding sample motion posture data; When the sample moving subject performs a designated movement task, the EEG signal of the sample moving subject is collected through an invasive brain-computer interface system, and corresponding sample EEG signal data is obtained; The sample motion posture data and sample EEG signal data are used as training samples to train the motion intention decoding model, wherein the motion posture data is used as annotations, and the motion intention decoding model is a deep learning model. The motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

[0106] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the configuration device, processing equipment and corresponding units of the above-described motion intention decoding model based on EEG signal data can refer to the following. Figure 1 The description of the configuration method of the movement intention decoding model based on EEG signal data in the corresponding embodiment will not be repeated here.

[0107] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0108] To this end, the present application provides a computer-readable storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the present application as follows: Figure 1 The steps of the configuration method of the motion intention decoding model based on EEG signal data in the corresponding embodiment, the specific operations can be referred to as follows Figure 1 The description of the configuration method of the movement intention decoding model based on EEG signal data in the corresponding embodiment will not be repeated here.

[0109] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] Due to the instructions stored in the computer readable storage medium, the present application can be executed as follows: Figure 1The steps of the configuration method of the motion intention decoding model based on EEG signal data in the corresponding embodiment, therefore, the present application can be realized as follows Figure 1 The beneficial effects that can be achieved by the configuration method of the motion intention decoding model based on EEG signal data in the corresponding embodiment are detailed in the previous description and will not be repeated here.

[0111] The above is a detailed introduction to the configuration method, device, processing equipment and computer-readable storage medium of the motion intention decoding model based on EEG signal data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the core idea of the present application; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for configuring a motion intention decoding model based on EEG signal data, characterized in that: The method comprises: During the process of the sample moving object performing the specified moving task, collecting a monitoring image of the sample moving object; Performing image recognition processing on the monitoring image to identify the motion posture of the sample motion object in the process of performing the specified motion task, and obtaining corresponding sample motion posture data; During the process of the sample motion subject performing the designated motion task, collecting the electroencephalogram (EEG) signal of the sample motion subject through an invasive brain-computer interface system and obtaining corresponding sample EEG signal data; The sample motion posture data and the sample EEG signal data are used as training samples to train a motion intention decoding model, wherein the motion posture data is used as annotations, the motion intention decoding model is a deep learning model, and the motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

2. The method according to claim 1, characterized in that The performing of image recognition processing on the monitoring image to identify the motion posture of the sample motion object in the process of performing the designated motion task and obtaining corresponding sample motion posture data includes: Using the Yolo-V8 algorithm to perform target detection processing on the surveillance image to obtain corresponding hand information, wherein the hand information includes hand position and hand size; Continuing to perform hand key point detection processing on the hand information through a key point detection algorithm to obtain corresponding hand key point information; The hand posture estimation network based on deep learning is used to perform data association between the hand information and the hand key point information to carry out hand posture estimation processing and obtain the sample motion posture data.

3. The method according to claim 1, characterized in that The specified motion task specifically includes the following motion process: After the first signal light turns on, touch the first sensor for 500ms; 500ms after the first sensor is touched, the first signal light is turned off and one of the three second signal lights is randomly turned on; After the target light is turned on, the target sensor corresponding to the target light among the three second sensors is touched for 200ms.

4. The method according to claim 3, characterized in that The sample motion gesture data specifically refers to data describing hand motion gestures, and the different hand motion primitives involved include grasping motion, upward motion, downward motion, leftward motion, rightward motion, forward motion, backward motion, and stop motion; The sample motion posture data is specifically configured with parameterized codes representing the different hand motion primitives decomposed into a time window of 200ms, and is expressed as follows: M={m1, m2,..., mx}, Wherein, M is the sample motion posture data, my is the time series data of the y-th time window, and y=1,2,...x.

5. The method according to any one of claims 1 to 4, characterized in that The method includes collecting EEG signals of the sample moving object through an invasive brain-computer interface system and obtaining corresponding sample EEG signal data during the sample moving object's execution of the designated moving task, including: During the process of the sample motion subject performing the designated motion task, collecting the EEG signal of the sample motion subject through the invasive brain-computer interface system, and obtaining corresponding first sample EEG signal data with a sampling frequency of 100 Hz for each signal channel; Processing the first sample EEG signal data into second sample EEG signal data with a sampling frequency of 200 Hz for each signal channel through downsampling processing; The second sample EEG signal data is subjected to bandpass filtering using a fourth-order Butterworth filter with a passband of 0.5 Hz to 50 Hz, and the corresponding sample EEG signal data is obtained.

6. The method according to claim 5, characterized in that The method further comprises: The sample EEG signal data is divided into time windows of 200 ms in size, and there are overlapping parts between adjacent time windows during the division process. The divided sample EEG signal data is represented as follows: E={e1,e2,...,eo}, Wherein, E is the sample EEG signal data, ep is the time series data of the p-th time window, and p=1, 2, ...o; The sample motion posture data and the sample EEG signal data are combined in units of the time window of 200ms window size, and the obtained data set as the training sample is expressed as follows: Q={(e1,m1), (e2,m2),..., (en,mn)}, Among them, Q is the data set serving as the training sample, n is the number of time series data pairs of the sample motion posture data and the sample EEG signal data, (eu, mu) serves as a paired neural signal segment and motion primitive label, and u=1,2,..n.

7. The method according to claim 1, characterized in that The motion intention decoding model includes the following configuration contents: The model inputs one-dimensional time series signal data, and the two-dimensional image signal obtained after frequency domain transformation is input into the first half of the model. The transformation process includes: using discrete Fourier transform to convert the signal from the time domain to the frequency domain, estimating the power spectral density by the square of the amplitude of the Fourier transform of the signal, and smoothing the estimated power spectral density by the Welch method; The first half of the model includes three two-dimensional convolutional layers and three maximum pooling layers. After each two-dimensional convolution process of the two-dimensional convolutional layer, the maximum pooling is performed by a maximum pooling layer, and then the next two-dimensional convolutional layer is input, until the third maximum pooling layer in the last position is output to the second half of the model; The second half of the model includes two GRU layers in series, which are used to extract temporal features. The output of the second GRU layer is processed in sequence by the flatten layer, the fully connected layer, the dropout layer, and the output layer to obtain the model output. The specifications of the three two-dimensional convolutional layers are 998×26×32, 497×11×64, and 246×3×64, respectively; the specifications of the three maximum pooling layers are 499×13×32, 248×5×64, and 123×1×64, respectively; The specifications of the flatten layer, the fully connected layer, the dropout layer and the output layer are 1×128, 1×64, 1×64 and 1×2 respectively.

8. A device for configuring a motor intention decoding model based on EEG signal data, characterized in that: The device comprises: A first acquisition unit is configured to acquire a monitoring image of the sample moving object while the sample moving object is performing a specified moving task; a recognition unit, configured to perform image recognition processing on the monitoring image to identify the motion posture of the sample motion object in the process of performing the specified motion task, and obtain corresponding sample motion posture data; a second acquisition unit, configured to acquire, through an invasive brain-computer interface system, an electroencephalogram (EEG) signal of the sample motion subject while the sample motion subject is performing the designated motion task, and obtain corresponding sample EEG signal data; A training unit is used to use the sample motion posture data and the sample EEG signal data as training samples to train a motion intention decoding model, wherein the motion posture data is used as a label, the motion intention decoding model is a deep learning model, and the motion intention decoding model is used to perform motion intention decoding processing on the motion intention of the corresponding motion object based on the EEG signal data input by the model.

9. A processing device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method according to any one of claims 1 to 7.

Images