Hand position estimation method and device based on brain and muscle electrical asynchronous time-frequency fusion

Abstract

The application provides a hand position estimation method and device based on brain and muscle asynchronous time-frequency fusion, and is applied to the technical field of biological signals. The method comprises the following steps: initial electroencephalogram data is obtained by performing sliding window segmentation on the preprocessed electroencephalogram signal of a target object by using a first time window; target electromyogram data is obtained by performing sliding window segmentation on the preprocessed electromyogram signal of the target object by using a second time window; the initial electroencephalogram amplitude and the initial electroencephalogram phase extracted from the initial electroencephalogram data are subjected to dimension transformation based on the number of channels and the length of the target electromyogram data, so that target electroencephalogram data is obtained; the target electroencephalogram data and the target electromyogram data are subjected to cross-level multi-dimensional feature extraction and fusion processing by using a feature fusion module in a hand position estimation model, so that target fusion features are obtained; and the target fusion features are subjected to feature mapping processing by using a regressor in the hand position estimation model based on a three-dimensional coordinate system of a hand, so that target hand position information of the target object is obtained.

Description

Technical Field

[0001] This invention relates to the field of biosignal technology, and more specifically to a method and apparatus for predicting hand position based on asynchronous time-frequency fusion of brain-myoelectricity. Background Technology

[0002] With the widespread adoption of human-computer interaction across various fields, gesture recognition has become one of the most important interaction methods. Since electroencephalography (EEG) and electromyography (EMG) signals are relatively easy to collect and not constrained by the application environment, existing technologies typically utilize control features in EEG signals and activity states in EMG signals to recognize and predict the gestures of target objects. However, current gesture recognition methods usually classify gestures, making it difficult to accurately predict hand positions based solely on EMG signals.

[0003] In the process of realizing the above-mentioned inventive concept, it was found through research that: in the related technologies, because the temporal synchronization between EEG signals and EMG signals is maintained only during the acquisition process, it is difficult to semantically align the EEG signals and EMG signals, which have inherent physiological delays. As a result, it is impossible to extract accurate features from EEG signals and EMG signals and fuse them to decode and obtain more accurate hand position information. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method and device for predicting hand position based on asynchronous time-frequency fusion of brain electromyography.

[0005] According to a first aspect of the present invention, a method for predicting hand position based on asynchronous time-frequency fusion of electroencephalography (EEG) and electromyography (EMG) is provided, comprising: performing sliding window segmentation on preprocessed EEG signals of a target object using a first time window to obtain initial EEG data; performing sliding window segmentation on preprocessed EMG signals of the target object using a second time window to obtain target EMG data, wherein the sliding window length of the first time window is greater than the sliding window length of the second time window; performing dimensional transformation on initial EEG amplitude and initial EEG phase extracted from the initial EEG data based on the number of channels and length of the target EMG data to obtain target EEG data, wherein the target EEG data includes the target EEG phase and target EEG amplitude after dimensional transformation; performing cross-level multidimensional feature extraction and fusion processing on the target EEG data and target EMG data using a feature fusion module within a hand position prediction model to obtain target fused features; and performing feature mapping processing on the target fused features using a regressor within the hand position prediction model based on a three-dimensional hand coordinate system to obtain target hand position information of the target object.

[0006] A second aspect of the present invention provides a hand position prediction device based on asynchronous time-frequency fusion of electroencephalography (EEG) and electromyography (EMG), comprising: a segmentation module, configured to perform sliding window segmentation on the preprocessed EEG signal of the target object using a first time window to obtain initial EEG data, and to perform sliding window segmentation on the preprocessed EMG signal of the target object using a second time window to obtain target EMG data, wherein the sliding window length of the first time window is greater than the sliding window length of the second time window; a transformation module, configured to perform dimensional transformation on the initial EEG amplitude and initial EEG phase extracted from the initial EEG data based on the number of channels and the length of the target EMG data to obtain target EEG data, wherein the target EEG data includes the target EEG phase and target EEG amplitude after dimensional transformation; an extraction and fusion module, configured to perform cross-level multi-dimensional feature extraction and fusion processing on the target EEG data and target EMG data using a feature fusion module within the hand position prediction model to obtain target fused features; and a mapping module, configured to perform feature mapping processing on the target fused features based on the hand three-dimensional coordinate system using a regressor within the hand position prediction model to obtain target hand position information of the target object.

[0007] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0008] A fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0009] A fifth aspect of the invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the above-described method.

[0010] The hand position prediction method and apparatus based on asynchronous time-frequency fusion of EEG and EMG according to the present invention segment the preprocessed EEG signal using a first time window with a longer sliding window length, and segment the preprocessed EMG signal using a second time window with a shorter sliding window length, to obtain initial EEG data and target EMG data in the time domain containing different numbers of data points. Thus, asynchronous EEG data and EMG data are constructed by taking into account the inherent physiological time delay of EEG and EMG.

[0011] Then, the initial EEG data in the time domain is transformed to obtain EEG amplitude and phase information in the frequency domain. Based on the number of channels and length of the target EMG data in the time domain, the initial EEG amplitude and initial EMG phase are transformed to obtain data in the same dimension that retains both the global information of the EEG and the data that reflects the brain's control of hand movement, as well as the original EMG response to the control of the EEG and the movement execution data generated by the EMG. This constructs asynchronous time-frequency input data containing more comprehensive features, so as to align the semantic information in the EEG data and the EMG data under the premise of considering the inherent physiological time delay of EEG and EMG.

[0012] Then, a pre-trained, robust, and reliable hand position prediction model with multiple layers of network modules is used to extract corresponding single-modal features from asynchronous time-frequency input data (target EEG phase, target EEG amplitude, and target EMG data). Cross-layer, multi-dimensional feature extraction and fusion processing is performed on multiple single-modal features to obtain multi-dimensional, multi-level target fusion features corresponding to hand position. A regressor is used to map the target fusion features based on a three-dimensional coordinate system to obtain accurate predicted hand position information. This achieves a complete, multi-dimensional integrated signal data processing from EEG signals to asynchronous time-frequency data construction to multi-level feature extraction, efficiently and accurately predicting the hand position of the target object, thus expanding the application scenarios and applicability of hand position prediction. Attached Figure Description

[0013] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0014] Figure 1 An application scenario diagram of the hand position prediction method based on asynchronous time-frequency fusion of brain electromyography according to an embodiment of the present invention is shown;

[0015] Figure 2 A flowchart of a hand position prediction method based on asynchronous time-frequency fusion of electroencephalography (EEG) according to an embodiment of the present invention is shown;

[0016] Figure 3 A schematic diagram illustrating the process of obtaining target electroencephalogram (EEG) data and target electromyogram (EMG) data during the application process according to an embodiment of the present invention is shown.

[0017] Figure 4 A schematic diagram of the target electroencephalogram (EEG) data and target electromyogram (EMG) data obtained during the application process according to an embodiment of the present invention is shown;

[0018] Figure 5 A flowchart illustrating the acquisition of target fusion features according to an embodiment of the present invention is shown;

[0019] Figure 6 A schematic diagram illustrating the entire process of obtaining the target hand position information of a target object according to an embodiment of the present invention is shown;

[0020] Figure 7 A flowchart illustrating the training of a hand position prediction model according to an embodiment of the present invention is shown;

[0021] Figure 8 A schematic diagram of initial EEG samples, initial EMG samples, and initial hand position samples during the training process according to an embodiment of the present invention is shown;

[0022] Figure 9 A schematic diagram is shown of the target training EEG data, target training EMG data, and target hand position data samples obtained during the training process according to an embodiment of the present invention;

[0023] Figure 10 A structural block diagram of a hand position prediction device based on asynchronous time-frequency fusion of electromyography according to an embodiment of the present invention is shown.

[0024] Figure 11 A block diagram of an electronic device based on a brain-myoelectric asynchronous time-frequency fusion method for hand position prediction according to an embodiment of the present invention is shown. Detailed Implementation

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0030] Human-computer interaction based on gesture recognition is currently a key research area. Multimodal gesture recognition based on electroencephalogram (EEG) and surface electromyography (EMG) signals offers numerous advantages, such as low susceptibility to environmental constraints, broad application prospects, robustness, and high accuracy in multidimensional feature analysis. Before a target performs a hand movement, the brain's supplementary motor cortex generates a slow preparatory potential to plan the action. Upon initiation of the hand movement, event-related desynchronization occurs, activating the primary motor cortex and decreasing μ / β rhythm energy in the contralateral brain, driving muscle contraction. After the hand movement ends, the sensorimotor cortex exhibits a strong β rhythm rebound, terminating the motor program, processing sensory feedback, and resetting the brain's neural and motor networks for the next action. Simultaneously, before hand movement, muscle contraction is typically accompanied by EMG signals reflecting the intensity and flexion / extension of the joints, as well as information about the hand's movement, shape, position, and orientation during the gesture.

[0031] However, due to the weak amplitude and low signal-to-noise ratio of EEG signals, and the potential for low signal quality in EMG signals due to hand tremors, accurate and robust gesture recognition is difficult to achieve. Furthermore, existing EEG- or EMG-based gesture recognition research primarily focuses on gesture category classification tasks with limited degrees of freedom, lacking research on sophisticated regression tasks such as hand position estimation and joint angle estimation.

[0032] Building upon this foundation, existing technologies for hand position prediction can extract specific information from single-modal signals from EEG and EMG signals respectively, and then analyze both types of specific information together to obtain the target's hand position. However, during movement, the brain typically generates EEG signals first, and as these signals are transmitted, the limb muscles begin to generate electrical activity, forming surface EMG signals, which then lead to hand position changes. Research during the development process revealed that related technologies, by only maintaining temporal synchronization between EEG and EMG signals during acquisition, struggle to semantically align the inherently physiologically delayed EEG and EMG signals. This results in a technical problem where accurate features cannot be extracted and fused from EEG and EMG signals to decode and obtain relatively accurate hand position information.

[0033] In view of this, embodiments of the present invention provide a hand position prediction method based on asynchronous time-frequency fusion of electroencephalography (EEG) and electromyography (EMG), comprising: performing sliding window segmentation on the preprocessed EEG signal of the target object using a first time window to obtain initial EEG data; performing sliding window segmentation on the preprocessed EMG signal of the target object using a second time window to obtain target EMG data, wherein the sliding window length of the first time window is greater than the sliding window length of the second time window; performing dimensional transformation on the initial EEG amplitude and initial EEG phase extracted from the initial EEG data based on the number of channels and length of the target EMG data to obtain target EEG data, wherein the target EEG data includes the target EEG phase and target EEG amplitude after dimensional transformation; performing cross-level multi-dimensional feature extraction and fusion processing on the target EEG data and target EMG data using a feature fusion module within the hand position prediction model to obtain target fused features; and performing feature mapping processing on the target fused features using a regressor within the hand position prediction model based on the three-dimensional coordinate system of the hand to obtain the target hand position information of the target object.

[0034] Figure 1 An application scenario diagram of the hand position prediction method based on asynchronous time-frequency fusion of brain electromyography according to an embodiment of the present invention is shown.

[0035] like Figure 1 As shown, the application scenario according to this embodiment may include a first EEG acquisition device 101, a first EMG acquisition device 102, a target object 103, and a processing device 104. The first EEG acquisition device 101 is used to acquire the EEG signals emitted by the target object 103. The first EMG acquisition device 102 is used to acquire the EMG signals emitted by the target object 103.

[0036] After acquiring EEG signals and EMG signals, the first EEG acquisition device 101 and the first EMG acquisition device 102 can interact with the processing device 104 to receive or send signals, etc.

[0037] The processing device 104 can be a receiver and processing device for receiving various electroencephalogram (EEG) signals and electromyogram (EMG) signals, such as receiving and processing EEG signals acquired by the first EEG acquisition device 101 and receiving and processing EMG signals acquired by the first EMG acquisition device 102 (this is just an example). The processing device 104 can analyze and process the received EEG and EMG signals and other data signals, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the hand position prediction method based on asynchronous time-frequency fusion of electroencephalography (EEG) and electromyography (EMG) provided in this embodiment of the invention can generally be executed by the processing device 104. Correspondingly, the hand position prediction device based on asynchronous time-frequency fusion of EEG and EMG provided in this embodiment of the invention can generally be housed in the processing device 104. The hand position prediction method based on asynchronous time-frequency fusion of EEG and EMG provided in this embodiment of the invention can also be executed by a receiver or receiver cluster that is different from the processing device 104 and capable of communicating with the first EEG acquisition device 101, the first EMG acquisition device 102, and / or the processing device 104. Correspondingly, the hand position prediction device based on asynchronous time-frequency fusion of EEG and EMG provided in this embodiment of the invention can also be housed in a processing device or processing device cluster that is different from the processing device 104 and capable of communicating with the first EEG acquisition device 101, the first EMG acquisition device 102, and / or the processing device 104.

[0039] It should be understood that Figure 1 The number of the first EEG acquisition device, the first EMG acquisition device, the target object, and the processing device shown is merely illustrative. Depending on the implementation requirements, any number of EEG acquisition devices, EMG acquisition devices, target objects, and processing devices can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-9 The hand position prediction method based on asynchronous time-frequency fusion of brain-myoelectricity in the disclosed embodiments is described in detail.

[0041] Figure 2 A flowchart of a hand position prediction method based on asynchronous time-frequency fusion of brain electromyography according to an embodiment of the present invention is shown.

[0042] like Figure 2 As shown, the hand position prediction method based on asynchronous time-frequency fusion of brain-myoelectricity in this embodiment includes operations S210~S240.

[0043] In operation S210, the preprocessed EEG signal of the target object is segmented by sliding window using the first time window to obtain initial EEG data, and the preprocessed EMG signal of the target object is segmented by sliding window using the second time window to obtain target EMG data.

[0044] The sliding window length of the first time window is greater than that of the second time window.

[0045] Initial multi-channel EEG and EMG signals can be obtained from the left motor region of the target subject and the moving right hand. Then, preprocessing such as filtering is performed on the multi-channel EEG and EMG signals to remove some interference information and retain the effective information. With equal step length, asynchronous sliding window segmentation is performed on the EEG and EMG signals using a first time window and a second time window with unequal sliding window lengths, thereby obtaining asynchronous initial EEG data and target EMG data. Both the initial EEG data and the target EMG data are time-domain data.

[0046] For example, the sliding window length of the first time window can be 1000 ms (equivalent to 500 data points), and the sliding window length of the second time window can be 200 ms (equivalent to 100 data points), with a step size of 10 ms (equivalent to 5 data points). Thus, the first time window with a step size of 10 ms and a sliding window length of 1000 ms can be used to segment multi-channel EEG signals, obtaining an initial EEG data set containing multiple initial EEG data points. Similarly, the second time window with a step size of 10 ms and a sliding window length of 200 ms can be used to segment multi-channel EMG signals, obtaining an initial EMG data set containing multiple target EMG data points.

[0047] In operation S220, based on the number of channels and length of the target EMG data, the initial EMG amplitude and initial EMG phase extracted from the initial EEG data are dimensionally transformed to obtain the target EMG data.

[0048] The target EEG data may include the target EEG phase and target EEG amplitude after dimensionality transformation.

[0049] The size of both the initial EEG data and the target EMG data is determined by the number of channels multiplied by the length. The initial EEG data in the time domain can be transformed to obtain multi-channel EEG data in the frequency domain. Then, the initial EEG amplitude and initial EEG phase are extracted from the multi-channel EEG data in the frequency domain. Based on the number of channels and the length of the target EMG data, dimensionality transformation is performed on the initial EEG amplitude and initial EEG phase to unify the length dimensions of the extracted EEG and EMG data, facilitating feature fusion.

[0050] For example, the preprocessed EEG signal can have 9 channels, the initial EEG data after sliding window segmentation has 9 channels, the EMG signal can have 5 channels, and the target EMG data after sliding window segmentation has 5 channels. After performing sliding window segmentation on the preprocessed EEG signal using the first time window, initial EEG data with a size of 9×500 (number of channels × length (data points)) can be obtained. After performing sliding window segmentation on the EMG signal using the second time window, target EMG data with a size of 5×100 can be obtained. Based on the size of the target EMG data, the initial EEG amplitude and initial EEG phase, both with a size of 9×500, extracted from the initial EEG data are transformed to obtain target EEG amplitude and target EEG phase, both with a size of 45×100.

[0051] In operation S230, the feature fusion module within the hand position prediction model is used to perform cross-level multi-dimensional feature extraction and fusion processing on the target EEG data and target EMG data to obtain the target fused features.

[0052] A pre-trained hand position prediction model can include network layers such as feature extraction networks (modules), feature fusion networks (modules), and regressors that can perform up-dimensional extraction, fusion, and mapping of data features.

[0053] Given the target EEG amplitude and phase in the frequency domain and the target EMG data in the time domain, these data are input into a pre-trained hand position prediction model. The hand position prediction model extracts corresponding multi-level monomodal EEG features and monomodal EMG features from the target EEG amplitude, phase, and EMG data, respectively. Then, the multi-level monomodal EEG features and monomodal EMG features are fused and mapped in multiple dimensions to obtain multi-dimensional features containing hand position information.

[0054] In operation S240, based on the three-dimensional coordinate system of the hand, the regressor in the hand position prediction model is used to perform feature mapping processing on the target fusion features to obtain the target hand position information of the target object.

[0055] A three-dimensional coordinate system for the hand can include the x-axis, y-axis, and z-axis. Based on this coordinate system, a regressor trained within the hand position prediction model is used to perform feature mapping on the obtained multi-dimensional target fusion features, thereby obtaining the specific coordinates of the hand position of the target object at a coarse-grained level, in order to predict the hand position of the target object.

[0056] According to an embodiment of the present invention, by using a first time window with a longer sliding window length to segment the preprocessed EEG signal and using a second time window with a shorter sliding window length to segment the preprocessed EMG signal, initial EEG data and target EMG data with different numbers of data points in the time domain are obtained, thereby constructing asynchronous EEG data and EMG data while taking into account the inherent physiological time delay of EEG and EMG.

[0057] Then, the initial EEG data in the time domain is transformed to obtain EEG amplitude and phase information in the frequency domain. Based on the number of channels and length of the target EMG data in the time domain, the initial EEG amplitude and initial EMG phase are transformed to obtain data in the same dimension that retains both the global information of the EEG and the data that reflects the brain's control of hand movement, as well as the original EMG response to the control of the EEG and the movement execution data generated by the EMG. This constructs asynchronous time-frequency input data containing more comprehensive features, so as to align the semantic information in the EEG data and the EMG data under the premise of considering the inherent physiological time delay of EEG and EMG.

[0058] Then, a pre-trained, robust, and reliable hand position prediction model with multiple layers of network modules is used to extract corresponding single-modal features from asynchronous time-frequency input data (target EEG phase, target EEG amplitude, and target EMG data). Cross-layer, multi-dimensional feature extraction and fusion processing is performed on multiple single-modal features to obtain multi-dimensional, multi-level target fusion features corresponding to hand position. A regressor is used to map the target fusion features based on a three-dimensional coordinate system to obtain accurate predicted hand position information. This achieves a complete, multi-dimensional integrated signal data processing from EEG signals to asynchronous time-frequency data construction to multi-level feature extraction, efficiently and accurately predicting the hand position of the target object, thus expanding the application scenarios and applicability of hand position prediction.

[0059] According to an embodiment of the present invention, the method of obtaining initial EEG data by sliding window segmentation of the preprocessed EEG signal of the target object using a first time window, and obtaining target EMG data by sliding window segmentation of the preprocessed EMG signal of the target object using a second time window includes the following operations.

[0060] According to an embodiment of the present invention, the initial electroencephalogram (EEG) signal and the initial electromyogram (EMG) signal are filtered to obtain filtered EEG signal and filtered EMG signal, respectively.

[0061] First, acquire multichannel initial EEG signals from the left motor region and multichannel initial EMG signals from the right hand. Then, use the decrease and rebound of the β power of the C3 channel in the EEG as criteria for identifying the active segment of the EEG signal, and extract the multichannel initial EEG signal of the active segment from the initial EEG signal. For the multichannel initial EMG signal, since the EMG signal usually does not have significant signal fluctuations (remains a stable signal) before hand movement, the initial EEG signal located after the active segment and exhibiting significant signal fluctuations can be used as the screening criterion for the active segment of the EMG signal, facilitating the extraction of the multichannel initial EEG signal of the active segment from the multichannel initial EMG signal.

[0062] The moment when the movement command is given to the target can also be used as a timestamp to extract the multi-channel initial EEG signal and multi-channel initial EMG signal of the active segment from the initial EEG signal and initial EMG signal.

[0063] Then, the extracted multi-channel initial EEG and EMG signals of the active segment are subjected to bandpass filtering and power frequency notch filtering. For example, a 4th-order Butterworth filter can be used to perform 0.5-100Hz bandpass filtering and 50Hz power frequency notch filtering on the initial EEG and EMG signals.

[0064] It can also directly perform bandpass filtering and power frequency notch filtering on the acquired multi-channel initial EEG signals and multi-channel initial EMG signals, and default the filtered signals as active segment signals.

[0065] According to an embodiment of the present invention, the filtered EEG signal and the filtered EMG signal are normalized respectively to obtain the processed EEG signal and the processed EMG signal.

[0066] Given a filtered EEG signal, the EEG signal located 1 second before the initial active segment EEG signal from the multi-channel initial EEG signal can be used as the baseline signal. This baseline signal is then used to rereference the filtered EEG signal to calibrate the initial EEG signal. Specifically, the baseline signal is subtracted from the initial active segment EEG signal to obtain the rereferenced signal, which is then used as the calibrated filtered EEG signal.

[0067] Then, the mean and variance of each channel in the multi-channel filtered EEG and multi-channel filtered EMG signals are calculated. The signals of each channel are then normalized so that the mean of each channel of the filtered EEG and filtered EMG signals is 0 and the variance is 1. This allows the EEG and EMG signals to be processed in the model in a more stable state under the same dimensions.

[0068] According to an embodiment of the present invention, the processed EEG signal is segmented using a first time window to obtain initial EEG data, and the processed EMG signal is segmented using a second time window to obtain target EMG data.

[0069] When performing sliding window segmentation on EEG and EMG signals, the cutoff times of the window lengths for the two signals must be the same. That is, when performing sliding window segmentation on EMG signals, the cutoff time of the second time window's sliding window length must be the same as the cutoff time of the first time window's sliding window length for EEG signals.

[0070] For example, the sliding window length of the first time window is 1000ms, the step size is 10ms, the length of the EEG signal processed is 0~1020ms, the segmentation starts from 0ms of the processed EEG signal, and the first initial EEG data of 0~1000ms, the second initial EEG data of 10~1010ms, and the third initial EEG data of 20~1020ms are obtained.

[0071] Then, a second time window is used to perform sliding window segmentation on the processed EMG signals. The sliding window length of the second time window is 200ms, the step size is 10ms, and the processing length of the EMG signals is 800~1020ms. The processed EMG signals are segmented backward using the segmentation cutoff times of the first initial EMG data (1000ms), the second initial EMG data (1010ms), and the third initial EMG data (1020ms), resulting in the first target EMG data (800~1000ms), the second target EMG data (810~1010ms), and the third target EMG data (820~1020ms). When the processing length of the EMG signals is 820~1100ms, the processed EMG signals are segmented backward using the segmentation cutoff times of the first initial EMG data (1000ms), the second initial EMG data (1010ms), and the third initial EMG data (1020ms), resulting in the target EMG data (820~1020ms).

[0072] According to embodiments of the present invention, preprocessing of the initial EEG and EMG signals by filtering and normalizing is performed to denoise and calibrate the acquired EEG and EMG signals, thereby obtaining high-quality processed EEG and EMG signals. Then, the processed EEG signals are processed using a first time window, and the processed EMG signals are processed using a second time window, resulting in initial EEG data and target EMG data with the same cutoff time but different data lengths after segmentation. This achieves preprocessing of the initial EEG and EMG signals. Furthermore, considering the inherent physiological temporal delays of EEG and EMG, asynchronous EEG and EMG data are constructed using the first and second time windows. This facilitates the extraction of more comprehensive and accurate features and semantic information from the asynchronous EEG and EMG data, thereby semantically aligning the EEG and EMG data.

[0073] According to an embodiment of the present invention, a method for obtaining target EEG data by performing dimensional transformation on the initial EEG amplitude and initial EEG phase extracted from the initial EEG data based on the number of channels and length of the target EMG data includes the following operations.

[0074] According to an embodiment of the present invention, the initial EEG data is processed by Fast Fourier Time-Frequency Transform to obtain the initial EEG amplitude and initial EEG phase.

[0075] The lengths of the initial EEG amplitude and initial EEG phase are the same as the length of the initial EEG data.

[0076] Specifically, the time window for the Fast Fourier Transform (FFT) can be a Hamming window, and the sliding window length of the Hamming window is the same as the length of the first time window used for sliding window processing of the initial EEG signal. For example, if the sliding window length of the first time window is 1000 ms (500 data points), then the sliding window length of the Hamming window used for the FFT of the initial EEG data is also 1000 ms (500 data points).

[0077] According to an embodiment of the present invention, the initial EEG amplitude and initial EEG phase are subjected to dimensionality transformation processing based on the number of channels and length of the target EEG data to obtain the target EEG data.

[0078] The target EEG data and the target EMG data are of the same length.

[0079] The initial EEG data size can be as shown in formula (1), the target EMG data size can be as shown in formula (2), and the target EEG data after dimensional transformation can be as shown in formula (3).

[0080] (1);

[0081] Here, c1 can be represented as the number of channels in the initial EEG data, and length1 can be represented as the length of the initial EEG data.

[0082] (2);

[0083] Here, c2 can be represented as the number of channels in the target electromyography data, and length2 can be represented as the length of the target electromyography data.

[0084] (3);

[0085] As can be seen from the above formulas (2) and (3), the length of the target EEG data and the length of the target EMG data after dimensional transformation are both length2, that is, the number of data points is the same.

[0086] According to an embodiment of the present invention, the initial EEG data is processed by Fast Fourier Transform to convert the time-domain EEG data into frequency-domain EEG data, and the initial EEG amplitude and initial EEG phase are determined. Then, based on the number of channels and length of the target EMG data, both the initial EEG amplitude and initial EMG phase are subjected to dimensionality transformation processing to obtain target EEG data and target EMG data of the same size and length. This results in asynchronous time-domain data (target EMG data) and frequency-domain data (target EEG amplitude and target EMG phase), which facilitates the alignment of semantic information in the EEG data and the EMG data while taking into account the inherent physiological temporal delays of EEG and EMG.

[0087] Figure 3 A schematic diagram illustrating the process of obtaining target EEG data and target EMG data during the application process according to an embodiment of the present invention is shown.

[0088] like Figure 3 As shown, Figure 3 This paper illustrates the process of using a first time window and a second time window to perform sliding window segmentation on EEG and EMG signals, followed by dimensional transformation to obtain target EEG and EMG data. First, the EEG signals are segmented using a first time window of 1000 ms to obtain initial EEG data. Then, the EMG signals are segmented using a second time window of 200 ms to obtain target EMG data. The initial EEG data has a size of 9×500 (number of channels × number of data points), and the target EMG data has a size of 5×100. A Fast Fourier Transform is then performed on the initial EEG data to extract initial EEG amplitude and initial EEG phase of size 9×500. Finally, based on the size of the target EMG data, a dimensional transformation is performed on the initial EEG amplitude and initial EEG phase to obtain target EEG amplitude and target EEG phase of size 45×100.

[0089] Figure 4 A schematic diagram of the target electroencephalogram (EEG) data and target electromyogram (EMG) data obtained during the application process according to an embodiment of the present invention is shown.

[0090] like Figure 4 As shown, Figure 4 The obtained target EEG and target EMG data are shown. The target EEG data after the first time window (1000ms) sliding window segmentation has 9 channels and a total length of 1000ms. The target EMG data after the second time window (200ms) sliding window segmentation has 5 channels and a total length of 200ms. The target EEG data can include five segments (0-0.2s segment 1, 0.2-0.4s segment 2, 0.4-0.6s segment 3, 0.6-0.8s segment 4, and 0.8-1.0s segment 5). These five segments correspond to the brain's process of receiving and parsing instructions, generating corresponding signals to control muscle movement based on the parsed instructions, and transmitting these control signals to the muscles. Simultaneously, the target muscles only begin to move and generate corresponding electromyographic (EMG) signals after receiving the corresponding EEG control signals. Therefore, there is an inherent physiological timing difference between EMG and EEG signals. By using first and second time windows of different lengths, the EEG and EMG signals are segmented and extracted in a targeted manner to comprehensively extract the semantic information from the EEG signals and align them temporally with the EMG signals.

[0091] According to an embodiment of the present invention, the method for obtaining target fused features by using the feature fusion module within the hand position prediction model to perform cross-level multidimensional feature extraction and fusion processing on target EEG data and target EMG data may include the following operations.

[0092] Figure 5 A flowchart illustrating the acquisition of target fusion features according to an embodiment of the present invention is shown.

[0093] like Figure 5 As shown, the method for obtaining target fusion features in this embodiment includes operations S510~S530.

[0094] According to embodiments of the present invention, a hand position prediction model may include a feature fusion module and a regressor. The feature fusion module may include a brain electromyography feature extraction submodule, a cross-level fusion subnetwork, and a fully connected network.

[0095] In operation S510, the EEG feature extraction submodule is used to perform dimensionality-upgrading and multi-level feature extraction and fusion processing on the target EEG amplitude, target EEG phase and target EMG data to obtain multi-level EEG features, multi-level EMG features and initial fusion features.

[0096] The EEG / EMG feature extraction submodule can include multiple subnetworks consisting of a dimensionality-upgrading module and multiple hierarchical modules. Different subnetworks are used to perform dimensionality-upgrading and feature extraction fusion on the target EEG amplitude, target EEG phase, and target EMG data, respectively, to obtain multi-level EEG features, multi-level EMG features, and initial fusion features after fusing the target EEG amplitude, target EEG phase, and target EMG data, and then transmit them to the cross-hierarchical fusion subnetwork.

[0097] In the S520 operation, the cross-level fusion subnetwork uses multi-level EEG features and multi-level EMG features to perform cross-level feature fusion processing on the initial fusion features to obtain multi-level total fusion features.

[0098] Upon receiving multi-level EEG features, multi-level EMG features, and initial fusion features, the cross-level fusion sub-network first uses the first-level features from the multi-level EEG and EMG features to perform feature fusion processing on the initial fusion features, obtaining the first-level fusion features. Then, it uses the second-level features from the multi-level EEG and EMG features to perform feature fusion processing on the obtained first-level fusion features, obtaining the second-level fusion features. This process is iteratively repeated to perform cross-level feature fusion, layer by layer concatenating and fusing the fine-grained features to coarse-grained semantic information extracted from the target EEG and target EMG data until the L-th level fusion feature is obtained. Finally, the L-th level fusion feature is concatenated and fused as a whole to obtain the multi-level total fusion feature, which is then sent to the fully connected network.

[0099] The total number of levels in the cross-level fusion sub-network is the same as the total number of levels in the brain electromyography feature extraction sub-module.

[0100] In the operation of S530, a fully connected network is used to perform feature semantic integration processing on the multi-level total fusion features to obtain the target fusion features.

[0101] Among them, the target fusion feature is represented by multi-dimensional features corresponding to the hand position, and the multi-level total fusion feature is represented by multi-dimensional features corresponding to EEG and EMG.

[0102] Fully connected networks can perform multi-dimensional feature transformation and mapping on the received multi-channel, multi-level total fusion features to obtain multi-dimensional target fusion features corresponding to the hand position.

[0103] According to an embodiment of the present invention, after obtaining asynchronous time-frequency input data, the feature fusion module in the pre-trained hand position prediction model is used to perform dimensionality upscaling and feature extraction processing on the target EEG phase, target EEG amplitude, and target EMG data, resulting in multi-level EEG features and multi-level EMG features (the feature levels of multi-level EEG features and multi-level EMG features can be from fine-grained to coarse-grained, from detailed features to semantic information). An initial fusion feature is obtained by fusing the target EEG phase, target EEG amplitude, and target EMG data. Then, a cross-level fusion subnetwork is used to perform cross-level feature fusion on the multi-level EEG features, multi-level EMG features, and initial fusion features, layer by layer superimposing and splicing the features from different levels, thereby obtaining a multi-level total fusion feature that integrates all fine-grained and coarse-grained fusion features. Finally, a fully connected network is used to convert the multi-channel multi-level total fusion feature into multi-dimensional features and map them to multi-dimensional features corresponding to the hand position, so that a regressor can extract the three-dimensional coordinate information for hand position prediction from the obtained target fusion features.

[0104] According to embodiments of the present invention, the brain electromyography feature extraction submodule may include a single-modality brain electronics network, a single-modality muscle electronics network, and a brain electromyography fusion subnetwork.

[0105] According to an embodiment of the present invention, the method for performing dimensionality-upgrading and multi-level feature extraction and fusion processing on target EEG amplitude, target EEG phase and target EMG data using the EEG / EMG feature extraction submodule to obtain multi-level EEG features, multi-level EMG features and initial fusion features includes the following operations.

[0106] According to an embodiment of the present invention, a single-modal brain electronic network is used to perform dimensionality-upgrading processing on the target EEG amplitude and the target EEG phase, respectively, to obtain dimensionality-upgraded EEG amplitude and dimensionality-upgraded EEG phase.

[0107] A single-modality EEG network may include a first upscaling module for upscaling the target EEG amplitude and a second upscaling module for upscaling the target EEG phase. The first and second upscaling modules are used to upscale the target EEG amplitude and phase, respectively, to transform the low-dimensional target EEG amplitude and phase into high-dimensional feature vectors. This increases the dimensionality and expressive power of the upscaled EEG amplitude and phase, facilitating accurate identification of key features from the upscaled EEG amplitude and phase.

[0108] Specifically, both the first and second dimensionality-upgrading modules can include a convolutional layer with T channels and a 1×1 kernel size, and a batch normalization layer. The number of channels can be set to 32. The convolutional layer in both modules can be used to perform cross-channel feature fusion on the target EEG amplitude and phase, respectively, thereby associating and integrating the detailed information in the target EEG amplitude and phase, and increasing their dimensionality to obtain intermediate-dimensional EEG amplitude and phase. Then, the batch normalization layer performs distribution standardization on the intermediate-dimensional EEG amplitude and phase to improve data stability.

[0109] According to an embodiment of the present invention, the amplitude and phase of the upgraded EEG are added together to obtain the upgraded EEG features.

[0110] By adding elements from the amplitude and phase of the upgraded EEG, an upgraded EEG feature containing all semantic features in the frequency domain can be obtained, which facilitates the extraction of fine-grained and coarse-grained EEG features from the upgraded EEG feature.

[0111] According to an embodiment of the present invention, multi-level feature processing is performed on the upgraded EEG features to obtain multi-level EEG features.

[0112] The unimodal EEG network may also include a first-level module for feature extraction processing of upgraded EEG features. This first-level module is of level L. Each layer in the first-level module can analyze and extract upgraded EEG features from fine to coarse, resulting in multi-level EEG features. Higher levels mean coarser feature granularity and more abstract semantic information, while lower levels mean finer feature granularity and more precise detailed feature information.

[0113] Specifically, the first-level module may include a convolutional layer with T channels and a k×k kernel size, a batch normalization layer, and a ReLU activation function layer. The number of channels can be set to 32, and the kernel size can be 3×3. The convolutional layer in the first-level module can be used to extract local features from the spliced ​​and added EEG features in a progressively coarse-grained manner, resulting in multi-level EEG feature extraction from fine to coarse granularity. Then, the batch normalization layer performs distribution normalization on the multi-level EEG features to eliminate interference. Finally, the activation function layer expresses the nonlinear mapping relationship between the multi-level EEG features and hand position, enabling the model to activate and express specific features based on the complex nonlinear correspondence between features in the pre-trained EEG signal and hand position, thus obtaining multi-level EEG features.

[0114] According to an embodiment of the present invention, a single-modal electromyography network is used to perform dimensionality upscaling and multi-level feature processing on the target electromyography data to obtain multi-level electromyography features.

[0115] A single-modal electromyography (EMG) network can include a third dimensionality-upgrading module for upgrading the target EMG data and a second-level module for multi-level feature extraction from the upgraded target EMG data. The third dimensionality-upgrading module transforms the low-dimensional target EMG data into a high-dimensional feature vector, resulting in upgraded EMG features. This increases the dimensionality and expressive power of the upgraded EMG features, facilitating accurate identification of key features. Each layer in the second-level module can analyze and extract upgraded EMG features from fine to coarse, resulting in multi-level EMG features. The first and second-level modules have the same number of layers; higher layers indicate coarser feature granularity and more abstract semantic information, while lower layers indicate finer feature granularity and more precise detailed feature information.

[0116] Specifically, the third dimensionality-upgrading module can also include a convolutional layer with T channels and a 1×1 kernel size, and a batch normalization layer, where the number of channels can also be set to 32. The convolutional layer in the third dimensionality-upgrading module can be used to perform cross-channel feature fusion on the target EMG data, thereby associating and integrating the detailed information in the target EMG data and increasing its dimensionality to obtain intermediate-dimensional EMG data. Then, the batch normalization layer performs distribution standardization processing on the intermediate-dimensional EMG data to improve the stability of the data.

[0117] The second-level module can include a convolutional layer with T channels and a k×k kernel, a batch normalization layer, and a ReLU activation function layer. The number of channels can be set to 32, and the kernel size can be 3×3. Specifically, the convolutional layer in the second-level module can be used to extract local features from the upgraded electromyographic features in a progressive manner, from fine to coarse, resulting in multi-level extracted electromyographic features ranging from fine to coarse granularity. Then, the batch normalization layer performs distribution normalization on the multi-level extracted electromyographic features to eliminate interference information. Finally, the activation function layer expresses the nonlinear mapping relationship between the multi-level extracted electromyographic features and hand position, enabling the model to activate and express specific features based on the complex nonlinear correspondence between features in the pre-trained electromyographic signal and hand position, thus obtaining multi-level electromyographic features.

[0118] According to an embodiment of the present invention, a brain-motor fusion subnetwork is used to fuse the target brain electrical amplitude, target brain electrical phase, and target electromyographic data to obtain initial fusion features.

[0119] The EEG / EMG fusion subnetwork may include a first fusion module for feature fusion of target EEG amplitude, target EEG phase, and target EMG data. First, the target EEG amplitude and target EEG phase are element-wise added to obtain composite EEG data. Then, the composite EEG data and the target EMG data are concatenated to obtain initial fused EMG data. The first fusion module then performs feature fusion processing on the initial fused EMG data to obtain initial fused features.

[0120] Specifically, the first fusion module may include a convolutional layer with T channels and a 1×1 kernel size, and a batch normalization layer. The number of channels can also be set to 32. The convolutional layer in the first fusion module can be used to perform cross-channel feature fusion on the initial fused EMG data, thereby associating and integrating detailed information and increasing the dimensionality of the initial fused EMG data to obtain intermediate fused EMG data. Then, the batch normalization layer performs distribution normalization processing on the intermediate fused EMG data to obtain stable initial fusion features.

[0121] According to an embodiment of the present invention, a single-modality EEG network in the EEG feature extraction submodule is used to perform multi-level feature extraction processing on the target EEG amplitude and phase. A single-modality myoelectronic network is then used to perform multi-level feature extraction on the target EMG data. Finally, an EEG fusion subnetwork is used to fuse the target EEG amplitude, target EEG phase, and target EMG data to obtain initial fused features. Thus, by utilizing the layer-by-layer interaction between the single-modality EEG network, the single-modality myoelectronic network, and the EEG fusion subnetwork in the EEG feature extraction submodule, consistent information on various features in EEG and EMG is accurately and comprehensively obtained. This facilitates cross-level fusion based on the initial fused features, using multi-level single-modality features to obtain complete feature semantic information of EMG and EEG from fine-grained to coarse-grained.

[0122] According to an embodiment of the present invention, the method for obtaining multi-level total fusion features by using multi-level EEG features and multi-level EMG features to perform cross-level feature fusion processing on the initial fusion features of the cross-level fusion sub-network may include the following operations.

[0123] According to an embodiment of the present invention, when the multi-level is L-level, the initial fusion features are subjected to cross-level feature fusion processing using the EEG features of the first level in the L-level EEG features and the EMG features of the first level in the L-level EMG features to obtain the fusion features of the first level; the fusion features of the first level are subjected to cross-level feature fusion processing using the EEG features of the (l+1)th level in the L-level EEG features and the EMG features of the (l+1)th level in the L-level EMG features to obtain the fusion features of the (l+1)th level, where L≥3 and l≥1.

[0124] The cross-layer fusion subnetwork can include a second fusion module consisting of a convolutional layer with T channels and a kernel size of k1×k2, a batch normalization layer, and a ReLU activation function layer, and a third fusion module consisting of a convolutional layer with T channels and a kernel size of k1×k2, a batch normalization layer, a ReLU activation function layer, and a Dropout layer (regularization layer). In the third fusion module, the number of channels T can be set to 32, and the kernel size can be 4×5.

[0125] The convolutional layers in the second fusion module can be used to perform cross-level and cross-modal feature fusion on any level of EEG features, any level of EMG features, and initial fused EMG data or fused features at any level. This allows for the fusion of features layer by layer, starting from fine-grained features, until coarse-grained semantic information features are fused. Then, an over-batch normalization layer is used to standardize the distribution of the convolutional features, making the fused features more stable. Finally, an activation function layer is used to express the nonlinear mapping relationship of the fused features, enabling the model to obtain feature semantic expressions with complex correlations to hand position based on the complex nonlinear correspondence between the fused features and hand position, thus obtaining the fused features at each level.

[0126] For example, when L is 4, four levels of EEG features are first extracted using a single-modality EEG network, and four levels of EMG features are extracted using a single-modality myoelectronic network. Cross-level feature fusion is then performed on the first-level EEG features, the first-level EMG features, and the initial fusion features to obtain the first-level fusion features. Cross-level feature fusion is then performed on the second-level EEG features, the second-level EMG features, and the first-level fusion features to obtain the second-level fusion features. Cross-level feature fusion is then performed on the third-level EEG features, the third-level EMG features, and the second-level fusion features to obtain the third-level fusion features. Cross-level feature fusion is then performed on the fourth-level EEG features, the fourth-level EMG features, and the third-level fusion features to obtain the fourth-level fusion features.

[0127] According to an embodiment of the present invention, the L-level fused features obtained after cross-level feature fusion processing are spliced ​​together to obtain the L-level total fused features.

[0128] After obtaining the fusion features of L levels, the fusion features of L levels can be concatenated first, and then added according to the elements to obtain the total concatenated features of L levels. Then, the total concatenated features of L levels are input into the third fusion module in the cross-level fusion sub-network for feature fusion to obtain the total fusion features of L levels.

[0129] The convolutional layers in the third fusion module can be used to perform cross-level and cross-modal feature fusion on the L-level concatenated total features composed of fused features from each level, and adjust the number of channels to perform deep fusion of the fused features from each level in the L-level concatenated total features. Then, an over-batch normalization layer is used to standardize the distribution of the deeply fused features to make the features more stable. Then, an activation function layer is used to express the nonlinear mapping relationship of the features, so that the model can obtain a feature semantic expression with a complex correlation with the hand position based on the complex nonlinear correspondence between the fused features and the hand position. Finally, a Dropout layer is used to constrain the feature fusion process to avoid overfitting, thereby obtaining the L-level total fused features.

[0130] The size of the initial fusion feature is the same as the size of the fusion feature at each level. For example, if the size of the target EEG data is 45×100 (number of channels × number of data points) and the size of the target EMG data is 5×100, and the size of the initial fusion feature is 50×100, then the size of the fusion feature at each level is also 50×100.

[0131] According to an embodiment of the present invention, the second fusion module in the cross-level fusion sub-network performs layer-by-layer detailed feature fusion of the EEG features of the (l+1)th level, the EMG features of the (l+1)th level, and the fusion features of the 1st level, to obtain L levels of fusion features. Then, the third fusion module performs overall cross-level feature fusion on the obtained L levels of fusion features, thereby obtaining a total fusion feature containing all fine-grained feature information and all coarse-grained semantic information. Based on the cross-level interactive fusion in the feature fusion process, the consistent information of various accurate and comprehensive features extracted from EEG and EMG can be fused from fine-grained to coarse-grained system multimodal fusion, thereby accurately predicting hand position information.

[0132] According to an embodiment of the present invention, a fully connected network may include a first fully connected layer and a second fully connected layer.

[0133] According to embodiments of the present invention, a method for performing feature semantic integration processing on multi-level total fusion features using a fully connected network to obtain target fusion features may include the following operations.

[0134] According to an embodiment of the present invention, the multi-level total fusion features are integrated using a first fully connected layer to obtain intermediate fusion features.

[0135] The first fully connected layer may include a first sub-fully connected layer with w1 neurons, a batch normalization layer, and a ReLU activation function layer. The second fully connected layer may include a second sub-fully connected layer with w2 neurons, a batch normalization layer, and a ReLU activation function layer. The number of neurons w1 and w2 in the two fully connected layers can be the same and set to 512. Alternatively, the number of neurons w1 and w2 in the two fully connected layers can be different, and the specific number of neurons can be set according to the training parameters.

[0136] According to an embodiment of the present invention, the intermediate fused features are processed by a second fully connected layer to obtain the target fused features.

[0137] Among them, the intermediate fusion feature is characterized by multidimensional fusion features corresponding to EEG and EMG, and the target fusion feature is characterized by multidimensional fusion features corresponding to hand position.

[0138] First, the first fully connected layer is used to globally integrate the multi-level total fusion features to obtain multi-dimensional fusion features corresponding to EEG and EMG. Then, the second fully connected layer is used to globally map the multi-dimensional fusion features corresponding to EEG and EMG to obtain the target fusion features corresponding to the hand position.

[0139] Among them, the fully connected network and the cross-level fusion sub-network can together constitute the cross-level fusion sub-module in the hand position prediction model.

[0140] According to an embodiment of the present invention, by using a fully connected network containing two fully connected layers to integrate and map the multi-level total fusion features, multi-dimensional and multi-level fusion features corresponding to the hand position can be obtained. This allows a regressor to perform feature mapping on the target fusion features based on a three-dimensional coordinate system, thereby obtaining accurate predicted hand position information.

[0141] The regressor of the hand position prediction model can include a third fully connected layer. The number of neurons in the third fully connected layer, w3, can be set to 3, corresponding to the three-dimensional coordinate system of the hand (X, Y, Z axes). Thus, through the mapping of the regressor, the predicted hand position information is obtained.

[0142] Figure 6 A schematic diagram illustrating the entire process of obtaining the target hand position information of a target object according to an embodiment of the present invention is shown.

[0143] like Figure 6 As shown, Figure 6This paper illustrates the entire process of feature extraction, fusion, and prediction of target EEG amplitude, target EEG phase, and target EMG data using a hand position prediction model. The target EEG amplitude and phase are input into the first and second dimensionality-upgrading modules of a single-modality EEG network. Dimensionality upgrading is performed through convolutional layers and batch normalization layers to obtain upgraded EEG amplitude and phase. The upgraded EEG amplitude and phase are then added together to obtain initial upgraded EEG features. These initial upgraded EEG features are input into the first-level module. In the first level, feature extraction is performed on the initial upgraded EEG features through convolutional layers, batch normalization layers, and ReLU activation function layers (referred to as activation function layers) to obtain the first-level EEG features. Then, in the second level, feature extraction is performed on the features obtained by element-wise addition of the first-level EEG features and the initial upgraded EEG features through convolutional layers, batch normalization layers, and ReLU activation function layers to obtain the second-level EEG features. This process is repeated to obtain three levels of EEG characteristics.

[0144] While processing the target EEG data, a single-modality electromyography (EMG) network is used to process the target EMG data. Dimensionality is increased through convolutional layers and batch normalization layers in the third dimensionality-increasing module to obtain initial dimensionality-increasing EMG features. These initial dimensionality-increasing EMG features are then input into the second-level module. In the first level, convolutional layers, batch normalization layers, and ReLU activation function layers are used to extract features from the initial dimensionality-increasing EMG features to obtain the first-level EMG features. Then, in the second level, convolutional layers, batch normalization layers, and ReLU activation function layers are used to extract features from the element-wise sum of the first-level EMG features and the initial dimensionality-increasing EMG features to obtain the second-level EMG features. This process is repeated to obtain three levels of EMG features.

[0145] The target EEG amplitude, target EEG phase, and target EMG data are fused using the overconvolutional layer and batch normalization layer in the first fusion module of the EEG-EMG fusion subnetwork to obtain the initial fusion feature f0.

[0146] The convolutional layers, batch normalization layers, and ReLU activation function layers in multiple second fusion modules within the cross-level fusion subnetwork are used to perform cross-level feature fusion on the first-level EEG features, first-level EMG features, and initial fusion features, resulting in the first-level fusion feature f1. Cross-level feature fusion is then performed on the second-level EEG features, second-level EMG features, and the first-level fusion feature, resulting in the second-level fusion feature f2. Finally, cross-level feature fusion is performed on the third-level EEG features, third-level EMG features, and the second-level fusion feature, resulting in the third-level fusion feature f3.

[0147] The three-level fusion features are concatenated to obtain the three-level concatenated total features. The convolutional layer, batch normalization layer, ReLU activation function layer and regularization layer (Dropout layer) in the third fusion module of the cross-level fusion sub-network are used to fuse the three-level concatenated total features to obtain the three-level total fusion features.

[0148] The first fully connected layer in the fully connected network integrates the total fused features from the three levels to obtain intermediate fused features. The second fully connected layer then performs global mapping on the intermediate fused features to obtain the target fused features. Finally, a regressor is used to perform feature mapping on the target fused features to obtain the estimated hand position information.

[0149] According to an embodiment of the present invention, the hand position prediction model is trained in the following manner, and the specific training method may include the following operations.

[0150] Figure 7 A flowchart of a training hand position prediction model according to an embodiment of the present invention is shown.

[0151] like Figure 7 As shown, the method for training a hand position prediction model in this embodiment includes operations S710 to S770.

[0152] In the S710 operation, the hand position prediction model to be trained and the training sample dataset are obtained.

[0153] The training sample dataset includes multiple initial EEG samples, multiple initial EMG samples, and multiple initial hand position samples.

[0154] Multiple initial EEG samples and multiple initial EMG samples can together constitute the training sample dataset X={EEG} i sEMG i}, where EEG i It can be characterized as i initial EEG samples, sEMG i This can be represented by i initial electromyography samples, and i initial hand position samples can be represented by Pos i This indicates that the initial hand position sample can include gesture position information corresponding to the three-dimensional coordinate system of the hand.

[0155] In operation S720, the initial EEG samples and initial EMG samples are preprocessed, and the preprocessed training EEG signals, training EMG signals and initial hand position samples are segmented by sliding windows using the first time window, the second time window and the third time window to obtain initial training EEG data, target training EMG data and target hand position data samples.

[0156] The step size of the first, second, and third time windows is the same.

[0157] Multiple initial EEG samples after preprocessing are segmented using a sliding window using a first time window to obtain multiple initial training EEG data. Multiple initial EMG samples after preprocessing are segmented using a second time window to obtain multiple target EMG data. Multiple initial hand position samples are segmented using a third time window to obtain multiple target hand position data samples. Any target hand position data sample obtained after the third sliding window segmentation can be used... To indicate, This can be represented as the x-axis information in the target hand position data sample with a step size of m. This can be represented as the y-axis information in the target hand position data sample with a step size of m. This can be characterized as the z-axis information in the target hand position data sample with a step size of m. Multiple initial training EEG data and multiple target training EEG data can be used... To indicate, This can be represented as i initial training EEG data. It can be represented as i initial training electromyographic data.

[0158] In the operation of S730, the initial training EEG data is transformed based on the target training EMG data to obtain the target training EEG data.

[0159] The target training EEG data and the target training EMG data are of the same length.

[0160] The dimensionality transformation process during training can refer to the dimensionality transformation process in the above application. Multiple initial training EEG data in the time domain are transformed into multiple training EEG data in the frequency domain using Fourier transform, from which multiple training EEG amplitudes and multiple training EEG phases are obtained. Based on the number of channels and length of the target training EEG data, dimensionality transformation is performed on the multiple training EEG amplitudes and multiple training EEG phases to obtain multiple target training EEG amplitudes. Training EEG phase with multiple targets .

[0161] In the S740 operation, the target training EEG data and target training EMG data are input into the hand position prediction model to be trained. The target training EEG data and target training EMG data are processed by cross-level multi-dimensional feature extraction and fusion using the feature fusion module to be trained, so as to obtain the training fusion features.

[0162] Multiple training EEG data and multiple training EMG data are input into the hand position prediction model to be trained, as described above. Feature extraction and fusion processes are then performed as described above to obtain multiple multi-level training EEG features. Multiple multi-level target training electromyographic characteristics And multiple initial training fused features f0', where, , l can be represented as the l-th level. Cross-level feature fusion processing is performed on any multi-level target training EEG feature, the corresponding multi-level target training EMG feature, and the initial training fusion feature to obtain multi-level training fusion features. Then, using a fully connected network as described above, the training fusion features from multiple levels are integrated and mapped to obtain the training fusion features.

[0163] In the operation of S750, based on the three-dimensional coordinate system of the hand, the regressor to be trained is used to perform feature mapping processing on the training fusion features to obtain the training hand position.

[0164] The training hand positions can be mapped using the regressor structure described above, and the resulting multiple training hand positions can be used... To indicate, among which, It can be represented as multiple x-axis positions of the training hand. This can be represented by multiple training hand y-axis positions. It can be represented as multiple z-axis positions of the training hand.

[0165] In the operation of S760, the target optimization loss value is obtained based on the optimization function and the training hand position and target hand position data samples.

[0166] The hand position prediction model to be trained compares and optimizes the multiple training hand positions output by multiple sample data with multiple target hand position data samples to obtain the target optimization loss value.

[0167] In the operation of S770, the model parameters of the hand position prediction model to be trained are adjusted according to the target optimization loss value to obtain the trained hand position prediction model.

[0168] According to embodiments of the present invention, a hand position prediction model is trained extensively using a large amount of EEG sample data, EMG sample data, and hand position sample data, similar to the application process. Simultaneously, an optimization function is used to calculate the model's loss value, allowing for parameter tuning based on the loss value, thereby obtaining the trained hand position prediction model. This achieves the construction of a hand position prediction model containing parallel interactive sub-networks, cross-layer fusion sub-networks, fully connected networks, and regressors. Through extensive training, the single-modality EEG network, single-modality EMG network, and EEG-EMG fusion sub-network can interact layer by layer, improving the model's ability to extract consistent information from various features. Combined with the optimization function, a hand position prediction model with high robustness and high accuracy is obtained, thus improving efficiency in practical applications.

[0169] According to an embodiment of the present invention, a method for preprocessing initial EEG samples and initial EMG samples, and using a first time window, a second time window, and a third time window to perform sliding window segmentation processing on the preprocessed training EEG signals, training EMG signals, and initial hand position samples to obtain initial training EEG data, target training EMG data, and target hand position data samples includes the following operations.

[0170] According to an embodiment of the present invention, the initial EEG sample and the initial EMG sample are filtered to obtain training filtered EEG signal and training filtered EMG signal.

[0171] The filtering process for multiple initial EEG samples and multiple initial EMG samples during training can be referred to the filtering process in the above application process, and will not be repeated here.

[0172] In addition to referring to the acquisition process in the above application process, the acquisition of the active segment EEG signal samples can also be determined by using the collected motion signal samples as a reference to determine the active segment signals of the EEG and EMG signal samples. The initial hand position sample can be extracted from the collected motion signal samples.

[0173] According to an embodiment of the present invention, based on the reference EEG signal extracted from the training filtered EEG signal, the active EEG signal extracted from the training filtered EEG signal is subjected to rereference processing to obtain a calibrated training EEG signal.

[0174] The acquisition of the reference signal and the processing of the rereference during training can be referred to the acquisition and processing in the above application process, and will not be repeated here.

[0175] According to an embodiment of the present invention, the calibration training EEG signal and the training filtered EMG signal are normalized respectively to obtain the processed training EEG signal and the processed training EMG signal.

[0176] The normalization process during training can refer to the process described above, and will not be repeated here.

[0177] According to an embodiment of the present invention, a sliding window segmentation process is performed on the processed training EEG signal using a first time window to obtain initial training EEG data, a sliding window segmentation process is performed on the processed training electromyography signal using a second time window to obtain target training electromyography signal, and a sliding window segmentation process is performed on the initial hand position sample using a third time window to obtain target hand position data sample.

[0178] The process and requirements for sliding window segmentation using the first and second time windows during training can be referenced from the above application process, and will not be repeated here.

[0179] During the sliding window segmentation of the initial hand position samples using the third time window, it is necessary to ensure that the cutoff time of the sliding window length for the first time window (processing training EEG signals) and the cutoff time of the sliding window length for the second time window (processing training EMG signals) are the same. The sliding window length of the third time window can be 1 ms, with the same step size as the first and second time windows.

[0180] According to an embodiment of the present invention, by filtering, rereferencing, and sliding window segmentation with different sliding window lengths on the training EEG signals, training EMG signals, and initial hand position samples before training, preprocessing of the initial EEG signals and initial EMG signals used for training is achieved. This allows the model to be trained to extract more comprehensive and accurate features and semantic information from asynchronous training EEG data and training EMG data, taking into account the inherent physiological time difference, and then train it.

[0181] Figure 8 A schematic diagram of initial EEG samples, initial EMG samples, and initial hand position samples during the training process according to an embodiment of the present invention is shown.

[0182] like Figure 8 As shown, Figure 8 This paper illustrates the process of using a first, second, and third time window to perform sliding window segmentation on the processed training EEG signals, processed training EMG signals, and initial hand position samples, followed by dimensional transformation to obtain target training EEG and target training EMG data. The acquired motion signal samples are used as a reference to determine the active segment signals of the EEG and EMG signal samples. A first time window with a length of 1000 ms is used to perform sliding window segmentation on the processed EEG signals, resulting in initial training EEG data of size 9×500. A second time window with a length of 200 ms is used to perform sliding window segmentation on the processed training EMG signals, resulting in target training EMG data of size 5×100. Then, a Fast Fourier Transform is performed on the initial training EEG data to extract the initial training EEG amplitude and phase of size 9×500. Finally, based on the size of the target training EMG data, a dimensional transformation is performed to obtain the target training EEG amplitude and phase of size 45×100.

[0183] Figure 9 A schematic diagram is shown of the target training EEG data, target training EMG data, and target hand position data samples obtained during the training process according to an embodiment of the present invention.

[0184] like Figure 9As shown, the target training EEG data after the first time window sliding window segmentation has a total length of 1000ms; the target training EMG data after the second time window sliding window segmentation has a total length of 200ms; and the target hand position data sample after the third time window sliding window segmentation. The target training EEG data can include 5 segments, corresponding to the brain's process of receiving instructions, parsing the instructions, generating signals to control muscle movement, and delivering them to the muscles. Simultaneously, the target muscle only begins to move and generate the corresponding EMG signal after receiving the corresponding EEG control signal, thus creating an inherent physiological time difference between the EMG and EEG signals. The hand position change only occurs when the muscle responds to the control of the EMG signal. Therefore, standardized and uniform segmentation and extraction of EEG signal samples, EMG signal samples, and initial hand position samples can be achieved through first, second, and third time windows of different lengths.

[0185] According to embodiments of the present invention, the optimization function may include a multi-objective optimization function and an effect evaluation function.

[0186] According to an embodiment of the present invention, a method for obtaining a target optimization loss value based on an optimization function and training hand position and target hand position data samples may include the following operations.

[0187] According to an embodiment of the present invention, a first optimized loss value is obtained based on a multi-objective optimization function and the mean square error between training hand position and target hand position data samples.

[0188] The first optimized loss value is calculated based on the mean square error between multiple sets of training hand position data and target hand position data samples obtained after inputting multiple training samples into the model.

[0189] The first optimization loss value can be calculated as shown in formula (4).

[0190] (4);

[0191] in, This can be represented as the first optimization loss value. It can be characterized as the mean square error between the x-axis coordinates of multiple training hand positions and the x-axis coordinates of the target hand position data samples. It can be characterized as the mean square error between the y-axis coordinates of multiple training hand positions and the y-axis coordinates of the target hand position data samples. It can be characterized as the mean square error between the z-axis coordinates of multiple training hand positions and the z-axis coordinates of the target hand position data samples.

[0192] According to an embodiment of the present invention, the model parameters of the hand position prediction model to be trained are adjusted based on the first optimized loss value to obtain an intermediate prediction model.

[0193] According to an embodiment of the present invention, an intermediate prediction model is used to perform cross-level multidimensional feature extraction, fusion, and feature mapping on multiple target training EEG data and multiple target training EMG data after preprocessing and sliding window segmentation, so as to obtain multiple optimized hand positions.

[0194] For example, the Adam optimizer can be used to optimize the training of the network, with the number of samples per batch set to 512 and the learning rate of the neural network set to [missing information]. The learning process proceeds sequentially for 8, 2, 3, and 3 rounds at each learning rate.

[0195] The intermediate prediction model obtained through training is used to perform prediction training on hand positions on multiple target training EEG data and multiple target training EMG data, as described above, to obtain multiple optimized hand positions.

[0196] According to an embodiment of the present invention, based on the effect evaluation function, a second optimized loss value is obtained according to the root mean square error between multiple optimized hand positions and multiple target hand position data samples after sliding window segmentation.

[0197] After obtaining multiple optimized hand positions, a secondary evaluation can be performed using an effectiveness evaluation function that includes the Pearson correlation coefficient and root mean square error.

[0198] The Pearson correlation coefficient can be expressed as shown in formula (5), and the root mean square error can be expressed as shown in formula (6).

[0199] (5);

[0200] Where PCC can be represented as the Pearson correlation coefficient, n can be represented as the total number of n optimized hand positions, and i can be represented as the i-th optimized hand position. This can be represented as the x-axis coordinate of the i-th optimized hand position. It can be represented as the average of the x-axis coordinates of multiple optimized hand positions. This can be represented as the x-axis coordinate of the i-th target hand position data sample. It can be represented as the average of the x-axis coordinates of multiple target hand position data samples.

[0201] (6);

[0202] RMSE can be represented as root mean square error.

[0203] According to an embodiment of the present invention, the model parameters of the intermediate prediction model are adjusted based on the second optimized loss value to obtain a trained hand position prediction model.

[0204] According to an embodiment of the present invention, multiple rounds of optimization using multiple optimization functions are used to comprehensively train and optimize the hand position prediction model to be trained, thereby obtaining an accurate, efficient, robust, and reliable hand position prediction model that can predict hand position on collected data in various application environments.

[0205] Furthermore, as can be seen from Tables 1 and 2 below, the comparison between the Pearson correlation coefficient and root mean square error of the triaxial position obtained using the method of the present invention is shown under different input data conditions.

[0206] Table 1

[0207]

[0208] In this context, the EEG frequency domain can be represented as the target EEG data, the sEMG time domain can be represented as the target EMG data, and the EEG time domain can be represented as the EEG data to be input in the time domain. As shown in Table 1, when the input data consists of frequency-domain EEG data and time-domain EMG data, the Pearson correlation coefficients estimated by the method of this invention on the x, y, and z axes are greater than those estimated by the method of this invention when the input data consists of both time-domain EEG and EMG data. In other words, the accuracy of the predicted location obtained by using frequency-domain EEG data and time-domain EMG data as input data is higher than the accuracy of the predicted location obtained by using only time-domain EEG and EMG data as input data.

[0209] Table 2

[0210]

[0211] As can be seen from Table 2, when the input data is frequency-domain EEG data and time-domain EMG data, the root mean square error (RMSE) on the x, y, and z axes predicted by the method of this invention is smaller than the RMSE on the x, y, and z axes predicted by the method of this invention when the input data is both time-domain EEG data and EMG data. In other words, the accuracy of the predicted location obtained by using frequency-domain EEG data and time-domain EMG data as input data is higher than the accuracy of the predicted location obtained by using time-domain EEG data and EMG data as input data.

[0212] Furthermore, as can be seen from Table 3 below, there is a comparison between the Pearson correlation coefficient and root mean square error obtained using the method of this invention and the methods of existing technologies (EarlyFusion, multimodal learning; EMG_Regression; EEGNet, deep neural network architecture designed for electroencephalogram signal processing; EEG_Regression).

[0213] Table 3

[0214]

[0215] Among them, when the input data are frequency domain EEG data and time domain EMG data, the Pearson correlation coefficient obtained by the method of the present invention is the largest and the root mean square error is the smallest. That is, compared with the prior art, the present invention is more accurate in predicting the gesture position.

[0216] Figure 10 A structural block diagram of a hand position prediction device based on asynchronous time-frequency fusion of brain electromyography according to an embodiment of the present invention is shown.

[0217] like Figure 10 As shown, the hand position prediction device based on asynchronous time-frequency fusion of brain-myoelectricity in this embodiment includes: a segmentation module 1010, a transformation module 1020, an extraction and fusion module 1030, and a mapping module 1040.

[0218] The segmentation module 1010 is used to perform sliding window segmentation on the preprocessed EEG signal of the target object using a first time window to obtain initial EEG data, and to perform sliding window segmentation on the preprocessed EMG signal of the target object using a second time window to obtain target EMG data. The sliding window length of the first time window is greater than the sliding window length of the second time window. The segmentation module 1010 can be used to perform the operation S210 described above, which will not be repeated here.

[0219] The transformation module 1020 is used to perform a dimensionality transformation on the initial EEG amplitude and initial EEG phase extracted from the initial EEG data based on the number of channels and length of the target EEG data, to obtain the target EEG data. The target EEG data includes the target EEG phase and target EEG amplitude after dimensionality transformation. The transformation module 1020 can be used to perform the operation S220 described above, which will not be repeated here.

[0220] The extraction and fusion module 1030 is used to perform cross-level multi-dimensional feature extraction and fusion processing on the target EEG data and the target EMG data using the feature fusion module within the hand position prediction model to obtain the target fused features. The extraction and fusion module 1030 can be used to perform the operation S230 described above, which will not be repeated here.

[0221] The mapping module 1040 is used to perform feature mapping processing on the target fusion features based on the hand's three-dimensional coordinate system using the regressor within the hand position prediction model to obtain the target hand position information of the target object. The mapping module 1040 can be used to execute the operation S240 described above, which will not be repeated here.

[0222] According to embodiments of the present invention, any plurality of modules among the segmentation module 1010, transformation module 1020, extraction and fusion module 1030, and mapping module 1040 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the segmentation module 1010, transformation module 1020, extraction and fusion module 1030, and mapping module 1040 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the segmentation module 1010, transformation module 1020, extraction and fusion module 1030, and mapping module 1040 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0223] Figure 11 A block diagram of an electronic device based on a brain-myoelectric asynchronous time-frequency fusion method for hand position prediction according to an embodiment of the present invention is shown.

[0224] like Figure 11 As shown, an electronic device according to an embodiment of the present invention includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0225] RAM 1103 stores various programs and data required for the operation of the electronic device. Processor 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Processor 1101 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1102 and / or RAM 1103. It should be noted that the programs may also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0226] According to embodiments of the present invention, the electronic device may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device may also include one or more of the following components connected to the I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0227] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0228] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 1102 and / or RAM 1103 and / or one or more memories other than ROM 1102 and RAM 1103 described above.

[0229] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the hand position prediction method based on asynchronous time-frequency fusion of electroencephalography (EEG) provided in the embodiments of the present invention.

[0230] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0231] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1109, and / or installed from the removable medium 1111. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0232] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by processor 1101, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0233] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A hand position estimation method based on brain-muscle asynchronous time-frequency fusion, characterized in that, include: Initial EEG data is obtained by sliding window segmentation of the preprocessed EEG signal of the target object using a first time window, and target EMG data is obtained by sliding window segmentation of the preprocessed EMG signal of the target object using a second time window, wherein the sliding window length of the first time window is greater than the sliding window length of the second time window. The initial EEG data is subjected to Fast Fourier Time-Frequency Processing to obtain initial EEG amplitude and initial EEG phase. Based on the number of channels and length of the target EMG data, the initial EEG amplitude and initial EEG phase are subjected to dimensional transformation to obtain target EEG data. The target EEG data includes target EEG phase and target EEG amplitude of the same length after dimensional transformation. The target EEG and EMG data are subjected to dimensionality-upgrading and multi-level feature extraction and fusion processing using the EEG / EMG feature extraction submodule within the feature fusion module of the hand position prediction model, resulting in multi-level EEG features, multi-level EMG features, and initial fused features. The cross-level fusion subnetwork within the feature fusion module uses the multi-level EEG and EMG features to perform cross-level feature fusion processing on the initial fused features, resulting in a multi-level total fused feature. Finally, the fully connected network within the feature fusion module performs feature semantic integration processing on the multi-level total fused feature to obtain the target fused feature. Based on the three-dimensional coordinate system of the hand, the regressor in the hand position prediction model is used to perform feature mapping processing on the target fusion features to obtain the target hand position information of the target object.

2. The hand position estimation method according to claim 1, characterized by, The process of obtaining initial EEG data by sliding window segmentation of the preprocessed EEG signal of the target object using a first time window, and obtaining target EMG data by sliding window segmentation of the preprocessed EMG signal of the target object using a second time window, includes: The initial EEG and initial EMG signals were filtered to obtain filtered EEG and filtered EMG signals, respectively. The filtered EEG signal and the filtered EMG signal are normalized respectively to obtain the processed EEG signal and the processed EMG signal. The processed EEG signal is segmented using a sliding window using the first time window to obtain the initial EEG data, and the processed EMG signal is segmented using a sliding window using the second time window to obtain the target EMG data.

3. The hand position estimation method according to claim 1, characterized by, The lengths of the initial EEG amplitude and the initial EEG phase are the same as the length of the initial EEG data.

4. The hand position estimation method according to claim 1, characterized by, The target fusion feature is characterized as a multi-dimensional feature corresponding to the hand position, and the multi-level total fusion feature is characterized as a multi-dimensional feature corresponding to EEG and EMG.

5. The hand position estimation method according to claim 1, characterized in that, The brain electromyography feature extraction submodule includes a single-modality brain electronics network, a single-modality muscle electronics network, and a brain electromyography fusion subnetwork; The brain-motor feature extraction submodule utilizes the target EEG amplitude, target EEG phase, and target EMG data to perform dimensionality-upgrading and multi-level feature extraction and fusion processing, resulting in multi-level EEG features, multi-level EMG features, and initial fused features, including: The target EEG amplitude and target EEG phase are respectively up-dimension processed using the single-modality brain-electronic network to obtain up-dimension EEG amplitude and up-dimension EEG phase; The amplitude and phase of the upgraded EEG are added together to obtain the upgraded EEG features; The upgraded EEG features are subjected to multi-level feature processing to obtain the multi-level EEG features; The target electromyographic data is subjected to dimensionality upscaling and multi-level feature processing using the single-modal electromyographic network to obtain multi-level electromyographic features; The target EEG amplitude, the target EEG phase, and the target EMG data are fused using the EEG fusion subnetwork to obtain the initial fusion features.

6. The hand position estimation method according to claim 1, wherein The cross-level fusion subnetwork utilizes the multi-level EEG features and the multi-level EMG features to perform cross-level feature fusion processing on the initial fusion features, obtaining multi-level total fusion features, including: In the case of a multi-level system with an L-level hierarchy The initial fusion features are subjected to cross-level feature fusion processing using the first level of EEG features in the L-level EEG features and the first level of EMG features in the L-level EMG features to obtain the first level of fusion features, where L≥3 and l≥1. The fusion features of the l-th level are obtained by using the EEG features of the (l+1)th level in the L-level EEG features and the EMG features of the (l+1)th level in the L-level EMG features to perform cross-level feature fusion processing on the fusion features of the l-th level. The L-level fused features obtained after cross-level feature fusion processing are spliced ​​together to obtain the total L-level fused features.

7. The hand position estimation method according to claim 1, wherein The fully connected network includes a first fully connected layer and a second fully connected layer; The step of using the fully connected network to perform feature semantic integration processing on the multi-level total fusion features to obtain the target fusion features includes: The first fully connected layer is used to integrate the multi-level total fusion features to obtain intermediate fusion features; The intermediate fusion features are processed by feature mapping using the second fully connected layer to obtain the target fusion features. The intermediate fusion features are characterized as multidimensional fusion features corresponding to EEG and EMG, and the target fusion features are characterized as multidimensional fusion features corresponding to hand position.

8. The hand position estimation method according to claim 1, characterized by, The hand position prediction model was trained using the following methods: Obtain the hand position prediction model to be trained and the training sample dataset, wherein the training sample dataset includes multiple initial EEG samples, multiple initial EMG samples, and multiple initial hand position samples; The initial EEG and initial EMG samples are preprocessed, and the preprocessed training EEG signals, training EMG signals, and initial hand position samples are segmented by sliding windows using the first time window, the second time window, and the third time window to obtain initial training EEG data, target training EMG data, and target hand position data samples. The step size of the first time window, the second time window, and the third time window is the same. Based on the target training electromyography data, the initial training electroencephalogram (EEG) data is subjected to dimensionality transformation to obtain the target training EEG data, wherein the target training EEG data and the target training electromyography data have the same length. The target training EEG data and the target training EMG data are input into the hand position prediction model to be trained. The target training EEG data and the target training EMG data are then subjected to cross-level multi-dimensional feature extraction and fusion processing using the feature fusion module to be trained, to obtain training fusion features. Based on the hand's three-dimensional coordinate system, the training fusion features are processed by the regressor to be trained to obtain the training hand position. Based on the optimization function, the target optimization loss value is obtained according to the training hand position and the target hand position data samples; Based on the target optimization loss value, the model parameters of the hand position prediction model to be trained are adjusted to obtain the trained hand position prediction model.

9. The hand position estimation method according to claim 8, wherein The initial EEG and initial EMG samples are preprocessed, and the preprocessed training EEG signals, training EMG signals, and initial hand position samples are segmented using the first, second, and third time windows to obtain initial training EEG data, target training EMG data, and target hand position data samples, including: The initial EEG and initial EMG samples were filtered to obtain the training filtered EEG and training filtered EMG signals, respectively. Based on the reference EEG signal extracted from the training filtered EEG signal, the active EEG signal extracted from the training filtered EEG signal is subjected to rereference processing to obtain the calibration training EEG signal. The calibration training EEG signal and the training filtered EMG signal are normalized respectively to obtain the processed training EEG signal and the processed training EMG signal. The first time window is used to perform sliding window segmentation on the processed training EEG signal to obtain the initial training EEG data. The second time window is used to perform sliding window segmentation on the processed training EMG signal to obtain the target training EMG signal. The third time window is used to perform sliding window segmentation on the initial hand position sample to obtain the target hand position data sample.

10. The hand position estimation method according to claim 8, wherein The optimization function includes a multi-objective optimization function and an effect evaluation function; The step of obtaining the target optimization loss value based on the optimization function, according to the training hand position and the target hand position data samples, includes: Based on the multi-objective optimization function, a first optimized loss value is obtained according to the mean square error between the training hand position and the target hand position data samples; Based on the first optimized loss value, the model parameters of the hand position prediction model to be trained are adjusted to obtain an intermediate prediction model; The intermediate prediction model is used to perform cross-level multidimensional feature extraction, fusion and feature mapping on multiple target training EEG data and multiple target training EMG data after preprocessing and sliding window segmentation to obtain multiple optimized hand positions. Based on the effect evaluation function, a second optimized loss value is obtained according to the root mean square error between the multiple optimized hand positions and the multiple target hand position data samples after sliding window segmentation. Based on the second optimized loss value, the model parameters of the intermediate prediction model are adjusted to obtain the trained hand position prediction model.

11. A hand position estimation device based on brain-muscle electrical asynchronous time-frequency, characterized in that, include: The segmentation module is used to perform sliding window segmentation on the preprocessed EEG signal of the target object using a first time window to obtain initial EEG data, and to perform sliding window segmentation on the preprocessed EMG signal of the target object using a second time window to obtain target EMG data, wherein the sliding window length of the first time window is greater than the sliding window length of the second time window; The transformation module is used to perform fast Fourier time-frequency processing on the initial EEG data to obtain initial EEG amplitude and initial EEG phase. Based on the number of channels and length of the target EMG data, the initial EEG amplitude and initial EEG phase are subjected to dimensional transformation to obtain target EEG data. The target EEG data includes target EEG phase and target EEG amplitude of the same length after dimensional transformation. The extraction and fusion module is used to perform dimensionality-upgrading and multi-level feature extraction and fusion processing on the target EEG data and the target EMG data using the EMG feature extraction submodule within the feature fusion module of the hand position prediction model, to obtain multi-level EEG features, multi-level EMG features, and initial fusion features; the cross-level fusion subnetwork within the feature fusion module uses the multi-level EEG features and multi-level EMG features to perform cross-level feature fusion processing on the initial fusion features, to obtain multi-level total fusion features; and the fully connected network within the feature fusion module performs feature semantic integration processing on the multi-level total fusion features to obtain the target fusion features; The mapping module is used to perform feature mapping processing on the target fusion features based on the hand three-dimensional coordinate system and using the regressor in the hand position prediction model to obtain the target hand position information of the target object.

Images