Prosthetic hand control mental load measuring method and system based on multi-source information fusion

Abstract

The invention provides a prosthetic hand control mental load measuring method and system based on multi-source information fusion, which can synthesize pupil change characteristics, electroencephalogram characteristics and functional near infrared spectrum multi-source information to construct a space-time convolutional neural network to extract high-identification characteristics. An attention mechanism multi-source information fusion model is established to fuse pupil, electroencephalogram and near infrared spectrum features, and the fusion model is input into a full-connection neural network, so that the quantification of the control mental load of the prosthetic hand of the amputee is realized, and an objective method is provided for measuring the usability of the prosthetic hand taking the mental load of the user as an evaluation index. The problem that current artificial hand usability evaluation depends on a subjective method and cannot be quantitatively and objectively evaluated is solved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent prosthetic hand control, and more specifically to a method, system and electronic device for measuring the mental workload of prosthetic hand manipulation based on multi-source information fusion. Background Art

[0002] Currently, patients with limb disabilities face great rehabilitation needs. Installing bionic prostheses is the most effective way to help amputees recover their motor functions, which is of great significance for improving the happiness of disabled people and maintaining social stability. The latest bionic prosthetic hands can restore perceptions such as touch, pain, and even heat, and use advanced algorithms to achieve highly intuitive control, thus approaching seamless integration between humans and machines; although these advances have generally improved users' performance in common control indicators such as prosthetic hand grasping completion rate and grasping time, the abandonment rate of prosthetic hands is still as high as 50%. How to improve the ease of use of prosthetic hands and reduce the abandonment rate has become a major problem that needs to be solved in upper limb prosthetic research.

[0003] Of course, appearance, function, and comfort are important factors that affect the abandonment rate of prosthetic hands. With technological innovation, the appearance of prosthetic hands is no different from that of human hands, their functions are gradually becoming more complete, and their comfort is constantly improving, but the abandonment rate of prosthetic hands is still high. After limb loss, since the pathway for transmitting proprioceptive information from the environment to the brain is cut off, patients must use a lot of additional cognitive compensation (vision, concentration, etc.) to complete the same action, making the originally intuitive action uncertain, greatly increasing the mental load of prosthetic hand control, resulting in the patient's efforts to control the prosthetic hand to achieve fine grasping, which discourages users and causes them to give up using prosthetic hands. However, current research on prosthetic hands only focuses on task execution, usually using an artificial experimental paradigm, in which users need to concentrate on completing tasks, which is often unrealistic in daily life, resulting in the product being unable to survive.

[0004] Improvements to prosthetic systems, whether through more intuitive control paradigms or perceptual feedback, are often quantified through performance in tasks that represent typical daily activities, with less consideration of user effort measures; while these metrics are useful, they do not reflect the user's ultimate preferences. Therefore, in order to reduce the abandonment rate of prosthetic hands, it is necessary to incorporate user effort measures of mental workload into the evaluation indicators of prosthetic hand control performance, fully demonstrate the ideal characteristics of prosthetic hands, effectively evaluate the ease of use of prosthetic hands, and provide targeted guidance for the improvement of prosthetic hands. However, the mental workload of prosthetic hand manipulation after amputation is regulated by brain plasticity, as well as additional influences such as phantom limb pain and weakened proprioception, and becomes extremely complex, including multiple factors such as mental concentration, emotional state, brain plasticity, motor sensation, and psychophysiological resonance. Existing mental workload measurement methods are no longer applicable.

[0005] Therefore, there is an urgent need for a patient-oriented, dedicated prosthetic hand manipulation mental workload measurement method that can objectively quantify the mental workload of patients while using prostheses. Summary of the invention

[0006] The present application provides a method and system for measuring the mental workload of prosthetic hand manipulation based on multi-source information fusion, which can objectively quantify the mental workload of patients during the use of prostheses and establish a prosthetic hand usability measurement method centered on the user's mental workload.

[0007] In the first aspect, the present application provides a method for measuring the mental workload of prosthetic hand manipulation based on multi-source information fusion, the method comprising: performing data preprocessing based on collected signal data, the signal data comprising EEG signals, fNIRS signals and pupil signal data; extracting preset features based on the signal data and performing multimodal fusion of the preset features to generate multimodal features; constructing a load measurement model and inputting multimodal features into the load measurement model; and outputting the probability of identifying the mental workload of prosthetic hand manipulation.

[0008] In an optional scheme of the first aspect, when collecting signal data, the method includes: collecting EEG signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload; collecting fNIRS signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload; collecting pupil signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload.

[0009] In an optional scheme of the first aspect, after collecting EEG signal data, the method includes: downsampling the collected EEG signal data to a first preset sampling rate; performing bandpass filtering processing of the EEG signal data at a preset filtering frequency; performing re-reference processing on the EEG signal data; and performing independent component analysis on the EEG signal data.

[0010] In an optional solution of the first aspect, after collecting fNIRS signal data, the method includes: downsampling the collected fNIRS signal data to a second preset sampling rate; and converting the collected fNIRS signal data into concentration changes of HbR and HbO by using a modified Beer-Lambert law, wherein the modified Beer-Lambert law is:

[0011] ,in, is the distance between the light source and the detector, For the wavelength The extinction coefficient of oxygenated hemoglobin is For the wavelength The extinction coefficient of deoxyhemoglobin is For the wavelength The extinction coefficient of oxygenated hemoglobin at For the wavelength The extinction coefficient of deoxyhemoglobin is For passing Wavelength after time The change in optical density at For passing Wavelength after time The change in optical density at The wavelength The differential path length factor, The wavelength The differential path length factor, and are the relative changes in the concentrations of oxyhemoglobin and deoxyhemoglobin, is the time interval between two adjacent samplings; a fourth-order IIR bandpass filter is applied to the converted HbR and HbO signals to extract the preset frequency signal and remove the breathing, blood pressure and heartbeat noise.

[0012] In an optional solution of the first aspect, after collecting pupil signal data, the method includes: using cubic spline interpolation to fill missing values ​​of the pupil data; and smoothing and filtering the pupil data by five-point cubic smoothing.

[0013] In an optional scheme of the first aspect, after the EEG signal data is processed, the method includes: performing time convolution along the time dimension of the EEG signal data to extract time information; performing convolution along the spatial dimension of the EEG signal data to extract spatial information; reducing the time dimension of the EEG feature to generate an EEG feature of a first preset dimension.

[0014] In an optional scheme of the first aspect, after the fNIRS signal data is processed, the method includes: using a first preset window size and a first preset step size to capture the rough time trend and local fine time features of the fNIRS signal data; extracting the spatial pattern of the multi-channel fNIRS signal data to generate fNIRS features of a second preset dimension.

[0015] In an optional scheme of the first aspect, after the pupil signal data is processed, the method includes: using a second preset window size and a second preset step size to capture the rough time trend and local fine time characteristics of the pupil signal data; performing time convolution along the time dimension of the input pupil signal data to shorten the time dimension of the pupil signal data, and generating pupil characteristics of a third preset dimension.

[0016] In an optional solution of the first aspect, after generating EEG, fNIRS and pupil features, the method includes:

[0017] The query, key and value vectors of EEG, fNIRS and pupil features are obtained through linear mapping. The attention value between the vectors is obtained by calculating the correlation between the query and key vectors of different features. The attention value and the value vectors of EEG, fNIRS and pupil features are weighted and concatenated to generate a multimodal feature that integrates EEG, fNIRS and pupil features.

[0018] In an optional solution of the first aspect, after obtaining the query, key and value vectors, the method includes: the calculation formulas of the query, key and value vectors of EEG, fNIRS and pupil features are: ,in is the feature vector, 3200 is the number of feature vectors of multimodal physiological signals, t is the feature dimension, is the parameter matrix of dimension transformation, is the transformed dimension, where ; Use each feature and The vector dot product of is used to calculate the attention value between features. The calculation formula is: ,in is the attention value of feature i to feature j. The softmax function plays a normalization role, mapping the dot product value to the interval (0,1) to form a weight;

[0019] The formula for calculating the weighted sum of the attention value and the value vectors of EEG, fNIRS and pupil features is:

[0020] ,in is the weighted and fusion result of feature i.

[0021] In an optional scheme of the first aspect, after splicing and generating a multimodal feature that fuses EEG, fNIRS and pupil features, the method further includes: constructing a multilayer perceptron classifier including a Dropout layer, a fully connected layer and a LogSoftMax layer, wherein the Probability of the Dropout layer is set to a preset value.

[0022] In an optional solution of the first aspect, the output of the multilayer perceptron classifier is:

[0023] , where y is the probability of identifying the mental workload of prosthetic hand manipulation output by the multi-layer perceptron classifier, m is the multimodal feature of EEG, fNIRS and pupil features, W is the weight matrix, and b is the bias.

[0024] In the second aspect, the present application provides a prosthetic hand manipulation mental workload measurement system based on multi-source information fusion for executing the method, including: a data acquisition module, used to collect EEG signal data, fNIRS signal data and pupil signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload, and then perform data preprocessing based on the collected signal data; a feature extraction module, used to process the EEG signal data, fNIRS signal data and pupil signal data and extract EEG features, fNIRS features and pupil features; a feature fusion module, used to perform multimodal fusion of the EEG features, fNIRS features and pupil features to generate multimodal features; a model processing module, used to construct a load measurement and input multimodal features into the load measurement, and then output the probability of identifying the mental workload of prosthetic hand manipulation.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate one or more embodiments of the present application and, together with the description, serve to explain the principles of the present application and to enable one of ordinary skill in the relevant art to make and use the present application.

[0027] Figure 1 This is a flow chart of an exemplary method for measuring the mental workload of prosthetic hand manipulation based on multi-source information fusion according to some embodiments of the present application.

[0028] Figure 2 It is a schematic diagram of an exemplary prosthetic hand manipulation mental workload measurement model based on multi-source information fusion according to some embodiments of the present application.

[0029] Figure 3 This is a schematic diagram of an exemplary EEG feature extraction process according to some embodiments of the present application.

[0030] Figure 4 is a schematic diagram of an exemplary fNIRS feature extraction process according to some embodiments of the present application.

[0031] Figure 5 This is a schematic diagram of an exemplary pupil feature extraction process according to some embodiments of the present application.

[0032] Figure 6 This is a schematic diagram of an exemplary self-attention mechanism feature fusion process according to some embodiments of the present application.

[0033] Figure 7 This is a flowchart of an exemplary self-attention mechanism feature fusion method according to some embodiments of the present application.

[0034] Figure 8 This is a schematic diagram of module connections of an exemplary prosthetic hand manipulation mental workload measurement system based on multi-source information fusion according to some embodiments of the present application.

[0035] Fig. 9 It is a schematic diagram of the structure of an exemplary electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present application will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application.

[0037] In response to the problems raised by the background technology, the present application provides a robust method for measuring the mental workload of prosthetic hand manipulation, objectively quantifies the mental workload of patients during the use of prostheses, establishes a prosthetic hand usability measurement method centered on the user's mental workload, and further improves the design and usage experience of prostheses.

[0038] Therefore, in this application, the mental workload of prosthetic hand manipulation is defined as the sum of the mental resources required to successfully manipulate the prosthetic hand to complete the grasping task. Specifically, the higher the mental workload, the stronger the attention and concentration required for the patient to complete the task accurately and effectively, that is, the worse the ease of use of the prosthetic hand.

[0039] In order to accurately measure the mental workload during the manipulation of a prosthetic hand, this application proposes to use pupil size changes as a representation of visual attention. The pupil changes during mental activities such as thinking, memory recall, and concentration. When performing complex cognitive tasks, the pupil usually dilates and remains for a long time, and returns to the baseline state after the task is completed. Therefore, pupil size changes, as an objective indicator of mental workload, are not only highly reliable, but also can reflect in real time the cognitive resources invested by the user in the process of manipulating the prosthesis. However, although pupil size changes can reflect mental workload, they are also affected by external factors such as light changes, anxiety, and stress, and there are certain interference factors; in order to overcome this problem, this application also proposes to combine other physiological signals to further improve the accuracy and stability of mental workload measurement.

[0040] Specifically, electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can serve as important aids, namely:

[0041] Electroencephalogram (EEG) can accurately capture the user's brain activation state when operating a prosthesis, reflecting the changes in mental workload. Since EEG has extremely high temporal resolution, it can monitor the patient's cognitive load fluctuations in real time, thereby providing efficient feedback for changes in mental workload during prosthetic manipulation. However, the spatial resolution of EEG is low, and it is difficult to provide detailed information on the activities of different brain areas.

[0042] Functional near-infrared spectroscopy (fNIRS) indirectly reflects the activity state of the brain by measuring changes in hemodynamics and oxygen saturation in brain tissue. fNIRS has good spatial resolution and can provide activation conditions of different brain regions. Especially under complex tasks, it can more accurately reflect the load of local brain regions. However, due to its relatively low temporal resolution, it is difficult to capture rapidly changing mental workload.

[0043] Therefore, in order to overcome the limitations of each single measurement method, this application proposes a comprehensive mental workload measurement scheme, which can achieve a more robust and comprehensive assessment of the mental workload of prosthetic hand manipulation by fusing the complementary features of pupil size changes, EEG and fNIRS. This method can fully utilize the advantages of different technologies in terms of temporal resolution, spatial resolution and interference resistance to ensure the accuracy and reliability of the measurement results.

[0044] refer to Figure 1 and Figure 2 As shown, Figure 1 A flowchart of an exemplary method for measuring mental workload of prosthetic hand manipulation based on multi-source information fusion in some embodiments of the present application is shown; Figure 2 A schematic diagram of an exemplary prosthetic hand manipulation mental workload measurement model based on multi-source information fusion in some embodiments of the present application is shown. Thus, the present application provides a prosthetic hand manipulation mental workload measurement method based on multi-source information fusion, the method comprising:

[0045] S1. Patients wore prosthetic hands and performed a series of prosthetic hand grasping tasks with different levels of mental workload. EEG signals, fNIRS signals and pupil signal data were collected synchronously, and data preprocessing was then performed based on the collected signal data.

[0046] Specifically, in the present application, an exemplary reference is made to using existing EEG signal acquisition equipment to collect EEG signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload, using existing fNIRS signal acquisition equipment to collect fNIRS signals of patients performing prosthetic hand grasping tasks under different levels of mental workload, and using existing eye tracking equipment to record pupil signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload, wherein the pupil signal data includes at least pupil signals in vertical and horizontal directions.

[0047] S2. Extracting preset features according to the signal data and performing multimodal fusion on the preset features to generate multimodal features.

[0048] In one example of the present application, after collecting EEG signal data, the present application exemplarily uses the EEG signal acquisition device of the first preset channel to collect EEG signal data of the patient performing the prosthetic hand grasping task under different levels of mental workload, and the sampling rate is set to the first preset sampling rate to ensure that the details of the changes in the EEG signal are accurately obtained at a higher time resolution. Then the collected EEG signal data is downsampled to the first adjusted sampling rate, and then the downsampled EEG signal data is subjected to bandpass filtering processing, re-reference processing and independent component (ICA) analysis at a preset filtering frequency, specifically:

[0049] Filtering is mainly used to remove environmental noise and power frequency interference. Specifically, EEG signal data is usually subject to various interferences, such as environmental noise, power frequency interference, etc. In order to ensure the quality of the signal, this application uses bandpass filtering to process the EEG signal.

[0050] The quality of EEG signal data may be affected by the reference electrode setting. In order to reduce the offset of the reference electrode to the signal, the present application adopts the re-reference technology, and exemplarily adjusts the reference mode of all electrodes to the average reference. Through this method, the common mode signal introduced by the reference electrode can be removed, so that the signal of each channel can more accurately reflect brain activity.

[0051] The non-EEG components in EEG signal data (such as electromyography, electrooculography, etc.) often interfere with signal analysis. In order to eliminate these artifacts, this application uses an independent component analysis method to process the signal. Independent component analysis can separate signal components from different sources, thereby effectively removing non-EEG signals such as electromyography artifacts (such as eye movement artifacts and facial muscle activities), ensuring that the analyzed signal mainly comes from brain activity.

[0052] Specifically, the EEG signal acquisition device, the first preset channel, the first preset sampling rate, the first adjusted sampling rate and the preset filtering frequency are set by the operating personnel according to actual needs and costs. In this example, the first preset channel is set to 32 channels for reference. The 32-channel EEG signal acquisition device can cover multiple areas of the brain and provide rich spatial information for subsequent analysis; the first preset sampling rate is set to 1000Hz for reference, and the sampling rate of the original EEG signal data is 1000Hz, but in order to reduce the complexity of data processing and take into account the signal quality, all collected EEG signals are first downsampled. The first adjusted sampling rate of the downsampling is set to 200Hz for reference. After downsampling, the time resolution of the signal is reduced to 200Hz, but it can still meet the analysis requirements of changes in mental workload and improve the efficiency of subsequent calculations; the preset filtering frequency is set to 0.1~50Hz for reference. This frequency band covers the main frequency bands of most EEG activities and can effectively filter out low-frequency drift and high-frequency noise.

[0053] Therefore, for each experiment, the participating patients performed the prosthetic hand grasping task under different levels of mental workload. At the beginning of the experiment, 10 seconds of resting signal collection was first performed, and then the task execution phase was entered. In order to ensure the representativeness and accuracy of the data, this application uses the 0-10 seconds EEG signal data at the beginning of the experiment as a sample for further analysis. At this time, the patient's mental workload is relatively low, so it can be used as a reference signal. Then, the dimension of the EEG signal data sample is (2000, 32), where 2000 represents the time dimension and 32 represents the channel dimension.

[0054] refer to Figure 3 As shown, Figure 3 An exemplary EEG feature extraction process schematic diagram of some embodiments of the present application is shown. In some examples of the present application, after the EEG signal data is processed, the method includes:

[0055] Temporal convolution is performed along the time dimension of the EEG signal data to extract time information.

[0056] Convolution is performed along the spatial dimension of the EEG signal data to extract spatial information.

[0057] The time dimension of the EEG feature is reduced to generate an EEG feature of a first preset dimension; wherein the first preset dimension is set by the operator according to actual needs, and the exemplary reference EEG feature dimension in this application is (100, 20).

[0058] In one example of the present application, after collecting fNIRS signal data, the present application exemplarily uses the fNIRS signal acquisition device of the second preset channel to collect fNIRS signal data of the patient performing the prosthetic hand grasping task under different levels of mental workload. The fNIRS signal acquisition device can detect the concentration changes of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in the brain through a near-infrared light source, thereby reflecting the blood oxygen metabolism of the brain area, and then inferring the mental workload level. Then, the sampling rate is set to the second preset sampling rate, and then the collected EEG signal data is downsampled to the second adjusted sampling rate, and then converted into the concentration changes of HbR and HbO through the modified Beer-Lambert law. The modified Beer-Lambert law is:

[0059] ,in, is the distance between the light source and the detector, For the wavelength The extinction coefficient of oxygenated hemoglobin is For the wavelength The extinction coefficient of deoxyhemoglobin is For the wavelength The extinction coefficient of oxygenated hemoglobin at For the wavelength The extinction coefficient of deoxyhemoglobin is For passing Wavelength after time The change in optical density at For passing Wavelength after time The change in optical density at The wavelength The differential path length factor, The wavelength The differential path length factor, and are the relative changes in the concentrations of oxyhemoglobin and deoxyhemoglobin, is the time interval between two adjacent samplings; a fourth-order IIR bandpass filter is applied to the converted HbR and HbO signals to extract the preset frequency signal and remove the breathing, blood pressure and heartbeat noise.

[0060] Among them, the specific fNIRS signal acquisition device, the second preset channel, the second preset sampling rate, the second adjusted sampling rate and the preset frequency are set by the operator according to actual needs and costs. In this example, the first preset channel is set to 8 channels for reference, and the second preset sampling rate is set to 150Hz for reference to ensure that the high time resolution signal of brain activity is obtained. The existing fNIRS signal data acquisition device uses near-infrared light to penetrate the brain tissue by placing multiple light sources and detectors on the scalp surface to measure the concentration change of hemoglobin. The sampling rate of the original fNIRS signal data is 150Hz. In order to reduce the computational burden of data processing and ensure the smoothness of the signal, this application performs downsampling processing on the collected original fNIRS signal data, and sets the second adjusted sampling rate of the downsampling to 50Hz for reference. After downsampling, it can not only reduce the amount of calculation, but also effectively process the data while maintaining sufficient signal accuracy. The bandpass filter frequency band used in this application is 0.01~0.50Hz, which includes the main frequency range of the dynamic activity of cerebral blood flow, and can effectively remove the interference caused by physiological fluctuations.

[0061] Therefore, for each experiment, the initial value of the fNIRS signal data may be affected by various external factors, so the signal needs to be baseline corrected. In each experiment, the 0-10 second fNIRS signal data at the beginning of the experiment is used as a sample for further analysis. During this period, the participants are in a relatively stable experimental environment, which can more accurately reflect the level of mental workload. At the same time, the first 3 seconds of the collected signal are baseline corrected to ensure that the subsequent data analysis starts with a stable baseline. The fNIRS signal data sample dimension after baseline correction is (500, 16), where 500 is the time dimension, and 16 is the number of 8-channel HbR and 8-channel HbO channels, which record the concentration changes of oxygenated and deoxygenated hemoglobin, respectively.

[0062] When performing baseline correction, the average value of the baseline segment of 0 to 3 seconds is subtracted from the signal of each channel to eliminate the signal drift in the early stage of the experiment and ensure that the data remains consistent throughout the experiment.

[0063] refer to Figure 4 As shown, Figure 4 An exemplary embodiment of some embodiments of the present application is shown. In some examples of the present application, after fNIRS signal data processing, the method includes:

[0064] A first preset window size and a first preset step size are used to capture the rough time trend and local fine time features of the fNIRS signal data;

[0065] The spatial pattern of the multi-channel fNIRS signal data is extracted to generate an fNIRS feature of a second preset dimension; wherein the first preset window size, the first preset step size, and the second preset dimension are set by the operator according to actual needs and costs. In this example, the first preset window size is set to (5, 1) for reference, the first preset step size is set to 5 for reference, and the second preset dimension is set to (100, 8) for reference.

[0066] In an example of the present application, after collecting pupil signal data, the present application exemplarily uses an eye tracking device to record the pupil signal data of the patient performing the prosthetic hand grasping task under different levels of mental load. By analyzing the changes in the pupil signal data, the mental load of the participant during the task execution can be objectively evaluated; and then the sampling rate is set to the third preset sampling rate. Since the eye tracking process may be affected by factors such as the environment and equipment, resulting in missing values ​​in some signals, it is necessary to fill the missing pupil signal data. In order to ensure the continuity and integrity of the pupil signal data, the present application exemplarily uses the cubic spline interpolation method to fill the missing values ​​to ensure that the time series of the filled data is continuous and does not produce large fluctuations; further, in order to remove the spike data that may be caused by instantaneous noise, equipment jitter, etc., it is necessary to smooth and filter the pupil signal data. The application exemplarily uses a five-point cubic smoothing method (i.e., a cubic smoothing filter with a window size of 5 data points) for smoothing, eliminating high-frequency noise and effectively removing noise interference, and retaining the true change trend of the pupil signal data.

[0067] The eye tracking device and the third preset sampling rate are specifically set by the operator according to actual needs and costs. In this example, the third preset sampling rate is set to 100 Hz for reference to ensure sufficient time resolution to reflect changes in brain activity and to simultaneously record signal data of pupil changes, so as to analyze pupil attention changes from multiple dimensions.

[0068] Therefore, for each experiment, the initial data collected by the eye tracking device will fluctuate initially due to the transient changes in the environment or equipment, so the pupil signal data needs to be baseline corrected. In each experiment, the pupil signal data from 0 to 10 seconds at the beginning of the experiment is used as a sample for further analysis. At the same time, the first 3 seconds of the collected signal are baseline corrected to ensure that the changes in subsequent data reflect the actual changes in mental workload caused by task execution, rather than the effects caused by initial environmental changes or equipment fluctuations. The sample dimension of the pupil signal data obtained is (1000, 2), where 1000 is the time dimension and 2 is the number of pupil signal data channels in the vertical and horizontal directions, which record pupil changes in the vertical and horizontal directions respectively.

[0069] refer to Figure 5 As shown, Figure 5 An exemplary embodiment of some embodiments of the present application is shown. In some examples of the present application, after the pupil signal data is processed, the method includes:

[0070] Using a second preset window size and a second preset step size to capture the rough time trend and local fine time characteristics of the pupil signal data;

[0071] Temporal convolution is performed along the time dimension of the input pupil signal data to shorten the time dimension of the pupil signal data and generate pupil features of a third preset dimension; wherein the second preset window size, the second preset step size and the third preset dimension are set by the operator according to actual needs and costs. In this example, the second preset window size is set to (2, 1) for reference, the second preset step size is set to 2 for reference, and the third preset dimension is set to (100, 4) for reference.

[0072] S3, constructing a load measurement model and inputting multimodal features into the load measurement model;

[0073] S4. Output the probability of identifying the mental workload of prosthetic hand manipulation.

[0074] Specifically, considering that EEG, fNIRS and pupils have different modalities, if only simple feature splicing is performed, due to the differences in signal patterns of different physiological signals, the features may interfere with each other and reduce the recognition accuracy; therefore, this application uses the self-attention mechanism for reference to perform multimodal fusion of deep spatiotemporal features learned from EEG, fNIRS and pupil signal data. The self-attention mechanism can learn the correlation between feature vectors, thereby increasing the weight of key features and reducing the impact of non-key features on the results, so as to reduce the mutual interference between signals of different modalities.

[0075] refer to Figure 6 and Figure 7 As shown, Figure 6 A schematic diagram of an exemplary self-attention mechanism feature fusion process in some embodiments of the present application is shown. Figure 7 A flowchart of an exemplary self-attention mechanism feature fusion method of some embodiments of the present application is shown. In some examples of the present application, after generating EEG, fNIRS and pupil features, the method includes:

[0076] S20, obtaining the query, key and value vectors of each feature vector through linear mapping; wherein the calculation formula of the query, key and value vector of the feature vector is:

[0077] ,in is the feature vector, 3200 is the number of feature vectors of multimodal physiological signals, t is the feature dimension, is the parameter matrix of dimension transformation, is the transformed dimension, where .

[0078] S21, by calculating the correlation between the query and key vectors of different features to represent the correlation between features, the attention value between vectors is obtained; wherein, the attention value of each feature is used and The vector dot product of is used to calculate the attention value between features. The calculation formula is:

[0079] ,in is the attention value of feature i to feature j. The softmax function normalizes the feature so that the dot product value is mapped to the interval (0, 1) to form a weight.

[0080] S22, weighting and concatenating the attention value and the value vector of each feature vector to generate a multimodal feature integrating EEG, fNIRS and pupil features; wherein the calculation formula for weighting and concatenating the attention value and the value vector of EEG, fNIRS and pupil features is:

[0081] ,in is the weighted and fusion result of feature i.

[0082] Furthermore, a multilayer perceptron classifier is constructed, which includes a Dropout layer, a fully connected layer and a LogSoftMax layer, wherein the Probability of the Dropout layer is set to a preset value, and the preset value is set by the operator according to actual needs. In this application, an exemplary reference is adopted to reduce the network complexity by using 0.5; wherein, the fully connected layer realizes the classification prediction of mental workload, and the Logsoftmax layer converts the prediction result into a probability value output.

[0083] Furthermore, the output of the multi-layer perceptron classifier is: , where y is the probability of identifying the mental workload of prosthetic hand manipulation output by the multi-layer perceptron classifier, m is the multimodal feature of EEG, fNIRS and pupil features, W is the weight matrix, and b is the bias.

[0084] Therefore, through the comprehensive measurement method of this application, this application not only provides a new idea for quantifying the mental workload during prosthetic manipulation, but also provides strong support for the design optimization and user experience improvement of future prosthetic hands. Prosthetic manufacturers and designers can further adjust the operation interface, control algorithm and feedback mechanism of prosthetics based on these mental workload data, so as to develop smart prosthetic products that are easier to use, more comfortable and meet user needs.

[0085] refer to Figure 8 As shown, Figure 8 A schematic diagram of module connections of an exemplary prosthetic hand manipulation mental workload measurement system based on multi-source information fusion in some embodiments of the present application is shown.

[0086] The present application also provides a prosthetic hand manipulation mental workload measurement system based on multi-source information fusion for executing the method, comprising:

[0087] The data acquisition module 101 is used to collect EEG signal data, fNIRS signal data and pupil signal data of the patient performing the prosthetic hand grasping task under different levels of mental workload, and then perform data preprocessing according to the collected signal data;

[0088] The feature extraction module 102 is used to process the EEG signal data, the fNIRS signal data and the pupil signal data and extract EEG features, fNIRS features and pupil features.

[0089] Specifically, the feature extraction module 102 includes an EEG feature extractor, an fNIRS feature extractor, and a pupil feature extractor, wherein:

[0090] The EEG feature extractor is mainly composed of two convolutional layers and a temporal pooling layer. The first convolutional layer is a temporal convolution, which performs temporal convolution along the temporal dimension of the input EEG signal data to extract temporal information. The dimension of the temporal convolution kernel is (20, 1), and the number is 20. The dimension of the temporal convolution kernel is set to (20, 1), indicating that the length of the convolution kernel is 20 time points, and the height of the convolution kernel is 1, indicating that each convolution operation is performed along the time axis of the EEG signal. Setting 20 convolution kernels can extract features of different time scales from the input EEG signal data and capture changes in different frequencies and waveforms. The second convolutional layer performs convolution along the spatial dimension of the input EEG data. The spatial dimension represents the signals of different brain electrode channels. The spatial convolution operation can identify the association between different electrode channels and extract global patterns across channels. The kernel length of the spatial convolution kernel is the same as the number of channels of the EEG data to capture the global spatial pattern. The convolution kernel dimension is (1, 32), which means that each convolution operation is performed along the spatial dimension of the EEG signal (i.e., 32 channels). There are 20 convolution kernels, and each convolution kernel slides on the spatial dimension to extract spatial features of different channel combinations. Then, an average pooling layer is used to reduce the temporal dimension of the EEG features. The time length of the pooling layer is 199 and the step size is 18, which means that each pooling operation will consider 199 consecutive time points and move with a step size of 18, so as to average and summarize the information in each time window to form a downsampled temporal feature. In this way, the EEG feature dimension obtained is (100, 20), which can effectively compress the dimension of the original EEG signal data while retaining key information, providing sufficient information for subsequent mental workload analysis, classification tasks or other downstream processing.

[0091] The fNIRS feature extractor 1022 mainly includes a time pooling layer, a time difference layer and a spatial convolution layer. The time pooling layer is used to capture the rough time trend of the fNIRS data. By downsampling the signal in the time dimension, the time pooling layer can smooth the original signal and extract a more macroscopic time change trend. The time pooling layer is used to capture the rough time trend of the fNIRS data. The window size is (5, 1), which means that the data of 5 time points are considered each time in the time dimension; the step size is 5, which means that each pooling operation will slide forward 5 time points. The time difference layering in parallel with the time pooling layer is used to capture the local fine temporal features in the fNIRS signal. The time difference layering can extract the local changes and short-term fluctuation characteristics of the signal by calculating the differences between consecutive time points. The time difference layering captures the local fine temporal features of the fNIRS data. The window size is (5, 1), which is the same as the time pooling layer, indicating that the differences between 5 adjacent time points are calculated each time; the step size is 5, which can be used for synchronous feature extraction in the time pooling layer; the outputs of the time pooling layer and the time difference layering will be merged along the convolution channel dimension, and the merged data will contain multi-level information of time pooling and time difference. The spatial convolution layer is used to extract spatial patterns from multi-channel fNIRS signal data. Since fNIRS signal data usually has multiple channels and there is cross-channel correlation in the spatial dimension, the spatial convolution layer can learn the spatial relationship between different channels and extract global spatial features. The convolution kernel dimension is set to (1, 16), which means that each convolution operation is performed along the 16 channels of the fNIRS signal data. Here, "1" represents that the convolution operation is performed along the time dimension and will not affect the data on the time axis. The number of convolution kernels is set to 8, and then the spatial feature patterns of different channel combinations are extracted, and the fNIRS feature dimension is (100, 8), which can provide sufficient feature representation for mental workload assessment and related analysis.

[0092] The pupil feature extractor 1023 mainly includes a time pooling layer, a time difference layer and a time convolution layer. The time pooling layer is used to extract a rough time trend from the pupil data. The pooling operation retains the main change trend in the signal by reducing the time resolution of the data. Similar to fNIRS feature extraction, the window size of the time pooling layer is set to (2, 1), indicating that each pooling operation considers the data of two consecutive time points in the time dimension; the step size of the time pooling layer is set to 2, indicating that each time the pooling window slides forward by 2 time points, thereby realizing downsampling in the time dimension. The time difference layer parallel to the time pooling layer is used to capture the local fine time features in the pupil signal. The time difference layer can extract the local changes and short-term fluctuation features of the signal by calculating the differences between consecutive time points. The window size of the time difference layer is set to (2, 1), which is the same as the time pooling layer, indicating that the difference between two adjacent time points is calculated each time; the step size of the time difference layer is set to 2, which can be used for synchronous feature extraction of the time pooling layer. The temporal convolution layer performs convolution operations along the time dimension of pupil data, and further reduces redundant information along the time dimension by learning local features along the time dimension, and extracts important temporal features. The convolution kernel processes along the time dimension to extract more abstract temporal features, which can enhance the performance of important patterns in the signal and reduce unnecessary noise; thus, the pupil signal feature dimension is obtained to be (100, 4), which provides highly refined input data for the subsequent mental workload assessment task.

[0093] A feature fusion module 103 is used to perform multimodal fusion of the EEG features, fNIRS features and pupil features to generate multimodal features;

[0094] The model processing module 104 is used to construct a load measurement and input multimodal features into the load measurement, thereby outputting a probability of identifying the mental load of prosthetic hand manipulation.

[0095] In some embodiments, reference Fig. 9 As shown, Fig. 9 The electronic device is a schematic diagram of a structure of an electronic device for implementing an embodiment of the present application. The electronic device includes: a memory 201 and a processor 202. The memory 201 stores a computer program that can be run on the processor 202. When the processor 202 executes the computer program, the method in the above embodiment is implemented. The number of the memory 201 and the processor 202 can be one or more.

[0096] The electronic device also includes:

[0097] The communication interface 203 is used to communicate with external devices and perform data exchange transmission.

[0098] If the memory 201, the processor 202 and the communication interface 203 are implemented independently, the memory 201, the processor 202 and the communication interface 203 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0099] Optionally, in a specific implementation, if the memory 201, the processor 202 and the communication interface 203 are integrated on a chip, the memory 201, the processor 202 and the communication interface 203 can communicate with each other through an internal interface.

[0100] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by the processor 202.

[0101] The embodiment of the present application also provides a chip, which includes a processor 202, which is used to call and execute instructions stored in the memory 201 from the memory 201, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0102] The embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor 202 and a memory 201. The input interface, the output interface, the processor 202 and the memory 201 are connected through an internal connection path. The processor 202 is used to execute the code in the memory 201. When the code is executed, the processor 202 is used to execute the method provided in the embodiment of the application.

[0103] It should be understood that the processor 202 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor 202 may be a processor 202 that supports the Advanced RISC Machines (ARM) architecture.

[0104] Further, the above-mentioned memory 201 may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0105] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0106] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for measuring the mental workload of prosthetic hand manipulation based on multi-source information fusion, characterized in that: The method comprises: Performing data preprocessing according to the collected signal data, wherein the signal data includes one or more of EEG signal, fNIRS signal and pupil signal data; Extracting preset features according to the signal data and performing multimodal fusion on the preset features to generate multimodal features; constructing a load measurement model and inputting the multimodal features into the load measurement model; and Output the probability of identifying the mental workload of prosthetic hand manipulation.

2. The method according to claim 1, characterized in that When collecting signal data, the method includes: Collect EEG signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload; Collect fNIRS signal data when patients perform prosthetic hand grasping tasks under different levels of mental workload; Pupil signal data were collected while patients performed prosthetic hand grasping tasks under different levels of mental workload.

3. The method according to claim 2, characterized in that After collecting EEG signal data, the method includes: Downsampling the collected EEG signal data to a first preset sampling rate; The EEG signal data is subjected to bandpass filtering at a preset filtering frequency; The EEG signal data is subjected to re-reference processing; The EEG signal data was subjected to independent component analysis.

4. The method according to claim 3, characterized in that After acquiring fNIRS signal data, the method includes: Downsampling the collected fNIRS signal data to a second preset sampling rate; The concentration changes of HbR and HbO are converted to the modified Beer-Lambert law, which is: ,in, is the distance between the light source and the detector, For the wavelength The extinction coefficient of oxygenated hemoglobin is For the wavelength The extinction coefficient of deoxyhemoglobin is For the wavelength The extinction coefficient of oxygenated hemoglobin at For the wavelength The extinction coefficient of deoxyhemoglobin is For passing Wavelength after time The change in optical density at For passing Wavelength after time The change in optical density at The wavelength The differential path length factor, The wavelength The differential path length factor, and are the relative changes in the concentrations of oxyhemoglobin and deoxyhemoglobin, is the time interval between two adjacent samplings; A fourth-order IIR bandpass filter is applied to the converted HbR and HbO signals to extract the preset frequency signal and remove the breathing, blood pressure and heartbeat noise.

5. The method according to claim 4, characterized in that After collecting pupil signal data, the method includes: The missing values ​​of pupil data were filled using cubic spline interpolation method; Pupil data were smoothed by five-point cubic smoothing.

6. The method according to claim 5, characterized in that After EEG signal data processing, the method includes: Performing time convolution along the time dimension of the EEG signal data to extract time information; Performing convolution along the spatial dimension of the EEG signal data to extract spatial information; The time dimension of the EEG feature is reduced to generate an EEG feature of a first preset dimension.

7. The method according to claim 6, characterized in that After fNIRS signal data processing, the method includes: A first preset window size and a first preset step size are used to capture the rough time trend and local fine time features of the fNIRS signal data; The spatial pattern of the multi-channel fNIRS signal data is extracted to generate fNIRS features of a second preset dimension.

8. The method according to claim 7, characterized in that After the pupil signal data is processed, the method includes: Using a second preset window size and a second preset step size to capture the rough time trend and local fine time characteristics of the pupil signal data; Temporal convolution is performed along the time dimension of the input pupil signal data to shorten the time dimension of the pupil signal data and generate pupil features of a third preset dimension.

9. The method according to claim 8, characterized in that After generating EEG, fNIRS, and pupil features, the method includes: Obtain query, key, and value vectors of EEG, fNIRS, and pupil features through linear mapping; By calculating the correlation between the query and key vectors of different features, the attention value between the vectors is obtained; The attention value and the value vectors of EEG, fNIRS and pupil features are weighted and concatenated to generate a multimodal feature that integrates EEG, fNIRS and pupil features.

10. The method according to claim 9, characterized in that After obtaining the query, key and value vectors, the method includes: The calculation formulas for the query, key, and value vectors of EEG, fNIRS, and pupil features are: ,in is the feature vector, 3200 is the number of feature vectors of multimodal physiological signals, t is the feature dimension, is the parameter matrix of dimension transformation, is the transformed dimension, where ; Use each feature and The vector dot product of is used to calculate the attention value between features. The calculation formula is: ,in is the attention value of feature i to feature j. The softmax function plays a normalization role, mapping the dot product value to the interval (0,1) to form a weight; The formula for calculating the weighted sum of the attention value and the value vectors of EEG, fNIRS and pupil features is: ,in is the weighted and fusion result of feature i.

11. The method according to claim 9, characterized in that After splicing and generating a multimodal feature that fuses EEG, fNIRS, and pupil features, the method further includes: Construct a multilayer perceptron classifier consisting of a Dropout layer, a fully connected layer, and a LogSoftMax layer, where the Probability of the Dropout layer is set to a preset value.

12. The method according to claim 11, characterized in that The output of the multi-layer perceptron classifier is: , where y is the probability of identifying the mental workload of prosthetic hand manipulation output by the multi-layer perceptron classifier, m is the multimodal feature of EEG, fNIRS and pupil features, W is the weight matrix, and b is the bias.

13. A prosthetic hand manipulation mental workload measurement system based on multi-source information fusion that implements the method according to any one of claims 1 to 12, characterized in that: include: A data acquisition module is used to collect EEG signal data, fNIRS signal data and pupil signal data of patients performing prosthetic hand grasping tasks under different levels of mental workload, and then perform data preprocessing based on the collected signal data; A feature extraction module, used for processing EEG signal data, fNIRS signal data and pupil signal data and extracting EEG features, fNIRS features and pupil features; A feature fusion module, used for multimodally fusing the EEG features, fNIRS features and pupil features to generate multimodal features; The model processing module is used to construct a load measurement and input multimodal features into the load measurement, thereby outputting the probability of identifying the mental load of prosthetic hand manipulation.

Images