Head-mounted Brain-Body Operation Ability Evaluation Method Based on Spatiotemporal Dynamic Fusion Network

Abstract

The present invention discloses a method for evaluating the head-mounted brain-body operation ability based on a spatio-temporal dynamic fusion network, belonging to the technical fields of artificial intelligence and human signal acquisition. It solves the problems that existing methods cannot comprehensively reflect the comprehensive state of an individual and are difficult to reveal the interaction and potential risks among multiple factors. The method includes collecting physiological parameter signals based on wearable devices, and the LSTM prediction model outputs attention and fatigue prediction results; in the present invention, the spatio-temporal dynamic fusion network model pre-constructed is used to analyze and process the graph structure data packet, which can accurately evaluate the working state signal characteristics of plateau high-risk operation personnel. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamically event-driven spiking neural network (deSNN) can convert them into spike sequence features to more accurately depict the eigenvalue. Iterative optimization enables the two to cooperate, improving the accuracy and reliability of the evaluation of the spatio-temporal dynamic fusion network model.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence and human signal acquisition, and particularly relates to a head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network. Background Art

[0002] For personnel engaged in high-risk work in high-altitude areas, the key to their safety and engineering stability lies in accurately evaluating their attention level, fatigue degree, and blood oxygen level. The low-oxygen environment in high-altitude areas can lead to a decrease in the partial pressure of inhaled oxygen, which in turn causes pulmonary circulation artery constriction, increases blood flow resistance, and leads to an increase in pulmonary artery pressure. Prolonged exposure to such an environment may cause an increase in the load on the right ventricle due to pulmonary hypertension, and then lead to right ventricular hypertrophy. These physiological changes may seriously affect the cognitive functions of the staff, including attention, reaction speed, and decision-making ability.

[0003] Traditional evaluation methods rely on the subjective self-evaluation of the staff and the monitoring of physiological indicators such as heart rate and respiration. These methods have limitations. Subjective evaluation is easily affected by emotions and cognitive biases and lacks objectivity; while single physiological indicators cannot comprehensively reflect the comprehensive state of an individual, especially in a variable environment such as the plateau, it is difficult to reveal the interaction between multiple factors and potential risks. In view of the above problems, we propose a head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network. Summary of the Invention

[0004] The purpose of the present invention is to provide a head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network for the deficiencies of the existing technology, and solve the problem that the existing methods using single physiological indicators cannot comprehensively reflect the comprehensive state of an individual, especially in a variable environment such as the plateau, it is difficult to reveal the interaction between multiple factors and potential risks.

[0005] The present invention is implemented as follows. A head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network, the head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network includes:

[0006] Collect physiological parameter signals based on a wearable device. Among them, the physiological parameter signals include electroencephalogram signals, photoplethysmogram signals, heart rate, blood oxygen, and heart rate. The wearable device includes a data acquisition module, and the data acquisition module is composed of a high-precision ADC module and a MAX86150 module;

[0007] Load the collected physiological parameter signals, preprocess the physiological parameter signals, obtain the preprocessed physiological parameter signals, and extract the features of the physiological parameter signals;

[0008] Obtain the physiological parameter signals after feature extraction, convert the physiological parameter signals into graph structure data packets, and obtain a set of graph structure data packets;

[0009] Load the graph structure data packet set, analyze and process the graph structure data packet based on a pre-constructed spatio-temporal dynamic fusion network model, and output the optimal temporal feature vector F of the graph data packet final , where the spatio-temporal dynamic fusion network model is a fusion of the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN, and STGNN is a graph neural network for processing spatio-temporal data;

[0010] Using the optimal temporal feature vector F final as the input, execute the LSTM prediction model, and the LSTM prediction model outputs the attention and fatigue prediction results.

[0011] The method for preprocessing physiological parameter signals includes:

[0012] Collect physiological parameter signals based on wearable devices, and identify electroencephalogram signals and photoplethysmogram signals in the physiological parameter signals;

[0013] Among them, collecting physiological parameter signals based on wearable devices is collected through a head-mounted brain-body operation ability evaluation system based on a spatio-temporal dynamic fusion network. The head-mounted brain-body operation ability evaluation system based on a spatio-temporal dynamic fusion network includes:

[0014] A data acquisition module that collects physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module;

[0015] Among them, the high-precision ADC module: consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter, and a front-end op-amp circuit. The sampling timing and accuracy are independently controllable. The electroencephalogram signal enters the low-noise differential input and output amplifier through the sampling electrode, is differentially amplified and then enters the ADC integrator, and finally outputs the AD conversion result after filtering at a specific frequency;

[0016] The MAX86150 module: an integrated biosensor module for photoplethysmogram (PPG) and electrocardiogram (ECG), including an internal LED;

[0017] The MCU main control module is used to transmit the data collected by the data acquisition module to the upper computer through Bluetooth;

[0018] The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the charging and discharging functions of the battery. The power management module consists of a power supply, a charging management, a regulated output, and a power protection.

[0019] Obtain electroencephalogram signals and preprocess the electroencephalogram signals;

[0020] Obtain the photoplethysmogram (PPG) signal and preprocess the PPG signal.

[0021] The method for preprocessing the electroencephalogram (EEG) signal includes:

[0022] Use a wearable device to extract five types of EEG signals, namely δ, θ, α, β, and γ, based on a 5-channel electrode.

[0023] Load the collected EEG signal and use a Butterworth filter to remove the low-frequency noise and high-frequency noise in the EEG signal. Among them, the processing methods for low-frequency noise and high-frequency noise in the EEG signal include standardization, principal component analysis for dimensionality reduction, and independent component analysis for separating independent source signals.

[0024] Obtain the EEG signal after removing low-frequency noise and high-frequency noise to get multi-channel time series data. The time series at time point t are {θ t} T , {δt} T , {αt} T , {bt} T and {γt} T , where T is the total number of time points.

[0025] The method for preprocessing the PPG signal includes:

[0026] Collect the dual-band PPG signal through a wearable device based on a dual-band bio-optical sensor.

[0027] Obtain the PPG signal and use a low-pass Butterworth filter to remove the high-frequency noise in the PPG signal, retaining the low-frequency hemodynamic information PPG low (t);

[0028] Remove short-term fluctuations and noise through mean filtering and remove spike noise through median filtering to obtain single-channel time series data {PPGt} T .

[0029] The method for extracting the characteristics of physiological parameter signals includes:

[0030] Load the multi-channel time series data of the preprocessed EEG signal and the single-channel time series data {PPGt} of the PPG signal T ;

[0031] Perform time-domain analysis on the multi-channel time series data of the EEG signal and the single-channel time series data {PPGt} of the PPG signal T respectively, and calculate the basic statistical characteristics of each channel signal.

[0032] Among them, the basic statistical features include the mean, variance, maximum value, and minimum value. The functional expressions of the mean, variance, maximum value, and minimum value are as follows:

[0033]

[0034] Among them, is the EEG mean, is the variance, the maximum value, the minimum value, is the data point of time t and channel i, and N is the number of channels, which is 5;

[0035] Use the fast Fourier transform to analyze the multi-channel time series data of electroencephalogram signals and the single-channel time series data of photoplethysmogram signals {PPGt} T of the spectral characteristics, where the spectral characteristics are as follows:

[0036]

[0037] Use the Morlet wavelet transform to obtain the time-frequency characteristics from low frequency to high frequency. The following formula is used to obtain the time-frequency characteristics from low frequency to high frequency:

[0038]

[0039] Among them, is the wavelet function at time t and frequency f.

[0040] The method of converting the physiological parameter signal into a graph structure data packet to obtain a set of graph structure data packets includes:

[0041] According to the type of the physiological parameter signal, classify the nodes into different types to realize the definition and initialization of the nodes;

[0042] Measure the difference between different nodes through the Euclidean distance, normalize the node distance, and complete the construction of the edges. For two nodes u and v, the Euclidean distance d uv The calculation formula is:

[0043]

[0044] Among them, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t;

[0045] Normalize the node distance using the following formula:

[0046]

[0047] Among them, max(d) and min(d) are the minimum and maximum values of the Euclidean distances between all node pairs;

[0048] Based on the relationships between nodes at each time t, construct an N × F matrix X, where N is the number of nodes and F is the feature dimension of each node. Each row represents the feature vector of a node. Then, construct an N × N adjacency matrix. Using the normalized value of the Euclidean distance, the element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, combine the node features X and the adjacency matrix A to form a complete graph structure data packet, denoted as (X, A, t).

[0049] The method for analyzing and processing the graph structure data packet based on the pre-constructed spatio-temporal dynamic fusion network model specifically includes:

[0050] Load the graph structure data packet and input it into the spatio-temporal dynamic fusion network model. In the spatio-temporal dynamic fusion network model, the STGNN network updates the feature vector of each node according to the graph convolution layer update formula to effectively capture spatio-temporal features. Among them, the update formula for the graph convolution layer at each time step is:

[0051] Among them, is the node feature matrix of the l-th layer, is the weight matrix of the graph convolution layer, is the bias vector, is the activation function (ReLU), contains self-loops, is 's degree matrix, H (0) is the initial input feature matrix of the network, and its dimension is N × F;

[0052] Aggregate the node features of the last layer at each time step to obtain a high-dimensional topological feature vector F (t) STGNN. The high-dimensional topological feature vector contains the spatio-temporal feature representation of each node at each time step and the topological structure information between nodes;

[0053] In the spatio-temporal dynamic fusion network model, the deSNN network normalizes the high-dimensional topological feature vector F (t) STGNN to obtain F ’(t) STGNN. F ’(t) STGNN is expressed as:

[0054]

[0055] Ensure that the eigenvalue is within an appropriate range, and define a threshold for each feature dimension:

[0056]

[0057] where μ is the mean of the feature vectors, σ is the standard deviation of the feature dimension, g is a coefficient used to adjust the position of the threshold relative to the mean and standard deviation, M is the dimension of the feature vector, and Ω is the firing threshold. To simulate the firing process of neurons, if the eigenvalue exceeds the threshold, record a firing event and the current timestamp. Each spike not only has a value (0 for no firing or 1 for firing), but also a time stamp indicating its position in the time series. Convert the firing events at all time steps into a spike train S(t)={(ti, 1)∣F′ (ti) STGNN , i > Ω,}

[0058] where the simulation of the neuron firing process is as follows:

[0059] Traverse each time step and check if there is a firing event. For each firing event, initialize V(0) to the resting potential V rest , and update the membrane potential using the Euler method, with the formula as follows:

[0060]

[0061] where V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt = t i+1 - t i is the actual time interval between consecutive time steps, and S(t i ) is the indicator function of the firing event (0 for no firing or 1 for firing). For time steps without firing, the membrane potential remains unchanged;

[0062] Collect the firing events of each neuron. To ensure the consistency of F (t) deSNN and F (t) STGNN on the time scale, perform a temporal aggregation operation. By calculating the interspike interval ISI i ={t i2 t i1 , t i3 t i2 , …}, where t ij is the time of the j-th firing of neuron i F. These data are further used to calculate statistical features such as firing frequency, mean, standard deviation, minimum, and maximum, and construct the feature vector F (t)deSNN = deSNN(S(t));

[0063] Based on the cross - validation iterative optimization process, output the optimal temporal - sequence feature vector F of the graph data packet final .

[0064] The method of outputting the optimal temporal - sequence feature vector F of the graph data packet based on the cross - validation iterative optimization process final includes:

[0065] Take F (t) deSNN as the input of F (t) STGNN and randomly and evenly divide the data set into Q subsets of similar size;

[0066] Define the MSE mean - square error function to evaluate the performance of the spatio - temporal dynamic fusion network model on the given data set, and perform the cross - validation (CV) loop process (f (1, Q)), divide the data set into Q 1 training set and 1 validation set, design an iterative function F (t_k) STGNN = Iterate(F (t_k-1) STGNN , STGNN, deSNN)k R, receive the model parameters of the current F (t_k-1) STGNN 、STGNN and deSNN, and return the updated feature vector;

[0067] Train the model on the training set, perform iterative update of the feature vector, and the iterative update process is as follows: F (0) STGNN as the input of STGNN to obtain the intermediate feature representation F (k-1) STGNN , then, use the current F (k-1) STGNN as the input of deSNN to obtain F (k-1) deSNN , adopt normalization to ensure the correct transfer of the feature vector, and then feedback F (k-1) deSNN to STGNN to update F (k) STGNN , for each iteration number k, execute the iterative function and collect the output F (k) STGNN；

[0068] Evaluate the model performance on the validation set, record the performance metrics, for each f, record the performance metrics, and calculate the average performance metric CV = 1 / K(∑ Kf=1 Performance f), compare the average performance metrics under different numbers of iterations, and select the number of iterations k with the best performance best =argmax k CV(K), and finally perform end-to-end training on the entire training set and the determined optimal k best Evaluate the performance of the final model on an independent test set to obtain the feature vector F that combines spatio-temporal information and statistical features final 。

[0069] The method for the LSTM prediction model to output attention and fatigue prediction results includes:

[0070] Load the optimal temporal feature vector F final , and perform data preprocessing on the optimal temporal feature vector F final to ensure that the F final feature vectors are on the same scale, and select the features most relevant to attention, fatigue, and blood oxygen from them;

[0071] where F final is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension at each time step, and the dimension of the input layer is N F, flatten the features at each time step into a part of the input vector to obtain an input vector with a dimension of NF;

[0072] Use an LSTM layer to process the data. Set the number of LSTM units included in this LSTM layer to H. H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are initial state zero vectors. Update the LSTM layer for time step t:

[0073]

[0074] where f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t 1, x t is the concatenation of the previous hidden state and the current input, i t is the output of the input gate, c t ’ is the candidate cell state, c tis the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment;

[0075] The hidden state h at the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which are used to predict the attention level, fatigue degree, and blood oxygen value respectively. The calculation of the output is as follows:

[0076]

[0077] Among them, the dimension of the Wout weight matrix is (3, H).

[0078] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0079] In the present invention, based on the pre-constructed spatio-temporal dynamic fusion network model, the graph structure data packet is analyzed and processed. The spatio-temporal dynamic fusion network model integrates the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN, and can accurately evaluate the working state signal characteristics of high-altitude high-risk operation personnel. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven spiking neural network (deSNN) can convert them into spike sequence features to more accurately depict the eigenvalue. Iterative optimization enables the two to cooperate, improving the accuracy and reliability of the evaluation of the spatio-temporal dynamic fusion network model.

[0080] In the present invention, the LSTM prediction model is used to output the attention and fatigue prediction results. Based on the LSTM prediction model, time series data can be processed, the attention, fatigue degree, and blood oxygen information can be mined, and it can be judged whether the operator can continue to work, avoiding safety accidents and ensuring the safety of high-altitude high-risk operations.

[0081] The present invention can comprehensively capture and analyze the physiological and psychological states of the staff to accurately judge whether they are suitable to continue to perform high-risk tasks. The present invention aims to improve the accuracy and reliability of the evaluation of the actual working ability of the staff through an innovative evaluation technology, so as to ensure the safety of the staff and the stability of the project. The head-mounted brain-body operation ability evaluation method based on the spatio-temporal dynamic fusion network not only integrates traditional physiological indicators, but also adds advanced physiological measurement technologies such as electroencephalogram (EEG) and photoplethysmogram (PPG) to comprehensively capture the physiological and psychological states of the staff. Through these technologies, the attention, fatigue degree, and blood oxygen level of the staff can be monitored more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 is the overall implementation process schematic diagram of the head-mounted brain-body operation ability evaluation method based on the spatio-temporal dynamic fusion network provided by the present invention.

[0083] Figure 2 Shows a schematic diagram of a head-mounted wearable device.

[0084] Figure 3 Shows a schematic diagram of the wearing effect of the head-mounted wearable device on the human body.

[0085] Figure 4 Shows a schematic diagram of the implementation process of the method for preprocessing physiological parameter signals.

[0086] Figure 5 Shows a schematic diagram of the implementation process of the method for converting physiological parameter signals into graph structure data packets to obtain a set of graph structure data packets.

[0087] Figure 6 Is a schematic diagram of the structure of the graph structure data packet provided by the present invention.

[0088] Figure 7 Shows a schematic diagram of the implementation process of the method for analyzing and processing graph structure data packets based on a pre-constructed spatio-temporal dynamic fusion network model.

[0089] Figure 8 Shows the implementation process schematic diagram of outputting the optimal temporal feature vector F of the graph data packet based on the cross-validation iterative optimization process. final Method implementation process schematic diagram.

[0090] Figure 9 Shows a schematic diagram of the architecture of a head-mounted brain-body operation ability evaluation system based on a spatio-temporal dynamic fusion network. Detailed implementation manner

[0091] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0092] Existing methods using a single physiological index cannot comprehensively reflect the comprehensive state of an individual. Especially in a variable environment such as the plateau, it is difficult to reveal the interaction and potential risks among multiple factors. To address the above problems, we propose a head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network. Briefly, when implementing the method, physiological parameter signals are first collected based on wearable devices, the physiological parameter signals are converted into graph structure data packets, and the graph structure data packets are analyzed and processed based on a pre-constructed spatio-temporal dynamic fusion network model. Then, with the optimal temporal feature vector F final as the input, an LSTM prediction model is executed, and the LSTM prediction model outputs attention and fatigue prediction results. In the embodiments of the present invention, the graph structure data packets are analyzed and processed based on a pre-constructed spatio-temporal dynamic fusion network model, and the spatio-temporal dynamic fusion network model integrates a spatio-temporal graph neural network STGNN and a dynamic event-driven spiking neural network deSNN, which can accurately evaluate the working state signal characteristics of high-risk plateau workers. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven spiking neural network (deSNN) can convert them into spike train features to more accurately characterize the feature values. Iterative optimization enables the two to cooperate, improving the accuracy and reliability of the spatio-temporal dynamic fusion network model evaluation.

[0093] The embodiments of the present invention provide a head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network, Figure 1 showing a schematic diagram of the overall implementation process of the head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network. The head-mounted brain-body operation ability evaluation method based on a spatio-temporal dynamic fusion network specifically includes:

[0094] S10, collecting physiological parameter signals based on wearable devices;

[0095] It should be noted that the physiological parameter signals include but are not limited to electroencephalogram signals, photoplethysmogram signals, heart rate, blood oxygen, heart rate; among them, the wearable device is a head-mounted wearable device, Figure 2 showing a schematic diagram of the head-mounted wearable device, and Figure 3 showing a schematic diagram of the human wearing effect of the head-mounted wearable device. The head-mounted wearable device uses 5-channel electrodes for EEG collection and has a high input impedance design to improve the EEG collection accuracy. In addition, a dual-band bioluminescence sensor integrating 660nm and 880nm is used to collect dual-band PPG data. To adapt to the collection requirements of different scenarios, a variety of electrodes are equipped, and users can adapt different electrodes according to different scenarios to achieve ideal detection accuracy and accuracy.

[0096] S20, loading the collected physiological parameter signals, preprocessing the physiological parameter signals, obtaining the preprocessed physiological parameter signals, and extracting the physiological parameter signal features;

[0097] S30. Obtain the physiological parameter signals after feature extraction, convert the physiological parameter signals into graph structure data packets, and obtain a set of graph structure data packets;

[0098] S40. Load the set of graph structure data packets, analyze and process the graph structure data packets based on a pre-constructed spatio-temporal dynamic fusion network model, and output the optimal temporal feature vector F of the graph data packets final , where the spatio-temporal dynamic fusion network model is a fusion of the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN, and the STGNN is a graph neural network for processing spatio-temporal data;

[0099] S50. Use the optimal temporal feature vector F final as the input, execute the LSTM prediction model, and the LSTM prediction model outputs the attention and fatigue prediction results.

[0100] In the embodiments of the present invention, the graph structure data packets are analyzed and processed based on a pre-constructed spatio-temporal dynamic fusion network model, and the spatio-temporal dynamic fusion network model (Spatio-Temporal Dynamic Fusion Network, STDFN) is a fusion of the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN (Dynamic Event-based Spiking Neural Network, deSNN), which can accurately evaluate the working state signal characteristics of plateau high-risk operators. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven spiking neural network (deSNN) can convert them into spike train features to more accurately characterize the feature values. Iterative optimization enables the two to cooperate, improving the accuracy and reliability of the evaluation of the spatio-temporal dynamic fusion network model.

[0101] The embodiments of the present invention provide a method for preprocessing physiological parameter signals, Figure 4 which shows a schematic implementation flow diagram of the method for preprocessing physiological parameter signals. The method for preprocessing physiological parameter signals specifically includes:

[0102] S101. Collect physiological parameter signals based on a wearable device, and identify the electroencephalogram signal and photoplethysmogram signal in the physiological parameter signals. The wearable device includes a data collection module, and the data collection module is composed of a high-precision ADC module and a MAX86150 module;

[0103] Among them, the collection of physiological parameter signals based on the wearable device is collected through a head-mounted brain-body operation ability evaluation system based on a spatio-temporal dynamic fusion network, Figure 9The schematic architecture diagram of a head-mounted brain-body operation ability evaluation system based on a spatio-temporal dynamic fusion network is shown. The head-mounted brain-body operation ability evaluation system based on the spatio-temporal dynamic fusion network includes:

[0104] A data acquisition module that acquires physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module;

[0105] Among them, the high-precision ADC module: It consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter, and a front-end operational amplifier circuit. The sampling timing and accuracy are independently controllable. The electroencephalogram signal enters the low-noise differential input and output amplifier through the sampling electrode, is differentially amplified and then enters the ADC integrator, and finally outputs the AD conversion result after filtering a specific frequency;

[0106] The MAX86150 module: A biosensor module that integrates photoplethysmogram (PPG) and electrocardiogram (ECG), including an internal LED;

[0107] An MCU main control module that is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth;

[0108] A power management module that is used to supply power to the data acquisition module and the MCU main control module and manage the charging and discharging functions of the battery. The power management module consists of a power supply, a charging management, a regulated output, and a power protection.

[0109] It should be noted that the high-precision ADC module outputs the final AD conversion result after filtering a specific frequency. The functional modules of each part are as follows:

[0110] (1) The low-noise differential input and output amplifier performs a first-stage amplification on the originally collected voltage, and the noise is as low as 0.98 uVpp.

[0111] (2) The amplified voltage signal is integrally amplified by the subsequent integrator. By changing the integration time, the sampling rate is directly affected, and a segmented sampling rate from 250 SPS (samples per second) to 16 KSPS is achieved.

[0112] (3) Digital signal filter: The digital filter receives the output signal of the integrating amplifier, weighs between the resolution and the sampling rate through adjustment and decimation, in order to obtain high-quality data sampling. A third-order sinc filter is integrated on each channel.

[0113] In this embodiment, the MAX86150 module is a biosensor module integrating photoplethysmogram (PPG) and electrocardiogram (ECG), including an internal LED, a photodetector, and low-noise electronics with ambient light suppression. It has a working voltage of 1.8V, and a separate voltage is used for the internal LED. The operating temperature ranges from -40°C to +85°C. The MAX86150 has a size of 3.3mm x 5.6mm x 1.3mm and is a 22-pin optical module that can be used in wearable devices. The data sampling rate of the MAX86150 for PPG ranges from 10 sps to 3200 sps, and the commonly used rates are 400 sps, 800 sps, and 1600 sps. It includes a 19-bit ADC and a proprietary discrete-time filter that can suppress 50Hz / 60Hz interference and slow-moving ambient noise involved.

[0114] The MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth.

[0115] In this embodiment, the MCU main control module uses an ARM Cortex-M4F 32-bit architecture microprocessor, paired with an RTOS (Real-time Operating System) operating system. Embedded technology is used to minimize power consumption to the greatest extent, and the usage time is extended by switching between the sleep mode and the startup mode. It integrates Bluetooth communication, WiFi communication, and 4G wireless communication, supports multiple wireless communication methods, has rich peripheral interfaces, and can change different configurations according to different usage scenarios. The highly integrated BGA (Ball Grid Array) package that only occupies a very small volume is suitable for wearable devices.

[0116] The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the charging and discharging functions of the battery. The power management module consists of a power supply, a charging management unit, a regulated output unit, and a power protection unit.

[0117] Among them, the power supply: A lithium battery is used to supply energy to the entire system, stably providing a voltage of 3.7V. The electrical energy of 480 mAh can support the system to run for a long time.

[0118] The charging management: The charging management realizes functions such as reverse connection protection of the lithium battery, a programmable charging current of up to 500 mA, indication of the charging status, and charging the battery within 1 hour.

[0119] The regulated output: The LDO converts the 3.7V of the lithium battery into 3.3V and 1.8V to supply to the MAX86150. The power supply ripple can be controlled within 8 uVrms, and the voltage change range with temperature is only within 0.3%. The maximum power consumption provided is 2W.

[0120] Power protection: The power protection circuit prevents phenomena such as overload, short circuit, overvoltage, undervoltage, and overcurrent during the use of the power supply, which seriously endanger the safety of battery use.

[0121] It should be noted that when the system is working, the power management module is responsible for supplying power to the entire hardware circuit and managing the charging and discharging functions of the battery; the high-precision ADC module is responsible for collecting tiny electroencephalogram (EEG) signals, and the MAX86150 module is responsible for collecting photoplethysmogram (PPG) signals; the main control transmits the data collected by the data acquisition module to the upper computer through Bluetooth. The power module provides the energy consumption required to support the operation of the entire system. During the working time, the high-precision ADC conversion module converts the collected voltage signal into a digital quantity and transmits it to the main control according to the agreed protocol, and the main control uses wireless communication to pack the digital quantity and send it to the upper computer.

[0122] S102, obtain the brain wave signal and preprocess the brain wave signal;

[0123] S103, obtain the photoplethysmogram signal and preprocess the photoplethysmogram signal.

[0124] In this embodiment, the method for preprocessing the brain wave signal includes:

[0125] Use a wearable device to extract five kinds of brain wave signals, namely δ, θ, α, β, and γ, according to 5-channel electrodes;

[0126] Load the collected brain wave signal, and use a Butterworth filter to remove the low-frequency noise and high-frequency noise in the brain wave signal. The low-frequency noise in the brain wave signal can be power frequency interference signals, and the high-frequency noise can be electromagnetic interference signals. Among them, the processing methods for low-frequency noise and high-frequency noise in the brain wave signal include normalization (the mean of each channel is 0 and the variance is 1), principal component analysis for dimensionality reduction, and independent component analysis for separating independent source signals. The processing of low-frequency noise and high-frequency noise in the brain wave signal is used to remove motion artifacts and electrooculogram artifacts;

[0127] Obtain the brain wave signal after removing the low-frequency noise and high-frequency noise to obtain multi-channel time series data. The time series at time point t are {θ t} T , {δt} T , {αt} T , {bt} T and {γt} T , where T is the total number of time points.

[0128] In this embodiment, the method for preprocessing the photoplethysmogram signal includes:

[0129] The wearable device acquires the dual - band photoplethysmogram (PPG) signal according to the dual - band bio - optical sensor, so as to adapt to its lower sampling rate and slower change characteristics;

[0130] Acquire the PPG signal, use a low - pass Butterworth filter to remove the high - frequency noise in the PPG signal, and retain the low - frequency hemodynamic information PPG low (t), and then remove short - term fluctuations, noise through mean filtering and remove spike noise through median filtering;

[0131] Remove short - term fluctuations, noise through mean filtering and remove spike noise through median filtering to obtain single - channel time - series data {PPGt} T 。

[0132] The embodiment of the present invention provides a method for extracting the characteristics of physiological parameter signals. The method for extracting the characteristics of physiological parameter signals specifically includes:

[0133] S201, load the multi - channel time - series data of the pre - processed electroencephalogram (EEG) signal and the single - channel time - series data {PPGt} of the PPG signal T ;

[0134] S202, perform time - domain analysis on the multi - channel time - series data of the EEG signal and the single - channel time - series data {PPGt} of the PPG signal T respectively, and calculate the basic statistical characteristics of each channel signal;

[0135] Among them, the basic statistical characteristics include mean, variance, maximum value, and minimum value. The functional expressions of mean, variance, maximum value, and minimum value are:

[0136]

[0137] Among them, is the EEG mean, is the variance, is the maximum value, is the minimum value, is the data point at time t and channel i, and N is the number of channels 5;

[0138] S203, use the fast Fourier transform to analyze the spectral characteristics of the multi - channel time - series data of the EEG signal and the single - channel time - series data {PPGt} T of the PPG signal, where the spectral characteristics are as follows:

[0139]

[0140] S204. Use the Morlet wavelet transform to obtain time-frequency features from low frequency to high frequency. The following formula is used to obtain the time-frequency features from low frequency to high frequency:

[0141]

[0142] where is the wavelet function at time t and frequency f.

[0143] In this embodiment, the features extracted by the above method, such as PPG t : [PPG low_t 、PPG medium_t 、PPG high_t 、PPG mt 、PPG st 2 、PPG max_t 、PPG min_t 、PPG FFT_t , α t : [α low_t 、α medium_t 、α high_t 、α mt 、α st 2 、α max_t 、α min_t 、α FFT_t , β t : [β low_t 、β medium_t 、β high_t 、βμ t 、βσ t 2 、β max_t 、β min_t 、β FFT_t , γ t : [γ low_t 、γ medium_t 、γ high_t 、γμ t 、γσ t 2 、γ max_t 、γ min_t 、γ FFT_t etc.

[0144] The embodiment of the present invention provides a method for converting a physiological parameter signal into a graph-structured data packet to obtain a set of graph-structured data packets. Figure 5 The schematic diagram of the implementation process of the method for converting a physiological parameter signal into a graph-structured data packet to obtain a set of graph-structured data packets is shown. The method for converting a physiological parameter signal into a graph-structured data packet to obtain a set of graph-structured data packets specifically includes:

[0145] S301. Classify the nodes into different types according to the type of physiological parameter signals to achieve node definition and initialization. For example, different frequency bands (δ, θ, α, β, γ) of EEG signals and PPG signals are regarded as nodes of different types, thus reflecting different types of physiological signals or signal characteristics.

[0146] S302. Measure the difference between different nodes through the Euclidean distance, normalize the node distance, and complete the construction of edges. For two nodes u and v, the Euclidean distance d uv The calculation formula is:

[0147]

[0148] Among them, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t;

[0149] In the embodiment of the present invention, in order to transform the distance into the interval [0, 1], the node distance is normalized, and the following formula is adopted:

[0150]

[0151] Among them, max(d) and min(d) are the minimum and maximum values of the Euclidean distances of all node pairs. In this way, a smaller distance (i.e., the nodes are more similar) corresponds to a larger value in the adjacency matrix;

[0152] S303. Based on the relationship between nodes at each time t, construct a matrix X of N F, where N is the number of nodes and F is the feature dimension of each node. Each row represents the feature vector of a node. Then, construct an adjacency matrix of N N. Use the normalized value of the Euclidean distance. The element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, combine the node features X and the adjacency matrix A to form a complete graph structure data packet, denoted as (X, A, t). The graph structure data packet is ready to be input into the spatio-temporal dynamic fusion network model. Generate a view of the graph structure data packet at a specific time t, as Figure 6 shown.

[0153] The embodiment of the present invention provides a method for analyzing and processing a graph structure data packet based on a pre-constructed spatio-temporal dynamic fusion network model. Figure 7 The figure shows a schematic implementation flowchart of the method for analyzing and processing a graph structure data packet based on a pre-constructed spatio-temporal dynamic fusion network model. The method for analyzing and processing a graph structure data packet based on a pre-constructed spatio-temporal dynamic fusion network model specifically includes:

[0154] S401. Load the graph structure data packet and input it into the spatio-temporal dynamic fusion network model. In the spatio-temporal dynamic fusion network model, the STGNN network updates the feature vector of each node according to the graph convolution layer update formula to effectively capture spatio-temporal features. Among them, the update formula of the graph convolution layer for each time step is:

[0155]

[0156] Among them, is the node feature matrix of the l-th layer, is the weight matrix of the graph convolution layer, is the bias vector, is the activation function (ReLU), contains self-loops, is the degree matrix of, H (0) is the initial input feature matrix of the network, and its dimension is N F;

[0157] S402. Aggregate the node features of the last layer for each time step to obtain the high-dimensional topological feature vector F (t) STGNN. The high-dimensional topological feature vector contains the spatio-temporal feature representations of each node (representing different EEG and PPG signals) for each time step and the topological structure information between nodes. For the high-dimensional feature vector of time step t and each node (representing different EEG and PPG signals), it reflects the spatio-temporal relationship between nodes and the dynamic characteristics of each node;

[0158] S403. The deSNN network in the spatio-temporal dynamic fusion network model normalizes the high-dimensional topological feature vector F (t) STGNN to obtain F ’(t) STGNN, F ’(t) STGNN is expressed as:

[0159]

[0160] Ensure that the eigenvalues are within an appropriate range. Define a threshold for each feature dimension:

[0161]

[0162] Among them, μ is the average value of the feature vectors, σ is the standard deviation of the feature dimensions, g is a coefficient used to adjust the position of the threshold relative to the average value and the standard deviation, M is the dimension of the feature vectors, Ω serves as the firing threshold, simulating the firing process of neurons. If the feature value exceeds the threshold, a firing event and the current timestamp are recorded. Each spike not only has a value (0 indicating no firing or 1 indicating firing), but also a time stamp indicating its position in the time series. The firing events at all time steps are converted into a spike train S(t)={(ti, 1)∣F′ (ti) STGNN , i > Ω,};

[0163] Among them, the process of simulating the firing of neurons is as follows:

[0164] Traverse each time step to check if there is a firing event. For each firing event, initialize V(0) to the resting potential V rest , and update the membrane potential using the Euler method. The formula is as follows:

[0165]

[0166] Among them, V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt = t i+1 - t i is the actual time interval between consecutive time steps, S(t i ) is the indicator function of the firing event (0 indicating no firing or 1 indicating firing). For time steps without firing, the membrane potential remains unchanged;

[0167] Collect the firing events of each neuron. To ensure the consistency of F (t) deSNN and F (t) STGNN on the time scale, perform a temporal aggregation operation. By calculating the inter-spike interval ISI i = {t i2 t i1 , t i3 t i2 , …}, t ij is the time of the j-th firing of neuron i F. These data are further used to calculate statistical features such as firing frequency, average value, standard deviation, minimum value, and maximum value, and construct the feature vector F (t) deSNN = deSNN(S(t));

[0168] S404, based on the cross-validation iterative optimization process, outputs the optimal temporal feature vector F of the graph data packetfinal .

[0169] In this embodiment, the spatiotemporal dynamic fusion network model STDFN is an advanced integrated framework that integrates STGNN (spatial-temporal graph neural network) and deSNN (dynamic event-driven spiking neural network), which is specifically used to process and analyze physiological data, aiming to extract features related to the attention level and fatigue of high-risk operators in plateaus. STGNN is a graph neural network specially designed to process spatiotemporal data. It can capture the spatiotemporal dependency and topological structure information in the data. STGNN aggregates the neighborhood information of nodes through graph convolution layers, thereby learning the spatiotemporal feature representation of nodes.

[0170] The embodiment of the present invention provides an optimal time series feature vector F of the output graph data packet based on a cross-validation iterative optimization process. final method, Figure 8 The optimal time series feature vector F of the output graph data packet is shown based on the cross-validation iterative optimization process. final Schematic diagram of the implementation process of the method, which outputs the optimal time series feature vector F of the graph data packet based on the cross-validation iterative optimization process final The methods include:

[0171] S4041, F (t) deSNN As F (t) STGNN The data set is randomly and evenly divided into Q subsets of similar size. In this embodiment, Q can be 5;

[0172] S4042, define the MSE mean square error function to evaluate the performance of the spatiotemporal dynamic fusion network model on a given data set and perform a cross validation (CV) cycle (f (1, Q)), divide the data set into Q 1 training set and 1 validation set, design an iterative function F (t_k) STGNN =Iterate(F (t_k-1) STGNN ,STGNN,deSNN)k R, receive the current F (t_k-1) STGNN , STGNN and deSNN model parameters, and return the updated feature vector;

[0173] S4043, train the model on the training set and perform iterative update of the feature vector. The iterative update process is as follows: (0) STGNN As the input of STGNN, the intermediate feature representation F is obtained (k-1) STGNN, then, use the current F (k-1) STGNN as the input of the deSNN to obtain F (k-1) deSNN , adopt normalization to ensure the correct transmission of the feature vector, and then feedback F (k-1) deSNN to the STGNN to update F (k) STGNN , for each iteration number k, execute the iteration function and collect the output F (k) STGNN；

[0174] S4044, evaluate the model performance on the validation set, record the performance metrics, for each f, record the performance metrics, calculate the average performance metric CV = 1 / K(∑ K f=1 Performancef) after all f are completed, compare the average performance metrics under different iteration numbers, and select the iteration number k with the best performance best = argmax k CV(K), and finally perform end-to-end training on the entire training set and the determined best k best to evaluate the performance of the final model on an independent test set, so as to obtain the feature vector F that combines spatio-temporal information and statistical features final .

[0175] It should be noted that evaluating the model performance on the validation set and recording the performance metrics are expressed as Performance = Evaluate(STDFN, Validation Set).

[0176] The embodiment of the present invention provides a method for the LSTM prediction model to output attention and fatigue prediction results. The method for the LSTM prediction model to output attention and fatigue prediction results specifically includes:

[0177] S501, load the optimal temporal feature vector F final , perform data preprocessing on the optimal temporal feature vector F final to ensure that the F final feature vectors are on the same scale, and select the features most relevant to attention, fatigue degree, and blood oxygen from them;

[0178] Among them, F final is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension at each time step, and the dimension of the input layer is N F, flatten the features at each time step into a part of the input vector to obtain an input vector with a dimension of NF;

[0179] S502 uses an LSTM layer to process data. The number of LSTM units included in this LSTM layer is set to H, where H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are zero vectors of the initial state. For the LSTM layer at time step t, the update is as follows:

[0180]

[0181] Among them, f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t 1, x t is the concatenation of the hidden state at the previous moment and the current input. i t is the output of the input gate, c t ’ is the candidate cell state, c t is the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment;

[0182] S503, the hidden state h at the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which are used to predict the attention level, fatigue degree, and blood oxygen value respectively. The calculation of the output is as follows:

[0183]

[0184] Among them, the dimension of the Wout weight matrix is (3, H).

[0185] In this embodiment, when training the LSTM prediction model, the mean square error is used to design a loss function. Since three values (attention level, fatigue degree, and blood oxygen value) are predicted, we design separate loss functions for each value and then combine them:

[0186]

[0187] Among them, y attention，I , y fatigue，i and y SpO2，i are the true values, y^ attention，i , y^ fatigue，i and y^ SpO2，iIt is the predicted value. Then, the model is trained using the Adam algorithm with the training set, and the validation set is used to adjust the hyperparameters and prevent overfitting. At the same time, an early stopping strategy is adopted to avoid overfitting. Finally, the trained model is deployed to the cloud for real-time monitoring and prediction of the attention level, fatigue degree, and blood oxygen value of the staff.

[0188] In the embodiment of the present invention, an LSTM prediction model is used to output the attention and fatigue prediction results. Based on the LSTM prediction model's ability to process time series data, mine attention, fatigue degree, and blood oxygen information, it can judge whether the operator can continue to work, avoid safety accidents, and ensure the safety of high-risk operations on the plateau.

[0189] Specifically, for the real-time evaluation of the head-mounted brain-body operation ability of the graph neural network, first, the EEG and PPG signals collected by the wearable device are processed. The EEG signal is filtered and denoised (including normalization, PCA dimensionality reduction, and ICA separation) by a Butterworth filter, and the PPG signal is processed by a low-pass Butterworth filter and mean and median filtering. Then, time-domain, frequency-domain, and wavelet transform feature extraction is performed.

[0190] Next, different frequency band EEG and PPG signals are used as nodes according to physiological parameters, and edges are constructed through Euclidean distance calculation and normalization processing. Based on the node relationship, a matrix X and an adjacency matrix A are constructed to form a graph data packet. Then, the graph data packet is input into the STDFN. Among them, the STGNN aggregates the node neighborhood information through the graph convolution layer to update the node feature vector and aggregates and normalizes it to obtain a high-dimensional topological feature vector.

[0191] Then, it is input into the deSNN. The deSNN simulates the neuron firing model to convert it into a pulse sequence feature vector, and the optimal feature vector is found through iterative optimization.

[0192] Finally, the feature vector is flattened and input into the LSTM prediction model. The LSTM updates the state according to the calculations of the forget gate, input gate, cell state update, and output gate. The output layer predicts the attention level, fatigue degree, and blood oxygen value based on the hidden state at the last moment. During training, the mean square error is used as the loss function, and the model is trained in combination with the Adam algorithm, and the parameters are adjusted through the validation set and the early stopping strategy to prevent overfitting.

[0193] In the embodiment of the present invention, it is proposed to use a head-mounted device to real-time monitor and evaluate the working status of high-risk operators in the plateau environment, ensuring the immediacy, high efficiency, and accuracy of the evaluation results. Moreover, in the head-mounted brain-body operation ability evaluation system based on the spatio-temporal dynamic fusion network, the hardware adopts a low-power design, flexible printed circuit board (FPC), and lightweight materials. The device can ensure long-term battery life while reducing the burden on the wearer. Multiple electrodes are equipped on the device, and users can select appropriate electrodes according to different environments to meet the needs of various scenarios.

[0194] In summary, the present invention provides a method for evaluating the head-mounted brain-body operation ability based on a spatio-temporal dynamic fusion network. In the embodiments of the present invention, the spatio-temporal dynamic fusion network model is used to analyze and process the graph structure data packet, and the spatio-temporal dynamic fusion network model integrates the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN, which can accurately evaluate the signal characteristics of the working state of plateau high-risk operators. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven spiking neural network (deSNN) can convert them into spike train features to more accurately characterize the eigenvalue. Iterative optimization enables the two to cooperate, improving the accuracy and reliability of the evaluation of the spatio-temporal dynamic fusion network model.

[0195] The present invention can comprehensively capture and analyze the physiological and psychological states of workers to accurately determine whether they are suitable to continue performing high-risk tasks. The purpose of the present invention is to improve the accuracy and reliability of the evaluation of the actual working ability of workers through an innovative evaluation technology, so as to ensure the safety of workers and the stability of the project. The method for evaluating the head-mounted brain-body operation ability based on the spatio-temporal dynamic fusion network not only integrates traditional physiological indicators, but also incorporates advanced physiological measurement technologies such as electroencephalogram (EEG) and photoplethysmogram (PPG) to comprehensively capture the physiological and psychological states of workers. Through these technologies, the attention, fatigue and blood oxygen level of workers can be monitored more accurately.

[0196] It should be noted that for the foregoing embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0197] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the above unit division is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection between devices or units can be in the form of telecommunications or other forms.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting the protection scope of the invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still, without conflict and without creative efforts, combine, add or delete the features in the embodiments of the present invention according to the circumstances or make other adjustments, so as to obtain different technical solutions that essentially do not deviate from the concept of the present invention, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. A head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network, characterized in that: The head-mounted brain-body operation ability assessment method based on the spatiotemporal dynamic fusion network includes: Collecting physiological parameter signals based on wearable devices, wherein the physiological parameter signals include brain wave signals, photoplethysmogram signals, heart rate, and blood oxygen; Loading the collected physiological parameter signals, preprocessing the physiological parameter signals, obtaining the preprocessed physiological parameter signals, and extracting features of the physiological parameter signals; Acquire the physiological parameter signal after feature extraction, convert the physiological parameter signal into a graph structure data packet, and obtain a graph structure data packet set; Load the graph structure data packet set, analyze and process the graph structure data packet based on the pre-built spatiotemporal dynamic fusion network model, and output the optimal time series feature vector F of the graph data packet final Among them, the spatiotemporal dynamic fusion network model STDFN integrates the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN. STGNN is a graph neural network used to process spatiotemporal data. STGNN can capture the spatiotemporal dependency and topological structure information in the data. STGNN aggregates the neighborhood information of nodes through graph convolution layers, thereby learning the spatiotemporal feature representation of nodes. With the optimal time series feature vector F final As input, execute the LSTM prediction model, and the LSTM prediction model outputs the attention and fatigue prediction results; Among them, the optimal time series feature vector F of the output graph data packet is final methods, including: F (t) deSNN As F (t) STGNN The input of F (t) deSNN represents the feature vector obtained by deSNN after processing the graph structure data packet at time step t, F (t) STGNN It represents the feature vector obtained by STGNN after processing the graph structure data packet at time step t, and randomly and evenly divides the data set into Q subsets; Define the MSE mean square error function to evaluate the performance of the spatiotemporal dynamic fusion network model on a given data set, perform a cross-validation CV cycle process f∈[1,Q], divide the data set into Q-1 training sets and 1 validation set, and design an iterative function F (t _k) STGNN =Iterate(F (t_k-1) STGNN ,STGNN,deSNN)k∈R, receiving the current F (t_k-1) STGNN , STGNN and deSNN model parameters, and return the updated feature vector; Train the model on the training set and perform iterative update of the feature vector. The iterative update process is as follows: (0) STGNN As the input of STGNN, the intermediate feature representation F is obtained (k-1) STGNN , then, using the current F (k-1) STGNN As the input of deSNN, we get F (k -1) deSNN , normalization is used to ensure the correct transmission of feature vectors, and then F (k-1) deSNN Feedback to STGNN to update F (k) STGNN , for each iteration number k, execute the iterative function and collect the output F (k) STGNN； Evaluate the model performance on the validation set, record the performance indicators, and for each f, record the performance indicators. Calculate the average performance indicator CV=1 / K(∑ K f=1 Performancef), compare the average performance indicators under different iteration numbers, and select the iteration number k with the best performance best =argmax k CV(K), finally, in the entire training set and the best k determined best The model is trained end-to-end and the performance of the final model is evaluated on an independent test set to obtain a feature vector F that integrates spatiotemporal information and statistical features. final .

2. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 1 is characterized by: The method for preprocessing a physiological parameter signal comprises: Based on the collection of physiological parameter signals by wearable devices, the brain wave signals and photoplethysmogram signals in the physiological parameter signals are identified; Wherein, the physiological parameter signals collected by the wearable device are collected by a head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network, and the head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network includes: Data acquisition module, which collects physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module; Among them, the high-precision ADC module: consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter and a front-end operational amplifier circuit. The sampling timing and accuracy are independently controllable. The brain wave signal enters the low-noise differential input and output amplifier from the sampling electrode, enters the ADC integrator after differential amplification, and outputs the final AD conversion result after filtering the specific frequency; MAX86150 module: Biosensor module with integrated photoplethysmogram (PPG) and electrocardiogram (ECG), including internal LED; MCU main control module, the MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth; Power management module: The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the charging and discharging functions of the battery. The power management module consists of power supply, charging management, voltage stabilization output, and power protection; Acquire brain wave signals and pre-process the brain wave signals; A photoplethysmogram signal is acquired and preprocessed.

3. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 2 is characterized by: The method for preprocessing brain wave signals comprises: Wearable devices are used to extract five types of EEG signals, namely, δ, θ, α, β and γ, based on 5-channel electrodes; Loading the collected brain wave signal, using Butterworth filter to remove low-frequency noise and high-frequency noise in the brain wave signal, wherein the low-frequency noise and high-frequency noise processing methods in the brain wave signal include standardization, principal component analysis dimensionality reduction, and independent component analysis to separate independent source signal methods; Obtain the EEG signal after removing low-frequency noise and high-frequency noise, and obtain multi-channel time series data. The time series at time point t are {θ t } T , {δt} T , {αt} T , {βt} T and {γt} T , T is the total number of time points.

4. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 2, characterized in that: The method for preprocessing a photoplethysmogram signal comprises: Collecting dual-band photoplethysmogram signals using a wearable device based on a dual-band bio-optical sensor; Obtain the photoplethysmogram signal, use a low-pass Butterworth filter to remove high-frequency noise in the photoplethysmogram signal, and retain the low-frequency hemodynamic information PPG low (t); The short-term fluctuations and noise are removed by mean filtering, and the spike noise is removed by median filtering to obtain single-channel time series data {PPGt} T .

5. The head-mounted brain-body performance evaluation method based on a spatiotemporal dynamic fusion network as described in any one of claims 2 to 4, characterized in that: The method for extracting physiological parameter signal features comprises: Load the preprocessed EEG signal multi-channel time series data and photoplethysmogram signal single-channel time series data {PPGt} T ; The multi-channel time series data of EEG signals and the single-channel time series data of photoplethysmogram signals {PPGt} T Perform time domain analysis and calculate the basic statistical characteristics of each channel signal; Among them, the basic statistical characteristics include mean, variance, maximum value, and minimum value. The function expressions of mean, variance, maximum value, and minimum value are: in, is the EEG mean, is the variance, Maximum value, Minimum value, is the data point at time t and channel i, N is the number of channels 5; Use Fast Fourier Transform to analyze multi-channel time series data of EEG signals and single-channel time series data of photoplethysmogram signals {PPGt} T The spectrum characteristics are as follows: Use Morlet wavelet transform to obtain low-frequency to high-frequency time-frequency features. The following formula is used to obtain low-frequency to high-frequency time-frequency features: in, is a wavelet function with time t and frequency f.

6. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 5, characterized in that: The method of converting the physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set comprises: According to the type of physiological parameter signal, the nodes are classified into different types to achieve node definition and initialization; The Euclidean distance is used to measure the differences between different nodes, and the node distance is normalized to complete the edge construction. For the two nodes u and v, the Euclidean distance d uv The calculation formula is: in, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t; The node distance is normalized using the following formula: Among them, max(d) and min(d) are the minimum and maximum values ​​of the Euclidean distances of all node pairs; Based on the relationship between nodes at each time t, construct N The matrix X of F, where N is the number of nodes, F is the feature dimension of each node, and each row represents the feature vector of a node. Then, construct N The adjacency matrix of N uses the normalized value of the Euclidean distance. The element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, the node feature X and the adjacency matrix A are combined to form a complete graph structure data packet, which is represented as (X, A, t).

7. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 1, characterized in that: The method for analyzing and processing graph structure data packets based on the pre-built spatiotemporal dynamic fusion network model specifically includes: Load the graph structure data package and input the graph structure data package into the spatiotemporal dynamic fusion network model. The STGNN network in the spatiotemporal dynamic fusion network model updates the feature vector of each node according to the graph convolution layer update formula to effectively capture the spatiotemporal features. The update formula of the graph convolution layer at each time step is: in, is the node feature matrix of the lth layer, is the weight matrix of the graph convolutional layer, is the bias vector, is the activation function (ReLU), Contains self-loops, yes The degree matrix, H (0) is the initial input feature matrix of the network, whose dimension is N F; Aggregate the node features of the last layer of each time step to obtain a high-dimensional topological feature vector F (t) In STGNN, the high-dimensional topological feature vector contains the spatiotemporal feature representation of each node at each time step and the topological structure information between nodes; The deSNN network in the spatiotemporal dynamic fusion network model is used to calculate the high-dimensional topological feature vector F. (t) STGNN performs normalization and obtains F ’(t) STGNN, F ’(t) STGNN is represented as: To ensure that the feature values ​​are within the appropriate range, define a threshold for each feature dimension: Among them, μ is the mean value of the feature vector, σ is the standard deviation of the feature dimension, g is a coefficient used to adjust the position of the threshold relative to the mean and standard deviation, M is the dimension of the feature vector, Ω is used as the threshold of emission, and the emission process of the neuron is simulated. If the feature value exceeds the threshold, a emission event and the current timestamp are recorded. Each pulse not only has a value, 0 for no emission or 1 for emission, but also a time stamp indicating its position in the time series. The emission events of all time steps are converted into a pulse sequence S(t)={(ti,1)|F′ (ti) STGNN ,i>Ω,}; Among them, the firing process of simulated neurons is: Traverse each time step and check whether there is a firing event. For each firing event, initialize V(0) to the resting potential V rest , the membrane potential is updated using the Euler method, and the formula is as follows: Where V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt=t i+1 -t i Since it is the actual time interval between consecutive time steps, S(t i ) is the indicator function of the firing event, 0 means no firing or 1 means firing, and for the time step without firing, the membrane potential remains unchanged; Collect the firing events of each neuron to ensure F (t) deSNN and F (t) STGNN performs temporal aggregation operations based on the consistency of time scale by calculating the time interval ISI between two emission events. i ={t i2- t i1 , t i3- t i2 , ...}, t ij is the time of the jth firing of neuron i∈F. These data are further used to calculate the firing frequency, mean, standard deviation, minimum and maximum statistical features to construct the feature vector F (t) deSNN =deSNN(S(t)); Based on the cross-validation iterative optimization process, the optimal time series feature vector F of the output graph data packet is final .

8. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 7, characterized in that: The method for the LSTM prediction model to output attention and fatigue prediction results includes: Load the optimal timing feature vector F final , for the optimal time series feature vector F final Perform data preprocessing to ensure F final The feature vectors are on the same scale, and the features most relevant to attention, fatigue, and blood oxygen are selected from them; Among them, F final It is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension of each time step, and the dimension of the input layer is N F, flatten the features of each time step into a part of the input vector, and obtain an input vector with dimension NF; Use an LSTM layer to process the data. Set the number of LSTM units contained in the LSTM layer to H. H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X. t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are the initial state zero vectors, and the LSTM layer at time step t is updated: Among them, f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t-1 , x t ] is the concatenation of the hidden state of the previous moment and the current input, i t is the output of the input gate, c t ' is the candidate cell state, c t is the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment; The hidden state h of the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which is used to predict the attention level, fatigue level and blood oxygen value respectively. The output is calculated as follows: Among them, the dimension of the Wout weight matrix is ​​(3, H).

Images