Head-mounted brain body operation capability assessment method based on space-time dynamic fusion network

Abstract

The invention discloses a head-mounted brain body workability assessment method based on a space-time dynamic fusion network, belongs to the technical field of artificial intelligence and human body signal acquisition, and solves the problems that an existing method cannot comprehensively reflect the comprehensive state of an individual, and interaction and potential risks among multiple factors are difficult to reveal. The method comprises the steps that physiological parameter signals are collected based on wearable equipment, and an attention and fatigue prediction result is output by an LSTM prediction model; according to the method, the graph structure data packet is analyzed and processed based on the pre-constructed space-time dynamic fusion network model, the working state signal characteristics of the plateau high-risk operating personnel can be accurately evaluated, the space-time graph neural network (STGNN) can capture the complex relation, the high-dimensional topological characteristics are extracted, and the high-risk working personnel can be accurately evaluated. The dynamic event-driven pulse neural network (deSNN) can convert the pulse sequence features into pulse sequence features so as to describe feature values more accurately, iterative optimization enables the pulse sequence features and the pulse sequence features to cooperate, and the accuracy and reliability of time-space dynamic fusion network model evaluation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and human body signal acquisition, and specifically relates to a head-mounted brain-body operation ability assessment method based on a spatiotemporal dynamic fusion network. Background Art

[0002] The key to the safety and engineering stability of personnel engaged in high-risk work at high altitudes is to accurately assess their attention level, fatigue and blood oxygen level. The low oxygen environment at high altitudes will cause a decrease in the partial pressure of oxygen in the inspired air, which will in turn cause pulmonary circulation artery constriction, increase blood flow resistance, and lead to increased pulmonary artery pressure. Long-term exposure to this environment may increase the load on the right ventricle due to pulmonary hypertension, leading to right ventricular hypertrophy. These physiological changes may seriously affect the cognitive function of workers, including attention, reaction speed and decision-making ability.

[0003] Traditional assessment methods rely on subjective self-evaluation of staff and monitoring of physiological indicators such as heart rate and breathing. These methods have limitations. Subjective evaluation is easily affected by emotions and cognitive biases and lacks objectivity. A single physiological indicator cannot fully reflect the comprehensive state of an individual, especially in a changeable environment such as the plateau. It is difficult to reveal the interaction and potential risks between multiple factors. To address the above problems, we proposed a head-mounted brain-body operational ability assessment method based on a spatiotemporal dynamic fusion network. Summary of the invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and provide a head-mounted brain-body performance assessment method based on a spatiotemporal dynamic fusion network, which solves the problem that the existing method uses a single physiological indicator that cannot fully reflect the comprehensive state of an individual, especially in a changeable environment such as the plateau, and it is difficult to reveal the interaction and potential risks between multiple factors.

[0005] The present invention is implemented in this way: a head-mounted brain-body operation ability assessment method based on a spatiotemporal dynamic fusion network, the head-mounted brain-body operation ability assessment method based on a spatiotemporal dynamic fusion network comprises: The wearable device collects physiological parameter signals, wherein the physiological parameter signals include brain wave signals, photoelectric volume pulse signal, heart rate, blood oxygen, and heart rate. The wearable device includes a data acquisition module, and the data acquisition module is composed of a high-precision ADC module and a MAX86150 module. Loading the collected physiological parameter signals, preprocessing the physiological parameter signals, obtaining the preprocessed physiological parameter signals, and extracting features of the physiological parameter signals; Acquire the physiological parameter signal after feature extraction, convert the physiological parameter signal into a graph structure data packet, and obtain a graph structure data packet set; Load the graph structure data packet set, analyze and process the graph structure data packet based on the pre-built spatiotemporal dynamic fusion network model, and output the optimal time series feature vector F of the graph data packet final , where the spatiotemporal dynamic fusion network model is a fusion of the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN. STGNN is a graph neural network used to process spatiotemporal data; With the optimal time series feature vector F final As input, the LSTM prediction model is executed, and the LSTM prediction model outputs the attention and fatigue prediction results.

[0006] The method for preprocessing a physiological parameter signal comprises: Based on the collection of physiological parameter signals by wearable devices, the brain wave signals and photoplethysmogram signals in the physiological parameter signals are identified; Wherein, the physiological parameter signals collected by the wearable device are collected by a head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network, and the head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network includes: Data acquisition module, which collects physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module; Among them, the high-precision ADC module: consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter and a front-end operational amplifier circuit. The sampling timing and accuracy are independently controllable. The brain wave signal enters the low-noise differential input and output amplifier from the sampling electrode, enters the ADC integrator after differential amplification, and outputs the final AD conversion result after filtering the specific frequency; MAX86150 module: Biosensor module with integrated photoplethysmogram (PPG) and electrocardiogram (ECG), including internal LED; MCU main control module, the MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth; Power management module: The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the battery charging and discharging functions. The power management module consists of power supply, charging management, voltage regulated output, and power protection.

[0007] Acquire brain wave signals and pre-process the brain wave signals; A photoplethysmogram signal is acquired and preprocessed.

[0008] The method for preprocessing brain wave signals comprises: Wearable devices are used to extract five types of EEG signals, namely, δ, θ, α, β and γ, based on 5-channel electrodes; Loading the collected brain wave signal, using Butterworth filter to remove low-frequency noise and high-frequency noise in the brain wave signal, wherein the low-frequency noise and high-frequency noise processing methods in the brain wave signal include standardization, principal component analysis dimensionality reduction, and independent component analysis to separate independent source signal methods; Obtain the EEG signal after removing low-frequency noise and high-frequency noise, and obtain multi-channel time series data. The time series at time point t are {θ t} T , {δt} T , {αt} T , {bt} T and {γt} T , T is the total number of time points.

[0009] The method for preprocessing a photoplethysmogram signal comprises: Collecting dual-band photoplethysmogram signals using a wearable device based on a dual-band bio-optical sensor; Obtain the photoplethysmogram signal, use a low-pass Butterworth filter to remove high-frequency noise in the photoplethysmogram signal, and retain the low-frequency hemodynamic information PPG low (t); The short-term fluctuations and noise are removed by mean filtering, and the spike noise is removed by median filtering to obtain single-channel time series data {PPGt} T .

[0010] The method for extracting physiological parameter signal features comprises: Load the preprocessed EEG signal multi-channel time series data and photoplethysmogram signal single-channel time series data {PPGt} T ; The multi-channel time series data of EEG signals and the single-channel time series data of photoplethysmogram signals {PPGt} T Perform time domain analysis and calculate the basic statistical characteristics of each channel signal; Among them, the basic statistical characteristics include mean, variance, maximum value, and minimum value. The function expressions of mean, variance, maximum value, and minimum value are:

[0011] in, is the EEG mean, is the variance, Maximum value, Minimum value, is the data point at time t and channel i, and N is the number of channels 5; Use Fast Fourier Transform to analyze multi-channel time series data of EEG signals and single-channel time series data of photoplethysmogram signals {PPGt} T The spectrum characteristics are as follows:

[0012] Use Morlet wavelet transform to obtain low-frequency to high-frequency time-frequency features. The following formula is used to obtain low-frequency to high-frequency time-frequency features:

[0013] in, is a wavelet function with time t and frequency f.

[0014] The method of converting the physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set comprises: According to the type of physiological parameter signal, the nodes are classified into different types to achieve node definition and initialization; The Euclidean distance is used to measure the differences between different nodes, and the node distance is normalized to complete the edge construction. For the two nodes u and v, the Euclidean distance d uv The calculation formula is:

[0015] in, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t; The node distance is normalized using the following formula:

[0016] Among them, max(d) and min(d) are the minimum and maximum values ​​of the Euclidean distances of all node pairs; Based on the relationship between nodes at each time t, construct N The matrix X of F, where N is the number of nodes, F is the feature dimension of each node, and each row represents the feature vector of a node. Then, construct N The adjacency matrix of N uses the normalized value of the Euclidean distance. The element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, the node feature X and the adjacency matrix A are combined to form a complete graph structure data packet, which is represented as (X, A, t).

[0017] The method for analyzing and processing graph structure data packets based on the pre-built spatiotemporal dynamic fusion network model specifically includes: Load the graph structure data package and input the graph structure data package into the spatiotemporal dynamic fusion network model. The STGNN network in the spatiotemporal dynamic fusion network model updates the feature vector of each node according to the graph convolution layer update formula to effectively capture the spatiotemporal features. The update formula of the graph convolution layer at each time step is: in, is the node feature matrix of the lth layer, is the weight matrix of the graph convolutional layer, is the bias vector, is the activation function (ReLU), Contains self-loops, yes The degree matrix, H (0) is the initial input feature matrix of the network, whose dimension is N F; Aggregate the node features of the last layer of each time step to obtain a high-dimensional topological feature vector F (t) In STGNN, the high-dimensional topological feature vector contains the spatiotemporal feature representation of each node at each time step and the topological structure information between nodes; The deSNN network in the spatiotemporal dynamic fusion network model is used to calculate the high-dimensional topological feature vector F. (t) STGNN performs normalization and obtains F ’(t) STGNN, F ’(t) STGNN is represented as:

[0018] To ensure that the feature values ​​are within the appropriate range, define a threshold for each feature dimension:

[0019] Among them, μ is the mean value of the feature vector, σ is the standard deviation of the feature dimension, g is a coefficient used to adjust the position of the threshold relative to the mean and standard deviation, M is the dimension of the feature vector, Ω is used as the threshold of emission, and the emission process of the neuron is simulated. If the feature value exceeds the threshold, a emission event and the current timestamp are recorded. Each pulse not only has a value (0 for no emission or 1 for emission), but also a time stamp indicating its position in the time series. The emission events of all time steps are converted into a pulse sequence S(t)={(ti,1)|F′ (ti) STGNN ,i>Ω,}; Among them, the firing process of simulated neurons is: Traverse each time step and check whether there is a firing event. For each firing event, initialize V(0) to the resting potential V rest, the membrane potential is updated using the Euler method, and the formula is as follows:

[0020] Where V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt=t i+1 -t i Since it is the actual time interval between consecutive time steps, S(t i ) is the indicator function of the firing event (0 for no firing or 1 for firing), and for the time step without firing, the membrane potential remains unchanged; Collect the firing events of each neuron to ensure F (t) deSNN and F (t) STGNN performs temporal aggregation operations based on the consistency of time scale by calculating the time interval ISI between two emission events. i ={t i2 t i1 , t i3 t i2 , ...}, t ij It is a neuron i The time of the jth issuance of F. These data are further used to calculate the issuance frequency, mean, standard deviation, minimum and maximum statistical features to construct the feature vector F (t) deSNN =deSNN(S(t)); Based on the cross-validation iterative optimization process, the optimal time series feature vector F of the output graph data packet is final .

[0021] The cross-validation iterative optimization process outputs the optimal time series feature vector F of the graph data packet. final methods, including: F (t) deSNN As F (t) STGNN The input is used to randomly and evenly divide the data set into Q subsets of similar size; Define the MSE mean square error function to evaluate the performance of the spatiotemporal dynamic fusion network model on a given data set and perform a cross validation (CV) cycle (f (1, Q)), divide the data set into Q 1 training set and 1 validation set, design an iterative function F (t_k) STGNN =Iterate(F (t_k-1) STGNN ,STGNN,deSNN)k R, receive the current F (t_k-1) STGNN , STGNN and deSNN model parameters, and return the updated feature vector; Train the model on the training set and perform iterative update of the feature vector. The iterative update process is as follows: (0) STGNN As the input of STGNN, the intermediate feature representation F is obtained (k-1) STGNN , then, using the current F (k-1) STGNN As the input of deSNN, we get F (k-1) deSNN , use normalization to ensure the correct transfer of feature vectors, and then use F (k-1) deSNN Feedback to STGNN to update F (k) STGNN , for each iteration number k, execute the iterative function and collect the output F (k) STGNN； Evaluate the model performance on the validation set, record the performance indicators, and for each f, record the performance indicators. Calculate the average performance indicator CV=1 / K(∑ K f=1 Performancef), compare the average performance indicators under different iteration numbers, and select the iteration number k with the best performance best =argmax k CV(K), finally, the best k is determined in the entire training set best The model is trained end-to-end and the performance of the final model is evaluated on an independent test set to obtain a feature vector F that integrates spatiotemporal information and statistical features. final .

[0022] The method for the LSTM prediction model to output attention and fatigue prediction results includes: Load the optimal timing feature vector F final , for the optimal time series feature vector F final Perform data preprocessing to ensure F final The feature vectors are on the same scale, and the features most relevant to attention, fatigue, and blood oxygen are selected from them; Among them, F final It is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension of each time step, and the dimension of the input layer is N F, flatten the features of each time step into a part of the input vector, and obtain an input vector with dimension NF; Use an LSTM layer to process the data. Set the number of LSTM units contained in the LSTM layer to H. H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X. t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are the initial state zero vectors, and the LSTM layer at time step t is updated:

[0023] Among them, f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t 1,x t ] is the concatenation of the hidden state of the previous moment and the current input, i t is the output of the input gate, c t ' is the candidate cell state, c t is the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment; The hidden state h of the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which is used to predict the attention level, fatigue level and blood oxygen value respectively. The output is calculated as follows:

[0024] Among them, the dimension of the Wout weight matrix is ​​(3, H).

[0025] Compared with the prior art, the embodiments of the present application have the following beneficial effects: In the present invention, graph structured data packets are analyzed and processed based on a pre-built spatiotemporal dynamic fusion network model, and the spatiotemporal dynamic fusion network model integrates the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN, which can accurately evaluate the working status signal characteristics of high-risk workers in plateaus. The spatiotemporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven pulse neural network (deSNN) can convert them into pulse sequence features to more accurately characterize the characteristic values. Iterative optimization enables the two to work together, thereby improving the accuracy and reliability of the spatiotemporal dynamic fusion network model evaluation.

[0026] In the present invention, the LSTM prediction model is used to output the attention and fatigue prediction results. Based on the LSTM prediction model, time series data can be processed to mine attention, fatigue and blood oxygen information, determine whether the operator can continue to work, avoid safety accidents, and ensure the safety of high-risk operations in the plateau.

[0027] The present invention can fully capture and analyze the physiological and psychological state of the staff to accurately determine whether they are suitable to continue to perform high-risk tasks. The present invention aims to improve the accuracy and reliability of the assessment of the actual work ability of the staff through an innovative assessment technology, thereby ensuring the safety of the staff and the stability of the project. The head-mounted brain-body work ability assessment method based on the spatiotemporal dynamic fusion network not only integrates traditional physiological indicators, but also adds advanced physiological measurement technologies such as electroencephalogram (EEG) and photoplethysmogram (PPG) to fully capture the physiological and psychological state of the staff. Through these technologies, the attention, fatigue and blood oxygen level of the staff can be more accurately monitored. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the overall implementation process of the head-mounted brain-body task ability assessment method based on the spatiotemporal dynamic fusion network provided by the present invention.

[0029] Figure 2 A schematic diagram of a head-mounted wearable device is shown.

[0030] Figure 3 A schematic diagram showing the effect of a head-mounted wearable device being worn on a human body is shown.

[0031] Figure 4 A schematic diagram of the implementation flow of the physiological parameter signal preprocessing method is shown.

[0032] Figure 5 The present invention shows a schematic diagram of the implementation process of a method for converting a physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set.

[0033] Figure 6 It is a structural schematic diagram of the graph structure data packet provided by the present invention.

[0034] Figure 7 The present invention shows a schematic diagram of the implementation process of a method for analyzing and processing graph structured data packets based on a pre-built spatiotemporal dynamic fusion network model.

[0035] Figure 8 The optimal time series feature vector F of the output graph data packet is shown based on the cross-validation iterative optimization process. final Schematic diagram of the implementation flow of the method.

[0036] Fig. 9The schematic diagram of the architecture of the head-mounted brain-body performance assessment system based on the spatiotemporal dynamic fusion network is shown. DETAILED DESCRIPTION

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0038] The existing methods using a single physiological index cannot fully reflect the comprehensive state of an individual, especially in a changeable environment such as the plateau, and it is difficult to reveal the interaction and potential risks between multiple factors. To address the above problems, we proposed a head-mounted brain-body performance evaluation method based on a spatiotemporal dynamic fusion network. In short, when the method is implemented, the physiological parameter signals are first collected based on the wearable device, and the physiological parameter signals are converted into graph structure data packets. The graph structure data packets are analyzed and processed based on the pre-built spatiotemporal dynamic fusion network model, and then the optimal time series feature vector F is used to evaluate the performance of the head-mounted brain-body performance. final As input, the LSTM prediction model is executed, and the LSTM prediction model outputs the attention and fatigue prediction results. In the embodiment of the present invention, the graph structure data packet is analyzed and processed based on the pre-built spatiotemporal dynamic fusion network model, and the spatiotemporal dynamic fusion network model integrates the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN, which can accurately evaluate the working status signal characteristics of high-risk plateau workers. The spatiotemporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven pulse neural network (deSNN) can convert it into pulse sequence features to more accurately characterize the feature values. Iterative optimization makes the two work together, which improves the accuracy and reliability of the spatiotemporal dynamic fusion network model evaluation.

[0039] The embodiment of the present invention provides a head-mounted brain-body operation ability assessment method based on a spatiotemporal dynamic fusion network. Figure 1 The overall implementation process diagram of the head-mounted brain-body operation ability assessment method based on the spatiotemporal dynamic fusion network is shown. The head-mounted brain-body operation ability assessment method based on the spatiotemporal dynamic fusion network specifically includes: S10, collecting physiological parameter signals based on the wearable device; It should be noted that the physiological parameter signals include but are not limited to brain wave signals, photoplethysmogram signals, heart rate, blood oxygen, heart rate; wherein the wearable device is a head-mounted wearable device, Figure 2 A schematic diagram of a head-mounted wearable device is shown, and Figure 3 The figure shows the human wearing effect of the head-mounted wearable device. The head-mounted wearable device uses 5-channel electrodes for EEG collection and is designed with high input impedance to improve the EEG collection accuracy. In addition, it integrates 660nm and 880nm dual-band bio-optical sensors to collect dual-band PPG data. In order to meet the collection needs of different scenarios, it is equipped with a variety of electrodes, and users can adapt different electrodes according to different scenarios to achieve ideal detection precision and accuracy.

[0040] S20, loading the collected physiological parameter signal, preprocessing the physiological parameter signal, obtaining the preprocessed physiological parameter signal, and extracting features of the physiological parameter signal; S30, acquiring the physiological parameter signal after feature extraction, converting the physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set; S40, load the graph structure data packet set, analyze and process the graph structure data packet based on the pre-built spatiotemporal dynamic fusion network model, and output the optimal time series feature vector F of the graph data packet final , where the spatiotemporal dynamic fusion network model is a fusion of the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN. STGNN is a graph neural network used to process spatiotemporal data; S50, with the optimal time series feature vector F final As input, the LSTM prediction model is executed, and the LSTM prediction model outputs the attention and fatigue prediction results.

[0041] In the embodiment of the present invention, the graph structure data packet is analyzed and processed based on the pre-built spatio-temporal dynamic fusion network model, and the spatio-temporal dynamic fusion network model (Spatio-Temporal Dynamic Fusion Network, STDFN) integrates the spatio-temporal graph neural network STGNN and the dynamic event-driven spiking neural network deSNN (Dynamic Event-basedSpiking Neural Network, deSNN), which can accurately evaluate the working status signal characteristics of high-risk workers in plateaus. The spatio-temporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven spiking neural network (deSNN) can convert them into pulse sequence features to more accurately characterize the characteristic values. Iterative optimization enables the two to work together, thereby improving the accuracy and reliability of the spatio-temporal dynamic fusion network model evaluation.

[0042] The embodiment of the present invention provides a method for preprocessing a physiological parameter signal. Figure 4 The figure shows a schematic diagram of the implementation process of the physiological parameter signal preprocessing method, wherein the physiological parameter signal preprocessing method specifically includes: S101, collecting physiological parameter signals based on the wearable device, identifying the brain wave signal and the photoelectric volume pulse signal in the physiological parameter signal, the wearable device includes a data acquisition module, and the data acquisition module is composed of a high-precision ADC module and a MAX86150 module; Among them, the physiological parameter signals collected by wearable devices are collected through a head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network. Fig. 9 The schematic diagram of the architecture of a head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network is shown. The head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network includes: Data acquisition module, which collects physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module; Among them, the high-precision ADC module: consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter and a front-end operational amplifier circuit. The sampling timing and accuracy are independently controllable. The brain wave signal enters the low-noise differential input and output amplifier from the sampling electrode, enters the ADC integrator after differential amplification, and outputs the final AD conversion result after filtering the specific frequency; MAX86150 module: Biosensor module with integrated photoplethysmogram (PPG) and electrocardiogram (ECG), including internal LED; MCU main control module, the MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth; Power management module: The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the battery charging and discharging functions. The power management module consists of power supply, charging management, voltage regulated output, and power protection.

[0043] It should be noted that the high-precision ADC module outputs the final AD conversion result after filtering the specific frequency. The functional modules of each part are as follows: (1) Low-noise differential input and output amplifiers amplify the original collected voltage in one stage, with noise as low as 0.98uVpp.

[0044] (2) The amplified voltage signal is integrated and amplified by the integrator at the rear. By changing the integration time, the sampling rate is directly affected, achieving a segmented sampling rate of 250SPS (samples per second) to 16KSPS.

[0045] (3) Digital signal filter: The digital filter receives the output signal of the integrating amplifier and balances the resolution and sampling rate through adjustment and extraction in order to obtain high-quality data sampling. A third-order sinc filter is integrated on each channel.

[0046] In this embodiment, the MAX86150 module is a biosensor module integrating photoplethysmogram (PPG) and electrocardiogram (ECG), including internal LED, photodetector and low-noise electronics with ambient light suppression, operating voltage 1.8V, and a separate voltage for internal LED, operating temperature -40℃ to +85℃, MAX86150 size 3.3mmx5.6mmx1.3mm, is a 22-pin optical module, can be used in wearable devices. The sampling rate of MAX86150 data sampling PPG ranges from 10sps to 3200sps, commonly used are 400sps, 800sps, 1600sps, including 19-bit ADC and proprietary discrete time filter, can suppress 50Hz / 60Hz interference and slow moving participating environmental noise.

[0047] MCU main control module, the MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth; In this embodiment, the MCU main control module adopts an ARM Cortex-M4F 32-bit architecture microprocessor, with an RTOS (Real-time Operating System) operating system, uses embedded technology to minimize power consumption, and switches between sleep mode and startup mode to extend the use time. It integrates Bluetooth communication, WiFi communication, and 4G wireless communication, supports multiple wireless communication methods, has a wealth of peripheral interfaces, and changes different configurations according to different usage scenarios. The highly integrated BGA (Ball Grid Array) package that only occupies a very small volume is suitable for wearable devices.

[0048] Power management module: The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the battery charging and discharging functions. The power management module consists of power supply, charging management, voltage regulated output, and power protection.

[0049] Among them, power supply: lithium battery is used to power the entire system, providing a stable voltage of 3.7V, and 480maH of electricity can support the system to run for a long time.

[0050] Charging management: Charging management implements functions such as positive and reverse polarity protection of lithium batteries, programmable charging current up to 500mA, indication of charging status, and full charging of the battery within 1 hour.

[0051] Regulated output: LDO converts the lithium battery 3.7V into 3.3V and 1.8V and provides it to MAX86150. The power ripple can be controlled within 8uVrms, and the voltage changes with temperature within 0.3%, providing a maximum power consumption of 2w.

[0052] Power protection: The power protection circuit prevents the power supply from being overloaded, short-circuited, over-voltage, under-voltage, over-current, and other phenomena that seriously endanger the safety of battery use during use.

[0053] It should be noted that when the system is working, the power management module is responsible for powering the entire hardware circuit and managing the battery charging and discharging functions; the high-precision ADC module is responsible for collecting tiny EEG signals, and the MAX86150 module is responsible for collecting photoplethysmogram (PPG) signals; the main control transmits the data collected by the data acquisition module to the host computer via Bluetooth. The power module provides the energy consumption required to support the operation of the entire system. During working hours, the high-precision ADC conversion module converts the collected voltage signal into a digital quantity and transmits it to the main control according to the agreed protocol. The main control uses wireless communication to package the digital quantity to the host computer.

[0054] S102, acquiring brain wave signals and preprocessing the brain wave signals; S103, obtaining a photoplethysmogram signal, and preprocessing the photoplethysmogram signal.

[0055] In this embodiment, the method for preprocessing brain wave signals includes: Wearable devices are used to extract five types of EEG signals, namely, δ, θ, α, β and γ, based on 5-channel electrodes; Load the collected brain wave signal, and use Butterworth filter to remove low-frequency noise and high-frequency noise in the brain wave signal. The low-frequency noise in the brain wave signal can be a power frequency interference signal, and the high-frequency noise can be an electromagnetic interference signal. The low-frequency noise and high-frequency noise processing methods in the brain wave signal include standardization (the mean of each channel is 0 and the variance is 1), principal component analysis dimensionality reduction, and independent component analysis separation of independent source signal methods. The low-frequency noise and high-frequency noise processing in the brain wave signal is used to remove motion artifacts and electrooculographic artifacts; Obtain the EEG signal after removing low-frequency noise and high-frequency noise, and obtain multi-channel time series data. The time series at time point t are {θ t} T , {δt} T , {αt} T , {bt} T and {γt} T , T is the total number of time points.

[0056] In this embodiment, the method for preprocessing the photoplethysmogram signal includes: The dual-band photoplethysmogram signal is collected by a wearable device based on a dual-band bio-optical sensor, thereby adapting to its lower sampling rate and slower changing characteristics; Obtain the photoplethysmogram signal, use a low-pass Butterworth filter to remove high-frequency noise in the photoplethysmogram signal, and retain the low-frequency hemodynamic information PPG low (t), then the short-term fluctuations,noise are removed by mean filtering and the spike noise is removed by median filtering; The short-term fluctuations and noise are removed by mean filtering, and the spike noise is removed by median filtering to obtain single-channel time series data {PPGt} T .

[0057] The embodiment of the present invention provides a method for extracting features of a physiological parameter signal, and the method for extracting features of a physiological parameter signal specifically includes: S201, loading the pre-processed EEG signal multi-channel time series data and photoplethysmogram signal single-channel time series data {PPGt} T ; S202, respectively, the multi-channel time series data of the EEG signal and the single-channel time series data of the photoplethysmogram signal {PPGt} T Perform time domain analysis and calculate the basic statistical characteristics of each channel signal; Among them, the basic statistical characteristics include mean, variance, maximum value, and minimum value. The function expressions of mean, variance, maximum value, and minimum value are:

[0058] in, is the EEG mean, is the variance, Maximum value, Minimum value, is the data point at time t and channel i, N is the number of channels 5; S203, using fast Fourier transform to analyze the multi-channel time series data of the EEG signal and the single-channel time series data of the photoplethysmogram signal {PPGt} T The spectrum characteristics are as follows:

[0059] S204, using Morlet wavelet transform to obtain low-frequency to high-frequency time-frequency features, and the following formula is used to obtain the low-frequency to high-frequency time-frequency features:

[0060] in, is a wavelet function with time t and frequency f.

[0061] In this embodiment, the features extracted by the above method such as PPG t :[PPGlow_t 、PPG medium_t 、PPG high_t 、PPG mt 、PPG st 2 、PPG max_t 、PPG min_t 、PPG FFT_t ], α t :[α low_t , α medium_t , α high_t , α mt , α st 2 , α max_t , α min_t , α FFT_t ], β t :[β low_t , β medium_t , β high_t , βμ t ,βσ t 2 , β max_t , β min_t , β FFT_t ], γ t :[γ low_t , γ medium_t , γ high_t , γμ t ,γσ t 2 , γ max_t , γ min_t , γ FFT_t ]wait.

[0062] The embodiment of the present invention provides a method for converting a physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set. Figure 5 The present invention shows a schematic diagram of the implementation process of a method for converting a physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set. The method for converting a physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set specifically includes: S301, classifying nodes into different types according to the types of physiological parameter signals, and implementing node definition and initialization. For example, different frequency bands (δ, θ, α, β, γ) of EEG signals and PPG signals are considered as different types of nodes, thereby reflecting different types of physiological signals or signal characteristics; S302, using the Euclidean distance to measure the differences between different nodes, normalizing the node distances, and completing the edge construction. For two nodes u and v, the Euclidean distance d uv The calculation formula is:

[0063] in, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t; In the embodiment of the present invention, in order to convert the distance to the interval [0, 1], the node distance is normalized using the following formula:

[0064] Among them, max(d) and min(d) are the minimum and maximum values ​​of the Euclidean distances of all node pairs, so that smaller distances (i.e., more similar nodes) correspond to larger adjacency matrix values; S303, based on the relationship between nodes at each time t, construct N The matrix X of F, where N is the number of nodes, F is the feature dimension of each node, and each row represents the feature vector of a node. Then, construct N The adjacency matrix of N uses the normalized value of the Euclidean distance. The element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, the node feature X and the adjacency matrix A are combined to form a complete graph structure data packet, represented as (X, A, t). The graph structure data packet is ready to be input into the spatiotemporal dynamic fusion network model. Generate a graph structure data packet view at a specific time t, such as Figure 6 shown.

[0065] The embodiment of the present invention provides a method for analyzing and processing graph structure data packets based on a pre-built spatiotemporal dynamic fusion network model. Figure 7 The schematic diagram of the implementation process of the method for analyzing and processing graph structure packets based on the pre-built spatiotemporal dynamic fusion network model is shown. The method for analyzing and processing graph structure packets based on the pre-built spatiotemporal dynamic fusion network model specifically includes: S401, load the graph structure data packet, input the graph structure data packet into the spatiotemporal dynamic fusion network model, and the STGNN network in the spatiotemporal dynamic fusion network model updates the feature vector of each node according to the graph convolution layer update formula to effectively capture the spatiotemporal features, wherein the update formula of the graph convolution layer at each time step is:

[0066] in, is the node feature matrix of the lth layer, is the weight matrix of the graph convolutional layer, is the bias vector, is the activation function (ReLU), Contains self-loops, yes The degree matrix, H (0)is the initial input feature matrix of the network, whose dimension is N F; S402, aggregate the node features of the last layer of each time step to obtain a high-dimensional topological feature vector F (t) In STGNN, the high-dimensional topological feature vector contains the spatiotemporal feature representation of each node (representing different EEG and PPG signals) at each time step and the topological structure information between nodes. For the high-dimensional feature vector of time step t and each node (representing different EEG and PPG signals), it reflects the spatiotemporal relationship between nodes and the dynamic characteristics of each node; S403, deSNN network in spatiotemporal dynamic fusion network model for high-dimensional topological feature vector F (t) STGNN performs normalization and obtains F ’(t) STGNN, F ’(t) STGNN is represented as:

[0067] To ensure that the feature values ​​are within the appropriate range, define a threshold for each feature dimension:

[0068] Among them, μ is the mean value of the feature vector, σ is the standard deviation of the feature dimension, g is a coefficient used to adjust the position of the threshold relative to the mean and standard deviation, M is the dimension of the feature vector, Ω is used as the threshold of emission, and the emission process of the neuron is simulated. If the feature value exceeds the threshold, a emission event and the current timestamp are recorded. Each pulse not only has a value (0 for no emission or 1 for emission), but also a time stamp indicating its position in the time series. The emission events of all time steps are converted into a pulse sequence S(t)={(ti,1)|F′ (ti) STGNN ,i>Ω,}; Among them, the firing process of simulated neurons is: Traverse each time step and check whether there is a firing event. For each firing event, initialize V(0) to the resting potential V rest , the membrane potential is updated using the Euler method, and the formula is as follows:

[0069] Where V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt=t i+1 -t i From is the actual time interval between consecutive time steps, S(t i) is the indicator function of the firing event (0 for no firing or 1 for firing), and for the time step without firing, the membrane potential remains unchanged; Collect the firing events of each neuron to ensure F (t) deSNN and F (t) STGNN performs temporal aggregation operations based on the consistency of time scale by calculating the time interval ISI between two emission events. i ={t i2 t i1 , t i3 t i2 , ...}, t ij It is a neuron i The time of the jth issuance of F. These data are further used to calculate the issuance frequency, mean, standard deviation, minimum and maximum statistical features to construct the feature vector F (t) deSNN =deSNN(S(t)); S404, based on the cross-validation iterative optimization process, output the optimal time series feature vector F of the graph data packet final .

[0070] In this embodiment, the spatiotemporal dynamic fusion network model STDFN is an advanced integrated framework that integrates STGNN (spatial-temporal graph neural network) and deSNN (dynamic event-driven spiking neural network), which is specifically used to process and analyze physiological data, aiming to extract features related to the attention level and fatigue of high-risk operators in plateaus. STGNN is a graph neural network specially designed to process spatiotemporal data. It can capture the spatiotemporal dependency and topological structure information in the data. STGNN aggregates the neighborhood information of nodes through graph convolution layers, thereby learning the spatiotemporal feature representation of nodes.

[0071] The embodiment of the present invention provides an optimal time series feature vector F of the output graph data packet based on a cross-validation iterative optimization process. final method, Figure 8 The optimal time series feature vector F of the output graph data packet is shown based on the cross-validation iterative optimization process. final Schematic diagram of the implementation process of the method, which outputs the optimal time series feature vector F of the graph data packet based on the cross-validation iterative optimization process final The methods include: S4041, F (t) deSNN As F (t) STGNN The data set is randomly and evenly divided into Q subsets of similar size. In this embodiment, Q can be 5; S4042, define the MSE mean square error function to evaluate the performance of the spatiotemporal dynamic fusion network model on a given data set and perform a cross validation (CV) cycle (f (1, Q)), divide the data set into Q 1 training set and 1 validation set, design an iterative function F (t_k) STGNN =Iterate(F (t_k-1) STGNN ,STGNN,deSNN)k R, receive the current F (t_k-1) STGNN , STGNN and deSNN model parameters, and return the updated feature vector; S4043, train the model on the training set and perform iterative update of the feature vector. The iterative update process is as follows: (0) STGNN As the input of STGNN, the intermediate feature representation F is obtained (k-1) STGNN , then, using the current F (k-1) STGNN As the input of deSNN, we get F (k-1) deSNN , use normalization to ensure the correct transfer of feature vectors, and then use F (k-1) deSNN Feedback to STGNN to update F (k) STGNN , for each iteration number k, execute the iterative function and collect the output F (k) STGNN； S4044, evaluate the model performance on the validation set, record the performance indicators, for each f, record the performance indicators, calculate the average performance indicator CV=1 / K(∑ K f=1 Performancef), compare the average performance indicators under different iteration numbers, and select the iteration number k with the best performance best =argmax k CV(K), finally, the best k is determined in the entire training set best The model is trained end-to-end and the performance of the final model is evaluated on an independent test set to obtain a feature vector F that integrates spatiotemporal information and statistical features. final .

[0072] It should be noted that the model performance is evaluated on the validation set, and the performance indicators are recorded as Performance = Evaluate (STDFN, Validation Set).

[0073] An embodiment of the present invention provides a method for an LSTM prediction model to output attention and fatigue prediction results. The method for an LSTM prediction model to output attention and fatigue prediction results specifically includes: S501, load the optimal time series feature vector F final , for the optimal time series feature vector F final Perform data preprocessing to ensure F final The feature vectors are on the same scale, and the features most relevant to attention, fatigue, and blood oxygen are selected from them; Among them, F final It is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension of each time step, and the dimension of the input layer is N F, flatten the features of each time step into a part of the input vector, and obtain an input vector with dimension NF; S502, using an LSTM layer to process data, setting the number of LSTM units contained in the LSTM layer to H, where H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are the initial state zero vectors, and the LSTM layer at time step t is updated:

[0074] Among them, f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t 1,x t ] is the concatenation of the hidden state of the previous moment and the current input, i t is the output of the input gate, c t ' is the candidate cell state, c t is the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment; S503, the hidden state h of the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which is used to predict the attention level, fatigue level and blood oxygen value respectively. The output is calculated as follows:

[0075] Among them, the dimension of the Wout weight matrix is ​​(3, H).

[0076] In this embodiment, the mean square error is used to design a loss function when training the LSTM prediction model. Since three values ​​(attention level, fatigue level, and blood oxygen level) are predicted, we design a separate loss function for each value and then combine them:

[0077] Among them, y attention，I ,y fatigue，i and SpO2，i is the true value, y^ attention，i ,y^ fatigue，i and y^ SpO2，i is the predicted value. Then, the model is trained with the Adam algorithm training set, and the validation set is used to adjust hyperparameters and prevent overfitting. The early stopping strategy is also used to avoid overfitting. Finally, the trained model is deployed to the cloud for real-time monitoring and prediction of the staff's attention level, fatigue, and blood oxygen level.

[0078] In the embodiment of the present invention, an LSTM prediction model is used to output attention and fatigue prediction results. Based on the LSTM prediction model, time series data can be processed to mine attention, fatigue and blood oxygen information to determine whether the operator can continue to work, avoid safety accidents, and ensure the safety of high-risk operations in the plateau.

[0079] Specifically, the head-mounted brain-body operation ability assessment of the graph neural network is real-time. First, the EEG and PPG signals collected by the wearable device are filtered and denoised by Butterworth filter (including standardization, PCA dimension reduction, and ICA separation), and the PPG signal is processed by low-pass Butterworth filter and mean and median filtering, and then the time domain, frequency domain and wavelet transform feature extraction are performed; Then, based on physiological parameters, EEG and PPG signals of different frequency bands are used as nodes, and edges are constructed by calculating and normalizing the Euclidean distance. Based on the node relationship, matrix X and adjacency matrix A are constructed to form a graph data packet. Then, the graph data packet is input into STDFN, where STGNN aggregates node neighborhood information through the graph convolution layer to update the node feature vector and aggregates and normalizes it to obtain a high-dimensional topological feature vector. Then it is input into deSNN, which simulates the neuron firing model to convert it into a pulse sequence feature vector, and finds the optimal feature vector through iterative optimization; Finally, the feature vector is flattened and input into the LSTM prediction model. The LSTM updates its state according to the calculation of the forget gate, input gate, cell state update and output gate. The output layer predicts the attention level, fatigue and blood oxygen value according to the hidden state at the last moment. During training, the mean square error is used as the loss function combined with the Adam algorithm to train the model, and the parameters and early stopping strategy are adjusted through the validation set to prevent overfitting.

[0080] In an embodiment of the present invention, it is proposed to use a head-mounted device to monitor and evaluate the working status of high-risk workers in a plateau environment in real time, to ensure the timeliness, efficiency and accuracy of the evaluation results. In the head-mounted brain-body work ability evaluation system based on the spatiotemporal dynamic fusion network, the hardware adopts low-power design, flexible printed circuit boards (FPCs) and lightweight materials. The device can reduce the burden on the wearer while ensuring long-term battery life. The device is equipped with a variety of electrodes, and users can use appropriate electrodes according to different environments to meet the needs of various scenarios.

[0081] In summary, the present invention provides a head-mounted brain-body work ability assessment method based on a spatiotemporal dynamic fusion network. In an embodiment of the present invention, the graph structure data packet is analyzed and processed based on a pre-built spatiotemporal dynamic fusion network model, and the spatiotemporal dynamic fusion network model integrates the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN, which can accurately assess the working status signal characteristics of high-risk workers in plateaus. The spatiotemporal graph neural network (STGNN) can capture complex relationships and extract high-dimensional topological features, and the dynamic event-driven pulse neural network (deSNN) can convert them into pulse sequence features to more accurately characterize the feature values. Iterative optimization enables the two to work together, thereby improving the accuracy and reliability of the spatiotemporal dynamic fusion network model evaluation.

[0082] The present invention can fully capture and analyze the physiological and psychological state of the staff to accurately determine whether they are suitable to continue to perform high-risk tasks. The present invention aims to improve the accuracy and reliability of the assessment of the actual work ability of the staff through an innovative assessment technology, thereby ensuring the safety of the staff and the stability of the project. The head-mounted brain-body work ability assessment method based on the spatiotemporal dynamic fusion network not only integrates traditional physiological indicators, but also adds advanced physiological measurement technologies such as electroencephalogram (EEG) and photoplethysmogram (PPG) to fully capture the physiological and psychological state of the staff. Through these technologies, the attention, fatigue and blood oxygen level of the staff can be more accurately monitored.

[0083] It should be noted that, for the above-mentioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be in the form of telecommunication or other forms.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. A head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network, characterized in that: The head-mounted brain-body operation ability assessment method based on the spatiotemporal dynamic fusion network includes: Collecting physiological parameter signals based on wearable devices, wherein the physiological parameter signals include brain wave signals, photoplethysmogram signals, heart rate, blood oxygen, and heart rate; Loading the collected physiological parameter signals, preprocessing the physiological parameter signals, obtaining the preprocessed physiological parameter signals, and extracting features of the physiological parameter signals; Acquire the physiological parameter signal after feature extraction, convert the physiological parameter signal into a graph structure data packet, and obtain a graph structure data packet set; Load the graph structure data packet set, analyze and process the graph structure data packet based on the pre-built spatiotemporal dynamic fusion network model, and output the optimal time series feature vector F of the graph data packet final , where the spatiotemporal dynamic fusion network model is a fusion of the spatiotemporal graph neural network STGNN and the dynamic event-driven pulse neural network deSNN. STGNN is a graph neural network used to process spatiotemporal data; With the optimal time series feature vector F final As input, the LSTM prediction model is executed, and the LSTM prediction model outputs the attention and fatigue prediction results.

2. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 1 is characterized by: The method for preprocessing a physiological parameter signal comprises: Based on the collection of physiological parameter signals by wearable devices, the brain wave signals and photoplethysmogram signals in the physiological parameter signals are identified; Wherein, the physiological parameter signals collected by the wearable device are collected by a head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network, and the head-mounted brain-body operation ability assessment system based on a spatiotemporal dynamic fusion network includes: Data acquisition module, which collects physiological parameter signals based on wearable devices. The data acquisition module consists of a high-precision ADC module and a MAX86150 module; Among them, the high-precision ADC module: consists of a low-noise differential input and output amplifier, an integrating amplifier, a digital filter and a front-end operational amplifier circuit. The sampling timing and accuracy are independently controllable. The brain wave signal enters the low-noise differential input and output amplifier from the sampling electrode, enters the ADC integrator after differential amplification, and outputs the final AD conversion result after filtering the specific frequency; MAX86150 module: Biosensor module with integrated photoplethysmogram (PPG) and electrocardiogram (ECG), including internal LED; MCU main control module, the MCU main control module is used to transmit the data collected by the data acquisition module to the host computer via Bluetooth; Power management module: The power management module is used to supply power to the data acquisition module and the MCU main control module and manage the charging and discharging functions of the battery. The power management module consists of power supply, charging management, voltage stabilization output, and power protection; Acquire brain wave signals and pre-process the brain wave signals; A photoplethysmogram signal is acquired and preprocessed.

3. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 2 is characterized by: The method for preprocessing brain wave signals comprises: Wearable devices are used to extract five types of EEG signals, namely, δ, θ, α, β and γ, based on 5-channel electrodes; Loading the collected brain wave signal, using Butterworth filter to remove low-frequency noise and high-frequency noise in the brain wave signal, wherein the low-frequency noise and high-frequency noise processing methods in the brain wave signal include standardization, principal component analysis dimensionality reduction, and independent component analysis to separate independent source signal methods; Obtain the EEG signal after removing low-frequency noise and high-frequency noise, and obtain multi-channel time series data. The time series at time point t are {θ t } T , {δt} T , {αt} T , {bt} T and {γt} T , T is the total number of time points.

4. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 2, characterized in that: The method for preprocessing a photoplethysmogram signal comprises: Collecting dual-band photoplethysmogram signals using a wearable device based on a dual-band bio-optical sensor; Obtain the photoplethysmogram signal, use a low-pass Butterworth filter to remove high-frequency noise in the photoplethysmogram signal, and retain the low-frequency hemodynamic information PPG low (t); The short-term fluctuations and noise are removed by mean filtering, and the spike noise is removed by median filtering to obtain single-channel time series data {PPGt} T .

5. The head-mounted brain-body performance evaluation method based on a spatiotemporal dynamic fusion network as described in any one of claims 2 to 4, characterized in that: The method for extracting physiological parameter signal features comprises: Load the preprocessed EEG signal multi-channel time series data and photoplethysmogram signal single-channel time series data {PPGt} T ; The multi-channel time series data of EEG signals and the single-channel time series data of photoplethysmogram signals {PPGt} T Perform time domain analysis and calculate the basic statistical characteristics of each channel signal; Among them, the basic statistical characteristics include mean, variance, maximum value, and minimum value. The function expressions of mean, variance, maximum value, and minimum value are: in, is the EEG mean, is the variance, Maximum value, Minimum value, is the data point at time t and channel i, N is the number of channels 5; Use Fast Fourier Transform to analyze multi-channel time series data of EEG signals and single-channel time series data of photoplethysmogram signals {PPGt} T The spectrum characteristics are as follows: Use Morlet wavelet transform to obtain low-frequency to high-frequency time-frequency features. The following formula is used to obtain low-frequency to high-frequency time-frequency features: in, is a wavelet function with time t and frequency f.

6. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 5, characterized in that: The method of converting the physiological parameter signal into a graph structure data packet to obtain a graph structure data packet set comprises: According to the type of physiological parameter signal, the nodes are classified into different types to achieve node definition and initialization; The Euclidean distance is used to measure the differences between different nodes, and the node distance is normalized to complete the edge construction. For the two nodes u and v, the Euclidean distance d uv The calculation formula is: in, is the eigenvalue of node u at time t, is the eigenvalue of node v at time t; The node distance is normalized using the following formula: Among them, max(d) and min(d) are the minimum and maximum values ​​of the Euclidean distances of all node pairs; Based on the relationship between nodes at each time t, construct N The matrix X of F, where N is the number of nodes, F is the feature dimension of each node, and each row represents the feature vector of a node. Then, construct N The adjacency matrix of N uses the normalized value of the Euclidean distance. The element Auv in the adjacency matrix A represents the connection strength between node u and node v. Finally, the node feature X and the adjacency matrix A are combined to form a complete graph structure data packet, which is represented as (X, A, t).

7. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 1, characterized in that: The method for analyzing and processing graph structure data packets based on the pre-built spatiotemporal dynamic fusion network model specifically includes: Load the graph structure data package and input the graph structure data package into the spatiotemporal dynamic fusion network model. The STGNN network in the spatiotemporal dynamic fusion network model updates the feature vector of each node according to the graph convolution layer update formula to effectively capture the spatiotemporal features. The update formula of the graph convolution layer at each time step is: in, is the node feature matrix of the lth layer, is the weight matrix of the graph convolutional layer, is the bias vector, is the activation function (ReLU), Contains self-loops, yes The degree matrix, H (0) is the initial input feature matrix of the network, whose dimension is N F; Aggregate the node features of the last layer of each time step to obtain a high-dimensional topological feature vector F (t) In STGNN, the high-dimensional topological feature vector contains the spatiotemporal feature representation of each node at each time step and the topological structure information between nodes; The deSNN network in the spatiotemporal dynamic fusion network model is used to calculate the high-dimensional topological feature vector F. (t) STGNN performs normalization and obtains F ’(t) STGNN, F ’(t) STGNN is represented as: To ensure that the feature values ​​are within the appropriate range, define a threshold for each feature dimension: Among them, μ is the mean value of the feature vector, σ is the standard deviation of the feature dimension, g is a coefficient used to adjust the position of the threshold relative to the mean and standard deviation, M is the dimension of the feature vector, Ω is used as the threshold of emission, and the emission process of the neuron is simulated. If the feature value exceeds the threshold, a emission event and the current timestamp are recorded. Each pulse not only has a value (0 for no emission or 1 for emission), but also a time stamp indicating its position in the time series. The emission events of all time steps are converted into a pulse sequence S(t)={(ti,1)|F′ (ti) STGNN ,i>Ω,}; Among them, the firing process of simulated neurons is: Traverse each time step and check whether there is a firing event. For each firing event, initialize V(0) to the resting potential V rest , the membrane potential is updated using the Euler method, and the formula is as follows: Where V(t) is the membrane potential at the current time step, V(t i+1 ) is the membrane potential at the next time step, Δt=t i+1 -t i Since it is the actual time interval between consecutive time steps, S(t i ) is the indicator function of the firing event (0 for no firing or 1 for firing), and for the time step without firing, the membrane potential remains unchanged; Collect the firing events of each neuron to ensure F (t) deSNN and F (t) STGNN performs temporal aggregation operations based on the consistency of time scale by calculating the time interval ISI between two emission events. i ={t i2 t i1 , t i3 t i2 , ...}, t ij It is a neuron i The time of the jth issuance of F. These data are further used to calculate the issuance frequency, mean, standard deviation, minimum and maximum statistical features to construct the feature vector F (t) deSNN =deSNN(S(t)); Based on the cross-validation iterative optimization process, the optimal time series feature vector F of the output graph data packet is final .

8. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 7, characterized in that: The cross-validation iterative optimization process outputs the optimal time series feature vector F of the graph data packet. final methods, including: F (t) deSNN As F (t) STGNN The input is used to randomly and evenly divide the data set into Q subsets of similar size; Define the MSE mean square error function to evaluate the performance of the spatiotemporal dynamic fusion network model on a given data set, perform a cross-validation (CV) cycle process (f∈1, Q)), and divide the data set into Q 1 training set and 1 validation set, design an iterative function F (t_k) STGNN =Iterate(F (t_k-1) STGNN ,STGNN,deSNN)k∈R, receiving the current F (t_k-1) STGNN , STGNN and deSNN model parameters, and return the updated feature vector; Train the model on the training set and perform iterative update of the feature vector. The iterative update process is as follows: (0) STGNN As the input of STGNN, the intermediate feature representation F is obtained (k-1) STGNN , then, using the current F (k-1) STGNN As the input of deSNN, we get F (k -1) deSNN , normalization is used to ensure the correct transmission of feature vectors, and then F (k-1) deSNN Feedback to STGNN to update F (k) STGNN , for each iteration number k, execute the iterative function and collect the output F (k) STGNN； Evaluate the model performance on the validation set, record the performance indicators, and for each f, record the performance indicators. Calculate the average performance indicator CV=1 / K(∑ K f=1 Performancef), compare the average performance indicators under different iteration numbers, and select the iteration number k with the best performance best =argmax k CV(K), finally, in the entire training set and the best k determined best The model is trained end-to-end and the performance of the final model is evaluated on an independent test set to obtain a feature vector F that integrates spatiotemporal information and statistical features. final .

9. The head-mounted brain-body operation ability assessment method based on spatiotemporal dynamic fusion network as claimed in claim 8, characterized in that: The method for the LSTM prediction model to output attention and fatigue prediction results includes: Load the optimal timing feature vector F final , for the optimal time series feature vector F final Perform data preprocessing to ensure F final The feature vectors are on the same scale, and the features most relevant to attention, fatigue, and blood oxygen are selected from them; Among them, F final It is a T N F matrix, T is the number of time steps, N is the number of physiological data at each time step, F is the feature dimension of each time step, and the dimension of the input layer is N F, flatten the features of each time step into a part of the input vector, and obtain an input vector with dimension NF; Use an LSTM layer to process the data. Set the number of LSTM units contained in the LSTM layer to H. H is a hyperparameter. At each time step t, the LSTM unit receives the flattened input vector X. t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , where h0 and c0 are the initial state zero vectors, and the LSTM layer at time step t is updated: Among them, f t represents the output of the forget gate, W if is the weight matrix of the forget gate, b f represents the bias vector [h t-1 , x t ] is the concatenation of the hidden state of the previous moment and the current input, i t is the output of the input gate, c t ' is the candidate cell state, c t is the updated cell state, o t is the output of the output gate, h t is the hidden state at the current moment; The hidden state h of the last time step after being processed by the LSTM layer T The dimension of the output layer is 3, which is used to predict the attention level, fatigue level and blood oxygen value respectively. The output is calculated as follows: Among them, the dimension of the Wout weight matrix is ​​(3, H).

Images