Electrophysiological and hemodynamic multimodal intelligent wearable system and data evaluation method

Abstract

The present invention discloses an electrophysiological and hemodynamic multimodal intelligent wearable system and data evaluation method, which belongs to the field of biomedical engineering technology. The system includes an optical module, a flexible and stretchable electrical signal module, and a central module. The optical module is used to collect hemodynamic signals by combining near-infrared light signals with hemodynamics, including oxygenated hemoglobin change signals, deoxygenated hemoglobin change signals, and pulse wave signals. The flexible and stretchable electrical signal module collects electrical signals from various parts through electrodes, including brain electroencephalogram signals, myoelectric signals, electrocardiogram signals, and electrooculogram signals. The central module is used to digitize and preprocess the hemodynamic signals and electrical signals from various parts, and then fuse the multimodal data, evaluate the data quality, and predict the dominant external events that affect the data quality. At the same time, the data quality and the dominant external events are output to the terminal to feedback and adjust the acquisition parameters of the optical module and the flexible and stretchable electrical signal module, thereby significantly improving the effectiveness and practicality of monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical engineering technology, and specifically relates to an electrophysiological and hemodynamic multimodal intelligent wearable system and a data evaluation method. Background Art

[0002] Traditional physiological signal detection methods typically detect a single signal, such as electroencephalography (EEG), electromyography (EMG), electrooculography (EOG), functional near-infrared MRI (fNIRS), and pulse wave pulsography. These detection methods have technical limitations and can only capture a single physiological signal. However, human activity involves changes at multiple levels. Therefore, to fully explore human functional activities, more diverse physiological multimodal detection technologies are needed.

[0003] With the continuous advancement of various physiological monitoring technologies, multimodal monitoring techniques have been developed in recent years, combining multiple technologies to assess physiological function. Among these, multimodal human monitoring methods, combining electroencephalography (EEG), electromyography (EMG), electrooculography (EOG), functional near-infrared cerebral isotope scanning (fNIRS), and pulse pulse prosthesis (PPG), combine the strengths of each, while overcoming their shortcomings. EEG offers the advantage of good temporal resolution, but its spatial resolution is limited; fNIRS offers better spatial resolution but poorer temporal resolution. Simultaneously, these signals from different dimensions provide complementary electrophysiological and hemodynamic information related to human activity. Therefore, multimodal monitoring methods based on electrophysiology and hemodynamics can address the shortcomings of individual monitoring technologies and complement their strengths. These methods can be used in a variety of applications, such as neurovascular coupling, motion monitoring, and cognitive assessment. They can help researchers gain a more systematic and in-depth understanding of the neural processes and functional mechanisms involved in cognitive tasks.

[0004] In the existing technology, as disclosed in the Chinese patent application with publication number CN112932474A, there is a rehabilitation training system based on cerebral blood oxygen and electromyographic signals, which includes: a synchronous information acquisition module, a processing and analysis module, and a feedback module, wherein: the synchronous information acquisition module is used to acquire the trainee's near-infrared cerebral blood oxygen and electromyographic signals; the processing and analysis module is used to process and analyze the cerebral blood oxygen signals and electromyographic signals synchronously acquired from the synchronous information acquisition module; the feedback module is used to determine whether the trainee's training exceeds the normal standard based on the cerebral blood oxygen signals and electromyographic signals processed and analyzed by the processing and analysis module, and if so, adjust the parameters of the training equipment. The above patent application uses photoelectric signal sensors and electrical signal sensors to detect signals, but the patent application simply detects cerebral blood oxygen signals and electromyographic signals, and does not expand to the field of multimodal detection of the whole body. The detected signals are not digitized and pre-processed at the acquisition end before being transmitted to the data processing system, resulting in large noise in the processed data. There is also no real-time noise detection of the collected data and feedback to the user to adjust the wearing of the device. Summary of the Invention

[0005] In view of the above, the purpose of the present invention is to provide an electrophysiological and hemodynamic multimodal intelligent wearable system and data evaluation method, which can process multiple physiological signals at the same time and obtain data with less noise, so as to obtain EEG signals, EMG signals, EO signals, ECG signals, blood oxygen signals and pulse wave signals more accurately. By extracting and fusing features from multimodal heterogeneous and heterogeneous data through algorithms, a comprehensive and integrated evaluation of the quality of multimodal signals is achieved, and at the same time, the dominant external events that affect the data quality are accurately marked, thereby improving the reliability of the data. The multimodal and multi-parameter closed-loop feedback module analyzes the data output results, evaluates the data acquisition quality, intelligently optimizes the equipment acquisition parameters, and intelligently recommends users to adjust the acquisition site, significantly improving the effectiveness and practicality of monitoring.

[0006] To achieve the above-mentioned purpose of the invention, an embodiment provides an electrophysiological and hemodynamic multimodal intelligent wearable system, comprising an optical module, a flexible and stretchable electrical signal module, and a central module, wherein at least one of the optical module and the flexible and stretchable electrical signal module is detachably connected to the central module;

[0007] The optical module is used to collect hemodynamic signals by combining near-infrared light signals with hemodynamics, including oxygenated hemoglobin concentration change signals, deoxygenated hemoglobin concentration change signals, and pulse wave signals;

[0008] The flexible and stretchable electrical signal module collects electrophysiological signals from various parts of the body through electrodes, including electroencephalogram (EEG) signals, electromyography (EMG) signals, electrocardiogram (ECG) signals, and electrooculogram (EOG) signals.

[0009] The central module is used to digitize and pre-process the hemodynamic signals and electrophysiological signals of various parts, and transmit the high-quality signals directly to the terminal, while multimodal data is integrated and evaluated for data quality and the dominant external events that affect the data quality are predicted. At the same time, the data quality and the dominant external events are also output to the terminal for feedback adjustment of the acquisition parameters of the optical module and the flexible stretchable electrical signal module and the equipment adjustment instructions.

[0010] In the system of the present invention, the optical module and the flexible and stretchable electrical signal module can be used independently in combination with the central module, or they can be used in combination. When the three are used in combination, the optical module and the flexible and stretchable electrical signal module are connected to the central module in a sandwich form.

[0011] Preferably, the optical module includes a plurality of transmitters, a plurality of receivers, and a signal transmission interface located on the first base circuit board;

[0012] The transmitter is used to transmit light signals of different near-infrared wavelengths to human tissue;

[0013] The receiver is used to receive the reflected signal formed after the light signal acts on human tissue, and uses the photoelectric signal processing algorithm to process the collected reflected signal data of different wavelengths according to brain functional activities and hemodynamic reactions, and the absorption and scattering rate of light by blood to obtain the corresponding hemodynamic signal;

[0014] The signal transmission interface is used to be detachably connected to the optical module interface of the central module and transmit the collection signal of the receiver.

[0015] Preferably, the flexible and stretchable electrical signal module includes a plurality of electrode patches, a hollow window, and a signal conduction interface connected to the second base circuit board, wherein some of the electrode patches can be flexibly stretched through the electrode wires and can be extended to be applied to various parts of the body;

[0016] The electrode patch is used to continuously monitor electrophysiological signals at various parts of the body, including electroencephalogram (EEG) signals on the surface of the brain, electromyography (EMG) signals from the skin, electrooculogram (EOG) signals from the eyebrows and eyes, and electrocardiogram (ECG) signals from the heart.

[0017] The hollow window is used to provide a light passage for the transmitter and receiver of the optical module;

[0018] The signal transmission interface is used to be detachably connected to the electrical module interface of the central module to transmit the monitored electrophysiological signals to the central module.

[0019] Preferably, the central module includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module;

[0020] The power supply module uses a lightweight mobile battery to provide power to the system, improving the portability of the device;

[0021] The optical module interface is used for detachable connection with the optical module;

[0022] The electrical module interface is used for detachable connection with the flexible and stretchable electrical signal module;

[0023] The digitization module is used to digitize the collected multimodal data at the front end;

[0024] The preprocessing module is used to preprocess the digitized multimodal data to improve the signal quality and transmit the preprocessed high-quality signal to the transmission module;

[0025] The multimodal data fusion and evaluation module is used to extract features from the digitized multimodal data and perform feature fusion using a Transformer model based on a multi-head attention mechanism. The module then evaluates data quality based on the fused features and predicts the dominant external events that affect the data quality.

[0026] The transmission module is used for transmitting the pre-processed hemodynamic signals, electrophysiological signals of various parts, data quality and leading external events to the terminal via wireless communication such as Bluetooth or WIFI, and receiving parameter adjustment information and equipment adjustment instructions fed back by the terminal;

[0027] The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module based on the received parameter adjustment information, and to adjust the equipment according to the equipment adjustment instructions.

[0028] Preferably, the hemodynamic signals and electrophysiological signals of various parts are pre-processed to obtain reliable signal data of various types and transmit them to the transmission module, including:

[0029] For EEG signals, filtering is performed, multi-channel independent component analysis is used to identify and remove eye artifacts and myoelectric interference, and the signal amplitude is normalized to enhance signal consistency;

[0030] For the electromyographic signal, bandpass filtering and full-wave rectification are performed, the electromyographic signal envelope is extracted using the moving average method, and the signal amplitude is mapped to a standardized range;

[0031] For ECG signals, a bandpass filter is used to remove baseline drift and high-frequency noise, wavelet transform is used to remove myoelectric interference, the Pan-Tompkins algorithm is used to detect the R wave peak, and the heartbeat cycle is converted into a binary sequence;

[0032] For the electrooculogram (EOG) signal, we calculated the static baseline and removed the drift, then band-pass filtered it, used principal component analysis to remove the interference of myoelectricity and environmental noise, and identified the eye movement events and gaze event codes.

[0033] For the oxygenated hemoglobin concentration change signal (DHbO) and the deoxygenated hemoglobin concentration change signal (DHb), the Beer-Lambert law was used to convert the light intensity signal into the hemodynamic concentration change signal (oxygenated hemoglobin concentration change signal and deoxygenated hemoglobin concentration change signal). A low-pass filter was then used to remove respiratory and heartbeat interference, and multi-channel independent component analysis (ICA) was used to remove motion artifacts. The signal was baseline-normalized to eliminate systematic errors.

[0034] For the pulse wave signal, a bandpass filter is used to filter the original light intensity signal, leaving the frequency band signal carrying the pulse change information. The signal baseline is calculated and the drift component is removed, and then the moving average method is used to smooth the signal.

[0035] Preferably, the original multimodal data with time synchronization and inconsistent sampling rates are cut into fixed-length windows:

[0036] EEG signals ;

[0037] electromyographic signals ;

[0038] electrooculogram ;

[0039] ECG signal ;

[0040] Oxyhemoglobin concentration change signal ;

[0041] Deoxyhemoglobin concentration change signal ;

[0042] Pulse wave signal ;

[0043] When extracting features from the multimodal data, the digitized EEG signals, EMG signals, EOG signals, ECG signals, oxyhemoglobin concentration change signals, deoxyhemoglobin concentration change signals, and pulse wave signals with inconsistent sampling rates are encoded separately to extract important feature information unique to each modality and related abnormal noise information, including:

[0044] EEG signals are extracted through convolutional neural networks to extract EEG frequency spatial features and abnormal signals ;

[0045] The electromyographic signal captures motion characteristics and abnormal signal characteristics through convolutional neural network ;

[0046] The electrooculogram (EOG) signal is extracted through a recurrent neural network to extract eye movement event-related pattern features and abnormal signal features. ;

[0047] ECG signals use autoencoders to extract the embedded features of the heartbeat cycle and abnormal signal features ;

[0048] The oxygenated hemoglobin concentration change signal is extracted based on the time convolution network to extract the blood oxygen fluctuation pattern characteristics and abnormal signal characteristics ;

[0049] Deoxyhemoglobin concentration change signal is extracted based on time convolution network to extract blood oxygen fluctuation pattern characteristics and abnormal signal characteristics ;

[0050] The pulse wave signal uses an autoencoder to capture the temporal relationship characteristics of the pulse beat and the abnormal signal characteristics ;

[0051] Extracted features The output sequence length is the same.

[0052] Preferably, a Transformer based on a multi-head attention mechanism is used for feature fusion, including:

[0053] All class features extracted from multimodal data are aligned and jointly encoded at the feature level, and feature splicing, linear transformation and normalization are performed to ensure the compatibility of different modal features. , and then input it into the Transformer model based on the multi-head attention mechanism, which dynamically adjusts the importance of different modalities using the multi-head attention mechanism, and uses the Transformer encoder layer to extract cross-modal joint features for all modal data.

[0054] Preferably, the data quality is assessed based on the fusion features and the dominant external events affecting the data quality are predicted, including:

[0055] After mapping the fusion features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate the data quality based on the mapped fusion features. The specific prediction signal quality is the sum of the duration of the noise events identified. The total data collection time is , take Score = Rate signal quality;

[0056] A classification algorithm is also used in the output layer to mark the dominant external events that affect data quality based on the mapped fusion features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise. This process evaluates data quality based on cross-modal joint features, which can more comprehensively and integratedly evaluate the quality of multimodal signals.

[0057] In the system of the present invention, the data transmission module and the parameter closed-loop feedback module cooperate to realize feedback adjustment of data acquisition. Specifically, the transmission module transmits hemodynamic signals, electrophysiological signals of various parts, data quality, and dominant external events to the terminal, and receives parameter adjustment information and equipment adjustment instructions fed back by the terminal. Among them, the acquisition parameters include the front-end amplification factor of the electrical signal and the transmitted light power of the optical signal. The equipment adjustment instructions include the loose status of the equipment, reminding the subject to keep the head still, and reminding the subject to wear a sunshade hat.

[0058] Front-end amplification of electrical signals: When the collected electrical signals are weak, the user is reminded to amplify the signals to a root mean square value of not less than 10 μA;

[0059] Transmitted optical power of the optical signal: When the optical signal is weak, the user is reminded to increase the transmitted optical power so that the light intensity at the receiving end is no less than A milliwatts and no more than B milliwatts. A is the minimum value of the hemodynamic signal calculated using the Bill-Lambert formula; B is the standard value for optical radiation safety set by the International Electrotechnical Commission (IEC).

[0060] Device loose status: When loose events dominate, remind the user to re-apply the device;

[0061] Remind the subject to keep the head still: When motion events dominate, remind the user to keep the head still;

[0062] Reminder to wear a light-shielding hat: When light noise dominates, remind users to cover the device's data collection area with black, opaque objects to prevent ambient light crosstalk.

[0063] The embodiment further provides a multimodal data evaluation method, which uses the above-mentioned electrophysiological and hemodynamic multimodal intelligent wearable system and includes the following steps:

[0064] Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module;

[0065] The central module is used to digitize and pre-process the collected multimodal data, and transmit high-quality multimodal data to the terminal. At the same time, the central module performs multimodal data fusion on the original data, evaluates the data quality, and predicts the dominant external events that affect the data quality. The data quality and dominant external events are also output to the terminal to feedback and adjust the acquisition parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention integrates processors for processing electrophysiological signals and optical signals into a wearable system, and integrates the acquisition, preprocessing, digitization, calculation, and transmission of electrophysiological and optical signals into a modular design. This allows the electrophysiological and optical signals detected by the flexible and stretchable electrical signal module and optical module to be directly preprocessed via a shorter path to the central module, avoiding information loss and noise caused by signal transmission. Cleaner electrophysiological and optical signals can be packaged and transmitted to the terminal according to the set communication requirements, allowing the terminal to play and store the preprocessed electrophysiological and optical signals in real time, allowing users to quickly obtain accurate physiological signal data. Comprehensively assessing data quality and noise detection through multimodal data improves the effectiveness and practicality of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A schematic diagram of the structure of a multimodal wearable system for electrophysiological and hemodynamic monitoring provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of the module functions of a multimodal wearable system for electrophysiological and hemodynamic monitoring provided by an embodiment of the present invention;

[0071] Figure 3 A schematic structural diagram of an optical module provided in an embodiment of the present invention;

[0072] Figure 4 A schematic structural diagram of a flexible and stretchable electrical signal module provided in an embodiment of the present invention;

[0073] Figure 5 A schematic structural diagram of a central module provided in an embodiment of the present invention;

[0074] Figure 6 A diagram showing the results of EEG signal processing provided by an embodiment of the present invention;

[0075] Figure 7 A diagram showing the electrocardiogram signal processing results provided by an embodiment of the present invention;

[0076] Figure 8A diagram showing the electromyographic signal processing results provided by an embodiment of the present invention;

[0077] Figure 9 This is a diagram of electrooculogram signal processing results provided by an embodiment of the present invention;

[0078] Figure 10 A diagram showing the result of processing the oxygenated hemoglobin concentration change signal provided by an embodiment of the present invention;

[0079] Figure 11 A diagram showing the result of processing a deoxyhemoglobin concentration change signal provided by an embodiment of the present invention;

[0080] Figure 12 A diagram showing the pulse wave signal processing results provided by an embodiment of the present invention;

[0081] Figure 13 Schematic diagram of multimodal data fusion and feedback implemented in the present invention;

[0082] Among them, there are central module 01, optical module 02, flexible stretchable electrical signal module 03, transmitters 100, 102, receivers 103-110, signal transmission interface 111, electrode patches 300-304, electrode wires 305, signal conduction interface 306, hollow window 307, central module shell 200, optical module interface 201, electrical module interface 202, shell bayonet 203 and 206, power supply module charging interface 204, device switch button 205, and work indicator LED light 207. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0084] like Figure 1 and Figure 2 As shown, the multimodal wearable system for electrophysiological and hemodynamic detection provided by the embodiment includes a central module 01, an optical module 02, and a flexible and stretchable electrical signal module 03. The flexible and stretchable electrical signal module 03 is on the skin-facing side, the optical module 02 is an interlayer, and the central module 01 is on the side away from the brain, forming a sandwich assembly. The optical module 02 and the flexible and stretchable electrical signal module 03 can be used independently in combination with the central module 01, or they can be used in combination. Among them, the central module 01 includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module.

[0085] In the embodiment, the optical module 02 is used to collect blood dynamic signals through near-infrared light signals, including oxygenated hemoglobin change signals, deoxygenated hemoglobin change signals, and pulse wave signals. Figure 3 As shown, the optical module 02 includes a plurality of transmitters 100 and 102 , a plurality of receivers 103 - 110 , and a signal transmission interface 111 on a first base circuit board.

[0086] Multiple emitters 100 and 102 are arranged with adhesive against the skin and are used to transmit light signals of different near-infrared wavelengths to human tissue. The emitters include, but are not limited to, LEDs and lasers, and each emits a light source with at least two wavelengths. Each emitter type can emit near-infrared light of different wavelengths, with the wavelengths of the near-infrared light ranging from 650 nanometers to 950 nanometers.

[0087] The multiple receivers 103-110 include, but are not limited to, photodiodes (PDs), avalanche photodiodes (APDs), and photomultiplier tubes (MPTs). These receivers are used to receive reflected signals from light signals striking human tissue. They can separate the reflected signals into different wavelengths of near-infrared light data based on the timing of light emission. Because human tissue absorbs and scatters light signals of different wavelengths differently, the receivers can also process the collected light signal data based on brain activity, hemodynamic responses, and the blood's absorption and scattering rates of light to derive corresponding hemodynamic signals. The collected near-infrared light data can be used to generate signals for detecting changes in oxyhemoglobin and deoxyhemoglobin in the brain, as well as pulse wave signals for detecting heart rate information. The signal transmission interface 111 is located on the side facing away from the skin, i.e., on a different side from the transmitter and receiver. It is used to detachably connect to the optical module interface of the central module and transmit the collected signals from the receiver. This signal transmission interface 111 can utilize, but is not limited to, magnetic connectors, board-to-board connectors, and wire-to-board connectors.

[0088] In the embodiment, the flexible and stretchable electrical signal module 03 is used to collect electrophysiological signals from various parts of the body through electrodes, including brain signals, myoelectric signals, electrocardiographic signals and electrooculographic signals. It has the characteristics of bending flexibility and extensibility, and can be attached to various parts of the body to collect electrophysiological signals according to the collection requirements. Figure 4As shown, the flexible and stretchable electrical signal module 03 includes a plurality of electrode patches 300-304 connected to the second base circuit board, an electrode wire 305, a signal transmission interface 306, and a hollow window 307. Specifically, a three-layer structure is adopted. The electrode patches 300-304 are attached to the skin through a conductive hydrogel and are present on the skin-closest side; the electrode wire 305 is present in the interlayer; the signal transmission interface 306, close to the central module 01 layer, transmits the signal to the central module 01. The signal transmission interface 306 can adopt, but is not limited to, a conductive foam connector, a magnetic connector, a board-to-board connector, and a wire-to-board connector. The hollow window 307 provides a light passage for the transmitter and receiver of the optical module 02, realizing unimpeded transmission of the light signal between the transmitter and receiver of the optical module. Some of the electrode patches can be flexibly stretched by the electrode wire. It can continuously detect EEG signals on the surface of the brain, extend to the muscles to be measured to measure myoelectric signals, or be placed near the eye to collect eye electricity.

[0089] In the embodiment, the central module is used to digitize and pre-process the hemodynamic signals and electrophysiological signals of various parts, and transmit the high-quality signals directly to the terminal, while the multimodal data is integrated and evaluated for data quality and the dominant external events that affect the data quality are predicted. At the same time, the data quality and the dominant external events are output to the terminal for feedback adjustment of the acquisition parameters of the optical module and the flexible stretchable electrical signal module. Figure 5 As shown, it includes a central module housing 200, an optical module interface 201, an electrical module interface 202, upper and lower shell snaps 203 and 206 to ensure the detachable central module housing, a work indicator LED light 207, a power supply module charging interface 204, and a device switch button 205.

[0090] The transmitter and receiver are located on the inner side of the optical module 02, that is, close to the brain side, and are connected to the outer transmission interface 111 of the optical module 02 through a flexible circuit board. The transmission interface 111 is connected to the central module 201. The central module 01 controls the light emission of the transmitters 100 and 102 in the optical module 02 in a time-sharing manner. At the same time, the interface 111 receives the emitted light signal emitted by the skin collected by the receivers 103-110 to obtain the hemodynamic signal. The optical module 02 is located between the electrical signal module 03 and the central module 01. Since the electrical signal module 03 and the central module 01 are connected to the interface 202 on the central module through the folded signal conduction interface 306, the electrical signal collected by the electrode patch can be transmitted to the central module 01 for processing.

[0091] In this embodiment, the central module is divided into a power supply module, a digitization module, a pre-processing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module according to its functions. The power supply module uses a lightweight mobile battery to provide power to the system.

[0092] The digitization module is used to digitize the collected multimodal data at the front end;

[0093] The preprocessing module is used to preprocess the digitized multimodal data and transmit the preprocessed high-quality signal to the transmission module. The preprocessing specifically includes:

[0094] The raw electroencephalogram (EEG) signals collected by the multimodal system usually contain noise and interference, so preprocessing is required to improve the signal quality. The preprocessing steps include amplification, filtering and artifact removal. Amplification is to enhance the amplitude of the signal so that it can be more easily recognized by the digitization module. Filtering usually involves low-pass filtering and high-pass filtering to remove power frequency interference (such as 50 / 60Hz) and motion artifacts. Artifact removal includes preferably using blind source separation (BSS) algorithm, fractal dimension analysis or multi-channel independent component analysis (ICA) to identify and remove artifacts and electromyographic interference, and also normalizing the signal amplitude to enhance signal consistency. The preprocessed signal, such as Figure 6 and sent to multiple terminals.

[0095] Electrocardiogram (ECG) signal acquisition also uses electrode patches in the flexible and stretchable electrical signal module 3. These patches are placed on the chest or limbs to detect the electrical activity of the heart. The electrode wire transmits the signal to the signal conduction interface, and then the signal is sent to the central module for preprocessing. The preprocessing of ECG signals includes noise filtering, baseline drift correction and R-peak detection. Noise filtering is intended to remove myoelectric interference and power line interference. Baseline drift correction is used to eliminate slow changes in the signal caused by breathing or other physiological changes. R-peak detection is to identify the QRS complex in the ECG signal, which is a key feature of the heart cycle. The preprocessed ECG signal, such as Figure 7 and sent to multiple terminals.

[0096] Electromyographic (EMG) signals are collected using electrode patches in a flexible and stretchable electrical signal module. These patches are placed on the surface of the muscle to detect the electrical activity of the muscle. The electrode wire transmits the signal to the signal conduction interface, and then the signal is sent to the central module for preprocessing. The preprocessing of electromyographic signals includes amplification, filtering and denoising. Amplification is to improve the signal-to-noise ratio of the signal, and filtering usually includes bandpass filtering to remove frequency components other than electromyographic signals. Denoising involves using a notch filter to remove power supply frequency interference, full-wave rectification, using the moving average method to extract the electromyographic signal envelope, and mapping the signal amplitude to a standardized range. The preprocessed EMG signal, such as Figure 8 and sent to the terminal.

[0097] Electrooculogram (EOG) signals are collected using electrode patches in a flexible and stretchable electrical signal module. These patches are placed around the eyes to detect the electrical signals generated by eye movements. The electrode wires transmit the signals to the signal transmission interface, and then the signals are sent to the central module for preprocessing. The preprocessing of EOG signals includes filtering and artifact removal. Filtering is used to remove low-frequency drift and high-frequency noise. Principal component analysis is used to remove electromyographic and environmental noise interference, identify eye movement events and gaze event encoding, and the preprocessed EOG signal, such as Figure 9 and sent to the terminal.

[0098] The acquisition of oxygenated hemoglobin concentration change signals and deoxygenated hemoglobin concentration change signals uses the transmitter and receiver in the optical module. The transmitter emits near-infrared light, which penetrates the scalp and skull, is absorbed and scattered by the brain tissue, and is then detected by the receiver and transmitted to the central module for digitization. The Beer-Lambert law is then used to convert the light intensity signal into an oxygenated hemoglobin concentration change signal and a deoxygenated hemoglobin concentration change signal, which are finally denoised through preprocessing. Preprocessing includes light intensity correction, motion artifact correction, and physiological noise filtering. Light intensity correction is used to eliminate errors caused by changes in light source intensity, motion artifact correction is used to remove artifacts caused by head movement, and physiological noise filtering helps to remove heartbeat and respiration-related noise. A low-pass filter is used to remove respiratory and heartbeat interference, multi-channel independent component analysis is used to remove motion artifacts, and the signal is baseline-normalized to eliminate system errors. Such as Figure 10 Oxyhemoglobin concentration change signal and Figure 11 The change in deoxyhemoglobin concentration is shown in a signal and sent to the terminal.

[0099] The collection of pulse wave signals (PPG) uses the transmitter and receiver in the optical module. The light emitted by the transmitter is absorbed by the skin and blood vessels. As the cardiovascular system beats, the intensity of the absorbed light changes. These changes are detected by the receiver and converted into electrical signals, which are then sent to the central module for preprocessing. The preprocessing of PPG signals includes filtering and denoising. Filtering is used to remove electromyographic and ambient light interference, while denoising involves using algorithms to identify and remove non-pulsatile changes, calculate the signal baseline and remove drift components, and use the moving average method to smooth the signal. The preprocessed PPG signal, such as Figure 12 and sent to the terminal.

[0100] In the embodiment, in the multimodal data fusion and evaluation module, as Figure 13 As shown, the digitized multimodal data is automatically extracted for features. Specifically, for the EEG signal, the original EEG signal is time-frequency transformed and then input into the convolutional neural network to extract the EEG frequency space and abnormal noise features.

[0101] For ECG signals, an autoencoder can be used to extract embedded features of heart rate, heart rate variability (HRV), and abnormal noise.

[0102] For electromyographic signals, convolutional neural networks can be used to extract movement characteristics of muscle activity, such as muscle activation patterns, force output, and abnormal noise.

[0103] For electrooculogram signals, recurrent neural networks can be used to extract event-related pattern features, such as blinking frequency, speed and direction of eye movements, and blinking habits.

[0104] For the change signals of oxyhemoglobin and deoxyhemoglobin, a temporal convolutional network is used to extract the blood oxygen fluctuation pattern characteristics and abnormal noise.

[0105] For pulse wave signals, an autoencoder is used to extract complex temporal relationship features such as heart rate, heart rate variability (HRV) and abnormal noise information.

[0106] In the embodiment, Figure 13 As shown in the figure, after extracting features, the multimodal data fusion and evaluation module also uses a Transformer model based on a multi-head attention mechanism to perform feature fusion, and then evaluates the data quality based on the fused features and predicts the dominant external events that affect the data quality.

[0107] Specifically, during feature fusion, all class features extracted from multimodal data are concatenated, linearly transformed, and normalized, and then input into the Transformer model based on the multi-head attention mechanism. The multi-head attention mechanism is used to dynamically adjust the importance of different modalities, and the Transformer encoder layer is used to extract cross-modal joint features for all modal data.

[0108] Specifically, during the evaluation and analysis, after mapping the fused features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate the data quality based on the mapped fused features, and a classification algorithm is used to mark the dominant external events that affect the data quality based on the mapped fused features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise.

[0109] In the embodiment, the transmission module also transmits the pre-processed hemodynamic signals, electrical signals of various parts, data quality, and dominant external events to the terminal via wireless communication, and receives parameter adjustment information and equipment adjustment instructions fed back by the terminal. The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module based on the received parameter adjustment information, and to adjust the equipment according to the equipment adjustment instructions. Among them, the acquisition parameters include the front-end amplification factor of the electrical signal and the emission light power of the optical signal; the equipment adjustment instructions include the loose state of the equipment, reminding the subject to keep the head still, and reminding the subject to wear a sunshade hat;

[0110] Front-end amplification factor for electrical signals: When the collected electrical signal is weak, the user is reminded to amplify the signal to a root mean square value of not less than 10 microamps;

[0111] Regarding the transmitted optical power of optical signals: When the optical signal is weak, the user is reminded to increase the transmitted optical power so that the light intensity at the receiving end is not less than A milliwatts and not greater than B milliwatts. A is the minimum value of the hemodynamic signal calculated using the Bill-Lambert formula; B is the standard value for optical radiation safety set by the International Electrotechnical Commission (IEC).

[0112] For loose device status: When loose events dominate, remind the user to re-apply the device;

[0113] Remind the subjects to keep their heads still: When motion events dominate, remind the users to keep their heads still;

[0114] Regarding the reminder to wear a light-blocking hat: When light noise dominates, remind users to cover the device's collection area with black, opaque objects to prevent ambient light crosstalk.

[0115] The embodiment further provides a multimodal data evaluation method, which uses the above-mentioned electrophysiological and hemodynamic multimodal intelligent wearable system and includes the following steps:

[0116] Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module;

[0117] The central module is used to digitize and pre-process the collected multimodal data and transmit it to the terminal. At the same time, the central module performs multimodal data fusion and data quality assessment on the original data and predicts the dominant external events affecting the data quality. The data quality and dominant external events are output to the terminal to feedback and adjust the acquisition parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

[0118] The system of the present invention is designed for real-time monitoring of EEG, EMG, EOG, near-infrared brain functional imaging and pulse wave signals. It uses a unique sandwich structure design, including an optical module, a flexible and stretchable electrical signal module and a central module, to ensure comfort and stability for long-term wear. The system comprehensively covers the needs of electrophysiological and hemodynamic monitoring through the integration of heterogeneous and heterogeneous multimodal acquisition devices. The optical module and the electrical signal module can be used independently or in combination with multimodality. The multi-parameter data fusion method realizes a comprehensive and integrated evaluation of the quality of multimodal signals through the feature integration and analysis of heterogeneous multi-source data, while accurately marking the dominant external events that affect the data quality, thereby improving the reliability of the data. The multi-parameter closed-loop feedback module analyzes the data output results, evaluates the data acquisition quality, intelligently optimizes the equipment acquisition parameters, and intelligently recommends users to adjust the acquisition site, significantly improving the effectiveness and practicality of monitoring.

[0119] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An electrophysiological and hemodynamic multimodal intelligent wearable system, characterized in that: The device comprises an optical module, a flexible and stretchable electrical signal module, and a central module, wherein at least one of the optical module and the flexible and stretchable electrical signal module is detachably connected to the central module; The optical module is used to collect hemodynamic signals through near-infrared light signals, including oxygenated hemoglobin concentration change signals, deoxygenated hemoglobin concentration change signals, and pulse wave signals; The flexible and stretchable electrical signal module collects electrophysiological signals from various parts of the body through electrodes, including electroencephalogram (EEG) signals, electromyography (EMG) signals, electrocardiogram (ECG) signals, and electrooculogram (EOG) signals. The central module is used to digitize and pre-process the hemodynamic signals and electrophysiological signals of various parts of the body, and transmit the high-quality signals directly to the terminal to perform multimodal data fusion, evaluate data quality, and predict the dominant external events that affect data quality. At the same time, it also outputs the data quality and dominant external events to the terminal to automatically feedback and adjust the acquisition parameters of the optical module and the flexible and stretchable electrical signal module and equipment adjustment instructions; Among them, multimodal data fusion and data quality assessment and prediction of dominant external events affecting data quality include: Feature extraction of multimodal data; A Transformer model based on a multi-head attention mechanism is used for feature fusion. This involves concatenating, linearly transforming, and normalizing all class features extracted from multimodal data before inputting them into the Transformer model based on a multi-head attention mechanism. The multi-head attention mechanism dynamically adjusts the importance of different modalities, and the Transformer encoder layer extracts cross-modal joint features from all modal data. Data quality is evaluated based on fused features and dominant external events that affect data quality are predicted, including: after mapping the fused features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate data quality based on the mapped fused features. Specifically, the total duration of noise events identified by the predicted signal quality is T_noise, the total length of data collection is T_total, and Score = (T_total-T_noise) / T_total is taken as the signal quality score. In addition, a classification algorithm is used in the output layer to mark dominant external events that affect data quality based on the mapped fused features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise.

2. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The optical module includes a plurality of transmitters, a plurality of receivers, and a signal transmission interface located on a first base circuit board; The transmitter is used to transmit light signals of different near-infrared wavelengths to human tissue; The receiver is used to receive the reflected signal formed after the light signal acts on human tissue, and uses the photoelectric signal processing algorithm to process the collected reflected signal data of different wavelengths according to brain functional activities and hemodynamic reactions, and the absorption and scattering rate of light by blood to obtain the corresponding hemodynamic signal; The signal transmission interface is used to be detachably connected to the optical module interface of the central module and transmit the collection signal of the receiver.

3. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The flexible and stretchable electrical signal module includes a plurality of electrode patches connected to the second base circuit board, a hollow window, and a signal conduction interface, wherein some of the electrode patches can be flexibly stretched through the electrode wires; The electrode patch is used to continuously monitor electrophysiological signals at various parts of the body; The hollow window is used to provide a light passage for the transmitter and receiver of the optical module; The signal transmission interface is used to be detachably connected to the electrical module interface of the central module to transmit the monitored electrophysiological signals to the central module.

4. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The central module includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module; The power supply module uses a lightweight mobile battery to provide power to the system; The optical module interface is used for detachable connection with the optical module; The electrical module interface is used for detachable connection with the flexible and stretchable electrical signal module; The digitization module is used to digitize the collected multimodal data at the front end; The preprocessing module is used to preprocess the digitized multimodal data and transmit the preprocessed high-quality signal to the transmission module; The multimodal data fusion and evaluation module is used to perform multimodal data fusion and evaluate data quality and predict dominant external events that affect data quality; The transmission module is used to transmit the pre-processed hemodynamic signals, electrophysiological signals of various parts, data quality and leading external events to the terminal through wireless communication, and receive the acquisition parameter adjustment information and equipment adjustment instructions fed back by the terminal; The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module based on the received parameter adjustment information, and to adjust the equipment according to the equipment adjustment instructions.

5. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: Pre-process the hemodynamic signals and electrophysiological signals of various parts and transmit them to the transmission module, including: For EEG signals, filtering is performed, multi-channel independent component analysis is used to identify and remove eye artifacts and myoelectric interference, and the signal amplitude is normalized to enhance signal consistency; For the electromyographic signal, bandpass filtering and full-wave rectification are performed, the electromyographic signal envelope is extracted using the moving average method, and the signal amplitude is mapped to a standardized range; For ECG signals, a bandpass filter is used to remove baseline drift and high-frequency noise, wavelet transform is used to remove myoelectric interference, the Pan-Tompkins algorithm is used to detect the R wave peak, and the heartbeat cycle is converted into a binary sequence; For the electrooculogram (EOG) signal, we calculated the static baseline and removed the drift, then band-pass filtered it. We used principal component analysis to remove the interference of myoelectricity and environmental noise, and identified the eye movement events and gaze event codes. For near-infrared brain functional imaging signals of oxygenated and deoxygenated hemoglobin concentration changes, the Beer-Lambert law was used to convert the light intensity signal into a hemodynamic concentration change signal. A low-pass filter was used to remove respiratory and heartbeat interference, and multi-channel independent component analysis was used to remove motion artifacts. The signals were baseline-normalized to eliminate systematic errors. The pulse wave signal was filtered using a bandpass filter, the signal baseline was calculated and the drift component was removed, and the moving average method was used to smooth the signal.

6. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: The digitized EEG signals, EMG signals, EO signals, ECG signals, oxyhemoglobin concentration change signals, deoxyhemoglobin concentration change signals, and pulse wave signals with inconsistent sampling rates are encoded separately, including: The EEG signal is processed through convolutional neural network to extract EEG frequency space features and abnormal signal features; The electromyographic signal captures motion characteristics and abnormal signal characteristics through convolutional neural networks; The electrooculogram (EOG) signals are used to extract eye movement event-related pattern features and abnormal signal features through a recurrent neural network. The ECG signal uses an autoencoder to extract the embedded features of the heartbeat cycle and abnormal signal features; The oxygenated hemoglobin concentration change signal and the deoxygenated hemoglobin concentration change signal are used to extract the blood oxygen fluctuation pattern characteristics and abnormal signal characteristics based on the time convolution network; The pulse wave signal uses an autoencoder to capture the temporal relationship characteristics of the pulse beat and the abnormal signal characteristics; The output sequences of the extracted features have the same length.

7. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The acquisition parameters include the front-end amplification factor of the electrical signal and the emission power of the optical signal; the equipment adjustment instructions include the loose state of the equipment, reminding the subject to keep the head still, and reminding the subject to wear a light-shielding hat; Front-end amplification factor for electrical signals: When the collected electrical signal is weak, the user is reminded to amplify the signal to a root mean square value of not less than 10 microamps; Regarding the transmitted optical power of optical signals: When the optical signal is weak, the user is reminded to increase the transmitted optical power so that the light intensity at the receiving end is not less than A milliwatts and not greater than B milliwatts. A is the minimum value of the hemodynamic signal calculated by the Bill-Lambert formula; B is the standard value for optical radiation safety set by the International Electrotechnical Commission. For loose device status: When loose events dominate, remind the user to re-apply the device; Remind the subjects to keep their heads still: When motion events dominate, remind the users to keep their heads still; Regarding the reminder to wear a light-blocking hat: When light noise dominates, remind users to cover the device's collection area with black, opaque objects to prevent ambient light crosstalk.

8. A multimodal data evaluation method, characterized in that: The method uses the electrophysiological and hemodynamic multimodal intelligent wearable system according to any one of claims 1 to 7, comprising the following steps: Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module; The central module is used to digitize and pre-process the collected multimodal data and transmit it to the terminal. At the same time, the central module performs multimodal data fusion on the original data, evaluates the data quality, and predicts the dominant external events that affect the data quality. The data quality and dominant external events are output to the terminal to feedback and adjust the acquisition parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

Images