Electrophysiology and hemodynamics multi-mode intelligent wearable system and data evaluation method

Abstract

The invention discloses an electrophysiology and hemodynamics multi-mode intelligent wearable system and a data evaluation method, and belongs to the technical field of biomedical engineering, the system comprises an optical module, a flexible stretchable electric signal module and a central module; the optical module is used for collecting hemodynamic signals including an oxyhemoglobin change signal, a deoxygenated hemoglobin change signal and a pulse wave signal by combining a near-infrared light signal with hemodynamics; the flexible stretchable electric signal module collects electric signals, including electroencephalogram signals, electromyographic signals, electrocardiosignals and electro-oculogram signals, of all parts through electrodes; the central module is used for digitalizing and preprocessing the hemodynamic signal and the electric signal of each part, fusing multi-modal data, evaluating data quality and predicting a dominant external event influencing the data quality; and meanwhile, data quality and dominant external events are output to the terminal to feed back and adjust acquisition parameters of the optical module and the flexible stretchable electric signal module, so that the effectiveness and practicability of monitoring are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of biomedical engineering technology, and specifically relates to an electrophysiological and hemodynamic multimodal intelligent wearable system and a data evaluation method. Background Art

[0002] Traditional physiological signal detection methods usually detect a single signal, such as: electroencephalogram (EEG) for detecting brain electrical signals, electromyogram (EMG) for detecting muscle electrical signals, electrooculogram (EOG) for detecting eye movement signals, functional near-infrared neuroradiography (fNIRS) for detecting brain hemodynamic signals, and pulse wave graph for detecting pulse wave signals. These detection methods have their technical limitations and can only collect a single physiological signal, while human activities involve changes at multiple levels. Therefore, in order to fully explore human functional activities, more diversified physiological multimodal detection technologies are needed.

[0003] With the continuous development of various physiological detection technologies, multimodal detection technologies that combine multiple technologies to detect physiological functions have been developed in recent years. Among them, the multimodal detection method of the human body that combines electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG), near-infrared functional brain imaging (fNIRS) and pulse wave (PPG) graph combines the advantages of both and makes up for the defects. EEG has the advantage of good temporal resolution in collecting EEG signals, but its spatial resolution is limited; fNIRS provides better spatial resolution, but poor temporal resolution. At the same time, these signals of different dimensions provide complementary electrophysiological and hemodynamic information related to human activities. Therefore, multimodal detection methods based on electrophysiology and hemodynamics can make up for the defects of a single detection technology, achieve complementary advantages, and can be used in various applications, such as neurovascular coupling, motion monitoring, cognitive function assessment, etc., which can help researchers to understand the neural processes and human functional mechanisms in cognitive tasks more systematically and deeply.

[0004] In the existing technology, a rehabilitation training system based on cerebral blood oxygen and electromyographic signals is disclosed in the Chinese patent application with publication number CN112932474A, which includes: a synchronous information acquisition module, a processing and analysis module and a feedback module, wherein: the synchronous information acquisition module is used to collect the trainee's near-infrared cerebral blood oxygen and electromyographic signals; the processing and analysis module is used to process and analyze the cerebral blood oxygen signals and electromyographic signals synchronously collected from the synchronous information acquisition module; the feedback module is used to determine whether the trainee's training exceeds the normal standard based on the cerebral blood oxygen signals and electromyographic signals processed and analyzed by the processing and analysis module, and if so, adjust the parameters of the training equipment. The above patent application uses photoelectric signal sensors and electrical signal sensors to detect signals, but the patent application simply detects cerebral blood oxygen signals and electromyographic signals, and does not expand to the field of multimodal detection of the whole body, nor does it digitize the detected signals at the acquisition end, pre-process them, and then transmit them to the data processing system, resulting in large noise in the processed data, and there is no real-time noise detection of the collected data, and feedback to the user to adjust the wearing of the device. Summary of the invention

[0005] In view of the above, the purpose of the present invention is to provide an electrophysiological and hemodynamic multimodal intelligent wearable system and data evaluation method, which can process multiple physiological signals at the same time and obtain data with less noise, so as to obtain EEG signals, EMG signals, EO signals, ECG signals, blood oxygen signals and pulse wave signals more accurately. Through the algorithm to extract and fuse features from multimodal heterogeneous data, a comprehensive and integrated evaluation of the quality of multimodal signals is achieved, and the dominant external events that affect the data quality are accurately marked, thereby improving the reliability of the data. The multimodal multi-parameter closed-loop feedback module analyzes the data output results, evaluates the data acquisition quality, intelligently optimizes the equipment acquisition parameters, and intelligently recommends users to adjust the acquisition site, significantly improving the effectiveness and practicality of monitoring.

[0006] To achieve the above-mentioned purpose of the invention, an electrophysiological and hemodynamic multimodal intelligent wearable system provided in an embodiment includes an optical module, a flexible and stretchable electrical signal module, and a central module, wherein at least one of the optical module and the flexible and stretchable electrical signal module is detachably connected to the central module; The optical module is used to collect hemodynamic signals by combining near-infrared light signals with hemodynamics, including oxygenated hemoglobin concentration change signals, deoxygenated hemoglobin concentration change signals, and pulse wave signals; The flexible and stretchable electrical signal module collects electrophysiological signals from various parts of the body through electrodes, including electroencephalogram signals, electromyography signals, electrocardiogram signals, and electrooculogram signals; The central module is used to digitize and pre-process the hemodynamic signals and electrophysiological signals of various parts, and directly transmit the high-quality signals to the terminal, while multimodal data is integrated and evaluated for data quality and the dominant external events that affect the data quality are predicted. At the same time, the data quality and the dominant external events are also output to the terminal to feedback and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module and the equipment adjustment instructions.

[0007] In the system of the present invention, the optical module and the flexible stretchable electrical signal module can be used independently in combination with the central module, or they can be used in combination. When the three are used in combination, the optical module and the flexible stretchable electrical signal module are connected to the central module in a sandwich form.

[0008] Preferably, the optical module comprises a plurality of transmitters, a plurality of receivers, and a signal transmission interface located on the first base circuit board; The transmitter is used to transmit light signals of different near-infrared wavelengths to human tissue; The receiver is used to receive the reflection signal formed after the light signal acts on human tissue, and uses the photoelectric signal processing algorithm to process the collected reflection signal data of different wavelengths according to brain function activities and hemodynamic reactions, and the absorption and scattering rate of light by blood to obtain the corresponding hemodynamic signal; The signal transmission interface is used to be detachably connected to the optical module interface of the central module and transmit the collection signal of the receiver.

[0009] Preferably, the flexible and stretchable electrical signal module comprises a plurality of electrode patches, a hollow window and a signal conduction interface connected to the second base circuit board, wherein some of the electrode patches can be flexibly stretched through electrode wires and can be extended to be applied to various parts of the body; The electrode patch is used to continuously monitor electrophysiological signals of various parts, including electroencephalogram signals on the surface of the brain, electromyography signals of the skin, electrooculogram signals at the eyebrows and eyes, and electrocardiogram signals of the heart; The hollow window is used to provide a light passage for the transmitter and receiver of the optical module; The signal transmission interface is used to be detachably connected to the electrical module interface of the central module to transmit the monitored electrophysiological signals to the central module.

[0010] Preferably, the central module includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module; The power supply module uses a lightweight mobile battery to provide power to the system, thereby improving the portability of the device; The optical module interface is used for detachably connecting with the optical module; The electrical module interface is used for detachably connecting with the flexible and stretchable electrical signal module; The digitization module is used to digitize the collected multimodal data at the front end; The preprocessing module is used to preprocess the digitized multimodal data to improve the signal quality, and transmit the preprocessed high-quality signal to the transmission module; The multimodal data fusion and evaluation module is used to extract features from the digitized multimodal data and to fuse features using a Transformer model based on a multi-head attention mechanism, and then evaluate the data quality based on the fused features and predict the dominant external events that affect the data quality; The transmission module is used for wireless communication such as Bluetooth or WIFI to transmit the pre-processed blood dynamic signals, electrophysiological signals of various parts, data quality and dominant external events to the terminal, and receive parameter adjustment information and equipment adjustment instructions fed back by the terminal; The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible and stretchable electrical signal module based on the received parameter adjustment information, and to adjust the equipment according to the equipment adjustment instructions.

[0011] Preferably, the hemodynamic signals and electrophysiological signals of various parts are preprocessed to obtain reliable signal data of various types and transmit them to the transmission module, including: For EEG signals, filtering is performed, multi-channel independent component analysis is used to identify and remove electrooculographic artifacts and electromyographic interference, and the signal amplitude is normalized to enhance signal consistency; For the electromyographic signal, bandpass filtering and full-wave rectification are performed, the electromyographic signal envelope is extracted using the moving average method, and the signal amplitude is mapped to a standardized range; For ECG signals, a bandpass filter is used to remove baseline drift and high-frequency noise, wavelet transform is used to remove myoelectric interference, the Pan-Tompkins algorithm is used to detect the R wave peak, and the heartbeat cycle is converted into a binary sequence; For the electrooculogram signal, the static baseline is calculated and the drift is removed, bandpass filtering is performed, and principal component analysis is used to remove the interference of electromyography and environmental noise, and to identify the encoding of eye movement events and gaze events; For oxygenated hemoglobin concentration change signals (DHbO) and deoxygenated hemoglobin concentration change signals (DHb), the Beer-Lambert law was used to convert the light intensity signals into hemodynamic concentration change signals (oxygenated hemoglobin concentration change signals, deoxygenated hemoglobin concentration change signals). Then, a low-pass filter was used to remove respiratory and heartbeat interference, and multi-channel independent component analysis (ICA) was used to remove motion artifacts. The signals were baseline-normalized to eliminate systematic errors. For the pulse wave signal, a bandpass filter is used to filter the original light intensity signal, leaving the frequency band signal carrying the pulse change information, and the signal baseline is calculated and the drift component is removed, and then the moving average method is used to smooth the signal.

[0012] Preferably, the original multimodal data with time synchronization and sampling rate inconsistency is cut into fixed time windows: EEG signals ; EMG ; Electrooculogram ; ECG signal ; Oxyhemoglobin concentration change signal ; Deoxyhemoglobin concentration change signal ; Pulse wave signal ; When extracting features from the above multimodal data, the digitized EEG signals, EMG signals, EOG signals, ECG signals, oxyhemoglobin concentration change signals, deoxyhemoglobin concentration change signals, and pulse wave signals with inconsistent sampling rates are encoded separately to extract important feature information unique to each modality and related abnormal noise information, including: EEG signals are extracted through convolutional neural networks to extract EEG frequency spatial features and abnormal signals ; The electromyographic signal captures the motion characteristics and abnormal signal characteristics through the convolutional neural network ; The electrooculogram signals are used to extract eye movement event-related pattern features and abnormal signal features through recurrent neural networks ; ECG signals use autoencoders to extract embedded features of heartbeat cycles and abnormal signal features ; The oxygenated hemoglobin concentration change signal is extracted based on the time convolution network to extract the blood oxygen fluctuation pattern characteristics and abnormal signal characteristics ; Deoxyhemoglobin concentration change signal extracts blood oxygen fluctuation pattern characteristics and abnormal signal characteristics based on time convolution network ; The pulse wave signal uses an autoencoder to capture the timing relationship characteristics of the pulse beat and the abnormal signal characteristics ; Extracted features The output sequence length is the same.

[0013] Preferably, a Transformer based on a multi-head attention mechanism is used for feature fusion, including: All class features extracted from multimodal data are aligned and jointly encoded at the feature level, and feature concatenation, linear transformation and normalization are performed to ensure the compatibility of features of different modalities. , and then input it into the Transformer model based on the multi-head attention mechanism, which dynamically adjusts the importance of different modalities using the multi-head attention mechanism, and uses the Transformer encoder layer to extract cross-modal joint features for all modal data.

[0014] Preferably, the data quality is evaluated based on the fusion features and the dominant external events affecting the data quality are predicted, including: After mapping the fused features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate the data quality based on the mapped fused features. The specific prediction signal quality is the sum of the duration of the noise events identified. The total data collection time is , take Score = Rate signal quality; In the output layer, a classification algorithm is also used to mark the dominant external events that affect data quality based on the mapped fusion features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise. This process evaluates data quality based on cross-modal joint features, which can more comprehensively evaluate the quality of multimodal signals.

[0015] In the system of the present invention, the cooperation of the data transmission module and the parameter closed-loop feedback module realizes the feedback adjustment of data acquisition. Specifically, the blood dynamic signal, the electrophysiological signal of each part, the data quality and the dominant external event are transmitted to the terminal through the transmission module, and the parameter adjustment information and the equipment adjustment instruction fed back by the terminal are received, wherein the acquisition parameters include the front-end amplification factor of the electrical signal and the emission light power of the optical signal, and the equipment adjustment instruction includes the loose state of the equipment, reminding the subject to keep the head still, and reminding to wear a sunshade cap; Front-end amplification of electrical signals: When the collected electrical signals are weak, the user is reminded to amplify the signals until the RMS value is not less than 10 μA; Transmitted optical power of optical signal: When the optical signal is weak, remind the user to increase the transmitted optical power so that the light intensity at the receiving end is not less than A milliwatts and cannot be greater than B milliwatts; A is the minimum value of the hemodynamic signal calculated by the Bill-Lambert calculation formula; B is the standard value for optical radiation safety of the International Electrotechnical Commission (IEC); Device loose status: When loose events dominate, remind the user to re-apply the device; Remind the subject to keep the head still: When the motion event is dominant, remind the user to keep the head still; Reminder to wear a light-shielding hat: When light noise is dominant, remind users to cover the device collection area with black opaque objects to prevent ambient light crosstalk.

[0016] The embodiment further provides a multimodal data evaluation method, which adopts the above-mentioned electrophysiological and hemodynamic multimodal intelligent wearable system and includes the following steps: Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module; The central module is used to digitize and preprocess the collected multimodal data, and transmit the high-quality multimodal data to the terminal. At the same time, the central module performs multimodal data fusion and data quality evaluation on the original data, and predicts the dominant external events that affect the data quality. The data quality and dominant external events are also output to the terminal to feedback and adjust the collection parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention integrates the processor for processing electrophysiological signals and optical signals into a wearable system, and integrates the collection, preprocessing, digitization, calculation and transmission of electrophysiological signals and optical signals into a modular design, so that the electrophysiological signals and optical signals detected by the flexible stretchable electric signal module and the optical module can be directly preprocessed through a shorter path to the central module, avoiding information loss and noise caused by signal transmission, and can package relatively clean electrophysiological signals and optical signals according to the set communication requirements. The terminal plays and stores the preprocessed electrophysiological signals and optical signals in real time, allowing users to quickly obtain accurate physiological signal data. Through multimodal data, the data quality and noise detection are comprehensively evaluated to improve the effectiveness and practicality of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of the structure of a multi-modal wearable system for electrophysiological and hemodynamic detection provided by an embodiment of the present invention; Figure 2A schematic diagram of the module functions of a multi-modal wearable system for electrophysiological and hemodynamic detection provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an optical module provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a flexible and stretchable electrical signal module provided in an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a central module provided by an embodiment of the present invention; Figure 6 A diagram showing the results of EEG signal processing provided by an embodiment of the present invention; Figure 7 A diagram showing the result of electrocardiogram signal processing provided by an embodiment of the present invention; Figure 8 A diagram showing the electromyographic signal processing results provided by an embodiment of the present invention; Fig. 9 A diagram showing the electrooculogram signal processing results provided by an embodiment of the present invention; Fig.10 A diagram showing the result of processing the oxygenated hemoglobin concentration change signal provided in an embodiment of the present invention; Fig.11 A diagram showing the result of processing a deoxyhemoglobin concentration change signal provided in an embodiment of the present invention; Fig.12 A pulse wave signal processing result diagram provided by an embodiment of the present invention; Fig.13 A schematic diagram of multimodal data fusion and feedback implemented in the present invention; Among them, there are central module 01, optical module 02, flexible and stretchable electrical signal module 03, transmitters 100, 102, receivers 103-110, signal transmission interface 111, electrode patches 300-304, electrode wires 305, signal conduction interface 306, hollow window 307, central module housing 200, optical module interface 201, electrical module interface 202, housing snap-ons 203 and 206, power supply module charging interface 204, device switch button 205, and work indication LED light 207. DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0021] like Figure 1 and Figure 2As shown, the multimodal wearable system for electrophysiological and hemodynamic detection provided by the embodiment includes a central module 01, an optical module 02, and a flexible and stretchable electrical signal module 03. The flexible and stretchable electrical signal module 03 is the skin-tight side, the optical module 02 is the interlayer and the central module 01 is the side away from the brain, forming a sandwich assembly. The optical module 02 and the electrical flexible and stretchable signal module 03 can be used independently in combination with the central module 01, or they can be used in combination. Among them, the central module 01 includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module.

[0022] In the embodiment, the optical module 02 is used to collect blood dynamic signals through near-infrared light signals, including oxygenated hemoglobin change signals, deoxygenated hemoglobin change signals, and pulse wave signals. Figure 3 As shown, the optical module 02 includes a plurality of transmitters 100 , 102 , a plurality of receivers 103 - 110 , and a signal transmission interface 111 on a first base circuit board.

[0023] Among them, multiple emitters 100, 102 are arranged on the skin with adhesive, and are used to emit light signals of different near-infrared wavelengths to human tissues, and the types include but are not limited to LEDs and lasers, and the emitters are light sources with no less than two wavelengths. Each type of emitter includes near-infrared light that can emit different wavelengths, and the wavelengths of the near-infrared light are distributed between 650 nanometers and 950 nanometers.

[0024] The types of multiple receivers 103-110 include but are not limited to photodiodes (PD), avalanche photodiodes (APD), and photomultiplier tubes (MPT). Multiple receivers are used to receive the reflected signals formed after the light signal acts on human tissue, and can divide the obtained reflected signals into near-infrared light data of different wavelengths reflected back according to the light emission timing. Since human tissue has different absorption and scattering effects on light signals of different wavelengths, the receiver can also collect light signal data of different wavelengths according to brain function activities and hemodynamic reactions and blood absorption and scattering rates of light, and can derive corresponding hemodynamic signals. Using the collected near-infrared light data, it is possible to obtain oxygenated hemoglobin change signals, deoxygenated hemoglobin change signals, and pulse wave signals for detecting heart rate information in the brain. The signal transmission interface 111 is on the side away from the skin, that is, on a different side from the transmitter and the receiver, and is used to be detachably connected to the optical module interface of the central module and transmit the collected signal of the receiver. The signal transmission interface 111 can use but is not limited to a magnetic suction connector, a board-to-board connector, and a wire-to-board connector.

[0025] In the embodiment, the flexible and stretchable electrical signal module 03 is used to collect electrophysiological signals from various parts of the body through electrodes, including electroencephalogram signals, electromyography signals, electrocardiogram signals and electrooculogram signals. It has the characteristics of bending flexibility and extensibility, and can be attached to various parts of the body to collect electrophysiological signals according to the collection requirements. Figure 4 As shown, the flexible and stretchable electrical signal module 03 includes a plurality of electrode patches 300-304, an electrode wire 305, a signal transmission interface 306, and a hollow window 307 connected to the second base circuit board. Specifically, a three-layer structure is adopted, and the electrode patches 300-304 are attached to the skin through conductive hydrogel and are present on the skin-closest side; the electrode wire 305 is present in the interlayer; the signal transmission interface 306, close to the central module 01 layer, transmits the signal to the central module 01, and the signal transmission interface 306 can be used but not limited to a conductive foam connector, a magnetic connector, a board-to-board connector, and a wire-to-board connector. The hollow window 307 provides a light port for the transmitter and receiver of the optical module 02, realizing the unimpeded transmission of the optical signal between the transmitter and the receiver of the optical module. Some of the electrode patches can be flexibly stretched through the electrode wire. It can continuously detect the EEG signals on the surface of the brain, extend to the muscles to be measured to measure the myoelectric signals, or place them near the eye to collect the eye electricity.

[0026] In the embodiment, the central module is used to digitize and pre-process the blood dynamic signals and electrophysiological signals of various parts, and directly transmit the high-quality signals to the terminal, while the multimodal data is integrated and the data quality is evaluated and the dominant external events affecting the data quality are predicted. At the same time, the data quality and the dominant external events are output to the terminal to feedback and adjust the acquisition parameters of the optical module and the flexible and stretchable electrical signal module. Figure 5 As shown, it includes a central module housing 200, an optical module interface 201, an electrical module interface 202, upper and lower shell snap-ons 203 and 206 to ensure the detachable central module housing, a work indicator LED light 207, a power supply module charging interface 204, and a device switch button 205.

[0027] The transmitter and receiver are located on the inner side of the optical module 02, that is, close to the brain side, and are connected to the outer transmission interface 111 of the optical module 02 through a flexible circuit board. The transmission interface 111 is connected to the central module 201, and the central module 01 controls the light emission of the transmitters 100 and 102 in the optical module 02 in a time-sharing manner, and at the same time receives the emission light signal emitted by the skin collected by the receivers 103-110 through the interface 111 to obtain the hemodynamic signal. The optical module 02 is located between the electrical signal module 03 and the central module 01. Since the electrical signal module 03 and the central module 01 are connected to the interface 202 on the central module through the folded signal conduction interface 306, the electrical signal collected by the electrode patch can be transmitted to the central module 01 for processing.

[0028] In the embodiment, according to the functional division, the central module is divided into a power supply module, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module. Among them, the power supply module uses a lightweight mobile battery to provide power for the system.

[0029] The digitization module is used to digitize the collected multimodal data at the front end; The preprocessing module is used to preprocess the digitized multimodal data and transmit the preprocessed high-quality signals to the transmission module. The preprocessing specifically includes: The raw electroencephalogram (EEG) signals collected by the multimodal system usually contain noise and interference, so preprocessing is required to improve the signal quality. The preprocessing steps include amplification, filtering and artifact removal. Amplification is to enhance the amplitude of the signal so that it can be more easily recognized by the digitization module. Filtering usually involves low-pass filtering and high-pass filtering to remove power frequency interference (such as 50 / 60Hz) and motion artifacts. Artifact removal includes preferably using blind source separation (BSS) algorithm, fractal dimension analysis or multi-channel independent component analysis (ICA) to identify and remove artifacts and electromyographic interference, and also normalizing the signal amplitude to enhance signal consistency. The preprocessed signal, such as Figure 6 and sent to multiple terminals.

[0030] Electrocardiogram (ECG) signal acquisition also uses electrode patches in the flexible and stretchable electrical signal module 3, which are placed on the chest or limbs to detect the electrical activity of the heart. The electrode wires transmit the signals to the signal conduction interface, and then the signals are sent to the central module for preprocessing. The preprocessing of ECG signals includes noise filtering, baseline drift correction and R peak detection. Noise filtering is designed to remove myoelectric interference and power line interference. Baseline drift correction is used to eliminate slow changes in the signal caused by breathing or other physiological changes. R peak detection is to identify the QRS complex in the ECG signal, which is a key feature of the heart cycle. The preprocessed ECG signal, such as Figure 7 and sent to multiple terminals.

[0031] Electromyographic (EMG) signals are collected using electrode patches in a flexible and stretchable electrical signal module, which are placed on the surface of the muscle to detect the electrical activity of the muscle. The electrode wires transmit the signals to the signal conduction interface, and then the signals are sent to the central module for preprocessing. The preprocessing of electromyographic signals includes amplification, filtering, and denoising. Amplification is to improve the signal-to-noise ratio of the signal, and filtering usually includes bandpass filtering to remove frequency components other than electromyographic signals. Denoising involves using a notch filter to remove power supply frequency interference, full-wave rectification, using the moving average method to extract the electromyographic signal envelope, and mapping the signal amplitude to a standardized range. The preprocessed EMG signal, such as Figure 8 and sent to the terminal.

[0032] Electrooculogram (EOG) signals are collected using electrode patches in a flexible and stretchable electrical signal module. These patches are placed around the eyes to detect the electrical signals generated by eye movements. The electrode wires transmit the signals to the signal conduction interface, and then the signals are sent to the central module for preprocessing. The preprocessing of EOG signals includes filtering and artifact removal. Filtering is used to remove low-frequency drift and high-frequency noise. Principal component analysis is used to remove electromyographic and environmental noise interference, identify eye movement events and gaze event encoding, and preprocessed EOG signals such as Fig. 9 and sent to the terminal.

[0033] The oxygenated hemoglobin concentration change signal and the deoxygenated hemoglobin concentration change signal are collected using the transmitter and receiver in the optical module. The transmitter emits near-infrared light, which penetrates the scalp and skull, is absorbed and scattered by the brain tissue, and is then detected by the receiver and transmitted to the central module for digitization. The Beer-Lambert law is then used to convert the light intensity signal into an oxygenated hemoglobin concentration change signal and a deoxygenated hemoglobin concentration change signal, which are finally denoised through preprocessing. Preprocessing includes light intensity correction, motion artifact correction, and physiological noise filtering. Light intensity correction is used to eliminate errors caused by changes in light source intensity, motion artifact correction is used to remove artifacts caused by head movement, and physiological noise filtering helps remove heartbeat and breathing-related noise. A low-pass filter is used to remove respiratory and heartbeat interference, and multi-channel independent component analysis is used to remove motion artifacts. The signal is baseline-normalized to eliminate system errors. Fig.10 Oxyhemoglobin concentration change signal and Fig.11 The deoxyhemoglobin concentration change signal is shown and sent to the terminal.

[0034] The pulse wave signal (PPG) is collected using the transmitter and receiver in the optical module. The light emitted by the transmitter is absorbed by the skin and blood vessels. As the cardiovascular system beats, the intensity of the absorbed light changes. These changes are detected by the receiver and converted into electrical signals, which are sent to the central module for preprocessing. The preprocessing of PPG signals includes filtering and denoising. Filtering is used to remove electromyographic and ambient light interference, while denoising involves using algorithms to identify and remove non-pulsatile changes, calculate the signal baseline and remove drift components, and use the moving average method to smooth the signal. The preprocessed PPG signal, such as Fig.12 and sent to the terminal.

[0035] In the embodiment, in the multimodal data fusion and evaluation module, as Fig.13 As shown, the digitized multimodal data is automatically extracted for features. Specifically, for the EEG signal, the original EEG signal is transformed in time and frequency and then input into the convolutional neural network to extract the EEG frequency space and abnormal noise features.

[0036] For ECG signals, autoencoders can be used to extract embedded features of heart rate, heart rate variability (HRV), and abnormal noise.

[0037] For electromyographic signals, convolutional neural networks can be used to extract movement characteristics of muscle activity, such as muscle activation patterns, force output, and abnormal noise.

[0038] For electrooculogram signals, recurrent neural networks can be used to extract event-related pattern features, such as blinking frequency, speed and direction of eye movement, and blinking habits.

[0039] For the change signals of oxygenated hemoglobin and deoxygenated hemoglobin, a temporal convolutional network is used to extract the blood oxygen fluctuation pattern characteristics and abnormal noise.

[0040] For pulse wave signals, an autoencoder is used to extract complex time series relationship features such as heart rate, heart rate variability (HRV) and abnormal noise information.

[0041] In the embodiment, Fig.13 As shown in the figure, after extracting features, the multimodal data fusion and evaluation module also uses a Transformer model based on a multi-head attention mechanism to perform feature fusion, and then evaluates the data quality based on the fused features and predicts the dominant external events that affect the data quality.

[0042] Specifically, during feature fusion, all class features extracted from multimodal data are concatenated, linearly transformed, and normalized, and then input into the Transformer model based on the multi-head attention mechanism. The multi-head attention mechanism is used to dynamically adjust the importance of different modalities, and the Transformer encoder layer is used to extract cross-modal joint features for all modal data.

[0043] Specifically, during the evaluation and analysis, after mapping the fused features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate the data quality based on the mapped fused features, and a classification algorithm is used to mark the dominant external events that affect the data quality based on the mapped fused features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise.

[0044] In the embodiment, the transmission module also transmits the pre-processed hemodynamic signals, electrical signals of various parts, data quality and dominant external events to the terminal through wireless communication, and receives parameter adjustment information and equipment adjustment instructions fed back by the terminal. The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module based on the received parameter adjustment information, and adjust the equipment according to the equipment adjustment instructions. Among them, the acquisition parameters include the front-end amplification factor of the electrical signal and the emission light power of the optical signal; the equipment adjustment instructions include the loose state of the equipment, reminding the subject to keep the head still, and reminding to wear a sunshade cap; Front-end amplification factor for electrical signals: When the collected electrical signal is weak, remind the user to amplify the signal until the RMS value is not less than 10 microamps; Regarding the transmitted optical power of the optical signal: when the optical signal is weak, remind the user to increase the transmitted optical power so that the light intensity at the receiving end is not less than A milliwatts and cannot be greater than B milliwatts; A is the minimum value of the hemodynamic signal calculated by the Bill-Lambert calculation formula; B is the standard value for optical radiation safety of the International Electrotechnical Commission (IEC); For loose device status: when loose events dominate, remind the user to re-apply the device; Remind the subjects to keep their heads still: When the motion event is dominant, remind the users to keep their heads still; Reminder to wear a light-shielding hat: When light noise is dominant, remind users to cover the device collection area with black opaque objects to prevent ambient light crosstalk.

[0045] The embodiment further provides a multimodal data evaluation method, which adopts the above-mentioned electrophysiological and hemodynamic multimodal intelligent wearable system and includes the following steps: Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module; The central module is used to digitize the collected multimodal data, pre-process it, and then transmit it to the terminal. At the same time, the central module performs multimodal data fusion and data quality assessment on the original data and predicts the dominant external events that affect the data quality. The data quality and dominant external events are output to the terminal to feedback and adjust the collection parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

[0046] The system of the present invention is designed for real-time monitoring of EEG, EMG, EOG, near-infrared brain functional imaging and pulse wave signals. The unique sandwich structure design, including optical modules, flexible and stretchable electrical signal modules and central modules, ensures comfort and stability for long-term wearing. The system integrates heterogeneous and heterogeneous multimodal acquisition devices to fully cover the needs of electrophysiological and hemodynamic monitoring. The optical module and the electrical signal module can be used independently or in combination with multimodality. The multi-parameter data fusion method integrates and analyzes the features of heterogeneous multi-source data to achieve a comprehensive and integrated evaluation of the quality of multimodal signals, while accurately marking the dominant external events that affect the data quality, thereby improving the reliability of the data. The multi-parameter closed-loop feedback module analyzes the data output results, evaluates the data acquisition quality, intelligently optimizes the equipment acquisition parameters, and intelligently recommends users to adjust the acquisition site, significantly improving the effectiveness and practicality of monitoring.

[0047] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An electrophysiological and hemodynamic multimodal intelligent wearable system, characterized in that: It includes an optical module, a flexible and stretchable electrical signal module, and a central module, wherein at least one of the optical module and the flexible and stretchable electrical signal module is detachably connected to the central module; The optical module is used to collect blood dynamic signals through near-infrared light signals, including oxygenated hemoglobin concentration change signals, deoxygenated hemoglobin concentration change signals, and pulse wave signals; The flexible and stretchable electrical signal module collects electrophysiological signals from various parts of the body through electrodes, including electroencephalogram signals, electromyography signals, electrocardiogram signals, and electrooculogram signals; The central module is used to digitize and pre-process the hemodynamic signals and electrophysiological signals of various parts, and directly transmit the high-quality signals to the terminal, while multimodal data is integrated and evaluated for data quality and the dominant external events that affect the data quality are predicted. At the same time, the data quality and the dominant external events are also output to the terminal to feedback and adjust the acquisition parameters of the optical module and the flexible stretchable electrical signal module and the equipment adjustment instructions.

2. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The optical module includes a plurality of transmitters, a plurality of receivers, and a signal transmission interface located on a first base circuit board; The transmitter is used to transmit light signals of different near-infrared wavelengths to human tissue; The receiver is used to receive the reflection signal formed after the light signal acts on human tissue, and uses the photoelectric signal processing algorithm to process the collected reflection signal data of different wavelengths according to brain function activities and hemodynamic reactions, and the absorption and scattering rate of light by blood to obtain the corresponding hemodynamic signal; The signal transmission interface is used to be detachably connected to the optical module interface of the central module and transmit the collection signal of the receiver.

3. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The flexible and stretchable electrical signal module includes a plurality of electrode patches connected to the second base circuit board, a hollow window and a signal conduction interface, wherein some of the electrode patches can be flexibly stretched through electrode wires; The electrode patch is used to continuously monitor the electrophysiological signals of various parts; The hollow window is used to provide a light passage for the transmitter and receiver of the optical module; The signal transmission interface is used to be detachably connected to the electrical module interface of the central module to transmit the monitored electrophysiological signals to the central module.

4. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The central module includes a power supply module, an optical module interface, an electrical module interface, a digitization module, a preprocessing module, a multimodal data fusion and evaluation module, a parameter closed-loop feedback module, and a transmission module; The power supply module uses a lightweight mobile battery to provide power to the system; The optical module interface is used for detachably connecting with the optical module; The electrical module interface is used for detachably connecting with the flexible and stretchable electrical signal module; The digitization module is used to digitize the collected multimodal data at the front end; The preprocessing module is used to preprocess the digitized multimodal data and transmit the preprocessed high-quality signal to the transmission module; The multimodal data fusion and evaluation module is used to extract features from the digitized multimodal data and to fuse features using a Transformer model based on a multi-head attention mechanism, and then evaluate the data quality based on the fused features and predict the dominant external events that affect the data quality; The transmission module is used to transmit the pre-processed hemodynamic signals, electrophysiological signals of various parts, data quality and dominant external events to the terminal through wireless communication, and receive the acquisition parameter adjustment information and equipment adjustment instructions fed back by the terminal; The parameter closed-loop feedback module is used to control and adjust the acquisition parameters of the optical module and the flexible and stretchable electrical signal module based on the received parameter adjustment information, and to adjust the equipment according to the equipment adjustment instructions.

5. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: Pre-process the blood dynamic signals and electrophysiological signals of various parts and transmit them to the transmission module, including: For EEG signals, filtering is performed, multi-channel independent component analysis is used to identify and remove electrooculographic artifacts and electromyographic interference, and the signal amplitude is normalized to enhance signal consistency; For the electromyographic signal, bandpass filtering and full-wave rectification are performed, the electromyographic signal envelope is extracted using the moving average method, and the signal amplitude is mapped to a standardized range; For ECG signals, a bandpass filter is used to remove baseline drift and high-frequency noise, wavelet transform is used to remove myoelectric interference, the Pan-Tompkins algorithm is used to detect the R wave peak, and the heartbeat cycle is converted into a binary sequence; For the electrooculogram signal, the static baseline is calculated and the drift is removed, bandpass filtering is performed, and principal component analysis is used to remove the interference of electromyography and environmental noise, and to identify the encoding of eye movement events and gaze events; For the near-infrared brain functional imaging signals of oxygenated hemoglobin concentration change signals and deoxygenated hemoglobin concentration change signals, the Beer-Lambert law was used to convert the light intensity signals into hemodynamic concentration change signals, a low-pass filter was used to remove respiratory and heartbeat interference, multi-channel independent component analysis was used to remove motion artifacts, and the signals were baseline-standardized to eliminate systematic errors; For the pulse wave signal, a bandpass filter is used to filter it, the signal baseline is calculated and the drift component is removed, and the moving average method is used to smooth the signal.

6. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: The digitized EEG signals, EMG signals, EOG signals, ECG signals, oxyhemoglobin concentration change signals, deoxyhemoglobin concentration change signals and pulse wave signals with inconsistent sampling rates are encoded separately, including: The EEG signal is used to extract EEG frequency space features and abnormal signal features through convolutional neural network; The electromyographic signal captures the motion characteristics and abnormal signal characteristics through the convolutional neural network; The electrooculogram signals are used to extract the eye movement event-related pattern features and abnormal signal features through a recurrent neural network; The ECG signal uses an autoencoder to extract the embedded features of the heartbeat cycle and abnormal signal features; The oxygenated hemoglobin concentration change signal and the deoxygenated hemoglobin concentration change signal are extracted based on the time convolution network to extract the blood oxygen fluctuation pattern characteristics and abnormal signal characteristics; The pulse wave signal uses an autoencoder to capture the temporal relationship characteristics of the pulse beat and the abnormal signal characteristics; The output sequences of the extracted features have the same length.

7. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: The Transformer based on the multi-head attention mechanism is used for feature fusion, including: After feature concatenation, linear transformation and normalization, all class features extracted from multimodal data are input into the Transformer model based on the multi-head attention mechanism. The multi-head attention mechanism is used to dynamically adjust the importance of different modalities, and the Transformer encoder layer is used to extract cross-modal joint features for all modal data.

8. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 4, characterized in that: Evaluate data quality based on fused features and predict leading external events that affect data quality, including: After mapping the fused features using at least one fully connected layer, a regression algorithm is used in the output layer to evaluate the data quality based on the mapped fused features. The specific prediction signal quality is the sum of the duration of the noise events identified. The total data collection time is , take Score = Rate signal quality; In the output layer, a classification algorithm is also used to mark the dominant external events that affect data quality based on the mapped fusion features, including weak electrical signals, weak optical signals, movement, blinking, looseness, and optical noise.

9. The electrophysiological and hemodynamic multimodal intelligent wearable system according to claim 1, characterized in that: The acquisition parameters include the front-end amplification factor of the electrical signal and the emission light power of the optical signal; the equipment adjustment instructions include the loose state of the equipment, reminding the subject to keep the head still, and reminding the subject to wear a sunshade cap; Front-end amplification factor for electrical signals: When the collected electrical signal is weak, remind the user to amplify the signal until the RMS value is not less than 10 microamps; Regarding the transmitted optical power of the optical signal: when the optical signal is weak, remind the user to increase the transmitted optical power so that the light intensity at the receiving end is not less than A milliwatts and cannot be greater than B milliwatts; A is the minimum value of the hemodynamic signal calculated by the Bill-Lambert calculation formula; B is the standard value of the International Electrotechnical Commission for optical radiation safety; For loose device status: when loose events dominate, remind the user to re-apply the device; Remind the subjects to keep their heads still: When the motion event is dominant, remind the users to keep their heads still; Reminder to wear a light-shielding hat: When light noise is dominant, remind users to cover the device collection area with black opaque objects to prevent ambient light crosstalk.

10. A multimodal data evaluation method, characterized in that: The method adopts the electrophysiological and hemodynamic multimodal intelligent wearable system according to any one of claims 1 to 9, and comprises the following steps: Perform multimodal data acquisition using at least one of an optical module and a flexible and stretchable electrical signal module; The central module is used to digitize the collected multimodal data, pre-process it, and then transmit it to the terminal. At the same time, the central module performs multimodal data fusion on the original data, evaluates the data quality, and predicts the dominant external events that affect the data quality. The data quality and the dominant external events are output to the terminal to feedback and adjust the collection parameters and equipment adjustment instructions of the optical module and flexible stretchable electrical signal module, and perform equipment adjustments.

Images