Method and system for detecting physical state of operator based on multiple physiological signals

Abstract

The embodiment of the invention provides a method and system for detecting the physical state of an operator based on multiple physiological signals. The method comprises the steps that multiple original physiological signals of the operator in an electric power site and an electric power operation scene are obtained; feature extraction is carried out based on the multiple original physiological signals and the electric power operation scene, and multiple modal features are obtained; wherein the multiple modal features comprise modal features corresponding to the multiple physiological signals respectively; performing multi-modal feature fusion on the multiple modal features to obtain fusion features; and based on the fusion features, performing multi-anomaly detection through a predetermined body state detection model to obtain an anomaly detection result. According to the scheme, the physical state of a worker can be detected in time, and safety accidents are reduced.

Description

Technical Field

[0001] This application relates to the fields of computer science and technology, and in particular to a method and system for detecting the physical condition of workers based on multiple physiological signals. Background Technology

[0002] The power industry is a core infrastructure supporting the nation's economic and social operations, and its safe operation highly depends on the standardized operation of frontline workers. Current power safety assurance systems primarily focus on the standardized supervision of operational procedures. However, in the high-intensity, high-risk working environment of power operations, workers are highly susceptible to various physical discomforts, which can lead to misoperation, safety accidents, and personal injury in severe cases. Therefore, monitoring the physical condition of power workers during operations and promptly detecting any abnormalities is crucial for further improving the safety level of on-site operations in the next-generation smart grid. How to detect the physical condition of workers has become a technical problem that needs to be solved. Summary of the Invention

[0003] This application aims to provide a method and system for detecting the physical condition of workers based on multiple physiological signals, which can detect the physical condition of workers in a timely manner and reduce the occurrence of safety accidents.

[0004] The technical solution of this application is implemented as follows: In a first aspect, embodiments of this application provide a method for detecting the physical state of workers based on multiple physiological signals, the method comprising: Acquire various raw physiological signals from workers at power sites and power operation scenarios; Based on the aforementioned multiple raw physiological signals and the power operation scenario, feature extraction is performed to obtain the multiple modal features; wherein, the multiple modal features include modal features corresponding to each of the multiple physiological signals; Multimodal feature fusion is performed on the aforementioned multiple modal features to obtain fused features; Based on the fusion features, multiple anomaly detections are performed using a pre-determined body state detection model to obtain anomaly detection results.

[0005] In the above scheme, feature extraction is performed based on the multiple original physiological signals and the power operation scenario to obtain the multiple modal features, including: The various raw physiological signals are preprocessed to obtain various physiological signals; wherein, the various physiological signals include electrocardiogram, photoplethysmography pulse wave, skin electrical activity and electroencephalogram. Based on the electrocardiogram, the photoplethysmography (PPG), the electrical activity of the skin, and the electroencephalogram (EEG), slide window slices of a preset length are performed to obtain signal segments of the electrocardiogram, PPG, EEG, and EEG, respectively. The signal segment is subjected to amplitude normalization and standardization processing to obtain the final signal segment; By using a modal convolutional backbone network and the power operation scenario, features are extracted from the electrocardiogram, the photoplethysmography pulse wave, the electrical activity of the skin, and the electroencephalogram to obtain the multiple modal features.

[0006] In the above scheme, the various raw physiological signals include raw electrocardiogram, raw photoplethysmography pulse wave, raw skin electrical activity, and raw electroencephalogram. The preprocessing of the various raw physiological signals yields a variety of physiological signals, including: Determine the acquisition time of the original electrocardiogram, the original photoplethysmogram, the original electrical skin activity, and the original electroencephalogram; Based on the acquisition time, the original ECG, the original photoplethysmography (PPG), the original electrical activity of the skin, and the original electroencephalogram (EEG) are aligned with their original timestamps to obtain the processed original ECG, processed original PPG, processed original electrical activity of the skin, and processed original EEG. The processed raw electrocardiogram, processed raw photoplethysmogram, processed raw electrodermal activity, and processed raw electroencephalogram are resampled to obtain the various physiological signals.

[0007] In the above scheme, the modal convolutional backbone network includes convolutional backbone networks corresponding to the electrocardiogram, the photoplethysmography pulse wave, the skin electrical activity, and the electroencephalogram. The method involves extracting features from the electrocardiogram, photoplethysmography (PPG), electrical skin activity, and electroencephalogram (EEG) using a modal convolutional backbone network and the power operation scenario, resulting in the various modal features, including: The temporal morphological features of the electrocardiogram, the photoplethysmography pulse wave, the skin electrical activity, and the electroencephalogram are extracted through the convolutional backbone network. The multi-scale convolutional branch modules in the convolutional backbone network are used to extract multi-scale features from the electrocardiogram, the photoplethysmography (PPG), the electrical activity of the skin, and the electroencephalogram (EEG), respectively, to obtain the multi-scale features of the electrocardiogram, PPG, EEG, and EEG respectively. Based on the weighted aggregation module in the convolutional backbone network, the temporal morphological features and the multi-scale features are fused to obtain the final fused features; The attention distribution along the temporal dimension is calculated on the final fused features to obtain the enhanced features; and the enhanced features are then subjected to modal aggregation to obtain cross-modal enhanced features. Based on the power operation scenario, the cross-modal enhancement features are subjected to interference weighting processing to obtain the multiple modal features.

[0008] In the above scheme, the step of fusing the multiple modal features to obtain fused features includes: Determine the time series corresponding to each of the multiple modal features; and determine the position encoding based on each time node in the time series; Based on the location encoding and the time series corresponding to each of the multiple modal features, the multiple modal features are concatenated to obtain concatenated features; Multiple mapping parameters are determined; and feature fusion is performed on the spliced ​​features based on the multiple mapping parameters to obtain the fused features.

[0009] In the above scheme, the step of performing multiple anomaly detection based on the fusion features using a pre-determined body state detection model to obtain anomaly detection results includes: The fatigue detection result is obtained by performing fatigue detection on the fused features through the fatigue recognition branch module in the body state detection model. The blood pressure abnormality detection branch module in the body state detection model is used to detect blood pressure abnormalities in the fused features to obtain blood pressure detection results. The normal state detection result is obtained by performing normal state detection on the fused features through the normal state recognition branch module in the body state detection model. The abnormal detection result is obtained by weighting the fatigue detection result, the blood pressure detection result, and the normal detection result.

[0010] In the above scheme, the step of weighting the fatigue detection result, the blood pressure detection result, and the normal detection result to obtain the abnormal detection result includes: Based on multiple raw physiological signals, the temporal characteristics of each of the multiple raw physiological signals are determined; Based on the temporal features of the various original physiological signals and their final fusion features, feature concatenation is performed to obtain a comprehensive information vector. The comprehensive information vector is subjected to nonlinear processing to obtain three weight data values; wherein the sum of the three weight data values ​​is 1. The fatigue detection result, the blood pressure detection result, and the normal detection result are weighted based on the three weighted data to obtain the abnormal detection result value.

[0011] Secondly, embodiments of this application provide a system for detecting the physical state of workers based on multiple physiological signals, comprising: an acquisition unit, a feature extraction unit, a fusion unit, and a detection unit; wherein, The acquisition unit is used to acquire various raw physiological signals of workers at the power site and the power operation scenario; The feature extraction unit is used to extract features based on the multiple original physiological signals and the power operation scenario to obtain the multiple modal features; wherein, the multiple modal features include modal features corresponding to each of the multiple physiological signals; The fusion unit is used to perform multimodal feature fusion on the multiple modal features to obtain fused features; The detection unit is used to perform multiple anomaly detection based on the fusion features and a pre-determined body state detection model to obtain anomaly detection results.

[0012] Thirdly, embodiments of this application provide a worker's physical condition detection device based on multiple physiological signals, the worker's physical condition detection device based on multiple physiological signals includes: a processor and a memory; wherein, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for causing a processor to perform the method described in the first aspect.

[0014] This application provides a method and system for detecting the physical state of workers based on multiple physiological signals. The method includes: acquiring multiple raw physiological signals of workers at a power site and the power operation scenario; extracting features based on the multiple raw physiological signals and the power operation scenario to obtain multiple modal features; wherein the multiple modal features include modal features corresponding to each of the multiple physiological signals; fusing the multiple modal features to obtain fused features; and performing multiple anomaly detection using a pre-determined physical state detection model based on the fused features to obtain anomaly detection results. By employing the above scheme, multiple modal features are obtained through feature extraction from multiple raw physiological signals and the power operation scenario. Subsequently, by fusing the multiple modal features, semantic interaction and information completion between different modalities are achieved. Based on the fused features, multiple anomaly detection using the physical state detection model can promptly detect the physical state of workers, accurately identifying and classifying common physiological states such as fatigue, arrhythmia, and abnormal blood pressure. This enables accurate identification of abnormal physical states of power workers, reducing the occurrence of safety accidents. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0016] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0017] Figure 1 This application provides an optional flowchart illustrating a method for detecting the physical state of workers based on multiple physiological signals, as an embodiment of the present application. Figure 2 This application provides a schematic diagram of the structure of a worker's physical condition detection system based on multiple physiological signals, as an embodiment of the present application. Figure 3 This application provides a schematic diagram of the structure of a worker's physical condition detection device based on multiple physiological signals. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0021] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0022] This application provides a method for detecting the physical state of workers based on multiple physiological signals. Figure 1 This application provides an optional flowchart illustrating a method for detecting the physical state of workers based on multiple physiological signals, which will be combined with... Figure 1 The steps shown are explained.

[0023] S101. Acquire various raw physiological signals of workers at the power site and power operation scenarios.

[0024] In some embodiments of this application, various primitive physiological signals include primitive electrocardiograms, primitive photoplethysmography pulse waves, primitive skin electrical activity, and primitive electroencephalograms.

[0025] In some embodiments of this application, the acquisition of multiple physiological signals, such as raw electrocardiogram (ECG), raw photoplethysmography (PPG), raw electrical skin activity (EDA), and raw electroencephalogram (EEG), provides data support for research on the detection of abnormal human states. Physiological signals, as objective reflections of the human body's state, can provide physiological characteristics for abnormal state detection from different perspectives. For example, abnormal heart rhythms can be detected by extracting features from ECG data, and fatigue can be detected using EEG signals. Just as ECG is typically used to detect abnormal states related to cardiac activity, a single physiological signal is often limited to the detection of a specific type of abnormality. However, there are close correlations between various physiological signals in the human body, and the same abnormal state is reflected to varying degrees in multiple physiological signals. Therefore, more and more research focuses on the information complementarity between different physiological signals, fusing features from multiple physiological signals to study the detection of abnormal states. Compared to single-modality fusion, multimodal signal fusion has a stronger advantage in improving abnormal state recognition. This is because a single physiological signal is difficult to capture the multidimensional features of abnormal human states. The method of fusing multimodal physiological signals can not only improve the limitations of single-modal signals in feature expression, but also enhance the sensitivity of the detection model to complex physiological state changes.

[0026] In some embodiments of this application, the method for detecting the physical condition of workers based on multiple physiological signals is applicable to the scenario of detecting the physical condition of workers in power fields.

[0027] In some embodiments of this application, the execution subject of the method for detecting the physical condition of workers based on multiple physiological signals is a physical condition detection device for workers based on multiple physiological signals.

[0028] In some embodiments of this application, various raw physiological signals of on-site power workers and power operation scenarios are acquired through data acquisition devices. Power operation scenarios include three types: ground inspection, high-altitude operations, and equipment maintenance.

[0029] S102. Based on multiple original physiological signals and power operation scenarios, feature extraction is performed to obtain multiple modal features; among them, the multiple modal features include the modal features corresponding to each of the multiple physiological signals.

[0030] In some embodiments of this application, multiple raw physiological signals are preprocessed to obtain multiple physiological signals; these multiple physiological signals include electrocardiogram (ECG), photoplethysmography (PPG), electrical skin activity (ESA), and electroencephalogram (EEG); based on the ECG, PPG, ESA, and EEG, sliding window slices are performed according to preset lengths to obtain signal segments for each of the ECG, PPG, ESA, and EEG; the signal segments are subjected to amplitude normalization and standardization to obtain the final signal segments; through a modal convolutional backbone network and a power operation scenario, features are extracted from the ECG, PPG, ESA, and EEG to obtain multiple modal features.

[0031] S103. Perform multimodal feature fusion on multiple modal features to obtain fused features.

[0032] In some embodiments of this application, the time series corresponding to each of the multiple modal features are determined; and based on each time node in the time series, a position code is determined; based on the position code and the time series corresponding to each of the multiple modal features, the multiple modal features are concatenated to obtain concatenated features; multiple mapping parameters are determined; and based on the multiple mapping parameters, the concatenated features are fused to obtain fused features.

[0033] S104. Based on fusion features, multiple anomalies are detected using a pre-determined body state detection model to obtain anomaly detection results.

[0034] In some embodiments of this application, fatigue detection is performed on the fused features by the fatigue recognition branch module in the body state detection model to obtain fatigue detection results; blood pressure abnormality detection is performed on the fused features by the blood pressure abnormality recognition branch module in the body state detection model to obtain blood pressure detection results; normal state detection is performed on the fused features by the normal state recognition branch module in the body state detection model to obtain normal detection results; and abnormality detection results are obtained by weighting the fatigue detection results, blood pressure detection results, and normal detection results.

[0035] For example, firstly, based on multiple publicly available physiological signal datasets and considering the complex working conditions at power operation sites, a multi-physiological signal dataset including ECG, PPG, EDA, and EEG was constructed to characterize the physical state of power workers in three typical working conditions: ground inspection, high-altitude operation, and equipment maintenance. To address the differences in structure and expressive power of different physiological signals, an adaptive dynamic gating mechanism was designed to fuse multi-modal signal features. Furthermore, a joint modeling mechanism with time-dependent awareness was constructed by combining Transformer and BiLSTM, further enhancing the model's sensitivity to multi-modal physiological signals. Based on this, a multi-task learning mechanism guided by auxiliary tasks was proposed to determine the abnormal physical state of power workers.

[0036] Understandably, by extracting features from various raw physiological signals and power operation scenarios, multiple modal features are obtained. Subsequently, by fusing these multiple modal features, semantic interaction and information completion between different modalities are achieved. Based on these fused features, a body state detection model can perform multi-anomaly detection, enabling timely detection of workers' physical conditions. This allows for accurate identification and classification of common physiological states such as fatigue, arrhythmia, and abnormal blood pressure, thereby accurately identifying abnormal physical conditions of power workers and reducing the occurrence of safety accidents.

[0037] In some embodiments of this application, S102 can be implemented by S201-S204, as follows: S201. Preprocessing multiple raw physiological signals separately to obtain multiple physiological signals; among them, multiple physiological signals include electrocardiogram, photoplethysmography pulse wave, skin electrical activity and electroencephalogram.

[0038] In some embodiments of this application, various primitive physiological signals include primitive electrocardiograms, primitive photoplethysmography pulse waves, primitive skin electrical activity, and primitive electroencephalograms.

[0039] In some embodiments of this application, the acquisition time of the original electrocardiogram (ECG), original photoplethysmography (PPG), original electrical skin activity (ESE), and original electroencephalogram (EEG) are determined. Based on the acquisition time, the original timestamps of the original ECG, PPG, ESE, and EEG are aligned to obtain the processed original ECG, processed original PPG, processed original ESE, and processed original EEG. The processed original ECG, processed original PPG, processed original ESE, and processed original EEG are resampled to obtain various physiological signals.

[0040] S202. Based on the electrocardiogram, photoplethysmography (PPG), electrical activity of the skin, and electroencephalogram (EEG), slide window slices are performed according to preset lengths to obtain signal segments of the electrocardiogram, PPG, EEG, and EEG, respectively.

[0041] S203. Perform amplitude normalization and standardization processing on the signal segment to obtain the final signal segment.

[0042] S204. Through a modal convolutional backbone network and a power operation scenario, features are extracted from electrocardiogram, photoplethysmography pulse wave, skin electrical activity, and electroencephalogram to obtain multiple modal features.

[0043] In some embodiments of this application, the modal convolutional backbone network includes convolutional backbone networks corresponding to electrocardiogram, photoplethysmography pulse wave, electrodermal activity, and electroencephalogram.

[0044] In some embodiments of this application, temporal morphological features of electrocardiogram (ECG), photoplethysmography (PPG), electrical activity of skin (EDS), and electroencephalogram (EEG) are extracted using a convolutional backbone network. Multi-scale feature extraction is performed on ECG, PPG, EDS, and EEG using multi-scale convolutional branch modules within the convolutional backbone network, yielding multi-scale features for each of these features. The temporal morphological features and multi-scale features are fused using a weighted aggregation module within the convolutional backbone network to obtain the final fused features. Attention distribution along the temporal dimension is calculated on the final fused features to obtain enhanced features. Modal aggregation is then performed on the enhanced features to obtain cross-modal enhanced features. Based on the power operation scenario, interference weighting is applied to the cross-modal enhanced features to obtain multiple modal features.

[0045] For example, this application constructs a multimodal physiological signal dataset of workers under complex working conditions in power field operations based on multiple publicly available and widely recognized physiological signal datasets to detect several abnormal physical states, including abnormal blood pressure, arrhythmia, and fatigue. The WESAD dataset was collected by Schmidt et al. at the University of Augsburg, Germany, in a laboratory environment. Subjects experienced various emotional and physiological stimuli, such as relaxation, stress, and fatigue, under controlled conditions. It includes three physiological signals: ECG, PPG, and EDA, categorized as fatigue, relaxation, etc. The FatigueSet dataset is a multimodal physiological signal dataset collected under a real fatigue-induced environment. Multiple sensors simultaneously record ECG, PPG, EEG, and EDA signals of subjects during continuous cognitive and motor tasks, categorized as fatigued and non-fatigued states. The CufflessBP dataset was collected by Kachuee et al. at Hanyang University, South Korea, under non-invasive cuffless measurement conditions. It includes ECG and PPG signals and uses a traditional cuffed blood pressure monitor as a reference to ensure effective labeling and verification of abnormal blood pressure states, including both abnormal and normal blood pressure states. Furthermore, the MIMIC-III dataset is a patient monitoring database from the long-term intensive care unit at Beth Israel Deaconess Medical Center in Massachusetts, USA. It covers various physiological signals such as ECG, PPG, and EEG, with recording periods ranging from several hours to several days, including physiological abnormalities such as arrhythmia and hypertension, demonstrating high medical reliability and completeness. This study employed a unified preprocessing workflow for the raw multimodal physiological signal data to ensure consistency in signal quality and feature distribution across different data sources.

[0046] To ensure consistency of multimodal data across time and statistical scales, this application employs a preprocessing workflow of uniform resampling followed by fixed-length sliding window slicing. First, each modality is aligned according to its original timestamp and then uniformly resampled to 128Hz. Subsequently, fixed-length sliding window slicing is performed on the aligned sequence: window length... Step length To ensure the validity of the segments, if the proportion of valid samples for any modality within a window is less than 95%, the entire window is discarded; segments shorter than a full window length are not padded and are ignored. Finally, the global mean is calculated for each modality on the training set. with standard deviation z-score standardization is consistently applied to the three subsets: train, val, and test. Based on this, to enhance the comparability of amplitudes across subjects, a linear scaling to [-1, 1] is performed on each window.

[0047] The environment at power work sites is complex, and different working conditions are often accompanied by multiple interference factors. The three common types of noise—power frequency electromagnetic interference (IF), electromyographic artifacts (EM), and baseline drift (BW)—may occur simultaneously in various scenarios. However, due to differences in the working environment and personnel movements, their relative intensity and impact on different modal signals vary. To more closely resemble actual conditions, this paper superimposes scenario-specific noise combinations onto multimodal physiological signals (ECG, PPG, EDA, EEG) to simulate signal pollution under real working conditions. In ground inspection scenarios, workers are mostly located in areas with concentrated transmission lines and high-voltage equipment, where there is a significant 50 / 60Hz power frequency electromagnetic field and its harmonic coupling, making power frequency interference (IF) the most prominent. In addition, prolonged walking and posture changes can also cause some baseline drift (BW), while occasional bending or operational movements may bring slight electromyographic interference (EMG). In high-altitude work scenarios, workers frequently engage in high-intensity activities such as climbing and pulling, accompanied by strong muscle contractions and electrode displacement, resulting in significant electromyographic (EMG) interference. Simultaneously, wind disturbances and postural instability cause baseline fluctuations, and nearby high-voltage lines may also superimpose power frequency interference. In equipment maintenance scenarios, personnel need to continuously bend over, turn around, and start / stop equipment, causing significant low-frequency disturbances; therefore, baseline drift interference (BW) is the most prominent. Furthermore, the exertion during operation also introduces intermittent EMG interference, while the operation of nearby equipment generates a certain degree of power frequency noise. The specific mathematical description is as follows: (1) in, This represents the synthesized physiological signal in scenario s; This represents the corresponding reference signal; , , These are power frequency electromagnetic interference, electromyography artifacts, and baseline drift noise, respectively. represents the relative weights (proportions) of the three types of noise in scene s.

[0048] The dominance of the three types of interference sources varies in different work scenarios. Therefore, different weights are assigned to power frequency interference, electromyographic noise, and baseline drift noise in the simulation. , and And always satisfy Because personnel are located in an area with dense electrical equipment, they are susceptible to the effects of 50 / 60Hz power frequency electromagnetic fields and their harmonic coupling. Therefore, power frequency interference is the main noise source, and [the system] is set accordingly. Electromuscular noise and baseline drift are relatively weaker, and [the system] is set accordingly. In high-altitude work scenarios, high-intensity movements such as climbing and pulling cause significant electromyographic interference; therefore, electromyographic noise is the primary noise source, and the following settings are implemented. The weights of the other two items are set as follows: In equipment maintenance scenarios, frequent bending and changes in posture can easily cause significant baseline drift; therefore, a baseline shift is set... ,at the same time The above weighting settings ensure that the dominant disturbance type is most prominent under each operating condition, while retaining the combined effects of other disturbances.

[0049] The preprocessed four-modal physiological signals were sliced ​​using a fixed-length sliding window, and any modality was recorded. One input segment is: (2) Where T is the time step of the window, which is set to 128 in this study; Let m be the number of channels for mode m. To avoid interference with model training caused by differences in physical quantities such as numerical range and signal amplitude between different modes, it is necessary to perform signal segment analysis for each mode. Each channel undergoes amplitude normalization and Z-score standardization, ensuring that all channels have approximately zero mean and unit variance within the sliding window. (3) in, and These represent the mean and standard deviation of the m-th channel within the current sliding window, respectively. This standardization step ensures consistency in the numerical scale of signals from different modalities, preventing the network from biasing certain modalities too high or too low during training. Furthermore, to ensure a uniform temporal structure for each modal feature, facilitating consistent temporal alignment and operational simplification in subsequent processing steps such as intra-modal multi-scale convolution extraction, attention mechanism modeling, and inter-modal gating fusion, the sliding window length T is uniformly set to a fixed value of 128. This processing preserves local temporal morphological features to the greatest extent possible while also providing structured input for subsequent encoding modules.

[0050] To fully explore the structural characteristics of different modal physiological signals, this application designs independent modal convolutional backbones (denoted as Backbonems) for four modalities: ECG, PPG, EDA, and EEG. These backbones are used to extract modality-specific temporal morphological features and uniformly map them to a fixed-dimensional representation space d. Each backbone consists of a set of standard one-dimensional convolutional processing steps, including convolutional layers (Conv1D), batch normalization (BN), the nonlinear activation function ReLU, and linear channel projection layers. The independent modal convolutional backbones are customized according to modal differences. For ECG signals, the duration of the key structural QRS wave is generally 0.06–0.12 seconds (corresponding to 8–15 sampling points), so the kernel size is set to 9 and the stride is set to 1 to extract local sharp waveform changes in the signal. For PPG signals, the pulse wave period typically ranges from 0.5 to 1.5 seconds (approximately 64–192 sampling points @ 128 Hz). To effectively capture the periodic fluctuation trend, a convolution kernel size of 64 and a stride of 2 are set. For EDA signals, the main frequency components are concentrated in the low-frequency band below 0.5 Hz, and the signal changes slowly. Therefore, a convolution kernel size of 128 and a stride of 4 are set to ensure that the convolution receptive field can cover a longer time window to extract trend information. For EEG signals, since they simultaneously contain multiple frequency band rhythms (e.g., alpha waves 8–12 Hz, beta waves 13–30 Hz, etc.), a multi-scale parallel convolution structure is used to extract high and low frequency features simultaneously. Within the same convolutional layer, three kernel sizes of 16, 32, and 64 are set in parallel to extract the corresponding frequency band rhythms. The above operations enable the sliding window input sequences of each modality to... After Backbonem encoding, all are uniformly mapped to a dimension of The standard representation completes the alignment between modalities in the structural dimension, providing a consistent input basis for subsequent cross-modal feature interaction and fusion.

[0051] To further enhance the temporal awareness of modality coding, this application introduces a multi-scale convolutional branch module in each modality backbone to simultaneously consider local details and global trends. Specifically, given a modality input... Each branch uses L parallel convolutional branches, with each branch having a different kernel size. With step size This yields feature representations at multiple scales. ,in These features are then further fed into a weighted aggregation module to obtain the final fused features. Its definition is as follows: (4) in, Indicates the first The attention weights for each branch are used to control the contribution of each scale feature to the final fusion result. The weights are calculated by performing global average pooling (GAP) on each scale feature and then feeding it into a lightweight multilayer perceptron (MLP). The calculation process is as follows: (5) (6) Formula (5) is a softmax weight normalization constraint, ensuring that the sum of all scale weights is 1, thus forming a normalized multi-scale weighted fusion. The final... It also includes local details extracted by short convolutional kernels and trend changes captured by long convolutional kernels.

[0052] To enhance the representational ability of multimodal physiological signals, this paper first introduces an intramodal self-attention mechanism on the fused features of each modality to improve the global dependency modeling in the temporal dimension. Given input features... Through three learnable linear transformation matrices Map these to a query, key, and value matrix respectively: (7) (8) (9) Then, the scaling dot product attention mechanism is used to calculate the attention distribution along the temporal dimension, and the weighted summation is used to obtain the enhanced features. : (10) in, The key vector dimension, scaling factor To stabilize gradients, softmax achieves attention normalization along the sequence dimension. This mechanism can effectively capture key moment dependencies in time series, amplify physiological segment features such as QRS complexes in ECG and rapid pulse rise edges in PPG, and enhance the discriminative power of modal representations.

[0053] To address the issue of single-modal attention being susceptible to noise interference in real-world scenarios (such as poor ECG contact and PPG being affected by motion artifacts), this paper further proposes a cross-modal attention mechanism (CMA) to achieve dynamic fusion of complementary information between modalities. A query matrix is ​​constructed centered on modality m. and with the remaining modes key-value pairs Interactive, aggregated cross-modal augmented features Represented as: (11) in, Let m be the query matrix for modality m. These are the keys and values ​​of other modalities, respectively; the weights assigned through softmax reflect the contribution of different modalities to the current modality; the summation operation ensures that the supplementary information from all modalities can be integrated into the enhanced features of modality m. This mechanism allows each modality to not only retain its own feature representation capabilities but also dynamically extract collaborative supplementary information from other modalities, thereby mitigating the uncertainty caused by single-modal noise. For example, if the PPG quality deteriorates due to motion interference, the stable trend components in the EEG and EDA signals can provide auxiliary information to achieve mutual reinforcement.

[0054] After completing self-attention modeling within a modality and cross-modal attention modeling between modalities, the features of each modality simultaneously contain temporal-series dependency information constructed within that modality and complementary enhancement information passed from other modalities. To achieve dynamic weight adjustment among these two types of information, this application introduces a gated fusion selector (GFS) mechanism to automatically determine whether to rely more on its own features or leverage auxiliary information from other modalities based on different operational contexts. The model takes the self-attention enhancement features and cross-modal enhancement features of each modality as input, first concatenating them along the channel dimension, and then compressing the temporal features into a channel representation through global average pooling. Subsequently, this representation is fed into a lightweight two-layer perceptron (MLP) to generate a normalized gate vector, which serves as the adjustment factor for the current modality fusion strategy. This gate vector is used to weightedly fuse the two types of feature sources, thereby generating the final gated fusion feature. This mechanism is dynamically adaptive: for example, when the ECG signal is clear and rhythmic, the model will be more inclined to retain its own structural features; while when the PPG signal is severely interfered with by hand movements, the model will increase its reliance on supplementary information from other modalities through a gating mechanism, thereby achieving complementary correction of information between modalities.

[0055] This application proposes a multimodal temporal gate network (MMT-GateNet) model for detecting bodily anomalies based on multimodal physiological signals. The MMT-GateNet model comprises three modules: adaptive gated feature extraction of multimodal physiological signals, multimodal feature fusion based on Transformer-BiLSTM, and multimodal anomaly detection guided by an auxiliary task. The model first independently encodes the temporal features of ECG, PPG, EEG, and EDA signals. Then, a cross-modal attention module enables semantic interaction and information completion between different modalities. A gating mechanism is introduced in the fusion layer to filter redundant information and enhance the expressive power of key modal features. To improve the model's ability to distinguish easily confused categories (such as fatigue and normal), an auxiliary task branch and an uncertainty-guided loss function are introduced to reinforce learning of small sample categories. Finally, the model supports fine-grained differentiation of four states (normal, fatigue, arrhythmia, and abnormal blood pressure).

[0056] In some embodiments of this application, S103 can be implemented by S301-S303, as follows: S301. Determine the time series corresponding to each of the multiple modal features; and determine the position encoding based on each time node in the time series; S302. Based on the location encoding and the time series corresponding to each of the multiple modal features, multimodal feature concatenation is performed on the multiple modal features to obtain the concatenated features; S303. Determine multiple mapping parameters; and perform feature fusion on the spliced ​​features based on the multiple mapping parameters to obtain fused features.

[0057] For example, the physiological signals of power workers often exhibit both slow, long-term evolution (such as fatigue accumulation) and sudden, rapid anomalies (such as abnormal QRS waveforms in arrhythmias or a sudden rise in blood pressure). This application employs a Transformer and BiLSTM cascaded structure to jointly model long-term dependencies and short-term dynamics. The Transformer captures long-term temporal dependencies through a multi-head attention mechanism, achieving information alignment and global interaction across the entire sequence. Subsequently, the BiLSTM extracts local high-frequency fluctuation features through gating units, thereby enhancing the model's adaptability to multi-scale changes in temporal features. To preserve location information, a learnable location code is first added to each time step, making the concatenated input feature sequence as follows: The input to the encoder is obtained by adding the concatenation features position by position. The formula is expressed as: (12) Where T is the number of time steps in a single window, D=4d is the channel dimension after concatenation, representing a learnable position encoding matrix, with each time step corresponding to a set of position vectors. This is then input into the Multi-Head Attention (MHA) module, which learns different substructures to capture sequences through multiple query-key-value subspaces. (13) in, Let h be the mapping parameters for the h-th attention head. is the scaling factor. All attention head outputs are concatenated and then linearly transformed. Finally, the attention result is added to the residual connection and normalized to obtain the Transformer output.

[0058] While the Transformer has significant advantages in modeling global dependencies, its equilibrium properties may be insufficient for representing local dynamics of transient changes. Therefore, this application introduces a Bidirectional Long Short-Term Memory (BiLSTM) network as a supplement to enhance the model's sensitivity to local fluctuations. This module recursively iterates from both the forward and backward directions of the time series, capturing the dynamic correlation between the current time step and its neighboring time steps. The forward and backward hidden states are then concatenated to form a locally enhanced representation for each time step, enabling the model to gain more nuanced dynamic perception while preserving global semantics. For the t-th time step of the input sequence, its forward and backward hidden states are represented as follows: (14) The bidirectional hidden states are concatenated to obtain the context-enhanced representation for each time step, as follows: (15) Where d represents the hidden state dimension of the unidirectional LSTM, and the concatenated representation of each time step is 2d-dimensional.

[0059] The next step is to converge the sequence into vectors. This is to drive the classifier. The results are aggregated into a sequence-level vector g. Average pooling is used by default; it is robust and has low overhead. Attention pooling can highlight key moments, but it is more prone to overfitting under strong noise conditions. Average pooling is used in the main results to ensure generalization consistency. Its formula is expressed as: (16) in, It is a learnable vector. Let g be the weight at time step t. Input g into a linear layer to obtain four classes of logits, and then use softmax to obtain the class probabilities. The main task employs multi-class cross-entropy loss. Its formula is expressed as: (17) (18) in, , For learnable parameters, It is the one-hot true label component.

[0060] Through the above-mentioned modeling strategy from global to local, the model introduces BiLSTM to enhance the characterization of dynamic sensitivity based on the semantic consistency features provided by Transformer, thereby achieving unified modeling of slow-changing processes (such as fatigue accumulation and blood pressure fluctuation trends) and sudden abnormalities (such as sudden changes in heart rhythm).

[0061] In some embodiments of this application, S104 can be implemented by S401-S403, as follows: S401. The fatigue detection results are obtained by performing fatigue detection on the fused features through the fatigue recognition branch module in the body state detection model.

[0062] S402. Blood pressure abnormality detection is performed on the fused features through the blood pressure abnormality identification branch module in the body state detection model to obtain the blood pressure detection result.

[0063] S403. The normal state detection branch module in the body state detection model is used to perform normal state detection on the fused features to obtain the normal detection result.

[0064] S404. The abnormal detection results are obtained by weighting the fatigue detection results, blood pressure detection results and normal detection results.

[0065] In some embodiments of this application, the temporal characteristics of each of the multiple original physiological signals are determined based on multiple original physiological signals; based on the temporal characteristics of each of the multiple original physiological signals and the final fusion characteristics of each of the multiple original physiological signals, feature splicing is performed to obtain a comprehensive information vector; the comprehensive information vector is subjected to nonlinear processing to obtain three weighted data values; wherein the sum of the three weighted data values ​​is 1; the fatigue detection result, blood pressure detection result and normal detection result are weighted based on the three weighted data to obtain the abnormal detection result value.

[0066] For example, the body anomaly detection model (i.e., body state detection model) introduces two parallel branch structures: an auxiliary task network and a relevance weight network. The core idea is to decouple the coupled supervision of the four-class classification into three binary classification sub-tasks oriented towards the key categories (fatigue, bp_abnormal, normal), with relevance weights adaptively applied to the samples. This determines which type of auxiliary supervision the current sample needs more. This not only alleviates the backbone bias caused by class imbalance, but also strengthens the discriminative features of samples near class boundaries.

[0067] Using the sequence-level vector g as a shared representation, three lightweight binary classification heads are established. The fatigue recognition branch receives global features from the fusion encoding module, processes them through a linear transformation and activation function, and outputs the probability that the sample belongs to a fatigued state. The blood pressure abnormality recognition branch uses the same structure and process to determine whether a sample has abnormal blood pressure characteristics. The normal state recognition branch is used to determine whether a sample belongs to a healthy state without abnormalities. The binary classification cross-entropy loss of each auxiliary head is: (19) in This indicates whether the current sample belongs to the positive class of the auxiliary head (e.g., when the true label is fatigue). (The other two ends are 0). It is optimized in parallel with the four-class classification supervision of the main task. The auxiliary head provides a strong directional gradient for a single class, which can alleviate the problem of small sample classes (such as bp_abnormal) being overwhelmed by the main task.

[0068] It is worth noting that not all samples need to be supervised by all auxiliary tasks simultaneously. To improve the effectiveness and flexibility of auxiliary supervision, a sample-level dynamic weight generation module was further designed, making the supervision of auxiliary tasks "sample-specific" and more closely aligned with the characteristics of the samples. Specifically, a small neural network is used to dynamically generate three weights based on the temporal characteristics of each sample, representing the importance or attention given to the sample in the three states of "fatigue," "abnormal blood pressure," and "normal," respectively.

[0069] In the model, three weights are dynamically generated for each sample, representing its importance in the three states of "fatigue," "abnormal blood pressure," and "normal," respectively. Specifically, the sample's temporal features (i.e., changes in physiological signals collected over a period of time) are first concatenated with the global features of the entire dataset to form a comprehensive information vector. This vector is then fed into a small neural network, undergoing a series of non-linear processing steps to output three weight values. The sum of these three values ​​is always equal to 1, representing the model's proportion of attention given to the three states. For example, if a sample's weights are 0.7, 0.2, and 0.1, it means the model considers the sample more likely to be in the "fatigue" state. To prevent the model from prematurely favoring one class during training (e.g., focusing only on the "normal" class while ignoring the "fatigue" class), a balancing mechanism is added. Initially, the weights for the three classes are assumed to be equal, with each class accounting for one-third. The model then dynamically adjusts these weights based on real-world data. In addition, a strategy of early guidance and late convergence was adopted. In the early stage of training, the influence of auxiliary modules on weights was enhanced to help the model learn to distinguish categories more quickly. In the later stage of training, the role of auxiliary modules was gradually weakened, allowing the main task to control the overall training, thereby improving the final accuracy and stability.

[0070] The specific implementation is as follows: The experimental environment for this application is configured as follows: the operating system is Ubuntu 18.04.6, the runtime environment is Python 3.8.10, and the deep learning framework is PyTorch 1.13.0. The hardware platform uses an NVIDIA Tesla T4 GPU, and CUDA 11.6 and cuDNN 8.3 are installed to support efficient parallel computing and deep learning model training.

[0071] The experimental results are as follows: Results of abnormal physical examination of power workers: To verify the training feasibility and classification performance reliability of the proposed MMT-GateNet model under various power operation scenarios, the training loss and validation loss were analyzed over the iterative process for four operating conditions (ground inspection, high-altitude operation, equipment maintenance, and mixed conditions), as well as the final classification confusion matrix on the training set. From the training dynamics, the loss curves under each operating condition showed a stable decreasing trend and gradually converged in the later stages. No significant overfitting was observed between the training and validation losses, indicating that the proposed model can be effectively trained in different operating scenarios and possesses good convergence and stability.

[0072] During training, both training and validation losses gradually decreased with increasing training epochs across all scenarios, converging after approximately 150 epochs without significant overfitting. Specifically, the validation loss in equipment maintenance and mixed scenarios was close to the training loss, indicating relatively stable model fitting ability in these two scenarios. In contrast, the validation loss decreased more slowly in the ground inspection scenario, suggesting that the training data may have limited its support for the model's generalization ability in this scenario. Regarding classification performance, the confusion matrices for different work scenarios show some differences in the recognition of each state category. In the ground inspection scenario, the accuracy rate for identifying fatigue was 72.0%, slightly lower than other categories, with some samples misclassified as normal or HR. In the high-altitude operation scenario, the accuracy rates for fatigue and HR states were high, but the normal state showed some confusion. In the equipment maintenance scenario, the overall classification accuracy for the four states was high, with normal and BP states reaching 96.3% and 96.8%, respectively. In mixed scenarios, the model's recognition accuracy for the four states is relatively balanced, ranging from 84.9% to 91.3%. However, some cross-classification errors still exist between BP and HR. These results indicate that the model's classification boundaries differ in different scenarios, and some categories are easily confused under specific conditions, which may be related to differences in signal patterns and sample distribution.

[0073] To evaluate the classification performance of the MMT-GateNet model under different power operation conditions on the test set, this application conducted a comprehensive analysis based on four metrics: accuracy, F1-score, precision, and recall. Overall, the model achieved high average performance across all four operation conditions (Ground, High, Service, and Combined), demonstrating its stability and reliability under various operating conditions. The anomaly detection results based on the MMT-GateNet model on the test set are shown in Table 1.

[0074] Table 1

[0075] Based on the classification performance distribution under various working conditions, the equipment maintenance (Service) condition exhibited the best classification performance, with an average F1-Score of 84.88%. The accuracy rates for both the normal and arrhythmia categories exceeded 89%, indicating that the model possesses high discriminative ability under this condition. This is mainly due to the obvious signal rhythm and fewer interference factors in the equipment maintenance scenario, which facilitates the full extraction of time-series features by the convolutional structure. Simultaneously, the gating mechanism further enhances the importance of key modal channels. In contrast, the high-altitude operation condition had a lower average F1-Score of only 80.11%, particularly poor performance in fatigue category recognition, with an F1-Score of 68.25%. This result indicates that in high-altitude scenarios, physiological signals are easily affected by factors such as changes in body position, strong light interference, and wind vibration disturbances, leading to increased signal fluctuations and artifacts, thus weakening the model's ability to learn fatigue-related features. The ground inspection and comprehensive conditions showed moderate classification performance, with average F1-Scores of 82.42% and 82.48%, respectively. While ground-based scenarios exhibit some noise interference, the overall signal stability remains high, allowing the model to maintain excellent recognition performance. The comprehensive working condition scenario, trained using multi-scenario signal fusion, demonstrates strong robustness and generalization ability in complex and diverse environments. Further analysis of the performance of each anomaly category under different working conditions reveals that arrhythmia shows the most stable recognition performance, with an F1-Score exceeding 88% in all four scenarios. This indicates that cardiac rhythm signals possess strong discriminative power in the time-frequency domain and can be accurately captured by convolution and attention mechanisms. In contrast, fatigue recognition remains challenging. Except for equipment maintenance, the F1-Score in the other three scenarios is below 72%, suggesting that the characteristics of multimodal signals under fatigue conditions exhibit certain uncertainties and interference sensitivity. Future research should focus on improving the model's ability to extract and discriminate weak fatigue features.

[0076] Model structure ablation experiment analysis To systematically evaluate the impact of each module on the performance of the MMT-GateNet model, this application conducted ablation experiments under ground inspection, high-altitude operation, equipment maintenance, and combined operating conditions, and combined feature visualization to assist in the analysis of the model's learning ability.

[0077] Experimental results show that the basic model (A) only possesses basic recognition capabilities under various operating conditions, with significantly insufficient classification accuracy, especially under complex conditions. After introducing the cross-modal gating fusion module (B), the model exhibits better suppression and weighted adjustment capabilities for interference information between different modes, resulting in a significant improvement in classification performance. This indicates that the module effectively enhances the interactive representation between multi-source signals. Further introduction of the temporal modeling module (C) enhances the model's ability to identify abnormal states at different time intervals, demonstrating the advantages of the Transformer-BiLSTM structure in capturing cross-temporal dependencies. The complete model (A+B+C+D) achieves optimal performance under all operating conditions. In equipment maintenance, the accuracy reaches 96.3%, the highest among all conditions, indicating high signal quality and ease of model extraction of stable modes in this scenario. In high-altitude operations and ground inspection, the accuracy reaches 86.1% and 88.6% respectively, maintaining a good level and verifying the model's robustness and stability under interference conditions. Furthermore, the model achieved an accuracy of 91.38% and an F1-Score of 90.74% under mixed operating conditions, demonstrating good generalization ability.

[0078] In the basic model (A), the distribution of samples is mixed and the boundaries between categories are unclear. In contrast, the complete model shows more compact sample aggregation and clearer boundaries in the latent space, indicating that the model's discriminative ability continuously improves with the gradual stacking of modules. In particular, after fusing the gating mechanism and auxiliary tasks, easily confused categories such as fatigue and abnormal blood pressure are better distinguished, demonstrating that this structural combination can effectively uncover key pattern differences. Cross-modal gating fusion enhances the signal co-modeling capability, temporal modeling strengthens the capture of state evolution processes, and auxiliary tasks improve the semantic consistency and training stability of the model. The combined synergy of the above modules effectively improves the recognition performance of the MMT-GateNet model under multiple working conditions and states, verifying the scientific validity and practicality of the proposed method.

[0079] Comparison with other models: To comprehensively evaluate the performance advantages of the proposed MMT-GateNet model in the task of detecting multimodal physiological signal anomalies in power workers, this application selects five representative mainstream methods as comparison models, including: the Auto-Decoder anomaly detection model based on autoencoders, the OCSVM single-class support vector machine, the CNN+LSTM structure integrating convolutional neural networks and long short-term memory networks, the pure convolutional neural network CNN, and the CNN+Transformer model incorporating the Transformer architecture. All models were trained and evaluated on the same four-class classification task test set, and the evaluation metrics included accuracy, F1-score, precision, and recall. The performance comparison of each model on the four-class classification task test set is shown in Table 2.

[0080] Table 2

[0081] As shown in Table 2, MMT-GateNet achieves the highest performance across all metrics in the four-class classification task, demonstrating strong overall classification capabilities. The model achieves 90.74% in F1-score and 90.85% in Recall, indicating that it not only comprehensively covers various sample types but also possesses high consistency prediction ability. The improvement in Recall performance, in particular, shows that the model has a high recognition effect on sample categories with blurred boundaries or indistinct features. In contrast, the Recall of CNN+Transformer is 85.66%. The difference mainly stems from MMT-GateNet's more effective modeling of weakly expressed categories such as fatigue. This capability comes from the modality gating mechanism introduced in the model, which dynamically adjusts the feature channels based on the actual contribution of different modalities to the classification task, thereby strengthening the expression of fatigue-related modalities such as EEG and EDA, and reducing confusion with the normal class. Meanwhile, the precision metric reached 91.15%, indicating that the model maintained a low false positive rate while recalling a large number of samples. This is closely related to its multi-scale attention structure, which can fuse multimodal features at different time scales, providing clearer discrimination boundaries, especially when dealing with time-varying categories such as blood pressure abnormalities. In identifying arrhythmias, MMT-GateNet's channel attention mechanism effectively improved the ability to perceive local abnormal features in ECG signals, allowing the model to retain the diagnostic value of key modalities while fusing multimodal data. For the normal category, the model reduced the risk of misclassifying abnormal categories as normal through a consistency modeling strategy among multimodalities, thereby boosting the overall accuracy to 91.38%. Compared to CNN+Transformer's 87.13%, this result reflects that MMT-GateNet has a more stable strategy design in controlling inter-category misclassification and optimizing sample boundary recognition, resulting in more balanced overall classification performance.

[0082] This application proposes a method for detecting abnormal physical states of personnel in multi-condition environments at power work sites. It integrates multimodal physiological signal feature modeling, gating fusion mechanisms, and auxiliary task-guided training strategies to accurately identify and classify common physiological states such as fatigue, arrhythmia, and abnormal blood pressure. The method has a unified structure, is easy to deploy, and can adapt to signal differences among different workers and in different work scenarios. Specific beneficial effects are as follows: (1) Multimodal physiological data modeling based on multiple working conditions: We constructed a multimodal physiological signal dataset covering three typical working conditions: ground inspection, high-altitude operation, and equipment maintenance. It integrates four types of signals: ECG, PPG, EDA, and EEG, comprehensively covering the physiological changes of workers under different working conditions. This dataset can support the model to jointly model and evaluate various abnormal states (such as fatigue, arrhythmia, and abnormal blood pressure), and improve the model's generalization ability in real and complex scenarios.

[0083] (2) Multimodal feature fusion structure based on gating mechanism: To address the problems of large differences in multimodal signal expression and susceptibility of some modes to interference, this invention introduces an intramodal self-attention mechanism and an intermodal cross-attention mechanism to extract temporal dependencies and complementary information, and designs a gating weight to dynamically adjust the modal contribution, thereby achieving key modality enhancement and redundant modality suppression. This fusion strategy improves the reliability of feature extraction and the separability of abnormal states.

[0084] (3) Multi-task recognition structure based on auxiliary task guidance: In addition to the main classification head, the present invention adds three auxiliary recognition branches for fatigue, abnormal blood pressure and normal state. Combined with the sample-level dynamic weight generation module, the model is guided to focus on samples with blurred boundaries and difficult-to-classify categories, which alleviates the class imbalance problem in the main task and improves the recognition ability of weak class samples.

[0085] Following the above process, this application ultimately constructed a multimodal physiological signal dataset for power operations. Specifically, for three typical working conditions—ground inspection, high-altitude operations, and equipment maintenance—four physiological states (abnormal heart rhythm, abnormal blood pressure, fatigue and health) and three noise levels (power frequency interference, electromyography interference, and baseline drift) were combined to generate a subset of four states containing 750 samples for each working condition, resulting in a total of 3000 noisy samples. A total of 9000 multimodal samples were obtained from the three working conditions, and these were further divided into a training set (N=2400), a validation set (N=300), and a test set (N=300) in an 8:1:1 ratio, thereby ensuring balanced coverage of data under different scenarios, states, and noise conditions, and the reproducibility of model evaluation.

[0086] Based on the above embodiments of the method for detecting the physical state of workers based on multiple physiological signals, this application also provides a system for detecting the physical state of workers based on multiple physiological signals, such as... Figure 2 As shown, Figure 2 This application provides a schematic diagram of a multi-physiological signal-based worker body state detection system. The system includes: an acquisition unit 201, a feature extraction unit 202, a fusion unit 203, and a detection unit 204. The acquisition unit 201 is used to acquire various raw physiological signals of workers at the power site and the power operation scenario; The feature extraction unit 202 is used to extract features based on the multiple original physiological signals and the power operation scenario to obtain the multiple modal features; wherein, the multiple modal features include modal features corresponding to each of the multiple physiological signals; The fusion unit 203 is used to perform multimodal feature fusion on the multiple modal features to obtain fused features; The detection unit 204 is used to perform multiple anomaly detection based on the fusion features and a pre-determined body state detection model to obtain anomaly detection results.

[0087] Based on the above embodiments of the method for detecting the physical condition of workers based on multiple physiological signals, this application also provides a device for detecting the physical condition of workers based on multiple physiological signals, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of a multi-physiological signal-based worker body condition detection device provided in an embodiment of this application. The device includes a processor 301 and a memory 302. The memory 302 stores a computer program; the processor 301 retrieves and runs the computer program from the memory to execute the multi-physiological signal-based worker body condition detection method as described in the above embodiment.

[0088] In the embodiments of this application, the processor 301 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0089] This application provides a computer-readable storage medium storing a computer program for implementing, when executed by a processor, a method for detecting the physical state of workers based on multiple physiological signals as described in any of the above embodiments.

[0090] For example, the program instructions corresponding to the method for detecting the physical state of workers based on multiple physiological signals in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to the method for detecting the physical state of workers based on multiple physiological signals in the storage media are read or executed by an electronic device, the method for detecting the physical state of workers based on multiple physiological signals as described in any of the above embodiments can be realized.

[0091] Furthermore, in the embodiments of this application, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0092] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0094] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0096] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0097] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0098] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0099] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0100] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for detecting a physical state of a worker based on a plurality of physiological signals, characterized by, The method comprises: acquiring a plurality of original physiological signals of a worker in a power field and a power operation scene; based on the plurality of original physiological signals and the power operation scene, feature extraction is performed to obtain the plurality of modal features; wherein the plurality of modal features include modal features corresponding to each of the plurality of physiological signals; performing multi-modal feature fusion on the plurality of modal features to obtain a fusion feature; based on the fusion feature, performing multi-abnormality detection through a pre-determined body state detection model to obtain an abnormality detection result.

2. The method of claim 1, wherein, The method comprises: performing preprocessing on the plurality of original physiological signals respectively to obtain a plurality of physiological signals; wherein the plurality of physiological signals include electrocardiogram, photoplethysmogram, electrodermal activity, and electroencephalogram; based on the electrocardiogram, the photoplethysmogram, the electrodermal activity, and the electroencephalogram, respectively performing sliding window slicing according to a pre-set length to obtain signal segments of the electrocardiogram, the photoplethysmogram, the electrodermal activity, and the electroencephalogram respectively; performing amplitude normalization and standardization processing on the signal segments to obtain final signal segments; performing feature extraction on the electrocardiogram, the photoplethysmogram, the electrodermal activity, and the electroencephalogram respectively through a modal convolution backbone network and the power operation scene to obtain the plurality of modal features.

3. The method of claim 2, wherein, The plurality of original physiological signals include original electrocardiogram, original photoplethysmogram, original electrodermal activity, and original electroencephalogram; The method comprises: determining the collection time of the original electrocardiogram, the original photoplethysmogram, the original electrodermal activity, and the original electroencephalogram respectively; based on the collection time, performing original timestamp alignment on the original electrocardiogram, the original photoplethysmogram, the original electrodermal activity, and the original electroencephalogram to obtain processed original electrocardiogram, processed original photoplethysmogram, processed original electrodermal activity, and processed original electroencephalogram; resampling the processed original electrocardiogram, the processed original photoplethysmogram, the processed original electrodermal activity, and the processed original electroencephalogram to obtain the plurality of physiological signals.

4. The method of claim 2, wherein, The modal convolution backbone network includes a convolution backbone network corresponding to each of the electrocardiogram, the photoplethysmogram, the electrodermal activity, and the electroencephalogram; The method comprises: extracting the time sequence pattern features of the electrocardiogram, the photoplethysmogram, the electrodermal activity, and the electroencephalogram respectively through the convolution backbone network; The multi-scale feature extraction is performed on the electrocardiogram, the photoplethysmogram, the electrodermal activity and the electroencephalogram through the multi-scale convolution branch module in the convolution backbone network, and the multi-scale features of the electrocardiogram, the photoplethysmogram, the electrodermal activity and the electroencephalogram are obtained. The time sequence features and the multi-scale features are fused based on the weighted aggregation module in the convolution backbone network to obtain final fusion features. The attention distribution in the time sequence dimension is calculated based on the final fusion features to obtain enhanced features, and the modal aggregation is performed on the enhanced features to obtain cross-modal enhanced features. The cross-modal enhanced features are subjected to interference weighting processing based on the power operation scene to obtain the multi-modal features.

5. The method of claim 1, wherein, The multi-modal feature fusion is performed on the multi-modal features to obtain fusion features, including: determining the time sequence corresponding to each of the multi-modal features, and determining the position encoding based on each time node in the time sequence; based on the position encoding and the time sequence corresponding to each of the multi-modal features, the multi-modal features are spliced to obtain spliced features; determining a plurality of mapping parameters, and performing feature fusion on the spliced features based on the plurality of mapping parameters to obtain the fusion features.

6. The method of claim 1, wherein, The fusion features are subjected to multi-abnormality detection through a pre-determined body state detection model to obtain an abnormality detection result, including: fatigue detection is performed on the fusion features through a fatigue recognition branch module in the body state detection model to obtain a fatigue detection result; blood pressure abnormality detection is performed on the fusion features through a blood pressure abnormality recognition branch module in the body state detection model to obtain a blood pressure detection result; normal state detection is performed on the fusion features through a normal state recognition branch module in the body state detection model to obtain a normal detection result; the fatigue detection result, the blood pressure detection result and the normal detection result are subjected to weighted processing to obtain the abnormality detection result.

7. The method of claim 6, wherein, The fusion features are subjected to multi-abnormality detection through a pre-determined body state detection model to obtain an abnormality detection result, including: based on a plurality of original physiological signals, determining the time sequence features of each of the plurality of original physiological signals; based on the time sequence features of each of the plurality of original physiological signals and the final fusion features of each of the plurality of original physiological signals, performing feature splicing to obtain a comprehensive information vector; nonlinear processing is performed on the comprehensive information vector to obtain three weight data values; wherein the sum of the three weight data values is 1; the fatigue detection result, the blood pressure detection result and the normal detection result are subjected to weighted processing based on the three weight data to obtain the abnormality detection result value.

8. A workman's physical state detection system based on multiple physiological signals, characterized by, including: an acquisition unit, a feature extraction unit, a fusion unit and a detection unit; wherein the acquisition unit is configured to acquire a plurality of original physiological signals of an operator in a power field and a power operation scene; The feature extraction unit is configured to perform feature extraction based on the multiple original physiological signals and the power operation scene to obtain the multiple modal features; the multiple modal features include modal features corresponding to the multiple physiological signals respectively; The fusion unit is configured to perform multi-modal feature fusion on the multiple modal features to obtain the fusion features; The detection unit is configured to perform multi-abnormality detection based on the fusion features by using a pre-determined body state detection model to obtain an abnormality detection result.

9. A workman's physical state detection device based on multiple physiological signals, characterized by, Comprising: a processor and a memory, The memory is configured to store a computer program; The processor is configured to call and run the computer program from the memory to perform the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, executable instructions are stored for causing the processor to perform the method of any one of claims 1 to 7 when executed.

Images