A multi-mode data-based wearable remote real-time monitoring system and method for power workers

Abstract

The application relates to a kind of power operating personnel wearable remote real-time monitoring system and method based on multi-mode data, including the real-time information acquisition module of operating personnel, multi-mode data real-time transmission module, feature extraction and early warning module based on transpose coding, data display and correction module connected in turn.The monitoring system proposed in the application obtains the behavior, environment, positioning information of user through wearable device, and carries out comprehensive judgment of safety risk from multiple directions;At the same time, a kind of multi-channel attention model network based on transpose coding is adopted, which can obtain multi-dimensional potential information in user behavior data;The practicability, reliability and robustness of the system are enhanced, and user behavior recognition classification and danger early warning can be better realized.

Description

Technical Field

[0001] This invention relates to a wearable remote real-time monitoring system and method for power workers based on multi-mode data, belonging to the field of artificial intelligence technology. Background Technology

[0002] The concept of recording daily human behavior or monitoring a user's health was first proposed in the mid-20th century using wearable products and electronic devices with specific functions. With the development of sensor technology, various lightweight and compact wearable sensors have emerged, such as inertial measurement unit (IMU) sensors, which can simultaneously acquire acceleration and angular acceleration data, and photoplethysmo graph (PPG) sensors, which can measure physiological parameters such as heart rate and blood oxygen saturation.

[0003] Leveraging IoT and cloud platform technologies, wearable devices can transmit signals and exchange information in real time. By establishing a connection between wearable devices and the cloud platform, the data collected by the wearable devices can be quickly transmitted to the cloud. Building a human body monitoring model in the cloud allows for convenient data analysis and alerts.

[0004] Currently, human behavior monitoring based on wearable devices mainly includes the following: collecting various effective human activity data, extracting effective feature values, researching human behavior recognition and classification methods, and establishing human behavior monitoring systems based on wearable devices. The vast majority of research on human behavior monitoring methods based on wearable devices revolves around one or more of these issues.

[0005] The main strategies for processing multimodal physiological signals are to leverage the mutual support of different signal categories, fuse complementary information, and ultimately improve recognition performance. Based on current research, modality fusion can be broadly categorized into four types: data-level fusion (sensor-level fusion), feature-level fusion, decision-level fusion, and model-level fusion.

[0006] In specialized power operation scenarios, more factors need to be considered, such as the portability of wearable devices, the diversity of data collected, and the recognition scenarios of detection algorithms. To improve the applicability of wearable remote real-time monitoring systems for power workers in complex environments, wearable devices need to collect as much multimodal information as possible, such as location data, proximity sensing, and physiological information. Furthermore, to improve the portability of the data acquisition devices, multiple functions will be integrated to reduce device size, while simultaneously reducing power consumption to extend usage time. Data will be transmitted to a cloud platform via network, ensuring reliable data transmission while analyzing and triggering alarms. A cloud platform with management capabilities will be built to provide real-time monitoring of staff, offering useful information to help power companies improve operational processes and increase work efficiency. Summary of the Invention

[0007] To address the specific monitoring needs of power scenarios and the problems of incomplete feature extraction and low utilization in this scenario, this invention proposes a behavior recognition method based on multimodal feature extraction and encoding, which is applied to a wearable remote real-time monitoring system for power workers.

[0008] The multimodal sensor data used in this invention is collected by wearable sensors worn on different parts of the human body. This means that multiple wearable sensors collect data simultaneously. Multimodal means that the wearable sensors include accelerometer sensors, gyroscope sensors, environmental electric field sensors, environmental gas sensors, and positioning sensors. They can simultaneously collect data from five modes: acceleration, angular velocity, environmental electric field, environmental gas, and positioning information, and can be extended to multiple modes.

[0009] This invention employs a convolutional network model, based on a squeeze and excitation (SE) module and a transposed encoding module, to realize multi-channel feature extraction and anomaly detection modules, fully capturing physiological information and achieving a higher recognition rate. This invention proposes a real-time behavior recognition method based on transposed encoding feature extraction.

[0010] This invention establishes a complete remote monitoring system, which specifically includes: wearable devices, a cloud platform, and a web interface.

[0011] The wearable device of this invention has the functions of data collection, data transmission, and command reception. By receiving commands from the cloud, it uses multiple sensors to collect data and transmits it to the cloud platform.

[0012] The cloud platform of this invention receives data from wearable devices, then processes the multimodal data through data preprocessing and data analysis to complete monitoring and alarm, and sends the visualization results to the web page.

[0013] The web interface of this invention features data display and administrator operation functions. Visualized results of data collected by wearable devices after processing, along with user alarm information, are displayed here. Administrators can use the web interface to correct data and issue commands to wearable devices.

[0014] This invention sets up various types of information to be collected based on the working characteristics of power personnel, such as proximity induction, hazardous gases, and location-based service (LBS) positioning, and proposes corresponding processing algorithms for the various types of data collected. Invention Overview:

[0016] The real-time behavior recognition method based on transposed coding feature extraction proposed in this invention, and the wearable remote real-time monitoring system for power workers it applies to, include a real-time information acquisition module for workers, a multimodal data preprocessing module, a transposed coding-based feature extraction and early warning module, and a data display and correction module connected in sequence. First, upon receiving a command, the sensor devices in the real-time information acquisition module collect various information. The multimodal data real-time transmission module transmits data wirelessly to a cloud database via TCP / IP protocol. The cloud server program determines the data type and processes environmental information, behavioral data, and location data accordingly. If environmental information exceeds an alarm threshold, an early warning is issued. For location data, the data type is determined; if it is LBS data, it is decoded and then transmitted to the multi-source data fusion and location module. For behavioral data, preprocessing is performed, including denoising, normalization, data fusion, and time-series-based extraction. Finally, a behavior recognition model is constructed and trained, followed by real-time behavior recognition. Finally, all early warning information obtained from the above steps is displayed in real-time for management.

[0017] This invention provides a feasible solution for a power worker monitoring system based on wearable devices. It can monitor user behavior categories in real time, making up for the shortcomings of single sensor data and simple deep learning networks in terms of adaptability and special behavior classification. Furthermore, it extracts comprehensive and effective features, improves feature utilization, and can be used for monitoring and management of power system workers. It also proposes a complete process, filling the gap in monitoring systems for special industries.

[0018] The technical solution of this invention is as follows:

[0019] A wearable remote real-time monitoring system for power workers based on multimodal data includes a real-time information acquisition module for workers, a multimodal data preprocessing module, a feature extraction and early warning module based on transpose encoding, and a data display and correction module connected in sequence.

[0020] The real-time information collection module for workers is used to collect real-time information through wearable devices, including environmental information, physiological information, and location information.

[0021] The multimodal data preprocessing module is used to: receive real-time information from wearable devices and preprocess it;

[0022] The transposed coding-based feature extraction and early warning module is used to: process environmental, physiological and location information data, and perform human behavior recognition through a transposed coding-based feature extraction algorithm;

[0023] The data display and correction module is used to: receive data from the data identification and early warning module, integrate various data, and display and alarm on the web page; at the same time, it provides an administrator operation interface on the web page for administrators to send instructions and correct data.

[0024] According to a preferred embodiment of the present invention, the real-time information acquisition module for workers includes an environmental information acquisition unit, a physiological information acquisition unit, a positioning information acquisition unit connected in parallel, and a data transmission unit connected serially thereto.

[0025] The environmental information acquisition unit is used to acquire environmental information; the physiological information acquisition unit is used to acquire physiological information; the location information acquisition unit is used to acquire location information; and the data transmission unit is used to remotely transmit the collected user behavior data to the cloud server's database in real time via the TCP / IP protocol.

[0026] More preferably, the environmental information acquisition unit includes an air quality detection unit and a proximity sensing unit;

[0027] The air quality detection unit is used to: read environmental data in the air, including the concentration of several toxic and harmful gases;

[0028] The proximity sensing unit is used to: read the electric field strength near the human body, and realize power plant monitoring and alarm;

[0029] More preferably, the physiological information acquisition unit includes an accelerometer, an angular velocity sensor, a temperature sensor, and a PPG sensor;

[0030] Accelerometers are used to acquire acceleration data along the X, Y, and Z axes; angular velocity sensors are used to acquire angular velocity data along the X, Y, and Z axes; temperature sensors are used to acquire body temperature data; and PPG sensors are used to measure heart rate and blood oxygen saturation.

[0031] More preferably, the positioning information acquisition unit includes a Global Positioning System (GPS) and a Wi-Fi base station information receiving system;

[0032] The Global Positioning System (GPS) is used to obtain GPS signals and Location-Based Services (LBS); the Wi-Fi base station information receiving system is used to obtain Media Access Control Address (MAC) and signal strength.

[0033] Further preferred, the preprocessing includes: sequentially performing denoising filtering, normalization, data fusion, and time series-based truncation.

[0034] According to a preferred embodiment of the present invention, the multimodal data preprocessing module includes a data receiving unit, a data preprocessing unit, and a data normalization unit connected in sequence.

[0035] The data receiving unit receives various data from the wearable device and packages them for transmission to the data preprocessing unit. The data preprocessing unit preprocesses different types of data and then transmits them to the data normalization unit. After normalization, the data is transmitted to the feature extraction and early warning module based on transpose encoding.

[0036] According to a preferred embodiment of the present invention, the feature extraction and early warning module based on transpose encoding includes an environmental information processing unit, a physiological information processing unit, and a location information processing unit connected in parallel; the environmental information processing unit is serially connected to an environmental information early warning unit; the physiological information processing unit is further serially connected to a parallel physiological information early warning unit and a physiological information recognition unit; and the location information processing unit is further serially connected to a location information decoding unit, a location information fusion unit, and an out-of-safety-zone early warning unit.

[0037] The environmental information processing unit is used to process environmental gas information and environmental voltage information, and to adjust the environmental gas concentration warning value and the environmental electric field voltage warning value.

[0038] The environmental information early warning unit is used to compare environmental information with preset early warning values, and to issue an alarm when the early warning value is exceeded.

[0039] The physiological information processing unit is used to process body temperature, heart rate, blood oxygen saturation, acceleration data of the X, Y, and Z axes, and angular velocity data of the X, Y, and Z axes, and to set personalized data warning thresholds according to different individual situations;

[0040] The physiological information early warning unit is used to judge the reasonable numerical range according to different individuals and output early warning information; physiological information early warning refers to analyzing the results obtained after processing the received physiological signals, and immediately alarming after identifying abnormalities; abnormality refers to the discovery of data outside the normal physiological and medical range by analyzing the wearer's heart rate, blood oxygen, and body temperature information and combining the wearer's vital signs.

[0041] The physiological information recognition unit achieves behavior classification and early warning of abnormal behavior through a multi-channel attention network model based on the squeeze excitation module and the transposed encoding module; whereby behavior classification refers to Human Activity Recognition (HAR).

[0042] The positioning information processing unit analyzes and processes the collected LBS and GPS positioning signals to parse the raw LBS base station data and combine the GPS and LBS data information to obtain the data positioning.

[0043] The location information decoding unit decodes LBS data to obtain visible data information; the location information fusion unit fuses data collected from different sources.

[0044] After confirming the location, the out-of-safe-zone warning unit compares it with the pre-defined safe zone and issues a warning if the location exceeds the safe zone range.

[0045] According to a preferred embodiment of the present invention, the multi-channel attention network model based on the squeezing excitation module and the transposed encoding module includes a serially connected multi-channel feature and anomaly monitoring module, a transposed encoding module, and an output module;

[0046] The multi-channel feature and anomaly monitoring module includes a noise superposition module, a data reconstruction module, an anomaly monitoring module, and a feature vector and anomaly information output module connected in series. The noise superposition module randomly selects channels of the multi-channel data to superimpose Gaussian noise and inputs it into the data reconstruction module. The data containing noise is then input into the data reconstruction module.

[0047] The data reconstruction module is used to extract a higher-level representation of the input signal. The data reconstruction module includes an input layer, a hidden layer, and an output layer. The input layer to the hidden layer is decoding, and the hidden layer to the output layer is encoding. The output of the hidden layer is used as the feature representation of the original input data. The feature vector of the hidden layer is input to the feature vector and anomaly information output module. The output layer is used as the input of the reconstructed data, and the reconstructed data is output to the anomaly monitoring module.

[0048] The anomaly monitoring module performs anomaly monitoring on the input data; the filtered abnormal data is input into the feature vector and anomaly information output module.

[0049] The feature vector and anomaly information output module obtains feature vectors from the data reconstruction module and anomaly data from the anomaly monitoring module. It combines the two data, removes the input feature vectors that are detected as anomalies, and finally outputs all data to the transpose encoding module in three channels.

[0050] According to a preferred embodiment of the present invention, the transpose encoding module includes a transpose module, an SE module, a layer normalization module, and an output module connected in series.

[0051] The transpose module transposes the input data across different axes, thereby redistributing different channel weights, reading different information from the three dimensions, and then inputting the transposed three types of channel information into the SE module.

[0052] The SE module is a squeezing and excitation network module;

[0053] The layer normalization module performs batch layer normalization on the output of the SE module and inputs the output data into the output module.

[0054] The output module inputs the output of the SE module into the fully connected layer, and then into the Softmax classifier to classify and identify the behavioral information, calculate the probability of each behavior, and obtain the behavior corresponding to the highest probability. This is the final behavior recognition result of the multi-channel attention network model based on the squeezing excitation module and the transpose encoding module.

[0055] According to a preferred embodiment of the present invention, the data display and correction module includes an information integration unit, which is serially connected to a physiological information display unit and an early warning information alarm unit, and then serially connected to a real-time data display unit and a user operation unit, and then serially connected to a parallel error information correction unit and an instruction issuing unit.

[0056] The information integration unit is used to integrate all environmental information, location information, physiological information and its early warning information;

[0057] The physiological information display unit is used to display physiological information in real time via a webpage;

[0058] The warning information alarm unit is used by the cloud server to notify the administrator and power workers of the warning information.

[0059] The real-time data display unit is used to display all environmental alarm information, location out-of-safe-zone warning information, and human behavior recognition information in real time via a web page.

[0060] The user operation unit is used to receive operations from users and administrators, including database calls, database modifications, and data alarm threshold modifications;

[0061] The instruction information issuing unit is used to transmit and control the wearable device with instructions from the administrator via the network.

[0062] A wearable remote real-time monitoring method for power workers based on multi-mode data, implemented through the aforementioned wearable remote real-time monitoring system for power workers based on multi-mode data, includes the following steps:

[0063] Step S1: Administrators select the monitoring scenario

[0064] Administrators can send commands to wearable devices via a web interface to control them and select different information collection frequencies for different usage scenarios.

[0065] Step S2: Sensors acquire multimodal information

[0066] The wearable device receives an administrator command from step S1 and controls the sensors to selectively collect the user's physiological information, environmental information, and location information. The physiological information includes acceleration data of the X, Y, and Z axes obtained by the accelerometer and angular velocity data of the X, Y, and Z axes obtained by the angular velocity sensor, as well as heart rate and body temperature. The environmental information includes proximity data and hazardous gas data. The location information includes GPS, LBS data, media access control address, and signal strength.

[0067] Step S3: Upload to cloud server

[0068] The multimodal information collected by the wearable device is transmitted to the cloud server in real time via the TCP / IP protocol;

[0069] Step S4: Determine the data type

[0070] The multimodal information stored on the cloud server is classified as follows: ambient gas concentration and electric field voltage are classified as environmental data; GPS, LBS, media access control address and signal strength data are classified as location data; and acceleration, angular velocity, temperature and heart rate data are classified as behavioral data.

[0071] Step S5: Physiological Information Preprocessing

[0072] Preprocessing physiological information specifically refers to: sequentially performing filtering, normalization, fusion, and time-series-based extraction.

[0073] Step S6: Environmental Information Early Warning

[0074] The system compares the pre-set warning value with the ambient gas concentration value collected by the ambient gas sensor to determine whether the value exceeds the limit; when the received value is greater than the pre-set warning value, an alarm is triggered.

[0075] Step S7: Build and train the behavior recognition model

[0076] The behavior recognition model is a multi-channel attention model network based on transposed coding, which includes a multi-channel feature extraction and anomaly detection module and a transposed coding module;

[0077] The user-defined parameters include: the shape of the input data, the shape of the reshaped data, and the superimposed noise parameters. Through iterative training, the parameters of the behavior recognition model are continuously optimized, and the trained behavior recognition model is finally obtained, as shown in Equation (I):

[0078]

[0079] In equation (I), A is the number of samples, M is the number of categories, and y ij The sign function is p, which takes the value 1 if the true class of sample i is equal to j, and 0 otherwise. ij The predicted probability that observed sample i belongs to category j;

[0080] Step S8: Real-time Behavior Recognition

[0081] The real-time collected and preprocessed physiological information is input into the trained multi-channel attention model network behavior recognition model based on transpose coding to perform real-time recognition of the current behavior and output the classification result corresponding to the physiological information.

[0082] Step S9: Location Data Analysis

[0083] The GPS signal and LBS signal are processed separately. Step S9 includes the following sub-steps:

[0084] Step S91: Data Decoding

[0085] The LBS signal is decoded. The LBS signal includes the communication base station code. The location information corresponding to the code is queried in the open source data warehouse to obtain the decoded base station location information.

[0086] Step S93: Multi-modal data fusion positioning

[0087] The parsed LBS data and GPS data are fused for positioning to obtain accurate positioning information.

[0088] Step S10: Real-time information display

[0089] The behavior recognition results output by the multi-channel attention model network based on transposed encoding are displayed through the behavior display unit in the data display and correction module.

[0090] According to a preferred embodiment of the present invention, in step S7, the parameters x, y, and z of the input data are set to be the length, width, and number of channels of the original input data, respectively.

[0091] Set the noise parameter L of the noise superposition module. L can be a random value of 0 or be Gaussian noise for superposition.

[0092] Set the initial parameters w and b for the data reconstruction module, where w is the network weight and b is the bias coefficient;

[0093] Set parameters x′, y′, and z′ to represent the length, width, and number of channels of the original input data after feature extraction by the data reconstruction module, respectively.

[0094] Set Batch_Size and Window_Size. Batch_Size refers to the number of behavior samples in each batch of the multi-channel feature extraction and anomaly detection module, and Window_Size refers to the length of the sample data.

[0095] The data reconstruction module in the multi-channel feature extraction and anomaly detection module has c1 neurons in the input layer, c2 neurons in the hidden layer, and c3 neurons in the output layer.

[0096] Set the input-output comparison function in the anomaly monitoring module to f, and the judgment threshold to α;

[0097] Set the input shape of the SE module to x′, y′, z′, which are the output shape parameters of the data reconstruction module, and the scaling parameter of its Excitation part is SERadio;

[0098] Step S71: Input raw data

[0099] Assume the input data is composed of the fusion of measurement input data from n triaxial accelerometers and triaxial angular velocity sensors, and the physiological information is of size Batch_Size×Window_Size×6n, where Window_Size is the data length and 6n is the number of data channels;

[0100] Step S72: Input data into the multi-channel feature extraction and anomaly detection module

[0101] Physiological information of size Batch_Size×Window_Size×6n is sent to the multi-channel feature extraction and anomaly detection module, which splits it into x, y, ..., z channel data and inputs it into the noise superposition module. The noise superposition module randomly selects channels of the multi-channel data to superimpose Gaussian noise and inputs it into the data reconstruction module.

[0102] The data reconstruction module is a three-layer network model, including an input layer, a hidden layer, and an output layer. The input layer has c1 neurons, the hidden layer has c2 neurons, and the output layer has c3 neurons. The relationship between c1, c2, and c3 is given by equation (II):

[0103] c1>c2≥c3 (II)

[0104] The final output size is The feature information is input into the anomaly monitoring module; the anomaly monitoring module reuses the network structure of the data reconstruction module, and obtains the reconstruction error by comparing the output layer data with the data output by the noise superposition module. The anomaly score obtained through the reconstruction error is calculated and compared with the judgment threshold α. If it exceeds the judgment threshold α, it is considered an abnormal behavior; the loss function of the reconstruction error is Equation (III):

[0105] L(x,z)=‖xz‖ 2 (III)

[0106] In equation (III), x is the original input layer data before noise is added, and z is the output layer data;

[0107] Step S73: Input data transpose encoding module

[0108] The size is The feature vector is input into the behavior recognition module; the original input shape is set as x×y×z, representing width, height and number of channels respectively. The original input is transposed along different axes to obtain y×z×x and z×x×y data; thereby redistributing different channel weights and reading information from three different dimensions. Then, the transposed and summed information from the three channels is input into the SE module.

[0109] Step S74: Input data into SE module

[0110] The SE module is a squeeze and excitation network module. It first utilizes the global average ensemble features to efficiently calculate channel attention, which includes a pooling layer, a fully connected layer, a convolutional layer, a fully connected layer, and a convolutional layer linked in sequence.

[0111] The attention mechanism that combines average pooling and max pooling operations is described by equation (IV):

[0112] Wc=σ(f{w 1,2}(AP(χ))+f{w1,w2}(MP(χ))) (IV)

[0113] In equation (IV), χ represents the tensor data of the three input SE modules. These represent average pooling and max pooling operations, respectively; σ is the sigmoid function, and w1 and w2 are network parameters used to adjust the scaling factors; Equation (IV) is a summary of Equation (V):

[0114] Wc=σ(w2ReLU(w1AP(χ))+w2ReLU(w1MP(χ)))(V)

[0115] Step S75: Additive Aggregation

[0116] The three attention channels output by the SE module are aggregated additively, and then Z-Pooling is used to preserve the feature representation while compressing the depth. The concatenated and pooled feature information is represented by Equation (VI):

[0117] Z_Pooling(χ) = [MaxPool 0d (χ),AvgPool 0d (χ)](VI)

[0118] In equation (VI), 0d represents the data in the 0th dimension where max pooling and average pooling occur;

[0119] Step S76: Layer Normalization

[0120] The aggregated output was batch-level normalized to obtain a 1×y×z output.

[0121] Step S77: Output the unit recognition result

[0122] The physiological information is converted from a vector to a scalar through an unfolding layer. After passing through a fully connected layer, the output information size is Batch_Size×Output_Length. The physiological information is then classified and identified by a Softmax classifier. The physiological information features of size are input into the Softmax classifier to solve for the probability of each behavior. The behavior corresponding to the highest probability is the final behavior recognition result of the multi-channel attention model network based on transpose encoding.

[0123] According to a preferred embodiment of the present invention, step S5 includes the following steps:

[0124] Step S51: Denoising Physiological Information

[0125] Denoising of physiological information is performed using wavelet thresholding.

[0126] Step S52: Normalization of physiological information

[0127] Mean normalization is performed to bring features of different dimensions to the same numerical order of magnitude.

[0128] Step S53: Multimodal data fusion

[0129] The collected physiological information is aligned according to timestamps;

[0130] Step S54: Time series-based extraction

[0131] Referring to user-preset parameters, including the size of the sliding window, the cutting frequency, and the step size of the sliding window, the physiological information processed in step S53 is processed by sliding window, so that the physiological information is input into the multi-channel attention model network based on transpose coding in the form of information blocks.

[0132] The beneficial effects of this invention are as follows:

[0133] 1. This invention proposes a wearable remote real-time monitoring system for power workers based on multi-mode data. The system processes and analyzes various types of data collected by wearable devices to monitor power workers.

[0134] 2. This invention proposes a hierarchical structure of wearable device—cloud platform—web page, with clear division of labor and close cooperation among each layer, which can clearly grasp various real-time information of workers and handle their anomalies.

[0135] 3. This invention collects specific data and designs unique algorithms for the special application scenarios of power operations. Addressing common problems encountered by power workers, it collects proximity induction information and harmful gas information, and designs special indoor and outdoor positioning algorithms and human activity recognition algorithms. This monitoring system is more targeted towards the power industry, can adapt to industry needs, and has higher accuracy and practical value. Attached Figure Description

[0136] Figure 1 This is a schematic diagram of the module composition and connection relationship of the wearable remote real-time monitoring system for power workers based on multi-mode data according to the present invention.

[0137] Figure 2 This is a flowchart illustrating the wearable remote real-time monitoring method for power workers based on multi-mode data, as described in this invention.

[0138] Figure 3 This is a schematic diagram of the working principle of the classification and recognition network of the feature extraction and early warning module based on transposed encoding of the present invention.

[0139] Figure 4 This is a schematic diagram of the multi-channel feature extraction and anomaly monitoring module of the present invention.

[0140] Figure 5 This is a schematic diagram of the transpose encoding module structure of the present invention.

[0141] Figure 6 This is a triaxial acceleration data graph used in this invention. Detailed Implementation

[0142] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments and accompanying drawings. Obviously, the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0143] Example 1

[0144] In the management of power system workers, there are many safety hazards. Common dangers include accidentally entering a high-voltage electric field, accidentally touching high-voltage electricity, exposure to toxic and harmful gases, or falls during work. These safety hazards are difficult to detect and report in a timely manner, resulting in inadequate handling and insufficient protection of workers' personal safety. To address these issues, designing a remote monitoring system based on wearable devices would be highly effective. Power workers can choose to wear a smart bracelet with a built-in heart rate monitoring module, a three-axis accelerometer, a three-axis angular velocity sensor, and a GPS and LBS positioning module to collect physiological and location information. It also carries a portable environmental information acquisition module that can collect the concentration of various toxic and harmful gases in the air and monitor multiple levels of high-voltage electricity. In the event of a fall, the smart bracelet can collect data from the x, y, and z-axis accelerometers. Figure 6 As shown, both the smart bracelet and the environmental information collection module can collect and transmit data to the cloud server in real time. Data is then stored and calculated on the cloud server to obtain corresponding identification results and early warning information. The power system operator management application platform can display the calculation results from the cloud platform in real time and provide feedback to management personnel, facilitating timely handling of emergencies.

[0145] A wearable remote real-time monitoring system for power workers based on multi-mode data, such as Figure 1 As shown, it includes a real-time information acquisition module for operators, a multimodal data preprocessing module, a feature extraction and early warning module based on transpose encoding, and a data display and correction module connected in sequence.

[0146] The real-time information collection module for workers is used to collect real-time information through wearable devices, including environmental information, physiological information, and location information.

[0147] The multimodal data preprocessing module is used to: receive real-time information from wearable devices and preprocess it;

[0148] The transposed coding-based feature extraction and early warning module is used to process environmental, physiological and location information data, and to identify human behavior through a transposed coding-based feature extraction algorithm.

[0149] The data display and correction module is used to: receive data from the data identification and early warning module, integrate various data, and display and alarm on the web page; at the same time, it provides an administrator operation interface on the web page for administrators to send commands and correct data.

[0150] Example 2

[0151] The difference between the wearable remote real-time monitoring system for power workers based on multi-mode data described in Embodiment 1 and the system is as follows:

[0152] The real-time information acquisition module for operators includes an environmental information acquisition unit, a physiological information acquisition unit, a positioning information acquisition unit connected in parallel, and a data transmission unit connected serially to them.

[0153] The environmental information acquisition unit is used to acquire environmental information; the physiological information acquisition unit is used to acquire physiological information; the location information acquisition unit is used to acquire location information; and the data transmission unit is used to remotely transmit the collected user behavior data to the cloud server's database in real time via the TCP / IP protocol.

[0154] The environmental information acquisition unit includes an air quality detection unit and a proximity sensing unit;

[0155] The air quality detection unit is used to read environmental data in the air, including the concentrations of several common toxic and harmful gases such as sulfur dioxide, carbon dioxide, carbon monoxide, and methane.

[0156] The proximity sensing unit is used to: read the electric field strength near the human body, and to sensitively and reliably monitor and alarm power plants with voltages of 220V, 10kV, and 35KV in the vicinity;

[0157] The air quality detection system starts with a sensor module that collects gas data from the air and then transmits the data to an STM32 microcontroller via RS-485 communication. The STM32 chip is the core of the entire air quality detection system, processing the returned air data. The proximity sensing system, built around the open-source JW5808 chip, utilizes a multi-level proximity sensing circuit controlled by a slide switch. This allows for sensitive and reliable monitoring and alarming of electric fields at nearby 220V, 10kV, and 35kV voltages.

[0158] The physiological information acquisition unit includes an accelerometer, an angular velocity sensor, a temperature sensor, and a PPG sensor;

[0159] Accelerometers are used to acquire acceleration data along the X, Y, and Z axes; angular velocity sensors are used to acquire angular velocity data along the X, Y, and Z axes; temperature sensors are used to acquire body temperature data; and PPG sensors are used to measure heart rate and blood oxygen saturation.

[0160] The location information acquisition unit includes a Global Positioning System (GPS) and a Wi-Fi base station information receiving system;

[0161] The Global Positioning System (GPS) is used to acquire GPS signals and location-based services (LBS) signals based on geographic location data; Wi-Fi base station information receiving systems are used to acquire Media Access Control Address (MAC) and signal strength.

[0162] Preprocessing includes: sequential denoising filtering, normalization, data fusion, and time-series-based truncation. This results in smooth, diverse, and uniformly dimensional multimodal data. The truncated behavioral data is then input into a feature extraction and early warning module based on transpose encoding.

[0163] The normalization refers to the mean normalization of numerical data in behavioral information, that is, transforming the values ​​of all similar sensors to the range of [-1, 1].

[0164] The data fusion refers to the process of aligning behavioral information carried by different sensors according to timestamps and then fusing them.

[0165] The time-series-based segmentation refers to segmenting the continuous time series collected by the sensor using a sliding window method to ensure that a complete set of motion data falls within a single sliding window.

[0166] The multimodal data preprocessing module includes a data receiving unit, a data preprocessing unit, and a data normalization unit connected in sequence.

[0167] The data receiving unit receives various data from wearable devices and packages them for transmission to the data preprocessing unit. The data preprocessing unit preprocesses different types of data and then transmits them to the data normalization unit. After normalization, the data is transmitted to the feature extraction and early warning module based on transpose coding.

[0168] The data receiving unit receives data collected in real time by the real-time information acquisition module of the operator via TCP / IP protocol and stores it in the cloud server database; the data preprocessing unit performs different data preprocessing algorithms for different data; the data normalization unit normalizes the data of different dimensions, that is, transforms the values ​​of all similar sensors to the range of [-1,1], and then transmits the processed information to the data display and correction module.

[0169] The feature extraction and early warning module based on transpose coding includes an environmental information processing unit, a physiological information processing unit, and a location information processing unit connected in parallel. The environmental information processing unit is serially connected to the environmental information early warning unit. The physiological information processing unit is further serially connected to the parallel physiological information early warning unit and physiological information recognition unit. The location information processing unit is further serially connected to the location information decoding unit, the location information fusion unit, and the out-of-safety-zone early warning unit.

[0170] The environmental information processing unit is used to process environmental gas information and environmental voltage information, taking into account the actual working environment of workers, and adjusting the environmental gas concentration warning value and environmental electric field voltage warning value according to the actual situation.

[0171] The environmental information early warning unit is used to compare environmental information with preset early warning values, and to issue an alarm when the warning value is exceeded.

[0172] The physiological information processing unit is used to process body temperature, heart rate, blood oxygen saturation, acceleration data of the X, Y, and Z axes, and angular velocity data of the X, Y, and Z axes, and to set personalized data warning thresholds according to different individual conditions;

[0173] The physiological information early warning unit is used to judge the reasonable numerical range according to different individuals and output early warning information; physiological information early warning refers to the analysis of the results obtained after processing the received physiological signals, and the immediate alarm after identifying abnormalities; abnormality refers to the discovery of data outside the normal physiological and medical range by analyzing the wearer's heart rate, blood oxygen, body temperature information, combined with the wearer's vital signs.

[0174] The physiological information recognition unit realizes behavior classification and early warning of abnormal behavior through a multi-channel attention network model based on the squeeze excitation module and the transposed encoding module; where behavior classification refers to Human Activity Recognition (HAR); through behavior recognition, the activity status of workers can be confirmed.

[0175] The positioning information processing unit analyzes and processes the collected LBS and GPS positioning signals, realizes the parsing of raw LBS base station data, and integrates GPS and LBS data information to obtain reasonable data positioning.

[0176] The location information decoding unit decodes LBS data to obtain visible data information. LBS base station positioning is a positioning method based on communication operator signal towers. It obtains the latitude and longitude information of the mobile phone SIM card through the signal towers, calculates the location point, and displays it on the map through an electronic map API to achieve the positioning purpose. Furthermore, LBS base station positioning is mandatory; the base station positioning server actively initiates positioning to the terminal, playing an important role in regulatory services.

[0177] The location information fusion unit merges data collected from different sources, making the positioning more accurate. LBS base station positioning is a ground-based positioning method; all signals originate from cell phone towers, and positioning can be achieved wherever a mobile phone can receive a signal. Base station positioning signals have strong penetrating power through ground buildings and tunnels, compensating for the shortcomings of satellite positioning. Secondly, LBS is mandatory; it can actively locate the device even when it doesn't request a location. LBS base station positioning also has strong monitoring capabilities. GPS positioning offers better accuracy, and fusion positioning can further enhance the precision of the positioning process.

[0178] After confirming its location, the out-of-safe-zone warning unit compares its position with the pre-defined safe zone and issues a warning if the location exceeds the safe zone's boundaries.

[0179] The multi-channel attention network model based on the squeezing excitation module and transpose encoding module includes a serially connected multi-channel feature and anomaly monitoring module, transpose encoding module, and output module;

[0180] The multi-channel feature and anomaly monitoring module includes a noise superposition module, a data reconstruction module, an anomaly monitoring module, and a feature vector and anomaly information output module connected in series. The noise superposition module randomly selects channels of the multi-channel data to superimpose Gaussian noise and inputs it into the data reconstruction module. Traditional non-superposition noise methods rely solely on minimizing reconstruction errors, which may result in the learned features being merely copies of the original input. To avoid this problem, the system proposed in this invention introduces a noise injection strategy to reduce the information contained in the original input. The data containing noise is then input into the data reconstruction module.

[0181] The data reconstruction module is used to extract higher-level representations of the input signal. It adjusts network parameters based on the error between the input and reconstructed signals, learns to fill in missing information, and learns the input data structure to make the extracted features more reflective of the original input. The goal of network training is to make the input and output as similar as possible, which is unsupervised learning. The data reconstruction module includes an input layer, hidden layers, and an output layer. The input-to-hidden-layer transition is decoding, and the hidden-to-output layer transition is encoding. The output of the hidden layer is used as the feature representation of the original input data. The feature vectors of the hidden layer are input to the feature vector and anomaly information output module. The reconstructed data, input from the output layer, is then output to the anomaly detection module.

[0182] The anomaly detection module monitors input data for anomalies. The training network reuses the three-layer network used by the data reconstruction module: input layer, hidden layer, and output layer. The network training method involves first training with normal data to learn a low-dimensional representation of the original dataset, thus reconstructing the original dataset to a certain extent. When using the test set, the network structure can encode and decode according to the format of normal data. The error between the input and output layers is within a certain range, referred to as the error distribution. When abnormal data is input, the error between the output and input layers exceeds this error distribution range. Therefore, data is considered abnormal if the error after encoding and decoding exceeds the error distribution range. The module inputs the selected abnormal data into the feature vector and outputs the anomaly information.

[0183] The feature vector and anomaly information output module obtains feature vectors from the data reconstruction module and anomaly data from the anomaly monitoring module. It combines the two data, removes the input feature vectors that are detected as anomalies, and finally outputs all data to the transpose encoding module in three channels.

[0184] The transpose encoding module includes a serially connected transpose module, an SE module, a layer normalization module, and an output module;

[0185] The transpose module transposes the input data along different axes, thereby redistributing different channel weights, reading different information from the three dimensions, and then inputting the transposed three channel information into the SE module.

[0186] The SE module is a squeeze-and-excitation network module; its high flexibility lies in its ability to be directly applied to existing network structures. It was proposed by Jie Hu et al. in their 2019 paper "Squeeze-and-Excitation Networks". It includes a pooling layer, a fully connected layer, a convolutional layer, a fully connected layer, and a convolutional layer linked in sequence. It utilizes global average ensemble features to efficiently compute channel attention.

[0187] The layer normalization module performs batch layer normalization on the output of the SE module and inputs the output data into the output module;

[0188] The output module takes the output of the SE module and inputs it into the fully connected layer, then into the Softmax classifier to classify and recognize the behavioral information, calculate the probability of each behavior, and obtain the behavior corresponding to the highest probability. This is the final behavior recognition result of the multi-channel attention network model based on the squeezing excitation module and the transpose encoding module.

[0189] The data display and correction module includes an information integration unit, which is serially connected to a physiological information display unit and an early warning information alarm unit, and then serially connected to a real-time data display unit and a user operation unit, and then serially connected to a parallel error information correction unit and an instruction issuance unit.

[0190] The information integration unit is used to integrate all environmental information, location information, physiological information, and early warning information;

[0191] The physiological information display unit is used to display physiological information in real time via a webpage;

[0192] The early warning information alarm unit is used by the cloud server to notify administrators and power workers of early warning information via SMS reminders and other means;

[0193] The real-time data display unit is used to display all environmental alarm information, location out-of-safe-zone warning information, and human behavior recognition information in real time via a web page.

[0194] The user operation unit is used to receive operations from users and administrators, including database calls, database modifications, and data alarm threshold modifications;

[0195] The instruction information distribution unit is used to transmit instructions from the administrator to the wearable device via the network and control it.

[0196] Example 3

[0197] A wearable remote real-time monitoring method for power workers based on multi-mode data is implemented through the wearable remote real-time monitoring system for power workers based on multi-mode data described in Embodiment 1 or 2, such as... Figure 2 As shown, the steps are as follows:

[0198] Step S1: Administrators select the monitoring scenario

[0199] Administrators can send commands to wearable devices via a web interface to control them and select different information collection frequencies for different usage scenarios.

[0200] Step S2: Sensors acquire multimodal information

[0201] The wearable device receives an administrator command from step S1 and controls the sensors to selectively collect the user's physiological information, environmental information, and location information. The physiological information includes acceleration data of the X, Y, and Z axes obtained by the accelerometer and angular velocity data of the X, Y, and Z axes obtained by the angular velocity sensor, as well as heart rate and body temperature. The environmental information includes proximity data and hazardous gas data. The location information includes GPS, LBS data, media access control address, and signal strength.

[0202] Step S3: Upload to cloud server

[0203] The multimodal information collected by the wearable device is transmitted to the cloud server in real time via the TCP / IP protocol;

[0204] Step S4: Determine the data type

[0205] The multimodal information stored on the cloud server is classified into environmental data (such as ambient gas concentration and electric field voltage), location data (such as GPS, LBS, media access control address, and signal strength data), and behavioral data (such as acceleration, angular velocity, temperature, and heart rate data). This allows for separate processing and analysis of different types of data.

[0206] Step S5: Physiological Information Preprocessing

[0207] Preprocessing physiological information specifically refers to: sequentially performing filtering, normalization, fusion, and time-series-based extraction.

[0208] Step S5 includes the following steps:

[0209] Step S51: Denoising Physiological Information

[0210] Denoising of physiological information is performed using wavelet thresholding.

[0211] Step S52: Normalization of physiological information

[0212] Mean normalization is performed to bring features of different dimensions to the same numerical order of magnitude.

[0213] Step S53: Multimodal data fusion

[0214] Align the various physiological information collected by the sensors according to timestamps;

[0215] Step S54: Time series-based extraction

[0216] Referring to user-preset parameters, including the size of the sliding window, the cutting frequency, and the step size of the sliding window, the physiological information processed in step S53 is processed by sliding window, so that the physiological information is input into the multi-channel attention model network based on transpose coding in the form of information blocks.

[0217] In step S51, the physiological information is denoised using the wavelet thresholding method, specifically:

[0218] Assume the actual measured signal is: f(t) = s(t) + e(t), t = 1, 2, ..., N, where s(t) is the original signal, f(t) is the noisy signal, e(t) is Gaussian white noise, and e(t) ~ N(0, σ 2), where σ is the noise intensity, and the denoising process is to remove the noise e(t) from the signal f(t) to obtain the best approximation of the original signal s(t);

[0219] First, discrete sampling is performed to obtain N'-point discrete signal f(x), x=0,1,2,…,N'-1, and its wavelet transform coefficients are shown in formula (1):

[0220]

[0221] Among them, W f (j,k) are wavelet coefficients, ψ(2 -j xk) is the scaling function, j is the scaling parameter, and k is the number of units the scaling function is shifted. Through the double scaling equations (2) and (3), the recursive implementation method of equation (1) is obtained:

[0222] S f (j+1,k)=S f (j,k)*h(j,k) (2)

[0223] W f (j+1,k)=S f (j,k)*g(j,k) (3)

[0224] Where the symbol "*" represents convolution, h and g represent low-pass and high-pass filters respectively, and S f (0,k) represents the original signal f(k), S f (j,k) represents the approximation coefficient at scale j, and the wavelet transform reconstruction formula is shown in formula (4):

[0225] S f (j-1,k)=S f (j,k)*h(j,k)+W f (j,k)*g(j,k) (4)

[0226] Secondly, the threshold is determined using a general threshold rule, and the threshold algorithm formula is shown in formula (5):

[0227]

[0228] Where σ=MAD / 0.6745, MAD is the median value of the absolute value of the wavelet decomposition coefficients of the first layer, 0.6745 is the adjustment coefficient of the standard deviation of Gaussian noise, and L is the size or length of the signal; then, the wavelet transform coefficients of the measured signal are subjected to nonlinear thresholding. For each high-frequency coefficient of the first to the Vth layer, a soft thresholding function is used to process it. The absolute value of the signal is compared with the threshold. Points less than the threshold are set to zero, and points greater than or equal to the threshold are shrunk to zero, becoming the difference between the value of that point and the threshold. The low-frequency coefficients of each layer are not processed. The soft thresholding function is shown in formula (6):

[0229]

[0230] Among them, W j,k For W f (j,k) is a simplified form where T is a threshold parameter greater than zero.

[0231] Finally, based on the low-frequency coefficients of the Nth layer of wavelet decomposition and the high-frequency coefficients of the 1st to Vth layers after quantization, wavelet reconstruction of the signal is performed according to formula (3) to obtain the denoised signal.

[0232] Step S6: Environmental Information Early Warning

[0233] Based on the pre-set warning values, the default values ​​are: carbon monoxide concentration not exceeding 0.08%, sulfur dioxide concentration not exceeding 0.0005%, carbon dioxide concentration not exceeding 1.5%, and methane concentration not exceeding 0.75%. These values ​​are compared with the ambient gas concentration values ​​collected by the current ambient gas sensors to determine if they exceed the limits. When a value is received that exceeds the pre-set warning value, an alarm is triggered.

[0234] Step S7: Build and train the behavior recognition model

[0235] The behavior recognition model is a multi-channel attention model network based on transposed coding, including a multi-channel feature extraction and anomaly detection module and a transposed coding module; such as Figure 3 , Figure 4 , Figure 5 As shown;

[0236] The user-defined parameters include: the shape of the input data, the shape of the reshaped data (length, width, and number of channels after feature extraction), and the superimposed noise parameters. Through iterative training, the parameters of the behavior recognition model are continuously optimized, and the trained behavior recognition model is finally obtained, as shown in Equation (I):

[0237]

[0238] In equation (I), A is the number of samples, M is the number of categories, and y ijThe sign function is 0 or 1; it takes the value 1 if the true class of sample i is equal to j, and 0 otherwise. ij The predicted probability that observed sample i belongs to category j;

[0239] Step S7: Set the parameters x, y, z of the input data to be the length, width, and number of channels of the original input data, respectively;

[0240] Set the noise parameter L of the noise superposition module. L can be a random value of 0 or be Gaussian noise for superposition.

[0241] Set the initial parameters w and b for the data reconstruction module, where w is the network weight and b is the bias coefficient;

[0242] Set parameters x′, y′, and z′ to represent the length, width, and number of channels of the original input data after feature extraction by the data reconstruction module, respectively.

[0243] Set Batch_Size and Window_Size. Batch_Size refers to the number of behavior samples in each batch of the multi-channel feature extraction and anomaly detection module, and Window_Size refers to the length of the sample data.

[0244] The data reconstruction module in the multi-channel feature extraction and anomaly detection module has c1 neurons in the input layer, c2 neurons in the hidden layer, and c3 neurons in the output layer.

[0245] Set the input-output comparison function in the anomaly monitoring module to f, and the judgment threshold to α;

[0246] Set the input shape of the SE module to x′, y′, z′, which are the output shape parameters of the data reconstruction module, and the scaling parameter of its Excitation part is SERadio;

[0247] like Figure 3 As shown, the specific implementation process is as follows:

[0248] Step S71: Input raw data

[0249] Assume the input data is composed of the fusion of measurement input data from n triaxial accelerometers and triaxial angular velocity sensors, and the physiological information is of size Batch_Size×Window_Size×6n, where Window_Size is the data length and 6n is the number of data channels;

[0250] Step S72: Input data into the multi-channel feature extraction and anomaly detection module

[0251] Physiological information of size Batch_Size × Window_Size × 6n is fed into the multi-channel feature extraction and anomaly detection module, such as... Figure 4 The data is split into x, y, ..., z channels and input into the noise superposition module. The noise superposition module randomly selects channels from the multi-channel data and superimposes Gaussian noise onto them. The data is then input into the data reconstruction module.

[0252] The data reconstruction module is a three-layer network model, including an input layer, a hidden layer, and an output layer. The input layer has c1 neurons, the hidden layer has c2 neurons, and the output layer has c3 neurons. The relationship between c1, c2, and c3 is given by equation (II):

[0253] c1>c2≥c3(II)

[0254] The final output size is The feature information is input into the anomaly monitoring module; the anomaly monitoring module reuses the network structure of the data reconstruction module, and obtains the reconstruction error by comparing the output layer data with the data output by the noise superposition module. The anomaly score obtained through the reconstruction error is calculated and compared with the judgment threshold α. If it exceeds the judgment threshold α, it is considered an abnormal behavior; the loss function of the reconstruction error is Equation (III):

[0255] L(x,z)= ‖xz‖ 2 (III)

[0256] In equation (III), x is the original input layer data before noise is added, and z is the output layer data;

[0257] Step S73: Input data transpose encoding module

[0258] The size is The feature vector is input into the behavior recognition module; the original input shape is set as x×y×z, representing width, height, and number of channels respectively. The original input is transposed along different axes to obtain x×y×z and z×y×z data; different channel weights are then redistributed to read information from three different dimensions. The transposed information is then summed and input into the SE module. The transposed encoding module based on three-channel attention can capture the interaction information between the spatial dimension, time, and multi-sensor modal dimensions of the input sensor tensor.

[0259] Step S74: Input data into SE module

[0260] The SE module is a squeeze and excitation network module. It first utilizes the global average ensemble features to efficiently calculate channel attention, which includes a pooling layer, a fully connected layer, a convolutional layer, a fully connected layer, and a convolutional layer linked in sequence.

[0261] The attention mechanism that combines average pooling and max pooling operations is described by equation (IV):

[0262] Wc=σ(f{w1,2}(AP(χ))+f{w1,w2}(MP(χ))) (IV)

[0263] In equation (IV), χ represents the tensor data of the three input SE modules. These represent average pooling and max pooling operations, respectively; σ is the sigmoid function, and w1 and w2 are network parameters used to adjust the scaling factors; Equation (IV) is a summary of Equation (V):

[0264] Wc=σ(w2ReLU(w1AP(χ))+w2ReLU(w1MP(χ))) (V)

[0265] The two fully connected layers in the SE module can be described by Equation (6), which uses two linear projections to assign corresponding weights to each channel.

[0266] Step S75: Additive Aggregation

[0267] The three attention channels output by the SE module are aggregated additively, and then Z-Pooling is used to preserve the feature representation while compressing the depth. The concatenated and pooled feature information is represented by Equation (VI):

[0268] Z_Pooling(χ) = [MaxPool 0d (χ),AvgPool 0d (χ)] (VI)

[0269] In equation (VI), 0d is the data in the 0th dimension where max pooling and average pooling occur; that is, Z_Pooling can reduce the 0th dimension of a given input tensor to 2 dimensions by aggregating two merged features. A tensor with shape x×y×z will be transformed into a tensor with shape 2×y×z through Z_Pooling.

[0270] Step S76: Layer Normalization

[0271] The aggregated output was batch-level normalized to obtain a 1×y×z output.

[0272] Step S77: Output the unit recognition result

[0273] The physiological information is converted from a vector to a scalar through an unfolding layer. After passing through a fully connected layer, the output information size is Batch_Size×Output_Length. The physiological information is then classified and identified by a Softmax classifier. The physiological information features of size are input into the Softmax classifier to solve for the probability of each behavior. The behavior corresponding to the highest probability is the final behavior recognition result of the multi-channel attention model network based on transpose encoding.

[0274] Step S8: Real-time Behavior Recognition

[0275] Real-time collected and preprocessed physiological information is input into a trained multi-channel attention model network behavior recognition model based on transpose coding for real-time recognition of the current behavior, and the classification results corresponding to the physiological information are output; including standing, sitting, lying down, walking, running, climbing stairs and user-defined specific behavioral activities;

[0276] Step S9: Location Data Analysis

[0277] Due to the encryption of LBS signals, location information cannot be directly obtained. Therefore, GPS signals and LBS signals are processed separately. Step S9 includes the following sub-steps:

[0278] Step S91: Data Decoding

[0279] The LBS signal is decoded. The LBS signal includes the communication base station code. The location information corresponding to the code is queried in the open source data warehouse to obtain the decoded base station location information.

[0280] Step S93: Multi-modal data fusion positioning

[0281] LBS base station positioning is performed by fusing parsed LBS data with GPS data. LBS base station positioning determines the user's location by calculating the distance between the base station and the user. The accuracy largely depends on the distribution and coverage of the base stations, and the error can exceed one kilometer. LBS base station positioning is a ground-based positioning method. All signals come from cell phone towers. Positioning can be performed wherever the mobile phone can receive a signal. Base station positioning signals have strong penetration in ground buildings and tunnels, which makes up for the shortcomings of GPS satellite positioning. At the same time, LBS base station positioning is mandatory. The base station positioning server actively initiates positioning to the terminal, which can better realize the monitoring function.

[0282] GPS positioning uses satellite signals, which have high accuracy. However, the positioning signal can be affected indoors, in underground garages, and in traffic tunnels, increasing positioning deviation. Even in areas with dense clouds, satellite signals can be limited, resulting in low positioning accuracy or positioning blind spots.

[0283] Neither GPS positioning alone nor base station positioning alone can meet the requirements of a specific environment. By using base station positioning in indoor areas without GPS signal to provide location information, the positioning blind spot can be reduced. At the same time, during the GPS module's satellite acquisition phase, base station positioning is used to determine the approximate location of the mobile phone, shortening the positioning time and obtaining accurate positioning information.

[0284] Step S10: Real-time information display

[0285] The behavior recognition results output by the multi-channel attention model network based on transposed encoding are displayed through the behavior display unit in the data display and correction module.

Claims

1. A wearable remote real-time monitoring system for power workers based on multi-mode data, characterized in that, It includes a real-time information acquisition module for operators, a multimodal data preprocessing module, a feature extraction and early warning module based on transpose encoding, and a data display and correction module, which are connected in sequence. The real-time information collection module for workers is used to collect real-time information through wearable devices, including environmental information, physiological information, and location information. The multimodal data preprocessing module is used to: receive real-time information from wearable devices and preprocess it; The transposed coding-based feature extraction and early warning module is used to: process environmental, physiological and location information data, and perform human behavior recognition through a transposed coding-based feature extraction algorithm; the transposed coding-based feature extraction and early warning module also includes a transposed coding module, which includes a transposed module, an SE module, a layer normalization module and an output module connected in series; The transpose module transposes the input data along different axes, thereby redistributing different channel weights, reading different information in the three dimensions, and then inputting the transposed three types of channel information into the SE module. The SE module is a squeezing and excitation network module; The layer normalization module performs batch layer normalization on the output of the SE module and inputs the output data into the output module. The output module inputs the output of the SE module into the fully connected layer, and then into the Softmax classifier to classify and identify the behavioral information, calculate the probability of each behavior, and obtain the behavior corresponding to the highest probability, which is the final behavior recognition result of the multi-channel attention network model based on the squeezing excitation module and the transpose encoding module. The data display and correction module is used to: receive data from the data identification and early warning module, integrate various data, and display and alarm on the web page; at the same time, it provides an administrator operation interface on the web page for administrators to send instructions and correct data.

2. The wearable remote real-time monitoring system for power workers based on multi-mode data according to claim 1, characterized in that, The real-time information acquisition module for operators includes an environmental information acquisition unit, a physiological information acquisition unit, a positioning information acquisition unit connected in parallel, and a data transmission unit connected in serial order therewith. The environmental information acquisition unit is used to acquire environmental information; the physiological information acquisition unit is used to acquire physiological information; the location information acquisition unit is used to acquire location information; and the data transmission unit is used to remotely transmit the collected user behavior data to the cloud server's database in real time via the TCP / IP protocol. The environmental information acquisition unit includes an air quality detection unit and a proximity sensing unit; The air quality detection unit is used to: read environmental data in the air, including the concentration of several toxic and harmful gases; The proximity sensing unit is used to: read the electric field strength near the human body, and realize power plant monitoring and alarm; The physiological information acquisition unit includes an accelerometer, an angular velocity sensor, a temperature sensor, and a PPG sensor; Accelerometers are used to acquire acceleration data along the X, Y, and Z axes; angular velocity sensors are used to acquire angular velocity data along the X, Y, and Z axes; temperature sensors are used to acquire body temperature data; and PPG sensors are used to measure heart rate and blood oxygen saturation. The location information acquisition unit includes a global positioning system and a Wi-Fi base station information receiving system; The Global Positioning System is used to acquire GPS signals and location-based services (LBS). The Wi-Fi base station information receiving system is used to obtain media access control address and signal strength; Preprocessing includes: noise reduction filtering, normalization, data fusion, and time series-based truncation.

3. The wearable remote real-time monitoring system for power workers based on multi-mode data according to claim 1, characterized in that, The multimodal data preprocessing module includes a data receiving unit, a data preprocessing unit, and a data normalization unit connected in sequence. The data receiving unit receives various data from wearable devices and packages them for transmission to the data preprocessing unit. The data preprocessing unit preprocesses different types of data and then transmits the data to the data normalization unit. After normalization, the data is transmitted to the feature extraction and early warning module based on transpose encoding.

4. The wearable remote real-time monitoring system for power workers based on multi-mode data according to claim 1, characterized in that, The feature extraction and early warning module based on transpose coding includes an environmental information processing unit, a physiological information processing unit, and a location information processing unit connected in parallel. The environmental information processing unit is serially connected to the environmental information early warning unit. The physiological information processing unit is further serially connected to the parallel physiological information early warning unit and physiological information recognition unit. The location information processing unit is further serially connected to the location information decoding unit, the location information fusion unit, and the out-of-safety-zone early warning unit. The environmental information processing unit is used to process environmental gas information and environmental voltage information, and to adjust the environmental gas concentration warning value and the environmental electric field voltage warning value. The environmental information early warning unit is used to compare environmental information with preset early warning values, and to issue an alarm when the early warning value is exceeded. The physiological information processing unit is used to process body temperature, heart rate, blood oxygen saturation, acceleration data of the X, Y, and Z axes, and angular velocity data of the X, Y, and Z axes, and to set personalized data warning thresholds according to different individual situations; The physiological information early warning unit is used to determine a reasonable range of values ​​for different individuals and output early warning information. Physiological information early warning refers to analyzing the results of processing received physiological signals and immediately issuing an alarm upon identifying abnormalities. Abnormalities refer to data that are found outside the normal physiological and medical range by analyzing the wearer's heart rate, blood oxygen, and body temperature information in combination with the wearer's physical signs; The physiological information recognition unit achieves behavior classification and early warning of abnormal behavior through a multi-channel attention network model based on the squeeze excitation module and the transposed encoding module; whereby behavior classification refers to human behavior recognition. The positioning information processing unit analyzes and processes the collected LBS and GPS positioning signals to parse the raw LBS base station data and combine the GPS and LBS data information to obtain the data positioning. The location information decoding unit performs LBS data decoding to obtain visible data information; The location information fusion unit fuses data collected from different sources; After confirming the location, the out-of-safe-zone warning unit compares it with the pre-defined safe zone and issues a warning if the location exceeds the safe zone range.

5. A wearable remote real-time monitoring system for power workers based on multi-mode data according to claim 4, characterized in that, The multi-channel attention network model based on the squeezing excitation module and the transposed encoding module includes a serially connected multi-channel feature and anomaly monitoring module, a transposed encoding module, and an output module. The multi-channel feature and anomaly monitoring module includes a noise superposition module, a data reconstruction module, an anomaly monitoring module, and a feature vector and anomaly information output module connected in series. The noise superposition module randomly selects channels of the multi-channel data to superimpose Gaussian noise and inputs it into the data reconstruction module. The data containing noise is then input into the data reconstruction module. The data reconstruction module is used to extract a higher-level representation of the input signal. The data reconstruction module includes an input layer, a hidden layer, and an output layer. The input layer to the hidden layer is decoding, and the hidden layer to the output layer is encoding. The output of the hidden layer is used as the feature representation of the original input data. The feature vector of the hidden layer is input to the feature vector and anomaly information output module. The output layer is used as the input of the reconstructed data, and the reconstructed data is output to the anomaly monitoring module. The anomaly monitoring module performs anomaly monitoring on the input data; the filtered abnormal data is input into the feature vector and anomaly information output module. The feature vector and anomaly information output module obtains feature vectors from the data reconstruction module and anomaly data from the anomaly monitoring module. It combines the two data, removes the input feature vectors that are detected as anomalies, and finally outputs all data to the transpose encoding module in three channels.

6. The wearable remote real-time monitoring system for power workers based on multi-mode data according to claim 1, characterized in that, The data display and correction module includes an information integration unit, which is serially connected to a physiological information display unit and an early warning information alarm unit, and then serially connected to a real-time data display unit and a user operation unit, and then serially connected to a parallel error information correction unit and an instruction issuance unit. The information integration unit is used to integrate all environmental information, location information, physiological information and its early warning information; The physiological information display unit is used to display physiological information in real time via a webpage; The warning information alarm unit is used by the cloud server to notify the administrator and power workers of the warning information. The real-time data display unit is used to display all environmental alarm information, location out-of-safe-zone warning information, and human behavior recognition information in real time via a web page. The user operation unit is used to receive operations from users and administrators, including database calls, database modifications, and data alarm threshold modifications; The instruction issuing unit is used to transmit and control the wearable device via the network the administrator's instructions.

7. A wearable remote real-time monitoring method for power workers based on multi-mode data, implemented by the wearable remote real-time monitoring system for power workers based on multi-mode data as described in any one of claims 1-6, characterized in that... The steps include the following: Step S1: Administrators select the monitoring scenario Administrators can send commands to wearable devices via a web interface to control them and select different information collection frequencies for different usage scenarios. Step S2: Sensors acquire multimodal information The wearable device receives an administrator command from step S1 and controls the sensors to selectively collect the user's physiological information, environmental information, and location information. The physiological information includes acceleration data of the X, Y, and Z axes obtained by the accelerometer and angular velocity data of the X, Y, and Z axes obtained by the angular velocity sensor, as well as heart rate and body temperature. The environmental information includes proximity data and hazardous gas data. The location information includes GPS, LBS data, media access control address, and signal strength. Step S3: Upload to cloud server The multimodal information collected by the wearable device is transmitted to the cloud server in real time via the TCP / IP protocol; Step S4: Determine the data type The multimodal information stored on the cloud server is classified as follows: ambient gas concentration and electric field voltage are classified as environmental data; GPS, LBS, media access control address and signal strength data are classified as location data; and acceleration, angular velocity, temperature and heart rate data are classified as behavioral data. Step S5: Physiological Information Preprocessing Preprocessing physiological information specifically refers to: sequentially performing filtering, normalization, fusion, and time-series-based extraction. Step S6: Environmental Information Early Warning The system compares the pre-set warning value with the ambient gas concentration value collected by the ambient gas sensor to determine whether the value exceeds the limit; when the received value is greater than the pre-set warning value, an alarm is triggered. Step S7: Build and train the behavior recognition model The behavior recognition model is a multi-channel attention model network based on transposed coding, which includes a multi-channel feature extraction and anomaly detection module and a transposed coding module; The user-defined parameters include: the shape of the input data, the shape of the reshaped data, and the superimposed noise parameters. Through iterative training, the parameters of the behavior recognition model are continuously optimized, ultimately resulting in the trained behavior recognition model, as shown in equation ( ). As shown in the image: （ ） Mode( In this context, A represents the number of samples, and M represents the number of categories. The sign function is set to 1 if the true class of sample i is equal to j, and 0 otherwise. Let i be the predicted probability that observed sample i belongs to category j; Step S8: Real-time Behavior Recognition The real-time collected and preprocessed physiological information is input into the trained multi-channel attention model network behavior recognition model based on transpose coding to perform real-time recognition of the current behavior and output the classification result corresponding to the physiological information. Step S9: Location Data Analysis The GPS signal and LBS signal are processed separately. Step S9 includes the following sub-steps: Step S91: Data Decoding The LBS signal is decoded. The LBS signal includes the communication base station code. The location information corresponding to the code is queried in the open source data warehouse to obtain the decoded base station location information. Step S93: Multi-modal data fusion positioning The parsed LBS data and GPS data are fused for positioning to obtain accurate positioning information. Step S10: Real-time information display The behavior recognition results output by the multi-channel attention model network based on transposed encoding are displayed through the behavior display unit in the data display and correction module.

8. A wearable remote real-time monitoring method for power workers based on multi-mode data, as described in claim 7, is characterized in that... Step S7: Set the parameters x, y, z of the input data to be the length, width, and number of channels of the original input data, respectively. Set the noise parameter L of the noise superposition module. L can be a random value of 0 or be Gaussian noise for superposition. Set the initial parameters w and b for the data reconstruction module, where w is the network weight and b is the bias coefficient; Setting parameters , , These represent the length, width, and number of channels of the original input data after feature extraction by the data reconstruction module; Set Batch_Size and Window_Size. Batch_Size refers to the number of behavioral samples in each batch of the multi-channel feature extraction and anomaly detection module, and Window_Size refers to the length of the sample data. The input layer of the data reconstruction module in the multi-channel feature extraction and anomaly detection module is configured as follows: There are 10 neurons, and the hidden layer has 100 neurons. There are neurons, and the output layer has _____ neurons. One neuron; Set the input-output comparison function in the anomaly monitoring module to f, and the judgment threshold to α; Set the input shape of the SE module to , , , which are the output shape parameters of the data reconstruction module, and the scaling parameter of its Excitation part is SERadio; Step S71: Input raw data Assume the input data is composed of fused measurement data from n triaxial accelerometers and triaxial angular velocity sensors, with a size of... Physiological information, For data length, It refers to the number of data channels; Step S72: Input data into the multi-channel feature extraction and anomaly detection module The size is The physiological information is sent to the multi-channel feature extraction and anomaly detection module, which splits it into x, y, z channel data and inputs it into the noise superposition module. The noise superposition module randomly selects channels of the multi-channel data and superimposes Gaussian noise, which is then input into the data reconstruction module. The data reconstruction module is a three-layer network model, including an input layer, hidden layers, and an output layer. The number of neurons in the input layer is... The number of neurons in the hidden layer is The output layer has One neuron, , and The relationship is expressed as ( ): （ ） The final output size is The feature information is input into the anomaly monitoring module; the anomaly monitoring module reuses the network structure of the data reconstruction module, and obtains the reconstruction error by comparing the output layer data with the data output by the noise superposition module. It then calculates the anomaly score obtained through the reconstruction error and compares it with the judgment threshold α. If the score exceeds the judgment threshold α, it is considered an abnormal behavior. The loss function of the reconstruction error is given by equation (…). ): （ ） Mode( In the diagram, x represents the original input layer data before noise is added, and z represents the output layer data. Step S73: Input data transpose encoding module The size is The feature vector is input into the behavior recognition module; the original input shape is set as... , representing width, height, and number of channels respectively. The original input is transposed along different axes to obtain , The data; thereby reallocating different channel weights, reading information from three different dimensions, and then inputting the transposed and summed three channel information into the SE module; Step S74: Input data into SE module The SE module is a squeeze and excitation network module. It first utilizes the global average ensemble features to efficiently calculate channel attention, which includes a pooling layer, a fully connected layer, a convolutional layer, a fully connected layer, and a convolutional layer linked in sequence. The attention mechanism combining average pooling and max pooling operations is described by equation ( ). ): （ ） Mode( )middle, To input tensor data from three SE modules, , and These represent average pooling and max pooling operations, respectively. It's the sigmod function. These are network parameters used to adjust the scaling factor; Equation ( ) is described as formula ( ): （ ） Step S75: Additive Aggregation The three attention channels output by the SE module are aggregated by addition, and then Z-Pooling is used to preserve the feature representation while compressing the depth. The concatenated and pooled feature information is represented by equation ( ). ): （ ） Mode( In this context, 0d represents the data in the 0th dimension where max pooling and average pooling occur. Step S76: Layer Normalization The aggregated output was then subjected to batch layer normalization to obtain... The output; Step S77: Output the unit recognition result The physiological information is converted from a vector to a scalar through the unfolding layer, and the output information size after passing through the fully connected layer is [value missing]. Physiological information is classified and identified using a Softmax classifier. Physiological information features of size are input into the Softmax classifier to solve for the probability of each behavior. The behavior corresponding to the highest probability is the final behavior recognition result of the multi-channel attention model network based on transpose coding.

9. A wearable remote real-time monitoring method for power workers based on multi-mode data, as described in claim 7 or 8, characterized in that, Step S5 includes the following steps: Step S51: Denoising Physiological Information Denoising of physiological information is performed using wavelet thresholding. Step S52: Normalization of physiological information Mean normalization is performed to bring features of different dimensions to the same numerical order of magnitude. Step S53: Multimodal data fusion The collected physiological information is aligned according to timestamps; Step S54: Time series-based extraction Referring to user-preset parameters, including the size of the sliding window, the cutting frequency, and the step size of the sliding window, the physiological information processed in step S53 is processed by sliding window, so that the physiological information is input into the multi-channel attention model network based on transpose coding in the form of information blocks.

Images