Multi-modal wearable sensing data real-time analysis method based on layered architecture

Abstract

The invention relates to the technical field of sensing signal processing, in particular to a multi-modal wearable sensing data real-time analysis method based on a layered architecture. The method comprises the following steps: acquiring sensor data, and respectively performing inertial navigation unit data acquisition, goniometer data acquisition and electrocardiogram data acquisition; performing data signal synchronization on the acquired sensor data; performing layered transmission on the synchronized data signal, performing sensor data processing on an edge layer and a fog layer in sequence, and transmitting the processed data to a cloud layer; performing sensor data modeling and comprehensive analysis on the cloud layer; and storing a data analysis processing result. According to the method, human body signals are collected, detection and classification of motion states, analysis of breathing states and monitoring of electrocardiosignals are included, different models and data processing are distributed to different levels for implementation according to different processing difficulties and different required calculation capabilities, and high efficiency of the whole model and algorithm is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of sensing signal processing, and in particular to a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture. Background Art

[0002] In the evaluation of the exercise ability of healthy people and the rehabilitation process of chronic disease patients, the responses of the cardio-pulmonary and circulatory systems during exercise play a crucial role. With the increase in cardiovascular diseases, diabetes, obesity and other chronic diseases, the pressure on the cardio-pulmonary system is also increasing. Real-time monitoring can more accurately evaluate an individual's endurance and metabolic efficiency, which is not only beneficial for athletes to design personalized training programs, but also of great significance for the health management of the general population and the rehabilitation of chronic disease patients.

[0003] The selection of sensors and the integration of multi-sensor data are crucial because wearable systems generate a large amount of different data from devices such as heart rate monitors, respiratory sensors and accelerometers, which requires effective data fusion for comprehensive analysis. Currently, the following problems generally exist in the sensing and monitoring devices on the market: most devices require professional medical background knowledge for operation and maintenance, and are not suitable for daily use by ordinary users; traditional sensing and monitoring devices are large in size and heavy in weight, making it difficult to carry around and not conducive to all-day monitoring; wearing these devices may affect the normal life and work of users and reduce user compliance; professional medical devices are expensive and difficult for ordinary users to afford.

[0004] Existing sensing and monitoring systems have exposed many deficiencies when dealing with the above challenges, especially in the construction of smart cities and large-scale data processing. Traditional data transmission methods cannot cope with the efficient transmission of a large amount of real-time data of users, often resulting in data loss or delay, seriously affecting the reliability of the system. At the same time, the lack of algorithms and tools specifically for big data analysis makes it difficult to extract valuable information from massive data. In addition, the scalability of existing systems is limited and it is difficult to adapt to the rapidly growing user needs in smart cities. In a complex environment such as a smart city, the introduction of a big data processing and transmission framework is particularly crucial.

[0005] The inaccuracy of data will affect the reliability of cardio-pulmonary function analysis. Therefore, a powerful distributed storage architecture is needed to ensure the integrity and protection of data during the entire collection, transmission and storage process, especially in a large-scale data environment. To address these challenges, by integrating edge, fog and cloud computing systems, using efficient computing, transmission capabilities and scalable storage architectures, the data processing efficiency can be optimized. This method can quickly analyze a large amount of data generated by wearable devices, thus supporting subsequent analysis and informed decision-making.

[0006] In view of this, the present invention proposes a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture to solve the above problems and meet the requirements of daily sensing monitoring and multi-user data transmission and analysis in the context of smart cities. Summary of the Invention

[0007] To solve the above-mentioned problems, the present invention provides a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture.

[0008] In a first aspect, a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture provided by the present invention adopts the following technical solutions: A real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture includes: Obtaining sensor data, wherein inertial navigation unit data acquisition, goniometer data acquisition, and electrocardiogram data acquisition are respectively performed; Synchronizing the data signals of the obtained sensor data; Transmitting the synchronized data signals in layers, wherein sensor data processing is sequentially performed in the edge layer and the fog layer and then transmitted to the cloud layer; sensor data modeling and comprehensive analysis are performed in the cloud layer; Storing the data analysis and processing results.

[0009] Further, the synchronizing of the data signals of the obtained sensor data includes achieving data synchronization during the transmission of multiple sensors of the same type, including data transmission synchronization of multiple inertial navigation units and data transmission synchronization of multiple goniometers. Among them, for the inertial navigation unit, the receiving moment is captured through an event timer, the propagation delay is calculated, and the channel asymmetry error is eliminated by using two-way delay measurement. And the integrity of the synchronized data is verified by fusing based on a Kalman filter; for the goniometer, the linear relationship between the master and slave clocks is fitted by the least squares method, and nonlinear correction is performed based on clock deviation compensation. The compensation formula is: a_corrected = a • [1 + γ(T - T 0 )], where γ is the crystal oscillator temperature coefficient and T 0 is the reference temperature.

[0010] Further, the sequentially performing sensor data processing in the edge layer and the fog layer includes performing inertial navigation unit data processing in the edge layer. Among them, the inertial navigation unit data is denoised using the Kalman filtering algorithm. For each dimension of the signal, the Kalman filtering algorithm is used for denoising, the system is predicted for the next moment, the state covariance is predicted according to the state transition matrix, and the Kalman gain is calculated according to the predicted covariance and the observation noise covariance for weighing the predicted value and the observed value, and the calculated Kalman gain is used to combine with the observed value Update the prediction state to obtain the optimal estimated state at time k , and at the same time update the optimal estimated covariance.

[0011] Furthermore, the sequential sensor data processing in the edge layer and the fog layer further includes processing the data of the goniometer in the edge layer. Specifically, the square roots of the data of two channels of each data segment of a single goniometer are calculated respectively, the signals of the two channels are combined, the continuous respiratory signal data are divided into multiple data segments according to time periods, the energy fusion algorithm is used for multiple goniometer data streams to generate a composite respiratory signal, and the dynamic allocation weight SNR of each goniometer signal within the sliding window is calculated.

[0012] Furthermore, the sequential sensor data processing in the edge layer and the fog layer further includes processing the electrocardiogram data in the edge layer. Specifically, the baseline drift correction is performed on the collected electrocardiogram data, the baseline drift below 0.5 Hz is separated by using a high-pass Butterworth filter, then the QRS complex wave is enhanced, and a band-pass filter is used to specifically enhance the energy of the QRS wave and suppress the electromyogram noise of 20 - 500 Hz and the low-frequency component of the T wave less than 5 Hz.

[0013] Furthermore, the sequential sensor data processing in the edge layer and the fog layer includes processing the data of the inertial navigation unit in the fog layer. Specifically, the acceleration data of the inertial navigation unit are processed to reduce the influence of gravity on motion classification, which is expressed as: where represents the transfer function, represents the complex frequency variable in the Laplace transform, refers to the quality factor, refers to the central angular frequency, is the filtered three-axis acceleration data, is the acceleration data after removing the influence of gravity.

[0014] Furthermore, the sequential sensor data processing in the edge layer and the fog layer further includes processing the data of the goniometer in the fog layer. Specifically, a hybrid filtering architecture is used to reduce the noise of the fused goniometer data, the time window length is dynamically adjusted, the low-frequency components of the respiratory signal are preferentially retained, and a Butterworth filter with a cut-off frequency of 10 Hz is used to eliminate the high-frequency interference of muscle tremors. After noise reduction, baseline correction and normalization are performed on the data. Morphological baseline extraction is adopted, and opening and closing operations are performed using a structural element matching the respiratory cycle to separate the slowly changing baseline drift. Then, respiratory dynamics analysis is carried out to reveal the hidden dynamic characteristics in respiration through the respiratory signal, and the maximum Lyapunov exponent is calculated to measure the sensitivity of the respiratory cycle to the initial conditions. The larger the value, the more unstable the rhythm.

[0015] Further, the processing of sensor data in the edge layer and the fog layer in sequence further includes processing electrocardiogram data in the fog layer. Among them, the QRS complex in the electrocardiogram data is detected. By introducing a dynamic integration window in the Pan-Tompkins algorithm, the window length is automatically adjusted according to the heart rate to avoid waveform overlap at high heart rates. Then, after performing square operation and moving integration respectively, time-frequency domain feature extraction is performed on the electrocardiogram signal, and a continuous RR interval sequence { } is calculated. The power spectral density is estimated by the Lomb-Scargle algorithm, and finally, respiratory-electrocardiogram coupling analysis is performed to establish a phase synchronization model between the respiratory signal D_norm(t) and the RR interval.

[0016] Further, the modeling and comprehensive analysis of sensor data in the cloud layer includes modeling inertial navigation unit data in the cloud layer. Among them, the modeling of inertial navigation unit data includes using a DCNN classifier for modeling, adopting 6 modules with increasing levels. The first 3 modules adopt a residual Inception structure, and the last 3 modules introduce dynamic deformable convolutions to capture irregular motion features. The number of output channels of each module gradually increases from 64 to 512, and an SE attention mechanism is added after convolution to strengthen the key feature response. Cross-modal interaction layers are inserted after the third and fifth modules, and the spatio-temporal correlation between acceleration and angular velocity is fused through a cross-attention mechanism.

[0017] Further, the modeling and comprehensive analysis of sensor data in the cloud layer further includes modeling angle meter data in the cloud layer. Among them, Fourier transform is performed using a distributed computing framework to convert the signal from the time domain to the frequency domain. A clustering algorithm is used to automatically identify the main frequency components in the frequency spectrum to determine the respiratory frequency. The amplitude spectrum of the signal is calculated, and the intermediate results are stored using a cloud distributed storage system. By detecting the peak or valley value of the respiratory waveform, the respiratory cycle, that is, the time interval between two adjacent breaths, is calculated. And the respiratory signal is decomposed into wavelet coefficients of different scales to analyze the waveform characteristics of the signal, and the data is comprehensively analyzed and stored in the long term. In summary, the present invention has the following beneficial technical effects: By collecting human signals, including the detection and classification of the motion state, the analysis of the respiratory state, and the monitoring of the electrocardiogram signal, and allocating these different models and data processing to different levels according to the processing difficulty and required computing power, the efficiency of the entire model and algorithm is ensured. Through the method of the present invention, real-time monitoring and analysis of the user's sensing signals can be achieved, a detailed health status report can be provided for the user, and the data management and analysis requirements in a multi-user environment can be supported, promoting the development of smart city health management. Description of the Drawings

[0018] Figure 1 Schematic diagram of a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture in Embodiment 1 of the present invention; Figure 2 Schematic diagram of the overall Apache-Flink structure in Embodiment 1 of the present invention. Detailed implementation manners

[0019] The present invention will be further described in detail below with reference to the accompanying drawings.

[0020] Embodiment 1 Refer to Figure 1 , a real-time analysis method for multi-modal wearable sensing data based on a hierarchical architecture in this embodiment includes: Obtain sensor data, wherein inertial navigation unit data acquisition, goniometer data acquisition, and electrocardiogram data acquisition are respectively performed; Synchronize the acquired sensor data; Transmit the synchronized data signals in layers, wherein sensor data processing is sequentially performed in the edge layer and the fog layer and transmitted to the cloud layer; sensor data modeling and comprehensive analysis are performed in the cloud layer; Store the data analysis and processing results.

[0021] Specifically: This embodiment is a multi-modal data hierarchical processing flow and a calculation and modeling process, which details the processing flow, modeling process, and result generation steps for obtaining multi-modal signals from 5 inertial navigation units, 3 goniometers, and electrocardiogram sensors worn by users, as Figure 1 shown.

[0022] S1. Data acquisition, The user wears all the sensors at the corresponding positions on the body and starts data acquisition.

[0023] S1.1 Inertial navigation unit data acquisition, Each user wears 5 inertial navigation units, respectively on the back, left upper arm, right upper arm, left thigh, and right thigh, comprehensively recording the physical information of all parts of the body during movement. The inertial navigation unit data is represented by , , corresponding to 5 inertial navigation units in sequence, where consists of three-dimensional ( ) acceleration signals, three-dimensional ( ) gyroscope signals, and three-dimensional ( ) magnetometer signals.

[0024] S1.2 Goniometer data acquisition, Each user wears three goniometers to record the angular changes of the chest cavity in two planes of space. These angular change parameters are represented by , and , corresponding to three different goniometers respectively, where . The time series of data acquisition is set as , and the sampling frequency is , where is the sampling period. The goniometer data matrix can be expressed as: S1.3 Electrocardiogram data acquisition, Each user wears a single-lead electrocardiogram sensor, and the electrode is aligned with the V2 lead on the chest. The signal is represented by , where , and the fixed sampling frequency of the electrocardiogram sensor is 500 Hz.

[0025] S2. Data transmission from sensors to the edge layer, All sensor data are transmitted to the mobile device via Bluetooth. For multiple sensors of the same type, data synchronization is achieved during transmission, including data transmission synchronization of multiple inertial navigation units and data transmission synchronization of multiple goniometers. The specific implementation form is as follows: The mobile device serves as the master device and also as the edge layer in the hierarchical transmission framework. It provides a clock reference to multiple inertial navigation units via Bluetooth connection. The slave devices synchronize according to the clock of the master device to ensure the synchronization of timestamps of different inertial navigation units during data transmission. Specifically: Designate the central inertial navigation unit as the master device (Master), and the remaining units as slave devices (Slave). The master device establishes a star topology network via the Bluetooth Low Energy 5.1 protocol to form a 1:M connection mode; The master device utilizes the 28-bit CLK clock built into the Bluetooth baseband layer to embed in the periodic broadcast packet: the local clock value of the master device, the timestamp of the last clock correction, and the clock drift compensation coefficient. The slave device captures the reception moment through the event timer, calculates the propagation delay, and uses two-way delay measurement to eliminate the channel asymmetry error; Based on the Kalman filter to fuse: the clock phase difference observation value, the crystal oscillator frequency stability, and the temperature drift characteristic, establish a second-order clock model: where represents the cumulative phase deviation of the slave device clock relative to the master device reference clock at time t, represents the initial deviation of the master and slave clocks measured through timestamp exchange at the start of the system, represents the theoretical operating frequency of the crystal oscillator of the inertial navigation unit, represents the frequency aging coefficient, represents the frequency drift acceleration coefficient.

[0026] Perform IEEE 1588-style boundary clock calibration every 100 ms, use CRC-16 to verify the integrity of synchronization data, and set the synchronization error threshold. When the limit is exceeded, trigger an emergency resynchronization.

[0027] The data transmission synchronization of the goniometer constructs a synchronization broadcast network through the extended broadcast channel of the mobile device. Each goniometer is dynamically assigned a dedicated time slot for transmitting angle data. The mobile device periodically sends a reference pulse signal as a hardware-level time reference. The goniometer obtains the master clock reference by listening to the broadcast packet and uses its hardware interrupt pin to capture the rising edge of the reference pulse to achieve preliminary synchronization.

[0028] Clock deviation compensation: The mobile device sends a synchronization packet containing three consecutive time reference points (t 1 , t 2 , t 3 ) to all goniometers every 50 ms. The goniometer records the reception time (t 1 ', t 2 ', t 3 ') according to the local crystal oscillator. The linear relationship between the master and slave clocks is fitted by the least squares method: Δt = a•t + b, where a represents the frequency drift rate and b is the initial offset. This model is nonlinearly corrected once an hour, and the compensation formula is: a_corrected = a•[1 + γ(T - T 0 )], where γ is the crystal oscillator temperature coefficient and T 0 is the reference temperature.

[0029] When the goniometer sends angle data, it embeds the predicted clock deviation Δt into the data packet header. After receiving the data, the mobile device dynamically adjusts the time slot allocation according to the difference between the current global time and the time stamp in the packet header.

[0030] S3. Edge layer data processing, modeling, and transmission, The mobile device, as the edge layer of the hierarchical network model, performs preliminary processing and modeling on all sensor data and transmits the data based on the hierarchical transmission framework.

[0031] S3.1 Edge layer data processing, modeling, S3.11 Edge layer inertial navigation unit data processing and modeling, First, perform denoising processing on the inertial navigation unit data. For each dimension of the signal, use the Kalman filter algorithm for denoising.

[0032] Predict the next moment of the system according to the state transition matrix: where is the optimal estimate at the previous moment, is the state transition matrix of the system, is the control input matrix of the system, is the control input of the system.

[0033] Predicted state covariance is: where, is the process noise covariance matrix.

[0034] According to the predicted covariance and the observation noise covariance, calculate the Kalman gain for weighing the predicted value and the observed value, and the calculation formula is: where, is the observation matrix, is the observation noise covariance matrix.

[0035] Using the calculated Kalman gain combining with the observed value update the predicted state to obtain the optimal estimated state at time k , and the calculation formula is: At the same time, update the optimal estimate covariance : Through the above Kalman filtering algorithm, the noise in the data can be effectively suppressed and a more accurate state estimate can be obtained.

[0036] The L2-norm is used to quantify the amplitude of the signal and reflect the overall strength of the signal. Using the L2-norm theory, calculate the data of the inertial navigation unit after filtering, and calculate the L2 norms of the acceleration and angular velocity respectively.

[0037] Next, sum the weighted L2 norms of the acceleration and angular velocity, integrate the acceleration and angular velocity information, and enhance the robustness of action classification.

[0038] where and are the weight coefficients. Set the threshold by calculating the statistic of the L2 norm of the static data : where is the mean of the L2 norm of the static data, is the standard deviation of the L2 norm of the static data, is the confidence coefficient. When the sum value of the L2 norm at the current moment is input , by comparing it with the set threshold, when it is less than the threshold, it is judged that the human body is in a static action; when it is greater than the threshold, it is judged that the human body is performing a dynamic action, realizing the classification of human static and dynamic actions.

[0039] S3.12 Edge layer goniometer data processing, Perform preliminary processing on the data on the edge layer device. Calculate the square root of the data of the two channels of each data segment of a single goniometer respectively, merge the signals of the two channels, and divide the continuous respiratory signal data into multiple data segments according to time periods. Each segment contains the signal change value within a certain time range. Assume that each segment of data is , the angle change value , where represents the k-th time period, expressed as: Use the energy fusion algorithm to generate a composite respiratory signal for the data streams of 3 goniometers, calculate the SNR of each goniometer signal within the sliding window, and dynamically allocate weights.

[0040] S3.13 Edge layer electrocardiogram data processing, First, perform baseline drift correction on the collected electrocardiogram data. Use a second-order high-pass Butterworth filter, because it has the maximum flatness characteristic in the passband and can accurately separate the baseline drift below 0.5 Hz (mainly due to respiratory movement and electrode contact impedance change) without affecting the QRS complex morphology. The transfer function is defined as: Then, enhance the QRS complex. Use a 5-15 Hz band-pass filter to specifically enhance the energy of the QRS wave (mainly distributed in 10-25 Hz), and suppress the myoelectric noise (20-500 Hz) and the low-frequency component of the T wave (<5 Hz). The transfer function satisfies: Finally, perform real-time heart rate estimation. Use the dynamic threshold peak detection algorithm to adaptively adjust the detection threshold within the sliding window to overcome the influence of signal amplitude fluctuation caused by movement.

[0041] S3.2 Edge layer data transmission, The edge device classifies the preprocessed data by theme, including: acceleration, angular velocity, goniometer data, and ECG data , and sends it to the Kafka cluster. The data is in Avro serialization format, with timestamps and device ID metadata attached. The topics are hashed and partitioned by device ID to ensure the sequentiality of data for the same user, while also supporting parallel Kafka consumption.

[0042] S4. Fog layer data processing, modeling, and transmission To achieve more efficient data processing and more accurate behavior analysis, a local server with medium computing power is selected as the fog layer node. As the core layer connecting edge devices and the cloud, the fog layer undertakes the task of in-depth analysis of the input data. Compared with edge devices, the fog layer has more powerful computing capabilities, capable of supporting complex signal processing algorithms and model training, thereby classifying human behaviors in more detail. By establishing and optimizing the behavior classification model in the fog layer, the dependence on cloud computing power can be effectively reduced, data transmission latency can be decreased, providing strong support for application scenarios with high real-time requirements.

[0043] S4.1 Fog layer data reception The Kafka consumer group in the fog computing layer consists of multiple fog computing nodes, responsible for real-time consumption of the preprocessed data sent from the edge layer to Kafka, including: acceleration, angular velocity, goniometer data, and ECG data , and the Kafka Topic is hashed and partitioned by device ID (for example, each user has an exclusive partition) to ensure that data for the same user is processed by the same consumer, maintaining temporal consistency. The data consumption process starts with a periodic poll() operation. The fog node pulls data in batches at 100ms intervals and converts the raw byte stream in Avro format into a structured object (including fields such as timestamps, device IDs, and data values) during the deserialization stage. Subsequently, the system dynamically adjusts the processing logic according to the real-time recognized human activity types (including "dynamic" and "static"): for dynamic activities, a 5-second sliding window (step size 1 second) is used to continuously calculate the correlation coefficient between the heart rate peak and the step frequency; for static activities, a 10-second rolling window is triggered to calculate the average respiratory rate.

[0044] S4.2 Inertial navigation unit data modeling in the fog layer Process the acceleration data of the inertial navigation unit to reduce the impact of gravity on motion classification. The specific processing method is as follows: Among them, represents the transfer function, represents the complex frequency variable in the Laplace transform, refers to the quality factor, refers to the central angular frequency is the filtered three-axis acceleration data, is the acceleration data after removing the gravity influence.

[0045] Calculate the sum of squares of the processed acceleration data, merge the three-axis acceleration data into one acceleration data, divide the continuous acceleration signal data into multiple data segments according to time periods, and each segment contains the signal change value within a certain time range. Let each segment of data be , where represents the k-th time period, expressed as: There are significant differences in different static actions in the inertial navigation unit, and the complexity of the classification model is small. Therefore, specific static behavior classification is selected in the fog layer. Acceleration and angular velocity are both inputs for constructing the designed deep convolutional neural network (DCNN) classifier. This architecture consists of two convolutional modules, and each module contains a convolutional operation, batch normalization, and ReLU activation function. The window size used for the convolutional operation is 5*5. In addition, this architecture also includes a dropout layer, a fully connected layer, and a softmax transfer function. The softmax transfer function classifies the activities by calculating the probabilities associated with each input in the softmax transfer function. The activity with the highest probability is selected as the prediction result. The learning rate decay factor is set to 0.01.

[0046] The network consists of four convolutional layers, and the number of output channels increases sequentially (18, 36, 72, and 100). There is a batch normalization layer and a max pooling layer after each convolutional layer. The kernel sizes of the convolutional layers are (3, 3), (2, 3), (2, 2), and (2, 2) respectively, and there are corresponding strides and paddings to preserve the spatial information. After the convolutional layers, the feature maps are flattened and input into two fully connected layers with 180 and 3 neurons respectively. A dropout layer with a probability of 0.2 is added before the first fully connected layer to prevent overfitting. This network is trained using an adaptive optimizer with a learning rate of 0.02, 20 epochs, and a batch size of 128. This architecture utilizes the spatial and temporal dependencies of sensor data to effectively classify human static activities.

[0047] S4.3 Processing of fog layer gyroscope data, Use a hybrid filtering architecture to denoise the fused gyroscope data, dynamically adjust the time window length, and preferentially retain the low-frequency components of the breathing signal. Adjust the window length according to the signal power spectral density (PSD) calculated in real time . Assign Gaussian weights to the data within the window: A Butterworth filter with a cut-off frequency of 10 Hz is adopted to eliminate high-frequency interferences such as muscle tremors. Fast convolution operations are achieved through preset filter coefficients. At the same time, Kalman optimization is carried out to establish a two-state model of respiratory movement (angle value + change rate), and the signal tracking ability is continuously optimized through a noise parameter adaptive mechanism. When sudden interferences are detected, the observation noise weight is automatically reduced to improve prediction stability.

[0048] State equation: Observation equation: The process noise covariance matrix and the observation noise variance are updated online: Where is the variance of the second derivative of the signal, is the forgetting factor.

[0049] After noise reduction, the data is further subjected to baseline correction and normalization. Morphological baseline extraction is adopted, and opening and closing operations are performed using a structural element matching the respiratory cycle to separate the slowly varying baseline drift. Define a flat structural element , with a length , being the current estimated value of the respiratory frequency.

[0050] Dynamic amplitude normalization is carried out. Within a sliding window, the signal is linearly mapped to the interval [-1, 1] to eliminate the sensor sensitivity difference while retaining the relative waveform morphology.

[0051] Subsequently, respiratory dynamics analysis is performed. The hidden dynamic characteristics in respiration are revealed through the respiratory signal, and the maximum Lyapunov exponent is calculated to measure the sensitivity of the respiratory cycle to the initial conditions. The larger the value, the more unstable the rhythm. Fast estimation is achieved by tracking the divergence rate of adjacent phase trajectories, avoiding complex matrix operations.

[0052] S4.4 Processing of electrocardiogram data in the fog layer, First, the QRS complexes in the electrocardiogram data are detected. Adaptive QRS detection is adopted, and a dynamic integration window is introduced in the Pan-Tompkins algorithm to automatically adjust the window length according to the heart rate, avoiding waveform overlap at high heart rates. The improved Pan-Tompkins algorithm consists of three steps. The first step is differential processing to enhance the QRS slope.

[0053] In the second step, a squaring operation is performed to highlight the high-frequency part.

[0054] In the third step, a moving integral is carried out, and a dynamic integral window is used to further smooth the noise.

[0055] Next, time-frequency domain feature extraction is performed on the electrocardiogram signal, and the continuous RR interval sequence { } is calculated, and the power spectral density P(f) is estimated by the Lomb-Scargle algorithm.

[0056] SDNN reflects the overall autonomic tone. The Lomb-Scargle method overcomes the problem of non-uniform sampling of RR intervals and accurately quantifies the LF / HF power ratio.

[0057] Finally, respiratory-electrocardiogram coupling analysis is carried out to establish a phase synchronization model between the respiratory signal and the RR interval: Multimodal fusion is performed, and the respiratory sinus arrhythmia (RSA) is quantified by the phase synchronization index ρ to reveal the vagus nerve-mediated cardiorespiratory coupling mechanism.

[0058] S4.5 Fog layer data transmission, After real-time processing and context-aware aggregation are completed at the fog computing layer, the system directs the data to the cloud computing layer through a hierarchical Kafka topic strategy. The fog node acts as a producer and divides the processed data into two types of streams, including the acceleration and angular velocity data of the inertial navigation unit, the respiratory signal after processing the goniometer data, and the processed ECG electrocardiogram data , which are published through one type of topic, serialized using Avro and appended with a compression flag (LZ4 algorithm compression rate 60%). Each message contains the device ID, the start and end timestamps of the time window, and the aggregated metrics in JSON format; the other type is the cold backup of the original data. For the original waveforms of sensors that need to be stored long-term (such as ECG segments), they are written into the / batch / data topic after being hashed and partitioned by user ID. The number of partitions dynamically matches the scale of the cloud storage cluster nodes to avoid write hotspots. During the transmission process, the fog layer ensures that the message is delivered only once through the idempotent producer of Kafka, and at the same time enables SSL encryption and ACL permission control to ensure the privacy compliance of medical data.

[0059] S5 Cloud layer data analysis, storage, and model building, The cloud layer plays a crucial role in the entire hierarchical algorithm. The central control system is selected as the cloud layer, which has powerful computing capabilities and storage space, can efficiently process large-scale data, support complex model training and inference, can achieve dynamic action classification, comprehensively analyze and long-term store other data, and meet the requirements of real-time, collaboration, and scalability.

[0060] S5.1 Cloud layer data reception The cloud computing layer pulls two types of data from the topics of the fog layer through the Kafka Connect service: the pre-aggregated summary data of the fog layer (including ), and the long-term archive of the original sensor data. The summary data is directly stored in the time series database for quick query, while the original data forms cold storage through distributed writing to the HDFS cluster, with a retention period of 6 months to support retrospective analysis. For batch processing tasks, the system uses Apache Spark to build an offline pipeline, triggering global feature extraction once an hour - for example, cross-user cardiopulmonary function correlation analysis (such as the distribution relationship between heart rate recovery rate and age), and at the same time training a personalized exercise prescription generation model through the XGBoost algorithm. The trained model is encapsulated as a microservice in ONNX format and deployed to the Kubernetes cluster, interacting with the front-end health management platform through the REST API.

[0061] S5.2 Cloud layer inertial navigation unit data modeling The dynamic activity classification task is executed in the cloud, leveraging the powerful computing power and distributed storage capabilities of cloud computing. Moreover, cloud training supports large-scale dataset processing, model parallel training, and real-time inference, meeting the high demand for computing resources in dynamic activity classification.

[0062] Based on the classifier developed for static actions, a new and more complex DCNN classifier is designed, expanding 4 basic convolutional modules to 6 modules with increasing levels. The first 3 modules adopt the residual Inception structure (including 1x1, 3x3, and 5x5 parallel convolutional kernels), and the last 3 modules introduce dynamic deformable convolutions to capture irregular motion features. The number of output channels of each module gradually increases from 64 to 512, and an SE attention mechanism is added after convolution to strengthen the key feature response. Cross-modal interaction layers are inserted after the third and fifth modules to fuse the spatio-temporal correlation of acceleration and angular velocity through the cross-attention mechanism: Among them, Q and K come from different sensor branches respectively. A multi-scale adaptive pooling layer is added before the fully connected layer, which combines three scales of 1x1, 3x3, and global average pooling, and dynamically weights and fuses features of different granularities through a gating network. An uncertainty calibration unit is added before the Softmax layer, which generates confidence weights based on Monte Carlo Dropout to suppress low-reliability predictions.

[0063] On the basis of the original adaptive learning rate, a hierarchical weight decay strategy is introduced. The decay coefficient of 0.001 is adopted for the underlying convolution kernels, and the fully connected layer in the upper layer is adjusted to 0.0005 to avoid overfitting. At the same time, gradient clipping and Lookahead optimization are integrated, and the parameter snapshot is updated every 5 steps to improve stability. Meta-learning-driven hyperparameter tuning is adopted, embedded in the MAML framework, and the generalization ability of the model is quickly evaluated through a small number of support set samples during the cloud training stage, and the learning rate decay coefficient and batch size are dynamically adjusted. A source-target feature decomposition module is constructed after the fully connected layer, and the target domain prototype is statistically calculated by using a sliding window, and cross-domain feature alignment is realized through the Wasserstein distance metric: where is the feature distance matrix between the source domain and the target domain.

[0064] An adaptive mechanism is adopted to improve the robustness of the model. This adaptive mechanism mainly consists of four parts: misclassification monitoring, misclassified sample storage, trigger condition detection, and adaptive retraining. During the test stage, as long as a difference is detected between the predicted result ( ) and the actual result , the misclassified data will be recorded.

[0065] Once the number of these errors reaches the predefined threshold , that is, , the classifier will start the adaptive retraining process to improve its accuracy.

[0066] This enhanced architecture realizes high-concurrency processing capabilities through the deep integration of structural complexity improvement and the adaptive mechanism, and meets the real-time response requirements relying on the elastic architecture of cloud computing.

[0067] S5.3 Cloud Angle Meter Data Modeling, The fitted respiration signal is obtained through multiple data processes, and the main features of the respiration signal, such as frequency, amplitude, period, waveform, etc., are analyzed through modeling in the cloud.

[0068] Frequency analysis: Use the distributed computing framework in the cloud to perform Fourier transform, converting the signal from the time domain to the frequency domain. Distributed computing can process large-scale data sets and improve computing efficiency.

[0069] Use clustering algorithms to automatically identify the main frequency components in the frequency spectrum, thereby determining the breathing frequency. The high computing power of the central control system supports the training and inference of complex models. The peak frequency corresponds to the main frequency of breathing.

[0070] Amplitude analysis: Calculate the amplitude spectrum of the signal and use the cloud-based distributed storage system to store intermediate results to ensure the high availability and reliability of the data.

[0071] Analyze the amplitude magnitudes of different frequency components to reflect the strength changes of breathing. Use the real-time data processing framework in the cloud to monitor the breathing signal in real time, detect abnormal situations in a timely manner and issue alarms.

[0072] Period analysis: By detecting the peaks or valleys of the breathing waveform, calculate the breathing period, that is, the time interval between two adjacent breaths. Period analysis can reveal the rhythm of breathing and detect abnormal situations such as apnea. Through the peak detection method, calculate the period of the signal. Let the period of the signal be , then there is: Among them, is the peak frequency in the frequency spectrum. Period analysis can identify the periodic changes of breathing and detect the rhythm of breathing.

[0073] Similarly, through the peak detection algorithm, calculate the inhalation time and the exhalation time , and calculate the duty cycle.

[0074] Waveform analysis: Decompose the breathing signal into wavelet coefficients of different scales, and analyze the waveform characteristics of the signal, such as the rise time, fall time and waveform shape of the breathing waveform. Wavelet transform can identify more subtle changes in the breathing waveform, such as abnormal breathing patterns such as wheezing and stridor. Through wavelet transform, analyze the waveform characteristics of the signal. Let the continuous wavelet transform of the signal be , then there is: Among them, is the scale parameter, is the translation parameter, is the mother wavelet function. A deep learning model is used to extract the detailed features of the respiratory signal and identify abnormal breathing patterns. The GPU resources in the cloud can accelerate the training and inference of the deep learning model.

[0075] S5.4 Comprehensive analysis and long-term storage of cloud data, The cloud layer is not only responsible for data modeling, but also needs to comprehensively analyze multi-source data and achieve long-term data storage. The implementation methods of comprehensive analysis and long-term storage are introduced separately below.

[0076] S5.41 Comprehensive analysis of cloud data, The comprehensive analysis of cloud data is to integrate multi-source data such as inertial navigation unit data and goniometer data, extract valuable insight information, and support decision-making and applications.

[0077] Perform statistical modeling on multi-source data, analyze the distribution characteristics, correlations, and change trends of the data. Implement data visualization to display the change trends of users' health data through charts. Generate personalized health suggestions based on the analysis results. If abnormal data is detected, the system will generate an alarm message and prompt the user to take further health measures. Conduct population health monitoring, combine data from multiple users, and perform statistical analysis of the population health status. This can be used to identify population health trends or the epidemic trends of certain diseases, providing data support for public health management.

[0078] The cloud layer supports real-time data analysis and uses a stream processing framework to achieve low-latency analysis. Distributed computing resources are adopted to ensure the scalability of analysis tasks and adapt to the growth of data scale.

[0079] S5.42 Long-term storage of cloud data, At the cloud computing layer, the long-term data storage system adopts a strategy that combines hierarchical storage architecture with automated lifecycle management to ensure the accessibility, security, and optimal cost of massive medical data. After the original sensor data (such as ECG waveforms, respiratory flow time series) is ingested through the / batch / data topic of Kafka, it is first written in a columnar storage format (Parquet) in slices to an object storage (such as AWS S3) and partitioned in both time and space dimensions - in the time dimension, it is divided by the collection date (such as year=2023 / month=08 / day=14), and in the space dimension, it is hashed and bucketed by the user's geographical location (such as region=us-east / bucket=05). This partitioning strategy significantly improves the query efficiency by time range or geographical attributes, while avoiding file overload in a single directory.

[0080] To balance storage costs and access performance, the system introduces a three-tier hot-warm-cold storage strategy: Hot storage layer: Retain data for the most recent 30 days and store it in a distributed file system supported by SSDs (such as HDFS) for immediate analysis with high-frequency access (such as retrieving the recent health trends of users).

[0081] Warm storage layer: Migrate data from 30 days to 1 year to a low-cost HDD cluster and adopt the Erasure Coding (EC coding) redundancy strategy to reduce the storage overhead by 40% while ensuring availability.

[0082] Cold storage layer: Compress data over 1 year into the Zstandard format (with a compression ratio of 5:1) and then archive it to a tape library or a service like Glacier. It only supports offline batch reading and is suitable for compliance audits or long-term scientific research analysis.

[0083] The data life cycle is dynamically managed by a policy engine: Based on access pattern analysis (such as the metadata access logs recorded by Apache Atlas), automatically demote low-frequency access data for storage; at the same time, set the retention period according to data sensitivity (such as ECG data containing personal health information) (such as 6 years stipulated by the EU GDPR). After the expiration, trigger an automated erasure process, and record an immutable audit log of the erasure process to a blockchain node for compliance traceability. The cloud layer uniformly manages metadata through a global data catalog (such as AWS Glue), annotates each data file with information such as the source device, acquisition parameters, and preprocessing version, and links with a permission system (such as Apache Ranger) to achieve column-level fine-grained access control (such as only allowing cardiologists to access the abnormal heart rate records of specific patients). The decoupled design of the storage layer and computing resources (such as performing direct S3 queries through Presto) enables data analysis without migrating data, further reducing the complexity of operation and maintenance.

[0084] As Figure 2 shown, the partition design of the Kafka topic (partition 1 to partition n) supports horizontal scaling, allowing the number of partitions to be dynamically increased according to data throughput to avoid a single-point bottleneck. The "topic: user1 - user n" at the user storage end corresponds to multi-tenant isolated storage to ensure fine-grained control of data permissions. The goniometer data uses adaptive Kalman filtering to fuse multi-sensor data at the edge layer and adjusts the weight coefficients in real time to reduce motion interference. The fog layer uses a lightweight convolutional network and achieves low-latency action recognition through model pruning, and synchronizes and updates with the cloud-trained model regularly. The QRS wave is located for the ECG data at the fog layer, and the disordered data stream is processed in combination with the Event Time mechanism of Flink to ensure medical-grade timing accuracy.

[0085] The denoising process of inertial sensor data is completed at the edge layer, reducing the amount of original data and consuming less computing resources in the fog layer. When an edge layer node fails, the partition is automatically reallocated. Combining with the Savepoint mechanism of Flink, seamless continuation of the processing offset is achieved and zero data loss occurs. The cloud storage is docked with the Flink Batch API, supporting offline analysis of historical data without affecting the throughput of real-time stream processing, and realizing T+0 and T+1 hybrid queries.

[0086] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited accordingly. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A multimodal wearable sensor data real-time analysis method based on a hierarchical architecture, characterized in that: include: Acquiring sensor data, including inertial navigation unit data acquisition, goniometer data acquisition and electrocardiogram data acquisition; Performing data signal synchronization on the acquired sensor data; The synchronized data signals are transmitted in layers, where sensor data is processed in the edge layer and fog layer in turn and transmitted to the cloud layer; sensor data modeling and comprehensive analysis are performed in the cloud layer; The data analysis and processing results are stored.

2. The multimodal wearable sensor data real-time analysis method based on a hierarchical architecture according to claim 1 is characterized in that: The data signal synchronization of the acquired sensor data includes realizing data synchronization for multiple sensors of the same type during transmission, including data transmission synchronization of multiple inertial navigation units and data transmission synchronization of multiple goniometers, wherein, for the inertial navigation unit, the receiving time is captured by an event timer, the propagation delay is calculated and the channel asymmetry error is eliminated by using a two-way delay measurement, and the integrity of the synchronization data is verified based on Kalman filter fusion; for the goniometer, the linear relationship between the master and slave clocks is fitted by the least squares method, and nonlinear correction is performed based on clock deviation compensation.

3. The multimodal wearable sensor data real-time analysis method based on a hierarchical architecture according to claim 2 is characterized in that: The sensor data processing is performed in the edge layer and the fog layer in turn, including inertial navigation unit data processing in the edge layer, and human body dynamic and static motion classification based on a set threshold value using the inertial navigation unit data, wherein the inertial navigation unit data is denoised using a Kalman filter algorithm, the signal of each dimension is denoised using a Kalman filter algorithm, the system is predicted at the next moment, the state covariance is predicted according to the state transfer matrix, and the Kalman gain is calculated according to the predicted covariance and the observed noise covariance. Used to weigh the predicted value and the observed value, using the calculated Kalman gain Combining Observations Update the predicted state to obtain the optimal estimated state at time k , and update the optimal estimated covariance at the same time.

4. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 3 is characterized in that: The method sequentially processes sensor data at the edge layer and the fog layer, and also includes processing goniometer data at the edge layer, wherein square roots are calculated for two channel data of each data segment of a single goniometer, the signals of the two channels are merged, the continuous breathing signal data is divided into multiple data segments according to time periods, an energy fusion algorithm is used for multiple goniometer data streams to generate a composite breathing signal, and the dynamic allocation weight SNR of each goniometer signal in the sliding window is calculated.

5. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 4 is characterized in that: The sensor data processing is performed in the edge layer and the fog layer in turn, and also includes ECG data processing in the edge layer, wherein the collected ECG data is corrected for baseline drift, a high-pass Butterworth filter is used to separate the baseline drift below 0.5 Hz, and then the QRS complex wave is enhanced, and a bandpass filter is used to specifically enhance the QRS wave energy, and suppress 20-500 Hz electromyographic noise and T wave low-frequency components less than 5 Hz.

6. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 5 is characterized in that: The sensor data processing is performed in the edge layer and the fog layer in turn, including the inertial navigation unit data processing in the fog layer, and the DCNN model is used for training and classification of static actions, wherein the acceleration data of the inertial navigation unit is processed to reduce the influence of gravity on the motion classification, which is expressed as: in, represents the transfer function, represents the complex frequency variable in the Laplace transform, Refers to the quality factor, Refers to the center angular frequency, is the filtered three-axis acceleration data, It is the acceleration data after removing the influence of gravity.

7. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 6 is characterized in that: The method sequentially processes sensor data in the edge layer and the fog layer, and also processes goniometer data in the fog layer, wherein a hybrid filtering architecture is used to reduce noise on the fused goniometer data, the time window length is dynamically adjusted, the low-frequency components of the respiratory signal are preferentially retained, and a Butterworth filter with a cutoff frequency of 10 Hz is used to eliminate high-frequency interference from muscle tremors. After noise reduction, the data is baseline corrected and normalized, morphological baseline extraction is used, and structural elements matching the respiratory cycle are used to perform opening and closing operations to separate slowly changing baseline drifts. Respiratory dynamics analysis is then performed to reveal the hidden dynamic characteristics of breathing through respiratory signals, and the maximum Lyapunov exponent is calculated to measure the sensitivity of the respiratory cycle to initial conditions, and a larger value indicates a more unstable rhythm.

8. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 7 is characterized in that: The sensor data processing is performed in the edge layer and the fog layer in turn, and also includes processing the ECG data in the fog layer, wherein the QRS peak group in the ECG data is detected, and the window length is automatically adjusted with the heart rate by introducing a dynamic integration window in the Pan-Tompkins algorithm to avoid waveform overlap under high heart rate; then, the ECG signal is subjected to time-frequency domain feature extraction after performing square operation and moving integration respectively, and the continuous RR interval sequence { }, the power spectral density is estimated by the Lomb-Scargle algorithm, and finally the respiratory-ECG coupling analysis is performed to establish a phase synchronization model between the respiratory signal D_norm (t) and the RR interval.

9. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 8, characterized in that: The sensor data modeling and comprehensive analysis are performed in the cloud layer, including inertial navigation unit data modeling in the cloud layer, wherein the inertial navigation unit data modeling includes modeling using a DCNN classifier, using 6 modules with increasing levels, the first 3 modules use a residual Inception structure, and the last 3 modules introduce dynamic deformable convolution to capture irregular motion features, the number of output channels of each module is gradually increased from 64 to 512, and an SE attention mechanism is added after the convolution to strengthen the key feature response, and a cross-modal interaction layer is inserted after the third and fifth modules, and the spatiotemporal correlation of acceleration and angular velocity is fused through a cross-attention mechanism.

10. The multimodal wearable sensor data real-time analysis method based on a layered architecture according to claim 9, characterized in that: The sensor data modeling and comprehensive analysis in the cloud layer also includes goniometer data modeling in the cloud layer, wherein a distributed computing framework is used to perform Fourier transform, the signal is converted from the time domain to the frequency domain, and a clustering algorithm is used to automatically identify the main frequency components in the frequency spectrum, so as to determine the respiratory frequency; the amplitude spectrum of the signal is calculated, and the intermediate results are stored in the cloud distributed storage system; the respiratory cycle, that is, the time interval between two adjacent breaths, is calculated by detecting the peak or valley value of the respiratory waveform; and the respiratory signal is decomposed into wavelet coefficients of different scales, the waveform characteristics of the signal are analyzed, and the data is comprehensively analyzed and stored for a long time.

Images