A multimodal analysis system for smart wearable cardiopulmonary exercise clothing

Abstract

The present invention discloses a multimodal analysis system for smart wearable cardiopulmonary exercise clothing. This system belongs to the technical field of smart wearables and includes a multimodal analysis algorithm, a multisensor integration system, and wearable clothing. The wearable clothing is equipped with an inertial measurement unit sensor, an electrocardiogram sensor, a rotation axis goniometer, and a data integration center module. The multimodal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling. The human behavior modeling is based on data processed by the inertial measurement unit sensor and uses a hierarchical network to identify the user's specific activities: dynamic and static classification is first performed through thresholds, and then specific activities are identified using pre-trained dynamic deep learning models and static deep learning models. At the same time, the system also adaptively updates the model based on the user's uniqueness. The present invention has the effect of improving the stability of multimodal modeling and achieving accurate monitoring of cardiopulmonary exercise function.

Description

Technical Field

[0001] The present invention relates to the technical field of smart wearables, and in particular to a multimodal analysis system for smart wearable cardiopulmonary exercise clothing. Background Art

[0002] At present, most daily health and sports wearable devices are in the form of single data category testing, such as smart bracelets, pressure detection shoes, etc., which lack comprehensive perception of various human physiological and motion data. The development of smart wearable clothing based on multimodal data fusion is one of the main solutions for this application.

[0003] Regarding the above-mentioned related technologies, the applicant found that the complexity of multi-sensor integration and multimodal data fusion modeling in cardiopulmonary exercise smart clothing technology is relatively high, which limits the reliability and accuracy of smart clothing in monitoring cardiopulmonary exercise, resulting in stability problems in the existing technology in multimodal modeling, and the inability to achieve accurate monitoring of cardiopulmonary exercise function. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a multimodal analysis system for cardiopulmonary exercise smart wearable clothing, which improves the stability of multimodal modeling and realizes accurate monitoring of cardiopulmonary exercise function.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A multimodal analysis system for smart wearable cardiopulmonary exercise clothing, including a multimodal analysis algorithm, a multi-sensor integration system, and wearable clothing. The wearable clothing is equipped with an inertial measurement unit sensor, an electrocardiogram sensor, a rotation axis goniometer, and a data integration center module.

[0007] The multimodal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling;

[0008] The human behavior modeling is based on data processed by the inertial measurement unit sensor and uses a layered network to identify the user's specific activities. It first uses threshold dynamic and static classification, and then uses pre-trained dynamic deep learning models and static deep learning models to identify specific activities. At the same time, the system will also adaptively update the model based on the user's uniqueness.

[0009] In order to efficiently divide the static and dynamic motion states, the data integration center module first calculates the Euclidean distance Q based on the acceleration data of the inertial measurement unit sensor. t for:

[0010] The three-axis components at time t, when Q t If the threshold is exceeded, it is considered as dynamic motion, otherwise it is considered as static motion;

[0011] The segmentation results directly determine the selection of subsequent classification models. Static actions use static deep learning models to identify activities in a static state, and dynamic movements use dynamic deep learning models to identify movements in a dynamic state.

[0012] Furthermore, in the dynamic and static activity recognition, the human behavior modeling uses the accelerometer and gyroscope data collected by the inertial measurement unit sensor, X = [x a ;x g ] are fed into two parallel convolutional streams to process features, and the convolution block is defined as:

[0013] ConvBlock(x)=σ(Conv1D(x,ω)+b)

[0014] ω is the SiLU activation function, ω and b are the convolution kernel weight and bias respectively. The convolution flow can extract local spatiotemporal features and generate feature representations for subsequent processing.

[0015] Furthermore, the feature y output by the convolutional flow a and y g Input to the sensor-level Transformer module to model the interaction between accelerometer and gyroscope features, the formula is:

[0016] Z=TransformerBlock(Concat[y a ,y g ])

[0017] The Transformer module mainly consists of a multi-head self-attention mechanism and a feedforward network, which is calculated as:

[0018] Z′ l =MSA(LN(Z l-1 ))+Z l-1

[0019] Z l =FFN(LN(Z′ l ))+Z′ l

[0020] LN is the layer normalization operation, l is the Transformer layer index, and the fused features are classified into different activity categories through the fully connected layer.

[0021] Furthermore, to enhance personalized recognition capabilities, the human behavior modeling module supports adaptive updates based on user characteristics. It collects user activity data during actual use and dynamically adjusts the human behavior modeling based on user-input annotation information. Specifically, it uses transfer learning technology to fine-tune static and dynamic deep learning models based on online collected data.

[0022] First, freeze the first few layers of feature extraction networks of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the last few layers of classification networks to adapt to the user's specific motion pattern.

[0023] Furthermore, in the training process, small batch incremental updates are adopted to update the user's historical data D user Based on, the loss function is defined as:

[0024]

[0025] f(x i ,θ) is the predicted value of the current static deep learning model or dynamic deep learning model, y i is the true label, l(f(x i ,θ),y i ) is the loss function;

[0026] Through the above mechanism, the system can continuously optimize the parameters of static deep learning models and dynamic deep learning models to improve personalized recognition accuracy.

[0027] Furthermore, the cardiopulmonary health analysis modeling is based on the processed ECG sensor data and axle goniometer data, using two-lead ECG sensor data to calculate ECG parameters during activity, including: heart rate and arrhythmia;

[0028] The axle goniometer fits the rise and fall of the chest to calculate breathing parameters, and covers the user's entire chest. It senses the rise and fall of the chest during natural breathing during daily activities and obtains angle changes. Through respiratory curve fitting and feature extraction, the respiratory curve is modeled and the breathing parameters during activity are calculated. The breathing parameters include respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

[0029] Furthermore, the method for fitting the axle goniometer data and extracting respiratory features is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is:

[0030]

[0031]

[0032] Filter the raw angle data collected by the axis goniometer to remove high-frequency noise and artifacts;

[0033] The core of the MAFF algorithm is to extract a matrix reflecting the characteristics of respiratory motion by calculating the minimum value of each pair of channels and further calculating the square root of the sum of the square differences between each column and its minimum value. The matrix can capture the phase differences and amplitude changes between different channels, thereby more accurately fitting the respiratory curve.

[0034] In order to improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By weighted fusion of multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weight of each channel;

[0035] Based on the fitted breathing curve, we extract the following key feature parameters:

[0036] Inspiratory time: The duration of the inspiratory phase is identified from the respiratory curve;

[0037] Expiratory time: the duration of the expiratory phase identified from the respiratory curve;

[0038] Respiratory time: the sum of inhalation time and exhalation time, that is, the time of a complete respiratory cycle;

[0039] Tidal volume: Estimates the amount of gas exchanged per breath by calculating the difference between the maximum and minimum tidal volumes during each respiratory cycle.

[0040] Furthermore, the inertial measurement unit sensors are arranged to cover the limbs and torso of the human body, thereby achieving comprehensive perception of the motion state;

[0041] The ECG sensor is placed close to the heart to obtain ECG signals and accurately reflect the electrical activity of the heart;

[0042] The axle goniometer senses the angle changes between the chest and abdomen, achieving multi-dimensional fitting of the synchronous rise and fall of the chest and abdomen during breathing, forming a complete respiratory motion model.

[0043] Furthermore, the multi-sensor integrated system uses a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensor, electrocardiogram sensor and shaft goniometer through the clock signal;

[0044] The clock signal is generated by the data integration center module and is expressed as clock signal f clk The data are distributed to the nodes of each inertial measurement unit sensor, ECG sensor and axis goniometer in the form of

[0045] f clk =1000Hz ensures the uniformity of data sampling time base, and realizes time synchronization and multimodal fusion of data.

[0046] Furthermore, the clock signal f clk Distributed in the nodes of each inertial measurement unit sensor, ECG sensor and shaft angle meter, the node receives f clk Then, a frequency division circuit is used to generate a clock signal that matches its sampling frequency;

[0047] For sensors of the same type, use the timestamp T in the clock signal clk As a benchmark, each sensor records the current timestamp when sampling to ensure data synchronization under the same time axis;

[0048] For different types of sensors, a segmented alignment mechanism is used, with the starting timestamp T of each second. align Alignment for benchmarks;

[0049] The data integration center module generates a global synchronization timestamp T every second. align =k·1S, different types of sensors align their sampling times to the closest T align , ensuring that the timestamp at the beginning of each second is consistent;

[0050] The data integration center module aggregates the collected data according to the timestamp and summarizes the time segments of different types of sensor data in T align To align the benchmarks, a multimodal fusion data stream is formed.

[0051] In summary, compared with the prior art, the above technical solution has the following beneficial effects:

[0052] (1) The present invention uses multiple inertial measurement unit sensors to sense the whole-body motion information of the human body, and combines multiple electrocardiogram sensors and axle goniometers to comprehensively obtain the cardiopulmonary status during activities. Through a multimodal analysis algorithm, the data of multiple sensors can be synchronized and transmitted efficiently and accurately, ensuring the precise monitoring and analysis of cardiopulmonary status and motion information. Users can choose offline or online recording methods according to their needs, conveniently and flexibly manage and analyze motion data, improve the stability of multimodal modeling, and realize accurate monitoring of cardiopulmonary exercise function.

[0053] (2) In terms of multi-sensor integration system, the hardware integration solution of the present invention realizes data acquisition, time synchronization and data transmission of multiple sensors through scientific design and advanced technology, ensuring the efficiency and accuracy of cardiopulmonary motion status monitoring. The inertial measurement unit sensor can obtain data from accelerometers, gyroscopes and magnetometers, providing rich information for comprehensive motion analysis. Its data sampling frequency is 60Hz, ensuring that subtle motion changes are captured. The ECG sensor can obtain ECG data with a sampling frequency of 500Hz. High-frequency sampling ensures the fine capture of ECG signals, especially during strenuous exercise, and can provide accurate cardiac electrical activity data. The axis goniometer can obtain multi-axis angle data with a sampling frequency of 100Hz, which is mainly used to monitor the angle changes of the chest cavity during breathing. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the overall structure of a wearable garment according to an embodiment of the present invention;

[0055] Figure 2 Flowchart of a multimodal analysis algorithm in an embodiment of the present invention.

[0056] Explanation of the accompanying symbols: 1. Inertial measurement unit sensor; 2. Electrocardiogram sensor; 3. Axle goniometer; 4. Data integration center module; 5. Data cable; 6. Inner layer of wearable clothing. DETAILED DESCRIPTION

[0057] The principles and features of the present invention are described below in conjunction with all the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0058] An embodiment of the present invention discloses a multimodal analysis system for smart wearable cardiopulmonary exercise clothing.

[0059] Reference Figure 1 and Figure 2 A multimodal analysis system for smart wearable clothing for cardiopulmonary exercise includes a multimodal analysis algorithm, a multi-sensor integration system and wearable clothing. The wearable clothing is equipped with an inertial measurement unit sensor 1, an electrocardiogram sensor 2, a rotation axis goniometer 3 and a data integration center module 4; the multimodal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling.

[0060] Human behavior modeling is based on data processed by the inertial measurement unit sensor 1 and uses a hierarchical network to identify the user's specific activities: first, dynamic and static classification is performed through thresholds, and then pre-trained dynamic deep learning models and static deep learning models are used to identify specific activities respectively. At the same time, the system will also adaptively update the model based on the user's uniqueness.

[0061] In order to efficiently divide the static and dynamic motion states, the data integration center module 4 first calculates the Euclidean distance Q based on the acceleration data of the inertial measurement unit sensor 1. t for:

[0062]

[0063] is the three-axis component of the accelerometer at time t, when Q t If the motion exceeds the set threshold, it is judged as dynamic motion; otherwise, it is judged as static motion.

[0064] The segmentation results directly determine the selection of subsequent classification models. Static actions use static deep learning models to identify activities in a static state, and dynamic movements use dynamic deep learning models to identify movements in a dynamic state.

[0065] In the dynamic and static activity recognition, the accelerometer and gyroscope data collected by the inertial measurement unit sensor 1 are X = [x a ;x g ] are fed into two parallel convolutional streams to process features, and the convolution block is defined as:

[0066] ConvBlock(x)=σ(Conv1D(x,ω)+b)

[0067] σ is the SiLU activation function, ω and b are the convolution kernel weight and bias respectively. The convolution flow can extract local spatiotemporal features and generate feature representations for subsequent processing.

[0068] The feature y output by the convolutional flow a and y g Input to the sensor-level Transformer module to model the interaction between accelerometer and gyroscope features, the formula is:

[0069] Z=TransformerBlock(Concat[y a ,y g ])

[0070] The Transformer module mainly consists of a multi-head self-attention mechanism and a feedforward network, which is calculated as:

[0071] Z′ l =MSA(LN(Z l-1 ))+Z l-1

[0072] Z l =FFN(LN(Z′ l ))+Z′ l

[0073] LN is the layer normalization operation, l is the Transformer layer index, and the fused features are classified into different activity categories through the fully connected layer.

[0074] To enhance personalized recognition capabilities, the data integration center module 4 supports adaptive updates based on user characteristics. It collects user activity data during actual use and dynamically adjusts human behavior modeling based on the annotation information input by the user. Specifically, based on the data collected online, it uses transfer learning technology to fine-tune the static deep learning model and the dynamic deep learning model.

[0075] First, freeze the first few layers of feature extraction networks of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the last few layers of classification networks to adapt to the user's specific motion pattern.

[0076] In the training process, small batch incremental updates are used to update the user's historical data D user Based on, the loss function is defined as:

[0077] f(x i ,θ) is the predicted value of the current static deep learning model or dynamic deep learning model, y i is the true label, l(f(x i ,θ),y i ) is the loss function;

[0078]

[0079] Through the above mechanism, the system can continuously optimize the parameters of static deep learning models and dynamic deep learning models to improve personalized recognition accuracy.

[0080] In order to verify the performance of the human body recognition modeling method in practical applications, the model was experimentally verified separately. The specific implementation steps are as follows:

[0081] Based on the setting of the clothing of the invention, 12 daily behaviors of 30 users were collected, including 6 static and 6 dynamic ones, namely: sitting, standing, lying on your back, lying down, lying on your left side, lying on your right side, walking, jogging, jumping up and down, squatting, going down stairs and going up stairs. In order to make the data generalizable, the ratio of males to females selected in the experiment was close to 1:1, with different heights, weights and ages, covering the age range of 20 to 45 years old. Each user repeated each behavior for 2 minutes. When collecting experimental data, there were no strict constraints on the standardization of the behavior, and it could be performed according to personal habits to ensure the diversity of the data set. Using this data set, training and testing were carried out on a PC with an Intel i8 core CPU and Intel 4090. In particular, in order to test the generalization ability of the algorithm, the test was trained on 20 random individuals from the 30 individuals in the data set, and the data of the remaining 10 individuals were used for testing. This was repeated 20 times, and the average accuracy, model training time and inference time / sample were calculated. The results are shown in Table 1:

[0082]

[0083]

[0084] Table 1 Performance comparison of different recognition algorithms

[0085] Table 1 shows that this method can achieve the best recognition accuracy and the highest generalization ability compared with other mainstream common models under the same experimental environment, while also requiring the shortest training time. Although the inference time of the proposed method is slightly higher than that of the CNN model, it is fully capable of meeting the real-time requirements in practice.

[0086] Cardiopulmonary fitness analysis modeling is based on processed ECG sensor 2 data and axle goniometer 3 data, using two-lead ECG sensor 2 data to calculate ECG parameters during activity, including heart rate and arrhythmia.

[0087] The axle goniometer 3 fits the rise and fall of the chest to calculate breathing parameters, and covers the user's entire chest. It senses the rise and fall of the chest during natural breathing during daily activities and obtains angle changes. Through respiratory curve fitting and feature extraction, a respiratory curve is modeled to calculate the respiratory parameters during activities. The respiratory parameters include respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

[0088] The method of fitting the data of the axis goniometer 3 and extracting the respiratory features is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is:

[0089]

[0090] The raw angle data collected by the shaft goniometer 3 is filtered to remove high-frequency noise and artifacts.

[0091] The core of the MAFF algorithm is to extract a matrix reflecting the characteristics of respiratory movement by calculating the minimum value of each pair of channels and further calculating the square root of the sum of the square differences between each column and its minimum value. The matrix can capture the phase differences and amplitude changes between different channels, thereby fitting the respiratory curve more accurately.

[0092] In order to improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By weighted fusion of multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weight of each channel.

[0093] Based on the fitted breathing curve, we extract the following key feature parameters:

[0094] Inspiratory time: The duration of the inspiratory phase identified from the respiratory curve.

[0095] Expiratory time: The duration of the expiratory phase identified from the respiratory curve.

[0096] Respiratory time: The sum of inhalation time and exhalation time, that is, the time of a complete breathing cycle.

[0097] Tidal volume: Estimates the amount of gas exchanged per breath by calculating the difference between the maximum and minimum tidal volumes during each respiratory cycle.

[0098] Table 2 below shows the key characteristic parameters and their corresponding data extracted after fitting the respiratory curve using the MAFF algorithm for four different breathing patterns. These breathing patterns include, but are not limited to, normal breathing, deep breathing, rapid breathing, and slow breathing. Each row of data corresponds to a breathing pattern and lists the specific values ​​of each characteristic parameter. The advantage of the MAFF algorithm is that it can more accurately quantify the key characteristics of the breathing pattern, avoiding bias caused by fitting errors, thereby improving the scientific nature and credibility of the research results.

[0099] Respiratory time (s) Inspiratory time (s) Exhalation time (s) Tidal volume (AU) Normal breathing 2.73±0.44 1.39±0.32 1.34±0.34 5.13±0.31 Take a deep breath 3.78±0.81 1.93±0.62 1.84±0.71 5.46±0.68 slow breathing 3.74±0.87 1.85±0.61 1.89±0.51 4.65±0.84 rapid breathing 2.24±0.32 1.03±0.36 1.21±0.44 3.57±0.26

[0100] Table 2 Feature parameters extracted based on fitting respiratory curve

[0101] Compared to other traditional respiratory curve analysis methods, the MAFF algorithm effectively reduces noise interference and sensitively captures subtle changes in the respiratory curve, thereby ensuring the accuracy of parameter extraction. This enables analysis based on these parameters to more reliably distinguish subtle differences between different respiratory forms, providing a solid foundation for subsequent in-depth research. This data can provide a deeper understanding of the physiological and biochemical differences between different respiratory forms, thus providing a scientific basis for the diagnosis and treatment of respiratory diseases.

[0102] The inertial measurement unit sensors 1 are arranged to cover the limbs and torso of the human body, achieving comprehensive perception of the motion state.

[0103] The ECG sensor 2 is placed close to the heart to acquire ECG signals and accurately reflect the electrical activity state of the heart.

[0104] The axle goniometer 3 senses the angle changes of the chest and abdomen, and realizes multi-dimensional fitting of the synchronous rise and fall of the chest and abdomen during breathing, forming a complete respiratory motion model.

[0105] The multi-sensor integrated system adopts a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensor 1, the electrocardiogram sensor 2 and the shaft goniometer 3 through the clock signal.

[0106] The clock signal is generated by the data integration center module 4 and is expressed as clock signal f clk The form is distributed to the nodes of each inertial measurement unit sensor 1, electrocardiogram sensor 2 and shaft angle meter 3,

[0107] f clk =1000Hz

[0108] Ensure the uniformity of data sampling time base to achieve time synchronization and multimodal fusion of data.

[0109] Clock signal f clk The nodes are distributed in each inertial measurement unit sensor 1, electrocardiogram sensor 2 and axis angle meter 3, and the nodes receive f clk Finally, a frequency division circuit is used to generate a clock signal that matches its sampling frequency.

[0110] For sensors of the same type, use the timestamp T in the clock signal clk As a benchmark, each sensor records the current timestamp when sampling to ensure data synchronization under the same time axis.

[0111] For different types of sensors, a segmented alignment mechanism is used, with the starting timestamp T of each second. align Align as a benchmark.

[0112] The data integration center module 4 generates a global synchronization timestamp T every second. align =k·1S, different types of sensors align their sampling times to the closest T align , ensuring that the timestamp at the beginning of each second is consistent.

[0113] The data integration center module 4 summarizes the collected data according to the timestamp, and divides the time segments of different types of sensor data into T align To align the benchmarks, a multimodal fusion data stream is formed.

[0114] Reference Figure 1 and Figure 2 The wearable clothing may also include smart wearable clothing and multiple sensors. The sensors are fixedly arranged in the interlayer of the clothing by sewing. The smart wearable clothing includes a top and shorts. The layer of the top that contacts the human body is the inner layer 6 of the wearable clothing, and the side of the wearable clothing away from the human body is the outer layer of the top; the layer of the shorts that contacts the human body is also the inner layer of the shorts, and the side of the wearable clothing away from the human body is the outer layer of the shorts.

[0115] The multiple sensors typically include five inertial measurement unit (IMU) sensors 1, two electrocardiogram (ECG) sensors 2, and three pairs of rotational goniometers 3. The five IMU sensors 1 are affixed to the lower left and right sleeves of the jacket, the middle of the abdomen, and the front of the left and right bottom edges of the shorts. The two ECG sensors 2 are affixed to the middle and left sides of the chest of the suit, and the rotational goniometers 3 are affixed to the inner layers of the suit's upper, mid-chest, and abdomen.

[0116] In this embodiment, the wearable clothing includes a top and shorts, both of which are made of sweat-absorbing and quick-drying tight-fitting fabrics, ensuring fit and comfort while also having excellent breathability. In terms of sensor integration, the hardware integration solution of the present invention realizes data acquisition, time synchronization and data transmission of multiple sensors through scientific design and advanced technology, ensuring the efficiency and accuracy of cardiopulmonary exercise status monitoring. The inertial measurement unit sensor 1 can obtain data from accelerometers, gyroscopes and magnetometers, providing rich information for comprehensive motion analysis. Its data sampling frequency is 60Hz, ensuring that subtle changes in movement are captured. The ECG sensor 2 can obtain ECG data with a sampling frequency of 500Hz. The high-frequency sampling ensures the fine capture of ECG signals, especially during strenuous exercise, and can provide accurate cardiac electrical activity data. The goniometer can obtain multi-axis angle data with a sampling frequency of 100Hz, and is mainly used to monitor the angle changes of the chest cavity during breathing.

[0117] An elastic belt is provided on one side of the inner layer 6 of the wearable clothing where the sensor is arranged. The sensor is fixed in the elastic cloth groove through the elastic belt and can be stably fixed on the body of users with different body shapes.

[0118] Multiple sensors are connected to the data integration center module 4 via data cables 5, which are routed within the garment's interlayer. The data integration center module 4 charges the sensors and receives, integrates, processes, and transmits their data. The silicone groove design is lightweight and does not interfere with the user's movements, while providing excellent protection.

[0119] The integrated multiple sensors solve the problem of time alignment and data synchronization acquisition of different types of sensors at the hardware level, including: time alignment and data synchronization between the five inertial measurement unit sensors 1, time alignment and data synchronization between the two electrocardiogram sensors 2, and time alignment and data synchronization between the three axle goniometers 3.

[0120] The multiple sensor integration system adopts high-precision clock synchronization circuit, and distributes the master clock signal to each sensor node through the data integration center module 4, ensuring that the data of multiple sensors are on the same time axis. The time alignment of multiple sensors is achieved by distributing a unified clock signal, realizing synchronous recording during acquisition.

[0121] The multi-sensor integrated system utilizes a high-precision clock synchronization circuit, distributing a master clock signal to each sensor node to ensure that each sensor's data is aligned on the same timeline. Time alignment of the five inertial measurement unit sensors (1) is achieved by distributing a unified clock signal, allowing data from multiple sensors to be recorded synchronously during acquisition. A similar approach is used to synchronize data from the two ECG sensors (2) and the three axle goniometers (3), ensuring that data from all sensors accurately matches timestamps during data transmission.

[0122] The multimodal analysis system preprocesses acquired sensor data and then implements multimodal modeling based on this processed data. This modeling integrates these data parameters to provide personalized analysis of the user's cardiorespiratory metrics across different activity states. This modeling includes both cardiorespiratory health analysis and human behavior modeling. Data preprocessing includes data segmentation and separate processing for different sensor types.

[0123] In this embodiment, each sensor's data is first segmented in seconds at different acquisition frequencies. The segmented data is then preprocessed separately based on the characteristics of each data type: the data from the inertial measurement unit sensor 1 is subjected to an extended Kalman filter, the ECG data is low-pass filtered, and the data from the axle goniometer 3 is baseline-removed and band-pass filtered.

[0124] The cardiopulmonary analysis model includes modeling and analysis of ECG parameters and axonometric goniometer 3 data.

[0125] In this embodiment, two-lead ECG data is used to calculate ECG parameters during activity, including heart rate and arrhythmia. A rotating axis goniometer 3 is used to fit the rise and fall of the chest to calculate respiratory parameters. The three rotating axis goniometers 3 cover the user's entire chest, sensing the rise and fall of the chest during breathing during activity, and obtaining accurate angle changes. Through data fitting and feature extraction, a respiratory curve is modeled to calculate respiratory parameters during activity, including respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow, and duty cycle.

[0126] The human action recognition model uses inertial sensing unit data to perform hierarchical human action recognition.

[0127] In this embodiment, the processed data of the inertial measurement unit sensor 1 is used to identify the specific activities of the user in a hierarchical manner: first, dynamic and static classification is performed based on the threshold to distinguish whether the current behavior is dynamic or static, and a further classification model is selected based on the result of the threshold classification. The second layer uses a pre-trained deep learning model to identify specific activities, which is divided into a dynamic deep learning model and a static deep learning model, which respectively identify dynamic activities and static activities to determine the specific activities of the user at this time. At the same time, the system will also adaptively update the model based on the uniqueness of the user to improve the effect. The adaptive update includes online re-learning of the model based on the differences between the user and the pre-trained human behavior model; and generating more data by generating a model based on the user's data to solve the problem of inaccurate classification caused by insufficient data when the user first uses it.

[0128] The present invention uses multiple inertial measurement units to sense the body's whole-body motion information, combined with multiple electrocardiogram sensors 2 and axle goniometers 3 to comprehensively acquire the cardiopulmonary status during activity. By solving the problem through hardware integration, data from multiple sensors can be efficiently and accurately synchronized and transmitted, ensuring precise monitoring and analysis of cardiopulmonary status and motion information. Users can choose offline or online recording methods according to their needs, allowing for convenient and flexible management and analysis of motion data.

[0129] The implementation principle of the multimodal analysis system for cardiopulmonary exercise smart wearable clothing in the embodiment of the present invention is as follows:

[0130] A multimodal analysis system for smart wearable cardiopulmonary exercise clothing is described. After the user puts on the clothing and starts the system, different sensors simultaneously collect data, align timestamps using a clock circuit, and transmit the data via wired transmission to a data integration center module 4. In the data integration center module 4, data segmentation and signal processing are performed, and the pre-processed data is then transmitted to a mobile device via a wireless gateway to implement multimodal data modeling and analysis.

[0131] Users can choose offline or online recording mode according to their needs, which allows for convenient and flexible management and analysis of sports data.

[0132] If recording is done offline (not connected to the mobile device), the collected data will be automatically saved in the Data Integration Center Module 4. The Data Integration Center Module 4 has built-in memory capable of storing a certain amount of data for subsequent analysis and review. This stored data can be exported via the USB port or synchronized to the device the next time it is connected to the mobile device.

[0133] When using the online recording method, the data of the data integration center module 4 is transmitted to the mobile device in real time through the form of a Bluetooth wireless gateway. The application on the mobile device receives and analyzes the data in real time and outputs the real-time analysis results. The data of the inertial measurement unit sensor 1 will be adaptively trained through a hierarchical human behavior recognition model for agile movement recognition and classification. These models are implemented by the adaptive hierarchical algorithm designed above, which can automatically identify different motion patterns, such as running, walking, jumping, etc., and count the duration and proportion of each motion, thereby providing a detailed motion analysis report. Since the sampling frequencies of different sensors are different, when a clock circuit is used to align timestamps, different types of data are integrated into a data packet in the form of seconds.

[0134] The data from the inertial measurement unit sensor 1 will be used to build a classifier and adaptive model update through the adaptive hierarchical algorithm model designed above, which will be used for user personalized motion recognition and classification. Specifically, the pre-processed motion data can be used to identify different motion patterns, such as running, walking, jumping, etc., through the above-mentioned trained classification model, and provide a detailed motion analysis report. At the same time, based on the individual differences of different users (including body shape and sports preferences, etc.) and the degree of matching with the recognition algorithm, the adaptive model update is performed to personalize the system to the user. The adaptive update includes online updating of the classifier based on the differences between the user and the pre-trained human behavior model; and based on the acquisition of new user data, solving the classification error problem caused by insufficient data when the user first uses it.

[0135] Data from ECG Sensor 2 is used to calculate ECG parameters, including heart rate and arrhythmia. Heart rate reflects exercise intensity and heart function. By monitoring changes in heart rate, the user's fitness level and training effectiveness can be assessed. Arrhythmia refers to abnormal heart rhythms, including tachycardia, bradycardia, or irregular heartbeats, which can affect effective blood circulation. During exercise, arrhythmia monitoring can help detect potential heart problems, prevent serious cardiac events, and ensure user safety.

[0136] The angle change data from the axle goniometer 3 is processed, feature extracted, and fitted to fit the chest respiratory curve. This data then calculates detailed respiratory parameters, including respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow, and duty cycle. These parameters can help assess the user's respiratory health, particularly during exercise, by monitoring respiratory changes in real time.

[0137] After the user completes their workout, the overall exercise data is uploaded to the cloud. The cloud server can store and analyze the data long-term, providing users with personalized health reports. For example, the user's overall heart rate changes during a single complete workout, whether there are any abnormalities, and recommendations for breathing methods during exercise based on heart rate and respiratory rate. For all individual exercises, the user's physiological parameters are analyzed over time, and guidance is provided.

[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal analysis system for smart cardiopulmonary exercise wearable clothing, characterized by: The invention comprises a multimodal analysis algorithm, a multi-sensor integration system and a wearable garment, wherein the wearable garment is provided with an inertial measurement unit sensor (1), an electrocardiogram sensor (2), a rotation axis goniometer (3) and a data integration center module (4); The multimodal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling; The human behavior modeling is based on the data processed by the inertial measurement unit sensor (1) and uses a hierarchical network to identify the specific activities of the user: first, dynamic and static classification is performed through thresholds, and then the specific activities are identified using a pre-trained dynamic deep learning model and a static deep learning model respectively. At the same time, the system will also adaptively update the model based on the uniqueness of the user; In order to efficiently divide the static and dynamic motion states, the data integration center module (4) first calculates the Euclidean distance based on the acceleration data of the inertial measurement unit sensor (1). for: , , is the three-axis component of the accelerometer at time 𝑡, when If the threshold is exceeded, it is considered as dynamic motion, otherwise it is considered as static motion; The segmentation results directly determine the selection of subsequent classification models. Static motion uses a static deep learning model to identify activities in a static state, and dynamic motion uses a dynamic deep learning model to identify movements in a dynamic state. The human behavior modeling in dynamic and static activity recognition, the accelerometer and gyroscope data collected by the inertial measurement unit sensor (1), Two parallel convolutional streams are fed into the feature processing, and the convolution block is defined as: is the SiLU activation function, and b are the convolution kernel weight and bias respectively. The convolution flow can extract local spatiotemporal features and generate feature representations for subsequent processing; The features output by the convolutional flow and Input to the sensor-level Transformer module to model the interaction between accelerometer and gyroscope features, the formula is: The Transformer module mainly consists of a multi-head self-attention mechanism and a feedforward network, which is calculated as: LN is the layer normalization operation, l is the Transformer layer index, and the fused features are classified into different activity categories through the fully connected layer; The human behavior modeling is to improve the personalized recognition capability. The data integration center module (4) supports adaptive updates based on user characteristics. It collects user activity data during actual use and dynamically adjusts the human behavior modeling based on the annotation information input by the user. Specifically, based on the data collected online, the static deep learning model and the dynamic deep learning model are fine-tuned using transfer learning technology; First, freeze the first few layers of the feature extraction network of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the classification network of the last few layers to adapt to the user's specific motion pattern; In the training process, small batch incremental updates are used to update the user's historical data. Based on, the loss function is defined as: is the predicted value of the current static deep learning model or dynamic deep learning model, is the true label, is the loss function.

2. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The cardiopulmonary health analysis modeling is based on processed ECG sensor (2) data and axle goniometer (3) data, and uses two-lead ECG sensor (2) data to calculate ECG parameters during activity, including: heart rate and arrhythmia; The axle goniometer (3) fits the rise and fall of the chest to calculate the breathing parameters, and covers the entire chest of the user, sensing the rise and fall of the chest during natural breathing in daily activities, obtaining the angle change, and modeling the breathing curve through breathing curve fitting and feature extraction to calculate the breathing parameters during activities. The breathing parameters include respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

3. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 2, characterized in that: The method for fitting the data of the axis goniometer (3) and extracting the respiratory features is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is: Filtering the raw angle data collected by the axis goniometer (3) to remove high-frequency noise and artifacts; The core of the MAFF algorithm is to extract a matrix reflecting the characteristics of respiratory motion by calculating the minimum value of each pair of channels and further calculating the square root of the sum of the square differences between each column and its minimum value. The matrix can capture the phase differences and amplitude changes between different channels, thereby more accurately fitting the respiratory curve. In order to improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By weighted fusion of multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weight of each channel; Based on the fitted breathing curve, we extract the following key feature parameters: Inspiratory time: The duration of the inspiratory phase is identified from the respiratory curve; Expiratory time: the duration of the expiratory phase identified from the respiratory curve; Respiratory time: the sum of inhalation time and exhalation time, that is, the time of a complete respiratory cycle; Tidal volume: Estimates the amount of gas exchanged per breath by calculating the difference between the maximum and minimum tidal volumes during each respiratory cycle.

4. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The inertial measurement unit sensors (1) are arranged to cover the limbs and trunk of the human body, thereby achieving comprehensive perception of the motion state; The ECG sensor (2) is placed close to the heart to obtain ECG signals and accurately reflect the electrical activity of the heart; The axis goniometer (3) senses the angle changes of the chest and abdomen, and realizes multi-dimensional fitting of the synchronous rise and fall of the chest and abdomen during breathing, forming a complete respiratory motion model.

5. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The multi-sensor integrated system adopts a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensor (1), the electrocardiogram sensor (2) and the axis goniometer (3) through the clock signal; The clock signal is generated by the data integration center module (4) and is The nodes of each inertial measurement unit sensor (1), electrocardiogram sensor (2) and axle goniometer (3) are distributed in the form of Ensure the uniformity of data sampling time base to achieve time synchronization and multimodal fusion of data.

6. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 5, characterized in that: The clock signal The nodes are distributed in each inertial measurement unit sensor (1), electrocardiogram sensor (2) and axis goniometer (3), and the nodes receive Then, a frequency division circuit is used to generate a clock signal that matches its sampling frequency; For sensors of the same type, use the timestamp in the clock signal As a benchmark, each sensor records the current timestamp when sampling to ensure data synchronization under the same time axis; For different types of sensors, a segment alignment mechanism is used, with the start timestamp of each second Alignment for benchmarks; The Data Integration Center module (4) generates a global synchronization timestamp every second S, different types of sensors align their sampling times to the closest , ensuring that the timestamp at the beginning of each second is consistent; The data integration center module (4) aggregates the collected data according to the timestamp, and sorts the time segments of different types of sensor data into To align the benchmarks, a multimodal fusion data stream is formed.

Images