Multi-modal analysis system for intelligent wearing clothes for cardiopulmonary exercise

Abstract

The invention discloses a multi-modal analysis system of intelligent wearing clothes for cardiopulmonary exercise, and belongs to the technical field of intelligent wearing, the multi-modal analysis system comprises a multi-modal analysis algorithm, a multi-sensor integration system and wearing clothes, and the wearing clothes are provided with an inertial measurement unit sensor, an electrocardio sensor, a rotating shaft goniometer and a data integration center module; the multi-modal analysis algorithm comprises human body behavior modeling and cardiopulmonary health analysis modeling; the human body behavior modeling is based on data processed by an inertial measurement unit sensor, and specific activities of a user are identified by using a hierarchical network: firstly, threshold dynamic and static classification is performed, and then specific activities are identified by respectively using a pre-trained dynamic deep learning model and a pre-trained static deep learning model; meanwhile, the system can adaptively update the model based on the uniqueness of the user. The method has the effects of improving the stability in the aspect of multi-modal modeling and achieving accurate monitoring of the cardiopulmonary movement function.

Description

Technical Field

[0001] The present invention relates to the technical field of smart wearables, and particularly to a multimodal analysis system for a cardiorespiratory exercise smart wearable garment. Background Art

[0002] Currently, most daily health exercise wearable devices are in the form of relatively single data category tests, such as smart bracelets, pressure detection shoes, etc., lacking comprehensive perception of various human physiological and exercise data. Developing smart wearable garments based on multimodal data fusion is one of the main solutions for this application.

[0003] In view of the above related technologies, the applicant has found that the complexity of multi-sensor integration and multimodal data fusion modeling in cardiorespiratory exercise smart garment technology is relatively high, thus restricting the reliability and accuracy of smart garments in monitoring cardiorespiratory exercise, resulting in stability problems in the existing technologies in multimodal modeling and being unable to achieve precise monitoring of cardiorespiratory exercise functions. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technologies, the present invention provides a multimodal analysis system for a cardiorespiratory exercise smart wearable garment, improving the stability in multimodal modeling and achieving precise monitoring of cardiorespiratory exercise functions.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A multimodal analysis system for a cardiorespiratory exercise smart wearable garment includes a multimodal analysis algorithm, a multi-sensor integration system, and a wearable garment. The wearable garment is provided with an inertial measurement unit sensor, an electrocardiogram sensor, a rotation angle measuring instrument, and a data integration center module;

[0007] The multimodal analysis algorithm includes human behavior modeling and cardiorespiratory health analysis modeling;

[0008] The human behavior modeling is based on the data processed by the inertial measurement unit sensor, and uses a hierarchical network to identify the specific activities of the user: first, through threshold dynamic and static classification, and then respectively use a pre-trained dynamic deep learning model and a static deep learning model to identify the specific activities. At the same time, the system will also adaptively update the model based on the uniqueness of the user;

[0009] In order to efficiently divide the static and dynamic motion states, the data integration center module first calculates the Euclidean distance Q based on the acceleration data of the inertial measurement unit sensor t as:

[0010] are the three-axis components of the acceleration at time, when Q t exceeds the set threshold, it is determined as dynamic motion, otherwise it is judged as static motion;

[0011] The partitioning result directly determines the selection of the subsequent classification model. Static actions use a static deep learning model to identify activities in a stationary state, and dynamic movements use a dynamic deep learning model to identify movements in a dynamic state.

[0012] Furthermore, in the human behavior modeling for dynamic and static activity recognition, the accelerometer and gyroscope data collected by the inertial measurement unit sensor, X = [x a ; x g are respectively fed into two parallel convolutional streams to process features. The convolutional block is defined as:

[0013] ConvBlock(x) = σ(Conv1D(x, ω) + b)

[0014] ω is the SiLU activation function, ω and b are the convolutional kernel weights and biases respectively. The convolutional streams can extract local spatio-temporal features and generate feature representations for subsequent processing.

[0015] Furthermore, the features y a and y g output by the convolutional streams are input into the Transformer module at the sensor level to model the interaction between accelerometer and gyroscope features. The formula is:

[0016] Z = TransformerBlock(Concat[y a , y g )

[0017] The Transformer module mainly consists of a multi-head self-attention mechanism and a feed-forward network. The calculation is:

[0018] Z′ l = MSA(LN(Z l-1 )) + Z l-1

[0019] Z l = FFN(LN(Z′ l )) + Z′ l

[0020] LN is the layer normalization operation, l is the Transformer layer index. The fused features are classified into different activity categories through a fully connected layer.

[0021] Further, the human behavior modeling aims to improve the personalized recognition ability. The data integration center module supports adaptive updates based on user characteristics. During actual use, it collects the user's activity data and dynamically adjusts the human behavior modeling in combination with the annotation information input by the user. Specifically, based on the online collected data, transfer learning technology is used to fine-tune the static deep learning model and the dynamic deep learning model.

[0022] First, freeze the first few layers of the feature extraction network of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the last few layers of the classification network to adapt to the user's specific motion pattern.

[0023] Further, in the training process, mini-batch incremental updates are adopted. Based on the user's historical data D user as the basis, the loss function is defined as:

[0024]

[0025] f(x i , θ) is the predicted value of the current static deep learning model or dynamic deep learning model, y i is the true label, and l(f(x i , θ), y i ) is the loss function;

[0026] Through the above mechanism, the system can continuously optimize the parameters of the static deep learning model and the dynamic deep learning model, and improve the personalized recognition accuracy.

[0027] Further, the cardiopulmonary health analysis modeling is based on the processed electrocardiogram sensor data and goniometer data. The electrocardiogram parameters during activities are calculated using the two-lead electrocardiogram sensor data, including: heart rate and arrhythmia;

[0028] The goniometer fits the chest undulation to calculate the respiratory parameters, covers the entire chest of the user, senses the chest undulation during natural breathing in daily activities, obtains the angle change, and through respiratory curve fitting and feature extraction, models the respiratory curve and calculates the respiratory parameters during activities. The respiratory parameters include respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation volume, average inspiratory flow rate, average expiratory flow rate, and duty cycle.

[0029] Further, the method for goniometer data fitting and respiratory feature extraction is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is:

[0030]

[0031]

[0032] Filter the original angle data collected by the rotary shaft goniometer to remove high-frequency noise and artifacts;

[0033] The core of the MAFF algorithm lies in calculating the minimum value of each pair of channels, and further calculating the square root of the sum of the squared differences between each column and its minimum value to extract a matrix that reflects the characteristics of respiratory movement. The matrix can capture the phase differences and amplitude changes between different channels, thus more accurately fitting the respiratory curve;

[0034] To improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By weighted fusion of multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weights of each channel;

[0035] Based on the fitted respiratory curve, we extract the following key characteristic parameters:

[0036] Inspiratory time: Identify the duration of the inspiratory phase from the respiratory curve;

[0037] Expiratory time: Identify the duration of the expiratory phase from the respiratory curve;

[0038] Respiratory time: The sum of the inspiratory time and the expiratory time, that is, the time of a complete respiratory cycle;

[0039] Tidal volume: Estimate the gas exchange volume of each breath by calculating the maximum and minimum differences within each respiratory cycle.

[0040] Furthermore, the inertial measurement unit sensors are arranged to cover the limbs and torso of the human body to achieve a comprehensive perception of the motion state;

[0041] The electrocardiogram sensor is arranged close to the heart to obtain the electrocardiogram signal, accurately reflecting the electrical activity state of the heart;

[0042] The rotary shaft goniometer senses the angular changes of the chest and abdomen, realizes the synchronous undulation of the chest and abdomen during multi-dimensional fitting of breathing, and forms a complete respiratory motion model.

[0043] Furthermore, the multi-sensor integration system adopts a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensors, electrocardiogram sensors and rotary shaft goniometers through the clock signal;

[0044] The clock signal is generated by the data integration center module and is distributed to the nodes of each inertial measurement unit sensor, electrocardiogram sensor and rotary shaft goniometer in the form of clock signal f clk so that,

[0045] f clk = 1000Hz ensures the unity of the data sampling time reference, realizes time synchronization and multi-modal fusion of data.

[0046] Further, the clock signal f clk is distributed at the nodes of each inertial measurement unit sensor, electrocardiogram sensor and rotary shaft angle measuring instrument, and the nodes receive f clk and then use a frequency division circuit to generate a clock signal matching its sampling frequency;

[0047] For sensors of the same type, the timestamp T clk in the clock signal is used as a reference. When each sensor samples, the current timestamp is recorded to ensure data synchronization under the unified time axis;

[0048] For sensors of different types, a segmented alignment mechanism is adopted, with the starting timestamp T align per second as the reference for alignment;

[0049] The data integration center module generates a global synchronization timestamp T align = k·1S per second. Different types of sensors align their respective sampling times to the closest T align to ensure that the timestamps at the beginning of each second are consistent;

[0050] The data integration center module summarizes the collected data according to the timestamps, aligns the time segments of data from different types of sensors with T align as the reference to form a multi-modal fusion data stream.

[0051] In summary, compared with the prior art, the beneficial effects of the above technical solution are:

[0052] (1) The present invention uses multiple inertial measurement unit sensors to sense the whole-body movement information of the human body, combines multiple electrocardiogram sensors and rotary shaft angle measuring instruments to comprehensively obtain the cardiopulmonary condition during activities, and through a multi-modal analysis algorithm, can efficiently and accurately synchronize and transmit the data of multiple sensors, ensuring the precise monitoring and analysis of the cardiopulmonary state and movement information. Users can choose offline or online recording methods according to their own needs, conveniently and flexibly manage and analyze the movement data, improve the stability in multi-modal modeling, and at the same time achieve the precise monitoring of the cardiopulmonary movement function.

[0053] (2) In terms of multi-sensor integration systems, the hardware integration solution of the present invention realizes data acquisition, time synchronization, and data transmission of multiple sensors through scientific design and advanced technology, ensuring the efficiency and accuracy of cardiopulmonary motion state monitoring. The inertial measurement unit sensor can obtain data of acceleration, gyroscope, and magnetometer, providing rich information for comprehensive motion analysis. Its data sampling frequency is 60 Hz, ensuring the capture of subtle motion changes. The electrocardiogram sensor can obtain electrocardiogram data with a sampling frequency of 500 Hz. The high-frequency sampling ensures the fine capture of electrocardiogram signals, especially during strenuous exercise, and can provide accurate cardiac electrical activity data. The shaft angle meter can obtain multi-axis angle data with a sampling frequency of 100 Hz, mainly used to monitor the angle changes of the chest cavity during breathing. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a schematic diagram of the overall structure of the wearable clothing in the embodiment of the present invention;

[0055] Figure 2 is a flowchart of the multi-modal analysis algorithm in the embodiment of the present invention.

[0056] Description of reference numerals: 1, inertial measurement unit sensor; 2, electrocardiogram sensor; 3, shaft angle meter; 4, data integration center module; 5, data line; 6, inner layer of wearable clothing. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The principles and features of the present invention will be described below in conjunction with all the drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0058] The embodiment of the present invention discloses a multi-modal analysis system for a cardiopulmonary motion intelligent wearable clothing.

[0059] Referring to Figure 1 and Figure 2 , a multi-modal analysis system for a cardiopulmonary motion intelligent wearable clothing includes a multi-modal analysis algorithm, a multi-sensor integration system, and a wearable clothing. The wearable clothing is provided with an inertial measurement unit sensor 1, an electrocardiogram sensor 2, a shaft angle meter 3, and a data integration center module 4; the multi-modal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling.

[0060] The human behavior modeling is based on the data processed by the inertial measurement unit sensor 1, and uses a hierarchical network to identify the specific activities of the user: first, through threshold dynamic and static classification, and then respectively use pre-trained dynamic and static deep learning models to identify specific activities. At the same time, the system will also adaptively update the model based on the uniqueness of the user.

[0061] To efficiently divide static and dynamic motion states, the data integration center module 4 first calculates the Euclidean distance Q based on the acceleration data of the inertial measurement unit sensor 1 t as follows:

[0062]

[0063] are the three-axis components of the accelerometer at time t. When Q t exceeds the set threshold, it is determined to be dynamic motion; otherwise, it is judged as static motion.

[0064] The division result directly determines the selection of the subsequent classification model. The static action uses a static deep learning model to identify activities in the static state, and the dynamic motion uses a dynamic deep learning model to identify motions in the dynamic state.

[0065] In human behavior modeling for dynamic and static activity recognition, the acceleration and gyroscope data collected by the inertial measurement unit sensor 1, X = [x a ; x g are respectively fed into two parallel convolutional streams to process features. The convolutional block is defined as:

[0066] ConvBlock(x) = σ(Conv1D(x, ω) + b)

[0067] σ is the SiLU activation function, ω and b are the convolutional kernel weights and biases respectively. The convolutional stream can extract local spatio-temporal features and generate feature representations for subsequent processing.

[0068] The features y a and y g output by the convolutional stream are input into the Transformer module at the sensor level to model the interaction between the accelerometer and gyroscope features. The formula is:

[0069] Z = TransformerBlock(Concat[y a , y g )

[0070] The Transformer module is mainly composed of a multi-head self-attention mechanism and a feed-forward network. The calculation is:

[0071] Z′ l = MSA(LN(Z l-1 )) + Z l-1

[0072] Z l = FFN(LN(Z′ l )) + Z′ l

[0073] LN is the layer normalization operation, l is the Transformer layer index, and the fused features are classified into different activity categories through a fully connected layer.

[0074] Human behavior modeling aims to improve personalized recognition ability. The data integration center module 4 supports adaptive updates based on user characteristics. During actual use, it collects the user's activity data and dynamically adjusts the human behavior modeling in combination with the annotation information input by the user. Specifically, based on the online collected data, transfer learning technology is used to fine-tune the static deep learning model and the dynamic deep learning model.

[0075] First, freeze the first few layers of the feature extraction network of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the last few layers of the classification network to adapt to the user's specific motion pattern.

[0076] During the training process, mini-batch incremental updates are adopted, with the user's historical data D user as the basis, and the loss function is defined as:

[0077] f(x i , θ) is the predicted value of the current static deep learning model or dynamic deep learning model, y i is the true label, and l(f(x i , θ), y i ) is the loss function;

[0078]

[0079] Through the above mechanism, the system can continuously optimize the parameters of the static deep learning model and the dynamic deep learning model, and improve the personalized recognition accuracy.

[0080] To verify the performance of this human recognition modeling method in actual applications, the model is separately experimentally verified. The specific implementation steps are as follows:

[0081] Based on the setting of the invention clothing, 12 daily behaviors of 30 users were collected, including 6 static and 6 dynamic ones, namely: sitting, standing, lying on your back, lying on your back, lying on your left side, lying on your right side, walking, jogging, jumping up and down, squatting, going down stairs and going up stairs. In order to make the data generalizable, the ratio of men and women selected in the experiment is close to 1:1, with different heights, weights and ages, and the age range covers 20 to 45 years old. Each user repeats each behavior for 2 minutes. When collecting experimental data, there is no strict constraint on the normativeness of the behavior, and it can be performed according to personal habits to ensure the diversity of the data set. Using this data set, training and testing are carried out on an Inteli8 core CPU and a PC equipped with Intel's 4090. In particular, in order to test the generalization ability of the algorithm, the test is trained with 20 random individuals out of the 30 individuals in the data set, and the data of the remaining 10 individuals are used for testing. Repeat 20 times, calculate the average accuracy, model training time and inference time / sample, and the results are shown in Table 1:

[0082]

[0083]

[0084] Table 1 Performance comparison of different recognition algorithms

[0085] From Table 1, we can see that this method can achieve the best recognition accuracy and the highest generalization ability compared with other mainstream common models under the same experimental environment, and the required training time is also the shortest. Although the reasoning time of the proposed method is slightly higher than that of the CNN model, it can fully meet the real-time requirements in practice.

[0086] Cardiopulmonary fitness analysis modeling is based on processed ECG sensor 2 data and axle goniometer 3 data, using two-lead ECG sensor 2 data to calculate ECG parameters during activity, including heart rate and arrhythmia.

[0087] The axle goniometer 3 fits the rise and fall of the chest to calculate the breathing parameters, and covers the entire chest of the user, senses the rise and fall of the chest during natural breathing in daily activities, obtains the angle change, and obtains the breathing curve through breathing curve fitting and feature extraction, and calculates the breathing parameters during activities. The breathing parameters include breathing rate, inhalation time, exhalation time, breathing time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

[0088] The method of fitting the data of the rotating shaft goniometer 3 and extracting the breathing characteristics is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is:

[0089]

[0090] Filter the original angle data collected by the rotary shaft goniometer 3 to remove high-frequency noise and artifacts.

[0091] The core of the MAFF algorithm lies in calculating the minimum value of each pair of channels, and further calculating the square root of the sum of the squared differences between each column and its minimum value, extracting a matrix that reflects the characteristics of respiratory motion. The matrix can capture the phase differences and amplitude changes between different channels, thus more accurately fitting the respiratory curve.

[0092] To improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By performing weighted fusion on multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weights of each channel.

[0093] Based on the fitted respiratory curve, we extract the following key characteristic parameters:

[0094] Inspiratory time: Identify the duration of the inspiratory phase from the respiratory curve.

[0095] Expiratory time: Identify the duration of the expiratory phase from the respiratory curve.

[0096] Respiratory time: The sum of the inspiratory time and the expiratory time, that is, the time of a complete respiratory cycle.

[0097] Tidal volume: Estimate the gas exchange volume of each breath by calculating the maximum and minimum differences within each respiratory cycle.

[0098] As shown in Table 2 below, under four different respiratory patterns, the key characteristic parameters extracted after fitting the respiratory curve according to the MAFF algorithm and their corresponding data are presented. These respiratory patterns include, but are not limited to, normal breathing, deep breathing, rapid breathing, and slow breathing. Each row of data corresponds to a respiratory pattern, and the specific values of each characteristic parameter are listed in detail. The advantage of the MAFF algorithm is that it can more accurately quantify the key characteristics of the respiratory pattern, avoiding the deviation caused by fitting errors, thereby enhancing the scientificity and credibility of the research results.

[0099] Respiration time (s) Inspiration time (s) Expiration time (s) Tidal volume (A.U.) Normal breathing 2.73±0.44 1.39±0.32 1.34±0.34 5.13±0.31 Deep breathing 3.78±0.81 1.93±0.62 1.84±0.71 5.46±0.68 Slow breathing 3.74±0.87 1.85±0.61 1.89±0.51 4.65±0.84 Rapid breathing 2.24±0.32 1.03±0.36 1.21±0.44 3.57±0.26

[0100] Table 2 Characteristic Parameter Table Extracted from the Fitted Respiratory Curve

[0101] Compared with other traditional respiratory curve analysis methods, the MAFF algorithm can effectively reduce noise interference and sensitively capture subtle changes in the respiratory curve, thus ensuring the accuracy of parameter extraction. This enables more reliable differentiation of the subtle differences between different respiratory patterns based on these parameters, providing a solid foundation for subsequent in-depth research. Through these data, the differences in different respiratory patterns at the physiological and biochemical levels can be deeply understood, providing a scientific basis for the diagnosis and treatment of respiratory-related diseases.

[0102] The arrangement of the inertial measurement unit sensor 1 covers the limbs and torso of the human body, achieving comprehensive perception of the motion state.

[0103] The arrangement of the electrocardiogram sensor 2 close to the heart obtains electrocardiogram signals, accurately reflecting the electrical activity state of the heart.

[0104] The rotary shaft goniometer 3 senses the angular changes of the chest and abdomen, realizes the synchronous undulation of the chest and abdomen during breathing through multi-dimensional fitting, and forms a complete respiratory motion model.

[0105] The multi-sensor integration system adopts a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensor 1, the electrocardiogram sensor 2, and the rotary shaft goniometer 3 through the clock signal.

[0106] The clock signal is generated by the data integration center module 4 and is distributed to the nodes of each inertial measurement unit sensor 1, electrocardiogram sensor 2, and rotary shaft goniometer 3 in the form of clock signal f clk The nodes of each inertial measurement unit sensor 1, electrocardiogram sensor 2, and rotary shaft goniometer 3,

[0107] f clk = 1000Hz

[0108] Ensure the unity of the data sampling time reference, and achieve time synchronization and multi-modal fusion of data.

[0109] The clock signal f clk is distributed at the nodes of each inertial measurement unit sensor 1, electrocardiogram sensor 2, and rotary shaft goniometer 3. After receiving f clk , the nodes use a frequency division circuit to generate a clock signal that matches their sampling frequency.

[0110] For sensors of the same type, the time stamp T clk in the clock signal is used as a reference. Each sensor records the current time stamp during sampling to ensure data synchronization under the unified time axis.

[0111] For sensors of different types, a segmented alignment mechanism is adopted, and alignment is performed based on the starting time stamp T align per second as a reference.

[0112] The data integration center module 4 generates a global synchronization timestamp T once every second. align = k·1S, and different types of sensors align their respective sampling times to the closest T align , ensuring that the timestamps at the beginning of each second are consistent.

[0113] The data integration center module 4 summarizes the collected data according to the timestamps, and aligns the time segments of data from different types of sensors with T align as the benchmark to form a multi-modal fusion data stream.

[0114] Refer to Figure 1 and Figure 2 , the wearable clothing can also include a smart wearable garment and multiple sensors. The sensors are fixedly arranged in the interlayer of the clothing in the form of thread sewing. The smart wearable garment includes a top and shorts. The layer of the top in contact with the human body is the inner layer 6 of the wearable garment, and the side of the wearable garment away from the human body is the outer layer of the top; the layer of the shorts in contact with the human body is also the inner layer of the shorts, and the side of the wearable garment away from the human body is the outer layer of the shorts.

[0115] The multiple sensors usually include five inertial measurement unit sensors 1IMU, two electrocardiogram sensors 2ECG, and three pairs of rotary shaft angle sensors 3. The five inertial measurement unit sensors 1 are fixed at the lower edge of the left sleeve of the top, the lower edge of the right sleeve, the middle of the abdomen, and the front sides of the lower left and right trouser edges of the shorts. The two electrocardiogram sensors 2 are fixedly arranged in the middle of the chest and the left side of the chest of the wearable garment. The angle sensors 3 are fixedly arranged in the inner interlayers of the upper chest, middle chest, and abdomen of the wearable garment.

[0116] In this embodiment, the wearable garment includes a top and shorts, both of which are made of sweat-absorbing and quick-drying tight-fitting fabrics, ensuring a good fit and wearing comfort, and at the same time having excellent breathability. In terms of sensor integration, the hardware integration solution of the present invention realizes the data acquisition, time synchronization, and data transmission of multiple sensors through scientific design and advanced technology, ensuring the efficiency and accuracy of cardiopulmonary motion state monitoring. The inertial measurement unit sensor 1 can obtain data of acceleration, gyroscope, and magnetometer, providing rich information for comprehensive motion analysis. Its data sampling frequency is 60Hz, ensuring that subtle motion changes are captured. The electrocardiogram sensor 2 can obtain electrocardiogram data, with a sampling frequency of 500Hz. The high-frequency sampling ensures the fine capture of electrocardiogram signals. Especially during strenuous exercise, it can provide accurate cardiac electrical activity data. The angle sensor can obtain multi-axis angle data, with a sampling frequency of 100Hz, mainly used to monitor the angle changes of the chest cavity during breathing.

[0117] On the side of the inner layer 6 of the wearable garment where the sensors are arranged, there is an elastic band. The sensors are fixed in the elastic cloth groove through the elastic band, and can be stably fixed on the bodies of users with different body shapes and postures.

[0118] Multiple sensors are connected to the data integration center module 4 in a wired connection form through the data line 5, and the transmission line is arranged inside the clothing interlayer. The data integration center module 4 completes the charging of multiple sensors and the reception, integration, processing, and transmission of multi-sensor data. The silicone groove design is lightweight and does not affect the user's movement, while providing good protection.

[0119] The multiple integrated sensors solve the problems of time alignment and data synchronous acquisition of different types of sensors at the hardware level, including: time alignment and data synchronization among the five inertial measurement unit sensors 1, time alignment and data synchronization between the two electrocardiogram sensors 2, and time alignment and data synchronization among the three rotary shaft angle gauges 3.

[0120] The multiple sensor integration system adopts a high-precision clock synchronization circuit. The master clock signal is distributed to each sensor node through the data integration center module 4 to ensure that the data of multiple sensors are on the same time axis. The time alignment of multiple sensors is achieved by distributing a unified clock signal, realizing synchronous recording during acquisition.

[0121] The multi-sensor integration system adopts a high-precision clock synchronization circuit. The master clock signal is distributed to each sensor node to ensure that the data of each sensor are on the same time axis. The time alignment of the five inertial measurement unit sensors 1 is achieved by distributing a unified clock signal, and the data of multiple sensors are synchronously recorded during acquisition. The data synchronization of the two electrocardiogram sensors 2 and the three rotary shaft angle gauges 3 also adopts a similar method to ensure that all sensor data can accurately match the time stamps during data transmission.

[0122] The multi-modal analysis system includes preprocessing the acquired sensor data, and then realizing multi-modal modeling based on the processed data. The modeling is achieved by fusing these data parameters and is ultimately used for personalized analysis of the cardiopulmonary indicators of the user in different activity states. The modeling includes cardiopulmonary health analysis modeling and human behavior modeling; data preprocessing includes data segmentation and separate processing of different types of sensors.

[0123] In this embodiment, each sensor segments the sensor data in seconds at different acquisition frequencies. For the segmented data, preprocessing is performed separately according to the characteristics of different types of data: the data of the inertial measurement unit sensor 1 is subjected to extended Kalman filtering, the electrocardiogram data is subjected to low-pass filtering, and the data of the rotary shaft angle gauge 3 is subjected to baseline removal and band-pass filtering;

[0124] The cardiopulmonary analysis model includes modeling and analysis of electrocardiogram parameters and the data of the rotary shaft angle gauge 3.

[0125] In this embodiment, two-lead ECG data are used to calculate ECG parameters during activities, including: heart rate and arrhythmia; the rotating axis goniometer 3 is used to fit the rise and fall of the chest to calculate the breathing parameters. The three rotating axis goniometers 3 cover the entire chest of the user, sense the rise and fall of the chest during breathing during activities, and obtain accurate angle changes. Through data fitting and feature extraction, the breathing curve is modeled to calculate the breathing parameters during activities, including: breathing rate, inhalation time, exhalation time, breathing time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

[0126] The human action recognition model uses inertial sensor unit data to perform hierarchical human action recognition.

[0127] In this embodiment, the processed data of the inertial measurement unit sensor 1 is used to identify the specific activities of the user in a hierarchical form: first, dynamic and static classification is performed based on the threshold to distinguish whether the current behavior is dynamic or static, and a further classification model is selected based on the result of the threshold classification. The second layer uses a pre-trained deep learning model to identify specific activities, which is divided into a dynamic deep learning model and a static deep learning model, which respectively identify dynamic activities and static activities to determine the specific activities of the user at this time. At the same time, the system will also adaptively update the model based on the uniqueness of the user to improve the effect. The adaptive update includes online relearning of the model based on the differences between the user and the pre-trained human behavior model; and generating more data by generating a model based on the user's data to solve the problem of inaccurate classification caused by insufficient data when the user first uses it.

[0128] The present invention uses multiple inertial measurement units to sense the whole body motion information of the human body, and combines multiple electrocardiograph sensors 2 and axle goniometers 3 to comprehensively obtain the cardiopulmonary status during activities; the problem is solved through hardware integration, and the data of multiple sensors can be synchronized and transmitted efficiently and accurately, ensuring the precise monitoring and analysis of cardiopulmonary status and motion information; users can choose offline or online recording methods according to their needs, so as to conveniently and flexibly manage and analyze motion data.

[0129] The implementation principle of a multimodal analysis system for cardiopulmonary exercise smart wearable clothing in an embodiment of the present invention is as follows:

[0130] A multimodal analysis system for smart wearable cardiopulmonary exercise clothing. After the user puts on the clothing, the system is started, and different sensors simultaneously collect data, complete the time stamp alignment in the form of a clock circuit, and transmit it to the data integration center module 4 through wired transmission. In the data integration center module 4, the data is first segmented and signal processed, and then the pre-processed data is transmitted to the mobile device as wireless gateway data to achieve multimodal data modeling and analysis.

[0131] Users can choose offline or online recording methods according to their own needs, facilitating flexible management and analysis of exercise data.

[0132] If the offline recording method is adopted, that is, not connecting to the mobile software during use, the collected data will be automatically saved in the data integration center module 4. The data integration center module 4 is built-in with a memory that can store a certain amount of data, facilitating subsequent analysis and viewing by users. The stored data can be exported through the USB interface or synchronized to the device when connecting to the mobile device next time.

[0133] When using the online recording method, the data of the data integration center module 4 is transmitted to the mobile device in real time in the form of a Bluetooth wireless gateway. The application on the mobile device receives and analyzes the data in real time and outputs real-time analysis results. The data of the inertial measurement unit sensor 1 will be adaptively trained through a hierarchical human behavior recognition model for action agility recognition and classification. These models are implemented through the above-designed adaptive hierarchical algorithm, which can automatically identify different exercise patterns, such as running, walking, jumping, etc., and count the duration and proportion of each exercise, so as to provide a detailed exercise analysis report. Due to the different sampling frequencies of different sensors, when using the clock circuit for timestamp alignment, different types of data are integrated into a data packet in the form of seconds.

[0134] The data of the inertial measurement unit sensor 1 will be used to construct a classifier and perform adaptive model updates through the above-designed adaptive hierarchical algorithm model for user personalized exercise recognition and classification. Specifically, the preprocessed exercise data can be used to identify different exercise patterns, such as running, walking, jumping, etc., through the above-trained classification model, and a detailed exercise analysis report is provided. At the same time, according to the matching degree between the individual differences of different users (including body type and exercise preferences, etc.) and the recognition algorithm, the adaptive model is updated to make the system adapt to users personally. The adaptive update includes online updating of the classifier according to the differences between the user and the pre-trained human behavior model; and based on the acquisition of new user data, solving the classification error problem caused by insufficient data when the user first starts using.

[0135] The data of the electrocardiogram sensor 2 will be used to calculate electrocardiogram parameters, including heart rate and arrhythmia. Heart rate can reflect exercise intensity and cardiac function status. By monitoring the changes in heart rate, the physical fitness level and training effect of users can be evaluated. Arrhythmia refers to abnormal heart rhythm, including tachycardia, bradycardia or irregularity, which may affect the effective circulation of blood. During exercise, the monitoring of arrhythmia can help detect potential heart problems in time, prevent serious heart events and ensure the safety of users.

[0136] The angular change data of the rotary shaft angle measuring instrument 3 are jointly used to fit the respiratory curve of the chest after being processed, feature extracted, and fitted and calculated, and then detailed respiratory parameters are calculated, including: respiratory rate, inspiratory time, expiratory time, respiratory time, tidal volume, minute ventilation volume, average inspiratory flow rate, average expiratory flow rate, and duty cycle. These parameters can help evaluate the user's respiratory health status, especially during exercise, to monitor respiratory changes in real time.

[0137] After the user completes the exercise, the overall exercise data is uploaded to the cloud. The cloud server can store and analyze the data for a long time and provide personalized health reports for the user. For example, the overall heart rate change of the user during a single complete exercise, whether there are any abnormalities, suggestions for the user's breathing pattern during exercise based on the heart rate and respiratory rate, etc.; for all the user's exercises, show the changes in the user's physiological parameters during exercise over time and give guiding suggestions.

[0138] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A multimodal analysis system for smart wearable cardiopulmonary exercise clothing, characterized by: The invention comprises a multi-modal analysis algorithm, a multi-sensor integrated system and a wearable garment, wherein the wearable garment is provided with an inertial measurement unit sensor (1), an electrocardiogram sensor (2), a rotation axis goniometer (3) and a data integration center module (4); The multimodal analysis algorithm includes human behavior modeling and cardiopulmonary health analysis modeling; The human behavior modeling is based on the data processed by the inertial measurement unit sensor (1), and uses a hierarchical network to identify the specific activities of the user: first, dynamic and static classification is performed through thresholds, and then the specific activities are identified using a pre-trained dynamic deep learning model and a static deep learning model respectively. At the same time, the system will also adaptively update the model based on the uniqueness of the user; In order to efficiently divide the static and dynamic motion states, the data integration center module (4) first calculates the Euclidean distance Q based on the acceleration data of the inertial measurement unit sensor (1). t for: is the triaxial component of the accelerometer at time t, when Q t If it exceeds the set threshold, it is judged as dynamic motion, otherwise it is judged as static motion; The division results directly determine the selection of subsequent classification models. Static actions use static deep learning models to identify activities in a static state, and dynamic movements use dynamic deep learning models to identify movements in a dynamic state.

2. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: In the human behavior modeling, in the dynamic and static activity recognition, the accelerometer and gyroscope data collected by the inertial measurement unit sensor (1), X = [x a ;x g ] are fed into two parallel convolutional streams to process features, and the convolutional block is defined as: ConvBlock(x)=σ(Conv1D(x,ω)+b) σ is the SiLU activation function, ω and b are the convolution kernel weight and bias respectively. The convolution flow can extract local spatiotemporal features and generate feature representations for subsequent processing.

3. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 2, characterized in that: The convolutional flow outputs the feature y a and g Input to the sensor-level Transformer module to model the interaction between accelerometer and gyroscope features, the formula is: Z=TransformerBlock(Concat[y a ,y g ]) The Transformer module is mainly composed of a multi-head self-attention mechanism and a feed-forward network, and is calculated as: Z′ l =MSA(LN(Z l-1 ))+Z l-1 FROM l =FFN(LN(Z′ l ))+Z′ l LN is the layer normalization operation, l is the Transformer layer index, and the fused features are classified into different activity categories through the fully connected layer.

4. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The human behavior modeling is to improve the personalized recognition capability. The data integration center module (4) supports adaptive updating based on user characteristics, collects user activity data during actual use, and dynamically adjusts the human behavior modeling in combination with the annotation information input by the user. Specifically, based on the data collected online, the static deep learning model and the dynamic deep learning model are fine-tuned using transfer learning technology; First, freeze the first few layers of feature extraction network of the static deep learning model or the dynamic deep learning model to maintain the generalization ability of the basic features, and adjust the last few layers of classification network to adapt to the user's specific motion mode.

5. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 4, characterized in that: In the training process, small batch incremental updates are used to update the user's historical data D user Based on, the loss function is defined as: f(x i , θ) is the predicted value of the current static deep learning model or dynamic deep learning model, y i is the true label, l(f(x i ,θ),y i ) is the loss function; Through the above mechanism, the system can continuously optimize the parameters of static deep learning models and dynamic deep learning models to improve personalized recognition accuracy.

6. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The cardiopulmonary health analysis modeling is based on processed ECG sensor (2) data and axle goniometer (3) data, and uses two-lead ECG sensor (2) data to calculate ECG parameters during activity, including: heart rate and arrhythmia; The axle goniometer (3) fits the rise and fall of the chest to calculate the breathing parameters, and covers the entire chest of the user, so as to sense the rise and fall of the chest during natural breathing in daily activities, obtain the angle change, and obtain the breathing curve through breathing curve fitting and feature extraction, and calculate the breathing parameters during activities. The breathing parameters include breathing frequency, inhalation time, exhalation time, breathing time, tidal volume, minute ventilation, average inspiratory flow, average expiratory flow and duty cycle.

7. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 6, characterized in that: The method for data fitting and respiratory feature extraction of the axle goniometer (3) is called multi-channel adaptive feature fusion MAFF. The formula of the MAFF algorithm is: Filtering the raw angle data collected by the shaft goniometer (3) to remove high-frequency noise and artifacts; The core of the MAFF algorithm is to extract a matrix reflecting the characteristics of respiratory motion by calculating the minimum value of each pair of channels and further calculating the square root of the sum of the square differences between each column and its minimum value. The matrix can capture the phase difference and amplitude change between different channels, thereby fitting the respiratory curve more accurately. In order to improve the accuracy of signal fitting, a weight optimization mechanism is introduced. By weighted fusion of multi-channel features, the MAFF algorithm uses an unconstrained minimization method to optimize the weight of each channel; Based on the fitted breathing curve, we extract the following key feature parameters: Inspiratory time: The duration of the inspiratory phase is identified from the respiratory curve; Exhalation time: the duration of the exhalation phase identified from the respiratory curve; Respiratory time: the sum of inhalation time and exhalation time, that is, the time of a complete respiratory cycle; Tidal volume: Estimates the amount of gas exchanged per breath by calculating the difference between the maximum and minimum tidal volumes during each respiratory cycle.

8. The multimodal analysis system for cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The inertial measurement unit sensors (1) are arranged to cover the limbs and trunk of the human body, thereby achieving comprehensive perception of the motion state; The ECG sensor (2) is arranged close to the heart to obtain ECG signals and accurately reflect the electrical activity state of the heart; The rotating shaft goniometer (3) senses the angle change of the chest and abdomen, realizes multi-dimensional fitting of the synchronous rise and fall of the chest and abdomen during breathing, and forms a complete breathing movement model.

9. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 1, characterized in that: The multi-sensor integrated system adopts a high-precision clock signal synchronization circuit to achieve time alignment and data synchronization of the inertial measurement unit sensor (1), the electrocardiogram sensor (2) and the shaft angle meter (3) through the clock signal; The clock signal is generated by the data integration center module (4) and is represented by the clock signal f clk The nodes of each inertial measurement unit sensor (1), electrocardiogram sensor (2) and axle goniometer (3) are distributed in the form of f clk =1000Hz Ensure the uniformity of data sampling time base and realize time synchronization and multi-modal fusion of data.

10. The multimodal analysis system of the cardiopulmonary exercise smart wearable clothing according to claim 9, characterized in that: The clock signal f clk The nodes are distributed in each inertial measurement unit sensor (1), electrocardiogram sensor (2) and axle angle meter (3), and the nodes receive f clk Afterwards, a frequency division circuit is used to generate a clock signal that matches its sampling frequency; For sensors of the same type, use the timestamp T in the clock signal clk As a benchmark, each sensor records the current timestamp when sampling to ensure data synchronization under the same time axis; For different types of sensors, a segment alignment mechanism is adopted, with the starting timestamp T of each second. align Alignment for benchmarks; The data integration center module (4) generates a global synchronization timestamp T every second. align = k·1S, different types of sensors align their sampling times to the closest T align , ensuring that the timestamp at the beginning of each second is consistent; The data integration center module (4) aggregates the collected data according to the timestamps and divides the time segments of different types of sensor data into T align To align the benchmarks, a multimodal fusion data stream is formed.

Images