A multi-modal physiological and motion monitoring system based on intelligent timing modeling

Abstract

The application discloses a kind of multi-modal physiological and motion monitoring systems based on intelligent timing modeling, it is related to motion monitoring technical field, including magnetic cilium sensing unit, signal acquisition module, wireless transmission module, signal processing and resolving module;Magnetic cilium sensing unit acquires magnetic signal, and is transmitted to signal acquisition module;Signal acquisition module converts magnetic signal into magnetic field intensity signal, and is transmitted to signal processing and resolving module by wireless transmission module;Signal processing and resolving module, according to magnetic field intensity signal, obtains monitoring result by deep learning resolution. The application can realize simultaneous sensing and decoupling of forces in different directions while reducing system energy consumption.

Description

Technical Field

[0001] This invention relates to the field of motion monitoring technology, and more specifically to a multimodal physiological and motion monitoring system based on intelligent time-series modeling. Background Technology

[0002] Wearable and adhesive monitoring technologies have evolved from "single-indicator, single-device" to "integrated systems across different locations and tasks." Early technologies relied heavily on channels such as PPG / ECG / EMG, pressure / deformation films, and IMUs for cardiopulmonary monitoring, electromyography, contact load estimation, and attitude estimation, respectively. While these devices performed stably in specific scenarios, they were susceptible to effects from adhesion tightness, sweat, temperature drift, and environmental electromagnetic fields during prolonged wear and strenuous exercise. Stacking multiple devices also increased wiring and packaging complexity and maintenance costs. With the development of flexible electronics and novel sensing materials, researchers have attempted to use flexible front-ends that conform to the skin's curvature to convert external forces into triaxial temporal signals, and to perform temporal modeling at the edge using structures such as BiLSTM and TCN. For engineering implementation, the system requires a closed-loop link from "front-end—acquisition—edge inference—wireless backhaul—online iteration," unifying sampling aperture, attitude / baseline estimation, edge-side lightweighting, and layered secure backhaul technology.

[0003] Existing technologies for wearable multimodal monitoring can be broadly categorized into three approaches: First, multi-device fusion, which involves collecting different modalities through multiple independent sensors and front-end circuits, then transmitting the raw data back via Bluetooth or other means, and using a larger model for fusion inference. This approach offers high accuracy for single tasks, but is limited by factors such as asynchronous sampling rates and clocks, differences in noise spectra, and channel delays. The time alignment and feature matching before fusion are costly, and the transmission of large amounts of raw data significantly increases power consumption and privacy costs. Second, mechanical decoupling, which relies on multi-cavity, multi-beam, or limiting structures to achieve directional separation at the physical level. This results in clear steady-state performance, but the multi-layer stacking leads to increased device thickness and stiffness, insufficient comfort and durability for long-term wear, and poor cross-part compatibility once the structure is finalized. Third, end-to-end data-driven approaches, which use 1D-CNN, BiLSTM, or Transformer to directly regress labels or physical quantities from sequences. This allows for rapid prototype development, but without physical priors and online calibration, it is sensitive to changes in wearing posture, temperature, and environmental background. Furthermore, it is difficult to achieve low-power real-time operation on the MCU when the model size is large. In summary, existing technologies share common shortcomings in terms of system coupling, energy efficiency, and interpretability. They are characterized by complex hardware size and wiring, high maintenance and consistency calibration costs, system errors introduced by clock drift and resolution differences, communication energy consumption becoming a bottleneck, structural fatigue affecting long-term stability, and significant performance fluctuations when purely data-driven migrations to new locations or populations. It is difficult to simultaneously meet the comprehensive goals of "low cost, low power consumption, interpretability, and easy cross-location expansion".

[0004] Therefore, how to achieve low-cost, low-power multi-directional force sensing motion monitoring is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a multimodal physiological and motion monitoring system based on intelligent temporal modeling. The system uses a "single flexible front end + unified three-axis acquisition" as the hardware module and combines a "mechanism prior + intelligent temporal modeling" software framework to achieve simultaneous sensing and decoupling of forces in different directions. It is deployed on the microcontroller in a low-power manner. The system of the present invention has significant advantages in hardware scale, assembly complexity and BOM cost, while retaining the scalability for cross-part applications.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multimodal physiological and motion monitoring system based on intelligent time-series modeling includes a magnetic ciliary sensing unit, a signal acquisition module, a wireless transmission module, and a signal processing and decoding module.

[0008] The magnetic cilia sensing unit collects triaxial magnetic signals and transmits them to the signal acquisition module;

[0009] The signal acquisition module converts the triaxial magnetic signal into a digital signal to obtain the magnetic field strength signal, and then transmits it to the signal processing and calculation module via the wireless transmission module.

[0010] The signal processing and resolution module uses deep learning to decouple and classify the magnetic field strength signal to obtain monitoring results.

[0011] Preferably, the magnetic cilia sensing unit employs a cilia array sensor made of soft magnetic material. The cilia array sensor includes a magnetic field sensing element and a cilia array fixed above it. The cilia array comprises magnetic cilia and an elastic substrate. The uniformly magnetized magnetic cilia are fixed within the elastic substrate, which is fixed above the magnetic field sensing element. Under external force, the magnetic cilia bend or compress, causing changes in the spatial orientation of their built-in magnetic dipoles, thereby generating coupled triaxial magnetic signals (Bx, By, Bz).

[0012] Preferably, the signal acquisition module uses a RA4M1 microcontroller connected to a magnetic field sensing element to acquire triaxial magnetic signals (Bx, By, Bz) caused by the deformation of magnetic cilia, and convert them into magnetic field strength signals.

[0013] Preferably, the signal processing and calculation module uses a BiLSTM model to decouple the magnetic field strength signal into force component signals, including normal force and tangential force. The specific process is as follows:

[0014] The continuously acquired magnetic field strength signal is divided into time series segments according to time sequence, and the input sequence is constructed using all time series segments;

[0015] The forward LSTM branch of the pre-trained BiLSTM model processes the input sequence from beginning to end, while the reverse LSTM branch of the BiLSTM model processes the input sequence from end to beginning. During processing, the hidden state at each time step is formed by concatenating the hidden states of the forward LSTM branch and the reverse LSTM branch.

[0016] The fully connected layers of the pre-trained BiLSTM model output estimates of the normal and tangential forces based on the hidden states at each time step, forming force component signals.

[0017] Preferably, during the training of the BiLSTM model, the mean absolute error (MAE) is used as the loss function to measure the difference between the predictive power and the true power. The Adam optimizer is used for optimization, and the initial learning rate is set to 0.001. During training, a regularization strategy is used to optimize the model. A Dropout layer is inserted between the hidden layers and fully connected layers of the BiLSTM model, with a dropout rate of 0.2. This causes 20% of the hidden units to be randomly turned off during training, thereby avoiding overfitting of the model to a specific force range or cilia deformation pattern. An early stopping mechanism is used to control the training iteration, and training is stopped based on the validation set loss.

[0018] The technical effect of the above solution is that the signal acquisition module uses a BiLSTM model to build a decoupling framework, which can convert stray coupled Hall signals into clear and physically meaningful electrical signals. This bridges the gap between the flexibility of the ciliary array sensor and the requirement for high-fidelity electrical signal reconstruction. By embedding the physical mechanism of ciliary deformation into a data-driven time model, the trade-off between rigid hardware or linear processing is avoided. Only a single, unmodified hardware platform is needed to fully realize the dual potential of the sensor in monitoring large mechanical events and weak physiological signals.

[0019] Preferably, the wireless transmission module uses an ESP32 chip, which is connected to the signal acquisition module to transmit magnetic field strength signals.

[0020] Preferably, the signal processing and resolution module identifies and classifies the magnetic field strength signal using a deep learning algorithm based on time-series modeling.

[0021] Preferably, the signal processing and resolution module employs a motion prediction model constructed based on a Temporal Convolutional Network (TCN). It identifies and classifies motion components based on force signals to obtain the motion type as the monitoring result. The motion prediction model comprises three stacked convolutional layers, an adaptive max-pooling layer, a fully connected layer, and a residual connection. The features extracted from the convolutional layers are compressed and represented by the adaptive max-pooling layer to obtain pooled features. The fully connected layer combines the pooled features with the magnetic field strength signal transmitted by the residual connection for classification to obtain the motion type. Each convolutional layer is followed by a ReLU activation function and a batch normalization layer. The residual connection allows only clear motion features to pass through while blocking interference signals that do not match the motion rhythm.

[0022] Preferably, the three convolutional layers employ dilated convolutions, progressively extracting signal features by setting different dilation factors. The ReLU activation function and batch normalization layers enhance the model's non-linear expressive power and accelerate training. Each convolutional layer is represented as follows:

[0023]

[0024] in, Indicates the extracted features; This represents the value of the input digital signal at time t; These are the weights of the convolution kernel; It is the length of the convolution kernel; It is the dilation factor. This expanded convolutional structure allows the model to expand its receptive field without increasing computational complexity, thereby better capturing long-term dependencies in the signal. The dilation factor grows exponentially with the number of layers, with the factor of each layer being twice that of the previous layer. This exponential growth allows the network to expand its receptive field without increasing computational burden, enabling it to accurately capture subtle action details while simultaneously filtering out large-scale coupling interference.

[0025] Preferably, the adaptive max pooling layer compresses time series data into a fixed-length feature representation, thereby solving the problem of inconsistent input signal lengths. The expression is as follows:

[0026]

[0027] Indicates aggregation characteristics; The i-th element of the j-th channel of the input signal is represented; N represents the number of samples in batch processing, which describes the number of windows for parallel processing in a single inference or training. The time step length of each window is T, and each time step contains three magnetic field components (Bx, By, Bz).

[0028] The technical effects of the above-mentioned technical solution are that the present invention uses a bidirectional LSTM network to achieve precise decoupling of normal force and tangential force, while a unified TCN can achieve high-precision cross-site signal recognition.

[0029] Preferably, the fabrication process of the ciliary array includes:

[0030] The Ecoflex prepolymer is cast to form a base layer and placed in a vacuum oven for about 15 minutes to remove entrained air.

[0031] The substrate layer is partially cured under a preset temperature condition to form an adhesive surface that promotes bonding.

[0032] Precisely align the Teflon (PTFE) microcavity mold with the adhesive surface;

[0033] Ecoflex and NdFeB micro powder were mixed in the target ratio (50 / 60 / 70 / 80% by weight) to form a magnetoelastic slurry, and then placed in a uniform strong magnetic field to make the particles align perpendicularly along the future cilia axis.

[0034] The magnetic elastomer slurry is injected into a Teflon mold and then vacuum degassed to eliminate residual air bubbles.

[0035] After complete curing at the preset temperature, gently remove the Teflon mold to obtain a vertically arranged array of fibers;

[0036] The uniformity of the array is ensured by eliminating the residual magnetism between the cilia in the cilia array through a uniform magnetization field.

[0037] Preferably, the magnetic field sensing element is a triaxial magnetometer, which can simultaneously capture rapid joint movements and weak physiological vibrations.

[0038] Preferably, the transducer core of the ciliary array sensor is fabricated on Ecoflex–NdFeB composite material using a low-temperature process with mold assistance and magnetic field alignment. This process creates a flexible magnetic response microstructure array that is compatible with both skin patch and compact ring packaging forms, allowing for reuse across different parts of the device without replacing the transducer core.

[0039] As can be seen from the above technical solution, compared with the prior art, this invention discloses a multimodal physiological and motion monitoring system based on intelligent temporal modeling. It aims to engineer multi-directional force sensing and recognition into an integrated, low-cost, and low-power wearable system. Without increasing complex mechanical structures or stacking multiple components, it achieves simultaneous sensing and decoupling of multi-directional forces from a single front end, cross-site reuse, and low power consumption at the edge. A unified network backbone, combined with lightweight individualization, supports rapid migration to multiple sites, meeting the requirements of large-scale manufacturing and engineering implementation. The system maintains stable performance and portability in different sites such as the ankle, knee, wrist, throat, and ring, possessing an ultra-wide working range and high pressure resolution, adapting to extreme conditions such as strenuous exercise and pathological vibrations. It demonstrates application potential for individualized follow-up and health management, and due to a significant reduction in the number of components, solder joints, and wiring, it exhibits significant advantages in BOM and assembly costs. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 A schematic diagram of the multimodal physiological and motion monitoring method based on intelligent temporal modeling provided by the present invention;

[0042] Figure 2 This is a schematic diagram of throat vibration signal acquisition provided by the present invention;

[0043] Figure 3 This is a schematic diagram of radial artery signal acquisition provided by the present invention;

[0044] Figure 4 This is a schematic diagram of knee joint signal acquisition provided by the present invention;

[0045] Figure 5 This is a schematic diagram of ankle signal acquisition provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the ankle signal classification results provided by the present invention;

[0047] Figure 7 This is a schematic diagram of the clustering results provided by the present invention;

[0048] Figure 8 This is a schematic diagram of the confusion matrix provided by the present invention;

[0049] Figure 9 Training / verification / test curves for different detection environments in the embodiments provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention discloses a multimodal physiological and motion monitoring system based on intelligent time-series modeling, including a magnetic ciliary sensing unit, a signal acquisition module, a wireless transmission module, and a signal processing and calculation module;

[0052] The magnetic cilia sensing unit collects magnetic signals and transmits them to the signal acquisition module;

[0053] The signal acquisition module converts the magnetic signal into a digital signal to obtain the magnetic field strength signal, and then transmits it to the signal processing and calculation module via the wireless transmission module.

[0054] The signal processing and resolution module uses deep learning to decouple and classify the magnetic field strength signal to obtain monitoring results.

[0055] Furthermore, the magnetic cilia sensing unit employs a cilia array sensor made of soft magnetic material. The cilia array sensor includes a magnetic field sensing element and a cilia array fixed above it. The cilia array comprises magnetic cilia and an elastic substrate. The uniformly magnetized magnetic cilia are fixed within the elastic substrate, which is then fixed above the magnetic field sensing element. Under external force, the magnetic cilia bend or compress, causing changes in the spatial orientation of their built-in magnetic dipoles, resulting in a change in the magnetic field strength signal.

[0056] Furthermore, the signal acquisition module uses a RA4M1 microcontroller connected to a magnetic field sensing element to acquire the magnetic signal B(t) caused by the deformation of magnetic cilia, and convert it into a digital signal to obtain the magnetic field strength signal.

[0057] Furthermore, the signal processing and computation module uses a BiLSTM model to decouple the magnetic field strength signal into force component signals, including normal force and tangential force. The specific process is as follows:

[0058] The continuously acquired magnetic field strength signal is divided into time series segments according to time sequence, and the input sequence is constructed using all time series segments;

[0059] The forward LSTM branch of the pre-trained BiLSTM model processes the input sequence from beginning to end, while the reverse LSTM branch of the BiLSTM model processes the input sequence from end to beginning. During processing, the hidden state at each time step is formed by concatenating the hidden states of the forward LSTM branch and the reverse LSTM branch.

[0060] The fully connected layers of the pre-trained BiLSTM model output estimates of the normal and tangential forces based on the hidden states at each time step, forming force component signals.

[0061] Furthermore, during the BiLSTM model training process, the mean absolute error (MAE) is used as the loss function to measure the difference between the predicted power and the true power. The Adam optimizer is used for optimization, and the initial learning rate is set to 0.001. During training, a regularization strategy is adopted to optimize the model. A Dropout layer is inserted between the hidden layers and fully connected layers of the BiLSTM model, with a dropout rate of 0.2. This randomly shuts down 20% of the hidden units during training, thereby avoiding overfitting of the model to a specific force range or cilia deformation pattern. An early stopping mechanism is used to control the training iterations, stopping training based on the validation set loss.

[0062] First, the BiLSTM architecture introduces a backward LSTM branch, enabling backtracking analysis of signal dependencies from the end of the sequence. This bidirectional modeling is crucial for capturing the asymmetric temporal dynamics of magnetociliary deformation: when a tangential force is applied (such as skin shearing), the cilia bend rapidly (forward temporal gradient), but their elastic recovery (after the force is removed) exhibits a slower hysteresis effect (backward temporal gradient). By fusing the hidden states of the forward and backward branches, the model successfully resolves this asymmetric dynamic characteristic.

[0063] Secondly, the input layer fully preserves the complete temporal characteristics of the Hall signal, rather than using isolated time points. Input sequence X∈ ^{N×100×3} (N is the batch size, 100 is the time step, and 3 is the triaxial Hall component) contains 100 consecutive sampling points (300 milliseconds), ensuring that the model can not only learn the instantaneous magnetic field value, but also grasp its rate of change (dB / dt) and interaxial correlation.

[0064] Finally, a task-specific regularization strategy is employed to improve the model's generalization ability under different variable loading conditions. A Dropout layer (dropout rate = 0.2) is inserted between the hidden layers and the fully connected output layer of the Bidirectional Long Short-Term Memory (BiLSTM) network, randomly disabling 20% ​​of the hidden units during training to avoid overfitting the model to specific force ranges (e.g., 0.1–0.5 N) or ciliary deformation patterns.

[0065] Furthermore, the wireless transmission module uses an ESP32 chip, which is connected to the signal acquisition module to transmit magnetic field strength signals.

[0066] Furthermore, the signal processing and resolution module identifies and classifies the magnetic field strength signal using a deep learning algorithm based on time-series modeling.

[0067] Furthermore, the signal processing and computation module constructs a motion prediction model based on a temporal convolutional network (TCN), identifies and classifies the force component signals, and obtains the action type as the monitoring result. The motion prediction model includes three convolutional layers, an adaptive max pooling layer, a fully connected layer, and a residual connection stacked sequentially. The features extracted by the convolutional layers are compressed and represented by the adaptive max pooling layer to obtain pooled features. The fully connected layer combines the pooled features and the magnetic field strength signal transmitted by the residual connection to classify the action type. Each convolutional layer is followed by a ReLU activation function and a batch normalization layer.

[0068] Furthermore, the three convolutional layers employ dilated convolutions, progressively extracting signal features by setting different dilation factors. ReLU activation functions and batch normalization layers enhance the model's non-linear expressive power and accelerate training. Each convolutional layer is represented as follows:

[0069]

[0070] in, Indicates the extracted features; This represents the value of the input digital signal at time t; These are the weights of the convolution kernel; It is the length of the convolution kernel; It is the expansion factor. This expanded convolutional structure allows the model to expand its receptive field without increasing computational complexity, thereby better capturing long-term dependencies in the signal.

[0071] Furthermore, the adaptive max-pooling layer compresses the time-series data into a fixed-length feature representation, thereby solving the problem of inconsistent input signal lengths. The expression is as follows:

[0072]

[0073] Indicates aggregation characteristics; Represents the i-th element of the j-th channel of the input signal; N is the number of samples in batch processing, used to describe the number of windows for parallel processing in a single inference or training, with a time step length of T for each window, and each time step containing three magnetic field components (Bx, By, Bz).

[0074] Furthermore, the fabrication process of the ciliary array includes:

[0075] The Ecoflex prepolymer is cast to form a base layer and placed in a vacuum oven for about 15 minutes to remove entrained air.

[0076] The substrate is partially cured at 80°C for 5 minutes to form an adhesive surface that promotes bonding.

[0077] Precisely align the Teflon (PTFE) microcavity mold with the adhesive surface;

[0078] Ecoflex and NdFeB micro powder were mixed in the target ratio (50 / 60 / 70 / 80% by weight) to form a magnetoelastic slurry, and then placed in a uniform strong magnetic field to make the particles align perpendicularly along the future cilia axis.

[0079] The magnetic elastomer slurry is injected into a Teflon mold and then vacuum degassed to eliminate residual air bubbles.

[0080] After complete curing at 80°C for 60 minutes, gently remove the Teflon mold to obtain a vertically arranged array of fibers.

[0081] The uniformity of the array is ensured by eliminating the residual magnetism between the cilia in the cilia array through a uniform magnetization field.

[0082] Furthermore, the magnetic field sensing element uses a triaxial magnetometer, which can simultaneously capture rapid joint movements and weak physiological vibrations.

[0083] Furthermore, the transducer core of the sensor is fabricated on Ecoflex–NdFeB composite material using a low-temperature process with mold assistance and magnetic field alignment to create a flexible magnetic response microstructure array. This array is compatible with both skin patch and compact ring packaging forms, allowing for reuse across different parts of the device without replacing the transducer core.

[0084] Furthermore, the signal acquisition module, wireless transmission module, and signal processing and resolution module are integrated on the MCU side. The preprocessing function of the signal acquisition module is implemented on the MCU side, and BiLSTM and TCN models are deployed to achieve lightweight temporal inference. Both models adopt 8-bit quantization and structured pruning. Combined with windowed inference and memory reuse strategies, the latency of a single inference is controlled at the millisecond level, and the power consumption is reduced to the tens of milliwatts level. The MCU side also adopts a baseline calibration and dynamic update mechanism for the signal processing and resolution module to eliminate signal baseline drift caused by changes in wearing posture, temperature drift, and environmental magnetic field interference. The MCU side also adopts an event-level summary backhaul strategy for the wireless transmission module to reduce bandwidth and privacy risks.

[0085] On the other hand, in one specific embodiment, the monitoring process using a multimodal physiological and motion monitoring system based on intelligent temporal modeling is as follows: Figure 1 As shown, specifically:

[0086] S1: Arrange the magnetic ciliary sensing unit on the part of the human body to be sampled;

[0087] S2: The magnetic ciliary sensing unit collects magnetic intensity signals;

[0088] S3: The signal acquisition module converts the magnetic intensity signal into a digital signal and transmits it to the signal processing and calculation module via the wireless transmission module;

[0089] S4: The signal processing and calculation module constructs a motion prediction model based on BiLSTM and TCN networks, calculates digital signals, and obtains external force information.

[0090] On the other hand, in one specific embodiment, a multimodal physiological and motion monitoring system based on intelligent temporal modeling is employed to accurately acquire human physiological and motion signals in multiple scenarios. Several magnetic ciliary sensing units are used to collect several sets of multimodal signals across body parts. The system can acquire signals from four typical human body parts during dynamic activities, including: Figure 2 The larynx vibration signal shown (used for sound detection, such as breathing, swallowing, and speech sounds), such as... Figure 3 The radial artery deformation signal shown (used for pulse detection and PTD waveform analysis), such as Figure 4 The knee joint motion signals shown (used to identify knee joint movements, such as bending and squatting) and such as Figure 5 The ankle movement signal shown is used for gait monitoring such as walking and running.

[0091] (1) Sound signal detection

[0092] Throat vibration signals can be used for sound signal detection. This detection scenario focuses on three types of actions: breathing, swallowing, and speaking. Cluster analysis is performed based on the signal characteristics of the sounds produced by these actions, and motion prediction models are used to achieve motion analysis and accurate identification.

[0093] The processing flow for laryngeal sound signal detection is as follows: A magnetic ciliary sensing unit is attached to the surface of the larynx to acquire a time series of triaxial magnetic field strength caused by phonation, swallowing, and respiration. The signal acquisition module converts the magnetic field strength time series into a digital signal and transmits it to the signal processing and resolution module. The signal processing and resolution module first uses a force decoupling unit based on BiLSTM to restore the digital signal of the magnetic field strength time series into the corresponding normal force sequence and tangential force sequence to characterize the micro-vibrations generated on the surface of the larynx. In the online recognition stage, the multi-channel time series composed of the above normal force and tangential force sequences is input into a motion prediction model based on a temporal convolutional network (TCN), which outputs action category labels such as breathing, swallowing, and speaking. Furthermore, the detection process also distinguishes between two types of speech content, indicating that it can be further subdivided into the speech content layer.

[0094] During the system modeling phase, dimensionality reduction and cluster analysis can be performed on the two-channel vibration sequences to verify the separability of different actions. The clustering results are as follows: Figure 7 As shown in Figure (D), the multi-channel clustering analysis of the sound signal reveals that, due to the uniqueness of the pronunciation, the two types of speech content are relatively clearly distinguished in the feature space (red and green clusters). One type of speech (reading text 1) is represented by the green cluster, and the other type of speech (reading text 2) is represented by the red cluster. Furthermore, "swallowing" and "breathing" are represented by the yellow and blue clusters, respectively, indicating that the sampling points are close to each other. This is because the airflow sound during shallow breathing is weak, similar to the preparatory sound of the larynx caused by saliva flow before swallowing; during the breathing recovery phase after rapid swallowing, the short-pulse swallowing sound is difficult to distinguish from the smooth sound of shallow breathing, resulting in blurred cluster boundaries. This reflects the difficulty of capturing the "transition phase" of an action through sound signals, and the distinction is even more challenging due to the simplicity of the action and the limited feature dimension.

[0095] Considering the temporal characteristics (action duration period) and frequency diversity (multi-syllable speech, single-frequency breathing) of sound signals, a motion prediction model is still used for feature extraction and classification. Different dilation factors are set to adapt to the multi-timescale features of sound signals: 1) For multi-frequency syllables in "speaking," a small dilation factor is used to focus on short-period features (such as syllable duration and intonation changes) to capture the structural information of speech content, which can more finely distinguish different speech contents; 2) For short-pulse sounds of "swallowing," a medium dilation factor is used to cover single-period features, enhancing the recognition of "instantaneous high-amplitude" signals; 3) For continuous and soft sounds of "breathing," a large dilation factor is applied to cover long-period features (such as breathing frequency and airflow stability) to maintain the temporal regularity of breathing patterns.

[0096] (2) Pulse signal detection

[0097] Radial artery deformation signals can be used for pulse signal detection. The pulse signal is an important physiological indicator reflecting human health, commonly used to assess cardiovascular health and exercise status. A typical pulse waveform consists of three characteristic components: the impact wave (P wave), the tidal wave (T wave), and the diastolic wave (D wave). The pulse signal is generated in three types of activity: rest, running, and sitting. By distinguishing pulse changes under different exercise states, an individual's health status can be assessed.

[0098] The processing flow for radial artery pulse signal detection is the same as that for laryngeal sound signal detection. Cluster analysis is performed on the pulse signals collected by the magnetic ciliary sensing unit from the human body in resting, running, and sitting positions to visually present the characteristic distribution. For example... Figure 7 As shown in Figure (C), based on the features in the figure, the three types of activities are relatively different. The focus is mainly on the interwoven edge samples in local areas. The pulse signal clustering results are shown in the upper right corner of the figure. "Running," "sitting," and "resting" are represented by yellow, green, and blue clusters, respectively. There is overlap between the edges of the yellow clusters and the clusters of the green and blue clusters. This is because during the initial stage of running (lower exercise intensity) and the deceleration stage (approaching walking), the pulse signal overlaps with the pulse signal from rapid posture adjustments while sitting (such as standing up) and rapid movements from a stationary state (such as suddenly standing up). In the clustering space, this reflects the feature confusion during the transition phase of the movement.

[0099] Considering the temporal characteristics (periodicity of heart rate / acceleration) and dynamic characteristics (pulse abrupt changes during running) of pulse signals, a TCN model is employed for feature extraction and action classification prediction. By dilating the convolutional layers to adapt to the pulse frequencies of different actions (e.g., a small dilation factor captures high-frequency pulses during running, while a large dilation factor captures stable features in a stationary state), the distinction between high-dynamic features of running, moderate-dynamic features of sitting, and low-dynamic features of stationary states is enhanced. Furthermore, an attention mechanism is introduced to focus on the overlapping regions between sitting and resting, and between sitting and running, thereby strengthening the learning of pulse features for subtle movements (such as posture adjustments while sitting and deceleration during running).

[0100] (3) Knee joint motion detection

[0101] The processing flow for knee joint motion detection is the same as that for throat sound signal detection. The knee joint plays a crucial supporting role in various human movements, including squatting, standing, and maintaining a static posture. Detecting knee joint motion signals is essential for scenarios such as motion posture analysis, rehabilitation training assessment (e.g., monitoring recovery after knee surgery), and daily activity recognition. This scenario focuses on four types of knee movements: bending, holding a static posture, squatting, and standing. Accurate motion recognition and status assessment are achieved by collecting and analyzing dynamic signals from the knee joint region.

[0102] The magnetic ciliary sensing unit can detect minute motion changes in the knee region and convert them into electrical signals. Using the collected knee joint signals, cluster analysis is performed, and the results are as follows: Figure 7 As shown in (A), it is clear that "bending over," "remaining still," "squatting," and "standing" exhibit significant inter-group separability differences, corresponding to the blue, yellow, green, and red clusters, respectively. Furthermore, there is partial overlap between the "squatting" and "bending over" regions. This is because, in the later stages of the squatting motion, the unidirectional movement trend of the knee joint resembles the signal characteristics of a small-amplitude bending motion, leading to feature overlap during clustering. This phenomenon reflects the correlation between the challenges in motion and signal acquisition.

[0103] (4) Ankle signal detection

[0104] The processing flow for ankle signal detection is the same as that for throat sound signal detection. The ankle is a vital part of the human body during walking and movement. Monitoring its signals is significant in gait analysis, exercise rehabilitation, and elderly movement monitoring. Common ankle signals include dynamic changes in various gait movements, such as walking, running, and turning. Accurate ankle signal monitoring can assess an individual's motor abilities and promptly identify movement disorders. The monitoring and classification results using the system of this invention are as follows: Figure 6 As shown, the horizontal axis represents time and the vertical axis represents magnetic signals. The figures in the figure represent the movement states determined by the magnetic signals within the corresponding time periods, namely standing, walking forward with small steps, walking forward with large steps, running, and walking backward.

[0105] A magnetic ciliary sensing unit was placed on the ankle to capture ankle motion signals under different movement states, reflecting pressure changes in the ankle joint during these states. To further explore the characteristic distribution of these signals under different movement states, [further details are needed]. Figure 5 Cluster analysis was performed on the signals acquired from the three channels (x, y, z) shown, and the results are as follows: Figure 7As shown in Figure (B), highly dynamic and large-amplitude movements can be observed. For example, "running" is represented by the red cluster in the figure, which is a relatively gentle and regular movement; "walking" is represented by the purple cluster; "moving forward" is represented by the yellow cluster; "walking backward" is represented by the blue cluster; and "walking in place" is represented by the green cluster, exhibiting very different ankle pressure change patterns and temporal characteristics. Therefore, clear boundaries naturally form during the clustering process, indicating that the time series signals of this movement have extremely high separability. On the other hand, clusters with inter-class overlap or difficulty in separation are mostly associated with movements with high motion similarity. For example, "walking" and "walking in place" have relatively similar ankle pressure change trends and temporal dynamics, resulting in low feature separability, making it difficult for clustering to completely separate them.

[0106] Figure 8 Tables (A)-(D) show the corresponding confusion matrices under the four detection environments described above, with the overall accuracy (ACC) annotated. Figure 8 In the middle (A), the confusion matrix is ​​shown. Figure 8 (B) is the confusion matrix in the ankle detection environment. Figure 8 In the middle (C), the confusion matrix is ​​shown. Figure 8 In the middle (D), the confusion matrix is ​​shown in the throat sound detection environment. Figure 9 The training / validation / test curves under four detection environments (A)-(D) are shown in the figure. The horizontal axis represents the training batch, and the vertical axis represents the accuracy and loss. Figure 9 In the middle (A), the curve is under the knee joint detection environment. Figure 9 (B) is the curve under the ankle detection environment. Figure 9 (C) represents the curve under pulse detection conditions. Figure 9 Figure (D) shows the curve in the laryngeal sound detection environment. Experimental results show that the system of the present invention achieves high performance in accuracy, precision, recall, and F1 score in the above four scenarios, verifying the robustness and generalization ability of the sensor-model pipeline for multimodal motion recognition tasks. From the perspective of specific scenarios: the sound signal achieved a perfect score of 1 in all four indicators, demonstrating the best performance. It can accurately classify breathing, swallowing, and two types of speech signals, completely overcoming the feature confusion problem between swallowing and breathing signals in the "action transition phase"; for knee joint motion detection (bending, staying still, squatting, standing), due to the significant differences in the biomechanical time characteristics between the four movements, all four metrics remain at a high level; the ankle signal and pulse signal detection scenarios show slightly lower metrics than the other two scenarios, which can be attributed to differences in signal features or action transition problems.

[0107] On the other hand, in one specific embodiment, the system of the present invention is applied to the classification and identification of tremor in Parkinson's disease (PD). Clinically, resting tremor in Parkinson's disease is typically concentrated in the 4-6 Hz frequency band. To ensure the clinical interpretability of the baseline data, this embodiment evaluates six frequency bands: 0 Hz, 3 Hz, 4 Hz, 5 Hz, 6 Hz, and 7 Hz. The 4-6 Hz band covers the classic frequency range, 3 Hz and 7 Hz are used as boundary conditions to detect the separability of frequency band edges, and 0 Hz represents a tremor-free baseline (DC).

[0108] A ring-shaped magnetic ciliary sensor worn on the index finger was used to induce metronome-guided simulated tremors in healthy adults. Subjects were seated with forearm support to minimize arm sway. Rhythmic oscillations were recorded under six conditions in independent trials: a 0 Hz baseline, 3 Hz, 4 Hz, 5 Hz, 6 Hz, and 7 Hz. During recording, small-amplitude rhythmic flexion and extension movements of the index finger were synchronized with the metronome rhythm, avoiding deliberate wrist movements, and the test tag was synchronized with the metronome start point. Triaxial sequential readings of approximately 333 Hz per axis (total frequency approximately 1.0 kHz) were acquired, and preprocessing and segmentation methods were consistent with those used for other body parts.

[0109] To visually demonstrate the separability of different categories, features were embedded using the Uniform Manifold Approximation and Projection (UMAP) technique, followed by k-means clustering analysis. Features in the 4–6 Hz band formed a tight manifold structure, while the 3 Hz and 7 Hz bands exhibited a boundary-like distribution. Since these oscillations may mimic physiological background fluctuations (such as muscle twitches and subtle postural adjustments), this highlights the challenge of distinguishing early / low-amplitude tremors from the true baseline signal.

[0110] To address coupling interference, such as electromagnetic crosstalk or timing mismatch noise, in the output of a triaxial Hall sensor and to achieve tremor state classification, this embodiment employs a Temporal Convolutional Network (TCN) as the motion prediction model. Unlike Bidirectional Long Short-Term Memory (BiLSTM) networks that directly output specific force signals for "explicit" decoupling, TCN embeds the decoupling logic directly into the classification process. This design is more effective in handling qualitative tasks such as tremor state recognition because it first eliminates coupling noise in the Hall signal. In the motion prediction model, if the first convolutional layer uses a dilation factor of 1, the second layer uses 2, and the third layer uses 4, the factor of each layer is twice that of the previous layer. This exponential growth allows the network to expand its receptive field without increasing computational burden, accurately capturing subtle tremor details (such as rapid 3-7 Hz rhythms) while simultaneously filtering out large-scale coupling interference.

[0111] Other key decoupling components of the TCN model include residual connections and three convolutional layers. Residual connections allow only clear tremor features to pass through, while blocking interfering signals that do not match the tremor rhythm. Each of the three convolutional layers is followed by a ReLU activation function and batch normalization, which extracts tremor-related Hall signal segments and distinguishes them from global interference. To handle variable-length inputs, an adaptive max-pooling layer retains the most informative peaks by taking the maximum value within the pooling span. Subsequently, the pooled features are input into a fully connected (FC) layer for classification, identifying small-amplitude, medium-amplitude, and large-amplitude tremors, aiding in subsequent Parkinson's disease severity diagnosis.

[0112] A validation set was constructed to verify the classification performance of the motion prediction model. A multi-class confusion matrix was constructed on the validation set, and core evaluation metrics were calculated based on the matrix. The core evaluation metrics include overall accuracy, precision, recall, and F1 score. The calculation showed that the model achieved an overall accuracy of 71.4%, precision of 75%, recall of 66.77%, and F1 score of 67.4%, which indicates that the model has the ability to effectively classify Parkinson's tremor.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0114] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal physiological and motion monitoring system based on intelligent temporal modeling, characterized in that, It includes a magnetic cilia sensing unit, a signal acquisition module, a wireless transmission module, and a signal processing and decoding module; The magnetic cilia sensing unit collects magnetic signals and transmits them to the signal acquisition module; The signal acquisition module converts the magnetic signal into a magnetic field strength signal, which is then transmitted wirelessly to the signal processing and calculation module. The signal processing and resolution module uses deep learning to decouple and classify the magnetic field strength signal to obtain monitoring results.

2. The multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 1, characterized in that, The magnetic cilia sensing unit uses a cilia array sensor made of soft magnetic material. The cilia array sensor includes a magnetic field sensing element and a cilia array fixed above it. The cilia array includes magnetic cilia and an elastic substrate. The magnetic cilia, which have been uniformly magnetized, are fixed in the elastic substrate, which is fixed above the magnetic field sensing element.

3. The multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 2, characterized in that, The signal acquisition module uses a RA4M1 microcontroller, which is connected to a magnetic field sensing element to acquire magnetic signals caused by the deformation of magnetic cilia and convert them into digital signals to obtain magnetic field strength signals.

4. The multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 1, characterized in that, The wireless transmission module uses an ESP32 chip, which connects to the signal acquisition module to transmit digital signals.

5. A multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 1, characterized in that, The signal processing and calculation module uses a BiLSTM model to decouple the magnetic field strength signal into force component signals, including normal force and tangential force.

6. A multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 5, characterized in that, The signal processing and computation module constructs a motion prediction model based on a temporal convolutional network. The motion prediction model is used to identify and classify force component signals to obtain the action type as the monitoring result. The motion prediction model consists of three convolutional layers, an adaptive max pooling layer, a fully connected layer, and a residual connection stacked in sequence. The features extracted by the convolutional layers are compressed and represented by the adaptive max pooling layer to obtain aggregated features. The fully connected layer combines aggregated features for classification to obtain action types; each convolutional layer is followed by a ReLU activation function and a batch normalization layer.

7. A multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 6, characterized in that, The three convolutional layers each employ dilated convolution, progressively extracting signal features by setting different dilation factors. Each convolutional layer is represented as follows: Where y(t) represents the extracted features; x(t) represents the value of the input digital signal at time t; w(i) is the weight of the convolution kernel; k is the length of the convolution kernel; and d is the dilation factor.

8. A multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 6, characterized in that, The expression for the adaptive max pooling layer is: Indicates aggregation characteristics; The i-th element of the j-th channel of the input signal is represented; N represents the number of samples in batch processing.

9. A multimodal physiological and motion monitoring system based on intelligent temporal modeling according to claim 2, characterized in that, The fabrication process of the ciliary array includes: Ecoflex prepolymer is cast to form a base layer and placed in a vacuum oven to remove entrained air; The substrate layer is partially cured under a preset temperature condition to form an adhesive surface that promotes bonding. Align the Teflon mold with the sticky surface; Ecoflex and NdFeB micro powder were mixed in the target ratio to form a magnetic elastomer slurry, and then placed in a uniform strong magnetic field so that the particles were arranged perpendicularly along the axis of the cilia. The magnetic elastomer slurry is injected into a Teflon mold and then degassed under vacuum. Complete curing is performed under preset temperature conditions, and demolding yields a vertically arranged array of fibers; Residual magnetism between cilia in the cilia array is eliminated by using a uniform magnetizing field.

Images