Multi-sensor fusion action recognition system for shoulder rehabilitation evaluation

Abstract

The invention belongs to a biological signal identification and data processing technology, and relates to a multi-sensor fusion action identification system for shoulder rehabilitation evaluation. Comprising a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module and an action recognition module, the flexible strain sensor comprises a plurality of sensing areas and connecting areas which are distributed around the rotator cuff area; the multichannel data acquisition module acquires multichannel sensing signals, and the multichannel sensing signals are preprocessed by the data preprocessing module. The action recognition module adopts a multi-modal feature fusion algorithm based on a bidirectional cross-modal interactive attention and time period specificity adaptive weight mechanism to realize dynamic correlation modeling between a strain sensing signal and inertial measurement data in a unified time sequence space, and cross-modal time sequence features are fused; capturing the global and local dependency relationship of the shoulder movement, and recognizing various rehabilitation actions. The method effectively overcomes the static limitation of the prior art, and remarkably improves the precision and robustness of shoulder rehabilitation evaluation.

Description

Technical Field

[0001] This invention belongs to the field of biosignal recognition and data processing technology, specifically relating to a multi-sensor fusion motion recognition system for shoulder rehabilitation assessment. Background Technology

[0002] With the accelerating aging of the population and the continuous increase in the number of patients with upper limb dysfunction caused by diseases such as stroke, frozen shoulder, and rotator cuff tears, shoulder rehabilitation assessment and training have become key issues that urgently need to be addressed in the field of rehabilitation medicine. Traditional shoulder function assessment mainly relies on the subjective judgment of physicians and simple measurement tools, which has limitations such as inconsistent assessment standards, poor reproducibility, and lack of objective quantitative indicators, making it difficult to provide accurate basis for the development of individualized rehabilitation programs.

[0003] In recent years, multimodal sensor fusion technology has been widely used in research on upper limb and shoulder movement recognition. Patent CN105963926A, published on September 28, 2016, proposes a multimodal hand function rehabilitation assessment system based on electromyography, force, and posture signals. It achieves multi-dimensional movement monitoring through multi-source parallel acquisition; however, the fusion method is merely feature splicing, lacking semantic interaction between modalities and a phased weight adaptive mechanism, making it difficult to characterize the complex multi-axial coupling characteristics of the shoulder joint. Patent CN106691478A, published on May 24, 2017, uses pressure, tactile, and inertial signals in parallel for hand function assessment, employing fixed or semi-static weights for weighted scoring. While it can fuse different modalities, it cannot dynamically adjust according to the movement stage or signal quality changes, and its robustness to noise and poor contact is insufficient. The invention patent CN120564254A, published on August 29, 2025, proposes a motion recognition system and method for upper limb exoskeletons. It achieves a breakthrough in exoskeleton control by fusing visual and inertial data through heterogeneous spatiotemporal graph convolution. However, its fusion mechanism relies on a fixed graph structure and static attention mechanism and does not design an adaptive weight update strategy for the stage characteristics and signal fluctuations of shoulder rehabilitation.

[0004] Existing multimodal fusion methods generally employ simple feature splicing or fixed weighted averaging, which are static fusion methods that lack mechanisms for dynamic weight adjustment based on action stage, signal quality, and individual differences. These static fusion approaches fail to establish deep cross-modal correlations, making it difficult to fully leverage the information complementarity between sensors and resulting in insufficient capture of complex motion features such as multi-axis shoulder rotation and muscle strain. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a multi-sensor fusion motion recognition system for shoulder rehabilitation assessment. It introduces a bidirectional cross-modal interactive attention and time-time specific adaptive weighting algorithm, which can realize dynamic correlation modeling of flexible strain and inertial signals at the feature layer. Modal weights are allocated in real time according to the motion stage and signal quality, thereby achieving high-precision, real-time, and automated recognition and quantitative assessment of shoulder rehabilitation movements. This effectively overcomes the static limitations of existing fusion technologies and significantly improves the accuracy and robustness of shoulder rehabilitation assessment.

[0006] The objective of this invention is achieved through the following technical solution: a multi-sensor fusion motion recognition system for shoulder rehabilitation assessment, comprising a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module, and a motion recognition module; The flexible strain sensor includes multiple sensing areas distributed around the rotator cuff region, and a connecting area connecting two adjacent sensing areas or the ends of the sensing areas; the multiple sensing areas are respectively located at the top of the rotator cuff, the anterior clavicle, the posterior deltoid muscle, and the axilla. During the multi-channel data acquisition process, the multiple sensing areas are connected in series; the multi-channel data acquisition module acquires the voltage divider signals of the multiple sensing areas, and also simultaneously acquires the inertial measurement data of the inertial measurement unit, and encapsulates the acquired voltage divider signals and inertial data into multi-channel sensing signals; wherein the inertial measurement data includes the accelerometer's rate of change of acceleration, the gyroscope's angular velocity, and the magnetometer's magnetic field change; The data preprocessing module is used to preprocess multi-channel sensor signals to eliminate dimensional differences and numerical biases. The motion recognition module employs a multimodal feature fusion algorithm based on bidirectional cross-modal interactive attention and time-time specific adaptive weighting mechanism for the preprocessed multi-channel sensor signals. This algorithm enables dynamic correlation modeling between flexible strain sensor signals and inertial measurement data within a unified temporal space. The bidirectional cross-modal interactive attention mechanism constructs semantic mapping and complementary relationships between modalities to adaptively adjust the dominance and interaction direction between modalities according to changes in shoulder movement characteristics. For the phased characteristics of shoulder rehabilitation movements, a time-time specific adaptive weighting mechanism dynamically allocates modal weights based on signal strength and quality at different movement stages, thereby achieving the fusion of cross-modal temporal features. The fused temporal features are then processed by a multi-layer feature encoder and a hybrid aggregation module to capture the global and local dependencies of shoulder movements, achieving high-precision recognition of various rehabilitation movements.

[0007] Preferably, the flexible strain sensor is made of a carbon black and silicone elastomer composite material, which is screen-printed onto a fabric substrate and cured in a vacuum environment; Each sensing area is a linear structure with a winding rectangular trajectory at the turning points; Each connecting region is a sheet-like or strip-like structure; The fabric base area where multiple connection areas are located, combined with the fabric base area where multiple sensing areas are located, is integrated as part of the garment's shoulder sleeve and embedded into the sewn area of ​​the garment's shoulder sleeve.

[0008] Preferably, the data preprocessing module preprocesses the multi-channel sensor signals, including: Adaptive differentiated low-pass filtering adaptively determines the filter order and cutoff frequency based on the spectral distribution characteristics of the sensing signals from different channels, thereby achieving independent filtering processing for the sensing signals from each channel. Data extraction for stable time periods: Construct a composite motion intensity index that integrates acceleration change rate, gyroscope angular velocity, and magnetometer magnetic field change. By combining the composite motion intensity index with an adaptive threshold, identify stable segments in multi-channel sensor signals and extract data for stable time periods. A two-stage extended Kalman filter architecture is used for attitude calculation. In the calibration stage, inertial data in a stationary state is collected and statistically analyzed to estimate the noise characteristics of the inertial measurement unit, thereby achieving noise characteristic modeling and obtaining the process noise matrix used to characterize the gyroscope integration error and zero-bias drift. And the observation noise matrix reflecting the measurement noise characteristics of the accelerometer and magnetometer. During the operation phase, adaptive adjustment of observation noise is achieved by combining composite motion intensity indicators. When the shoulder is in a stable state, the constructed observation noise matrix is ​​used. This represents the observation noise; when the shoulder enters the dynamic movement phase, the observation noise matrix will be... The observed noise is represented by the scaled-up representation.

[0009] Furthermore, when extracting data during stable time periods, the 25th percentile of the composite exercise intensity index sequence was used. As a statistical benchmark, the dynamic threshold Defined as: ; When the composite exercise intensity index is continuously lower than the dynamic threshold and the duration exceeds the first preset time, the corresponding continuous time interval is determined to be a stable time period.

[0010] Preferably, the processing procedure of the action recognition module includes: Multimodal features are independently encoded using a flexible strain modal encoder and an inertial modal encoder to independently encode the sensing signals of different modes. The flexible strain modal encoder consists of two layers of one-dimensional convolution and one layer of feedforward network to extract the dynamic change pattern of local deformation of the shoulder and obtain flexible strain modal features. The inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer to capture the global dynamic features of attitude change and obtain inertial modal features. Modal and temporal location encoding injects modal identifiers and temporal location information into flexible strain modal features and inertial modal features respectively, and maps flexible strain modal features and inertial modal features to a temporal feature space of the same dimension. Cross-modal dynamic feature interaction is achieved through a bidirectional cross-modal interactive attention mechanism to realize dynamic feature interaction between flexible strain modes and inertial modes; Time-specific adaptive weighting: The action recognition module introduces a time-specific gating weighting mechanism at the time series modeling level, which dynamically adjusts the modality fusion ratio according to the changes in action stage features. Signal quality weighting is used to design an adaptive fusion mechanism based on signal quality at the global level. With signal-to-noise ratio and channel reliability as the core indicators, the fusion weights of flexible strain mode and inertial mode are dynamically adjusted to obtain a global feature vector after signal quality weighted fusion. Global semantic aggregation and average pooling are used to input the global feature vector, which is obtained by weighted fusion of signal quality, into a multi-layer stacked encoder. Each encoder layer includes a multi-head self-attention module and a feedforward network structure. The encoder captures the continuous features of the action evolution over time and the correlation between previous and subsequent frames through progressive attention calculation and nonlinear feature transformation. It extracts features from the temporal trajectory, stage division and cross-modal correlation of the entire action process and constructs a unified global semantic representation to achieve feature aggregation and form a fixed-dimensional action representation vector. After the action representation vector is mapped to the preset action category space through a fully connected layer, the recognition results of multiple shoulder rehabilitation actions are output.

[0011] The present invention has the following advantages and effects compared with the prior art. 1. More Comprehensive and Stable Motion Monitoring: The system adopts a flexible strain sensor integrated with the clothing design, taking into account the physiological movement patterns of the shoulder to achieve a highly body-fitting wearing method. The sensors are distributed at multiple points along the shoulder and sleeve, comprehensively covering the main movement directions of the shoulder joint, significantly reducing measurement blind spots and improving monitoring accuracy. The sensor material is treated with carbon black-organic silicone elastomer composite and polydimethylsiloxane (PDMS) curing, possessing high sensitivity, flexibility, and moisture and sweat resistance, maintaining stable signal and low noise output under prolonged wear and multi-movement scenarios.

[0012] 2. More Efficient and Reliable Multimodal Data Acquisition: A multi-channel acquisition platform is built around a microcontroller to achieve synchronous acquisition and real-time transmission of flexible strain and inertial measurement data. The system effectively suppresses noise through analog-to-digital converter sampling and active filtering, and transmits the data to the host computer via Bluetooth, ensuring data continuity and accuracy, and providing high-quality input for motion recognition.

[0013] 3. More Adaptive and Robust Data Processing: The system incorporates adaptive mechanisms across signal processing and attitude estimation. Adaptive multi-channel filtering automatically determines the cutoff frequency of each channel, and the adaptive stable segment extraction algorithm identifies high-confidence intervals based on dynamic thresholds, ensuring data purity and continuity. Attitude estimation employs a two-stage adaptive extended Kalman filter (AEKF), updating the process and observation noise matrices in real time during operation to achieve 50 Hz attitude estimation. This triple adaptive strategy enables the system to dynamically adjust parameters according to changes in signal characteristics and the environment, significantly improving accuracy, stability, and robustness.

[0014] 4. More Accurate Multimodal Fusion Recognition: This system is based on a multimodal Transformer architecture with a bidirectional cross-modal interactive attention mechanism and a time-specific adaptive weighting mechanism. It achieves dynamic correlation modeling of flexible strain sensing signals and inertial measurement signals within a unified temporal space. The algorithm establishes semantic mapping and complementary relationships between the two modalities through a bidirectional attention mechanism, adaptively adjusting the direction of information interaction and modal dominance according to the movement stage, thereby accurately capturing the multidimensional features of complex shoulder movements. The time-specific adaptive weighting module dynamically allocates modal weights based on signal quality and stage changes, enabling the model to strengthen key modal responses at different movement stages. The fusion stage combines signal quality-driven global weight optimization to effectively suppress noise interference and enhance model robustness. A hybrid aggregation strategy of classification identifiers (CLS tokens) and global average pooling further enhances the global feature representation capability, achieving high-precision recognition of subtle movements and multi-axis rotations. The system exhibits excellent stability and generalization performance under complex postures and different subject conditions, providing accurate and reliable technical support for real-time monitoring and quantitative evaluation of shoulder rehabilitation training.

[0015] 5. Real-time Intelligent Interaction with the Host Computer: The platform can directly load trained models, perform instant classification and visualization feedback on collected data, and output strain waveforms, posture curves, and recognition results in real time. The system supports data storage and online model iteration, constructing an intelligent closed loop of "collection—processing—recognition—feedback—optimization," providing low-latency, quantitative, and scalable intelligent assessment and individualized intervention support for rehabilitation training. Attached Figure Description

[0016] Figure 1 This is a structural block diagram of the multi-sensor fusion motion recognition system for shoulder rehabilitation assessment in an embodiment of the present invention; Figure 2 This is a schematic diagram of the layout of the flexible strain sensor and inertial measurement unit in the shoulder sleeve area; Figure 3 This is a structural design drawing of a flexible strain sensor; Figure 4 This is a schematic diagram of a flexible strain sensor printed on a fabric substrate; Figure 5 This is a schematic diagram of the equivalent circuit of the sensing area when collecting sensor signals. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0018] Example This embodiment provides a multi-sensor fusion motion recognition system for shoulder rehabilitation assessment. It uses a flexible strain sensor and an inertial measurement unit (IMU) to perform multimodal fusion to achieve shoulder motion recognition and assessment, and is used for motion monitoring and quantitative assessment during rehabilitation training.

[0019] like Figure 1 , Figure 2 As shown, the system in this embodiment specifically includes a flexible strain sensor 1, an inertial measurement unit 2, a multi-channel data acquisition module, and a host computer, wherein the host computer is equipped with a data preprocessing module and an action recognition module.

[0020] I. Flexible Strain Sensor The flexible strain sensor includes multiple independent sensing areas 11 distributed around the shoulder cuff region, and a connecting area 12 connecting two adjacent sensing areas or the ends of the sensing areas, such as... Figure 3 As shown, this embodiment has four sensing areas, with four independent sensing areas respectively located at the top of the rotator cuff, the anterior clavicle, the posterior deltoid muscle, and the axilla. This arrangement of the sensing areas is based on the physiological structure and movement characteristics of the shoulder joint, and can detect major movements such as shoulder abduction, elevation, flexion, internal rotation, extension, and external rotation, achieving comprehensive coverage of multi-directional shoulder movements, effectively reducing blind spots, and improving the accuracy and stability of motion monitoring.

[0021] Furthermore, the sensing area and connecting area of ​​the flexible strain sensor can be optimized in terms of length and aspect ratio to reduce the impact of the connecting area resistance on the overall sensing accuracy. Specifically, the optimized design is as follows: the sensing area is designed as a wavy rectangular trajectory with a length of 30mm, a width of 1.5mm, and a spacing of 1mm at the bends. Its larger aspect ratio significantly increases the effective resistance change ratio of the sensing area, thereby improving strain sensitivity. The connecting area is designed as a length of 80mm and a width of 35mm. Its smaller aspect ratio effectively reduces the resistance value of the connecting area, decreasing its proportion in the overall resistance change, thereby reducing interference from non-sensing areas and improving measurement accuracy and stability.

[0022] The flexible strain sensor is fabricated using a carbon black and silicone elastomer composite material. The material is screen-printed onto a fabric substrate and then cured in a vacuum environment. The conductive material is prepared in the following mass ratio: 3.0g carbon black, 30.3g silicone elastomer, 45.0g silicone oil, and 1.6g curing agent. After stirring at 400rpm for 30 minutes, the paste is screen-printed onto the fabric substrate and cured in a 100°C vacuum oven for 1 hour. See also... Figure 3 , Figure 4 Each sensing area 11 is a linear structure, preferably 30mm × 1.5mm, with a rectangular meandering trajectory at the bends, to increase the conductive path and strain accumulation effect of the sensing area, reduce interference, and improve signal stability, thereby improving sensitivity and resolution. Each connecting area is a sheet or strip structure, and the connecting area 12 can completely cover the corresponding fabric substrate 13 to form Figure 3 , Figure 4 The irregular black area, containing multiple connecting areas, combines with the fabric base area containing multiple sensing areas to form a shoulder sleeve that is embedded into the seam area of ​​the garment's shoulder sleeve.

[0023] In terms of garment construction, the flexible strain sensor employs a flexible fabric design that conforms to the shoulder and sleeve curves. Its edge shape matches the seams of the garment, allowing it to naturally embed into the seam area of ​​the shoulder and sleeve without being directly compressed or cut by the seams, thus avoiding performance damage. A highly elastic, form-fitting fabric is used as the base, enabling the sensor to closely conform to the shoulder curve and maintain a stable position during movement, reducing measurement errors caused by slippage or wrinkles. Balancing physiological adaptability, comfort, and engineering feasibility, it ensures stable data acquisition and practical wearability.

[0024] To further enhance performance, the fabricated flexible strain sensor was immersed twice in a 1% PDMS (polydimethylsiloxane) ethanol solution and then cured in stages to form a uniform and dense protective layer on the surface. This protective layer possesses excellent electrical insulation and flexibility, effectively isolates humidity and sweat interference, stabilizes the conductive network, reduces hysteresis, and significantly improves signal stability and long-term durability.

[0025] II. Multi-channel data acquisition module The multi-channel data acquisition module, centered on a microcontroller, uses four channels (ADC1-ADC4) of an analog-to-digital converter (ADC) to acquire voltage divider signals (i.e., strain electrical signals) from four independent sensing areas. Simultaneously, it acquires inertial measurement data from the inertial measurement unit (IMU) via an IIC bus (Inter-Integrated Circuit), specifically including the accelerometer's rate of change of acceleration, the gyroscope's angular velocity, and the magnetometer's magnetic field changes, achieving real-time fusion of dual-mode signals. A flexible strain sensor detects local shoulder deformation, while the IMU provides overall attitude information; their combination enables comprehensive monitoring of shoulder movement. To ensure signal quality, an active low-pass filter with a cutoff frequency of 5 Hz is connected in series at the sensor output to suppress high-frequency noise. The microcontroller connects via serial port to Bluetooth, encapsulating the acquired voltage divider signals and inertial data into multi-channel sensing signals at a 50 Hz sampling frequency. These multi-channel sensing signals are then encapsulated with analog low-pass filtering and a synchronization timestamp before being transmitted to the host computer's data preprocessing module, achieving real-time fusion of local and overall shoulder motion information.

[0026] During multi-channel data acquisition, four sensing areas are connected in series and connected to a reference resistor. This forms a voltage divider circuit, which reflects local deformation in real time through voltage changes, such as... Figure 5 As shown. The series structure can superimpose signal strength and suppress noise interference, ensuring the stability and reliability of the measurement. The reference resistor can be set to 500 Ω.

[0027] The equivalent resistance of the sensing area of ​​the flexible strain sensor increases with increasing tensile deformation and decreases with increasing compressive deformation. Together with a reference resistor, they form a voltage divider circuit. Based on the voltage divider principle and the reference resistor, the resistance change of each sensing area in the flexible strain sensor is calculated using the following formula: ; ; ; ; in Indicates external power supply. This represents the voltage to ground obtained by averaging the 20-point values ​​sampled from the four connection areas of the flexible strain sensor using an analog-to-digital converter (ADC); the four calculated resistance values... ~ The deformation of the four sensing areas of the flexible strain sensor under the current posture was characterized.

[0028] III. Data Preprocessing Module The data preprocessing module is used to preprocess multi-channel sensor signals, including adaptive differentiated Butterworth low-pass filtering, stable time period data extraction, and extended Kalman filter attitude calculation. This is used to filter out noise, fuse multi-source data and extract stable time period data, and adaptively adjust observation noise to eliminate dimensional differences and numerical biases, thereby improving the accuracy and stability of subsequent analysis and identification. The data preprocessing module's processing mainly includes the following parts: (1) Adaptive Differentiated Low-Pass Filtering For multi-channel sensor signals, the data preprocessing module employs a Butterworth low-pass filtering method that combines differentiation and adaptive mechanisms. Based on the spectral distribution characteristics of different channel sensor signals, the filter order and cutoff frequency are adaptively determined to achieve independent filtering processing for each channel sensor signal. This effectively suppresses high-frequency noise while maximizing the preservation of the low-frequency effective components of shoulder movement.

[0029] The data preprocessing module first performs a Fast Fourier Transform (FFT) on the sensor signals from each channel and calculates the power spectral density: ; in, Indicates the signal at a frequency point The power spectral density on Represents the frequency index in the Fast Fourier Transform The corresponding frequency point, It is the complex spectrum sequence of the original time series (i.e., the original sensing signals of each channel) after fast Fourier transform; Represents the normalization coefficient. The number of FFT sampling points. The sampling frequency.

[0030] Define a power spectrum curvature function in logarithmic coordinates to detect inflection points in energy changes: ; ; ; ; Within a preset frequency band [0.2, 10] Hz, the frequency corresponding to the point of maximum curvature is selected as the adaptive cutoff frequency: ; in, It represents the power spectrum curvature value, used to detect inflection points in energy changes and measure the degree of local curvature of the curve; This represents the first-order difference approximation (slope) of the power spectrum curve in logarithmic coordinates. This represents the second-order difference approximation (rate of curvature) of the power spectrum curve in logarithmic coordinates. Frequency index The corresponding logarithmic coordinate position of the frequency point Frequency index The corresponding logarithmic coordinate position of the frequency point Frequency index The corresponding logarithmic coordinate position of the frequency point; Indicates the signal at the frequency index The corresponding logarithmic power spectral density at the frequency point Indicates the signal at the frequency index The corresponding logarithmic power spectral density at the frequency point Indicates the signal at the frequency index The logarithmic power spectrum value at the corresponding frequency point; This represents the points on the frequency and power spectrum curves in an algebraic coordinate system, used to obtain a smooth energy change trend; This indicates the adaptive cutoff frequency.

[0031] The data preprocessing module can automatically adjust the filtering parameters according to the signal energy distribution, eliminating the need for manual threshold setting. Experimental results show that the optimal cutoff frequencies for the flexible strain sensor, accelerometer, gyroscope, and magnetometer are 0.5 Hz, 2 Hz, 3 Hz, and 1 Hz, respectively, effectively balancing signal smoothness and dynamic response. The cutoff frequency for each channel's sensing signal is adaptively set using FFT. Specifically, a second-order low-pass filter structure is used for the multi-channel sensing signal detected by the flexible strain sensor to maintain smoothness, while a fourth-order filter structure is used for the overall attitude information detected by the inertial measurement unit to enhance cutoff characteristics. Furthermore, to ensure signal phase consistency and time alignment, bidirectional zero-phase filtering is employed, meaning the filtering process uses bidirectional computation to achieve zero phase delay.

[0032] (2) Data extraction during stable time periods To eliminate short-term disturbances and improve the reliability of attitude calculation, the data preprocessing module constructs a composite motion intensity index that integrates the rate of change of acceleration, the angular velocity of the gyroscope, and the magnetic field change of the magnetometer. The composite motion intensity index integrates the acceleration, angular velocity, and magnetic field change characteristics of the inertial measurement unit to comprehensively characterize the overall motion state of the shoulder. By combining the composite motion intensity index with an adaptive threshold, stable segments in the multi-channel sensor signals are identified, and data from the stable time period (which can be referred to as the stable segment) is extracted to achieve effective screening of low-noise, high-confidence data.

[0033] For each sampling time Define three types of characteristic quantities: rate of change of acceleration. angular velocity amplitude With the rate of change of magnetic field During the static calibration phase, the standard deviations of three characteristic quantities—acceleration rate of change, angular velocity amplitude, and magnetic field rate of change—were calculated respectively. , , The composite exercise intensity index was obtained by normalizing the index. : ; This composite motion intensity index can comprehensively reflect the overall motion intensity of the shoulder at any given time; the lower the value, the more stable the posture.

[0034] To achieve adaptive judgment under different experimental conditions, this embodiment introduces a percentile-based dynamic threshold mechanism to determine whether a stable time period has appeared in the multi-channel sensing signal. The 25th percentile of the composite motion intensity index sequence is used as the threshold. As a statistical benchmark, the dynamic threshold Defined as: ; When the composite exercise intensity index continuously below the dynamic threshold If the duration exceeds the first preset time (e.g., 5 seconds), the corresponding continuous time interval is determined to be a stable time period.

[0035] To avoid over-segmentation caused by short-term fluctuations or minor posture adjustments, this embodiment further introduces a temporal constraint mechanism. This mechanism requires that the duration of a stable time period be no less than a first preset time (e.g., 5 seconds) to filter out pseudo-stable segments caused by sensor noise or brief periods of stillness. When the interval between adjacent stable time periods is less than a second preset time (e.g., 0.3 seconds), the adjacent stable time periods are automatically merged into one stable time period to reduce over-segmentation and maintain temporal continuity. The data from the stable time periods are labeled and stored independently for subsequent posture calculation and rehabilitation movement recognition. The stable time period recognition mechanism used in this embodiment can automatically adjust the threshold according to signal fluctuation characteristics, eliminating the need for manual threshold setting and maintaining stable recognition performance under different subjects, movement amplitudes, and environmental conditions.

[0036] (3) Adaptive Extended Kalman Filter Finally, the data preprocessing module uses a two-stage extended Kalman filter architecture for attitude estimation. During the filtering process, the observation noise matrix is ​​dynamically updated according to the motion state to improve the robustness and stability of attitude estimation.

[0037] The first stage is the calibration stage, which involves collecting inertial data under static conditions and performing statistical analysis to estimate the noise characteristics of each flexible strain sensor and inertial measurement unit, realizing noise characteristic modeling, obtaining the process noise matrix used to characterize the gyroscope integration error and zero bias drift, as well as the observation noise matrix reflecting the measurement noise characteristics of the accelerometer and magnetometer.

[0038] For a gyroscope, the theoretical output in a stationary state should be close to zero angular velocity. Let the number of collected N samples be... , Its mean is The formula for calculating the gyroscope noise variance is: ; For an accelerometer, the theoretical output at rest should be: , Let N be the acceleration due to gravity, and let the number of collected samples be N. Its mean is The formula for calculating the accelerometer noise variance is: ; Based on the aforementioned noise variance, the process noise matrix is ​​constructed by adaptively setting the variance of static data. With observation noise matrix : ; Among them, the process noise matrix Characterizing gyroscope integration error and zero bias drift, observation noise matrix This reflects the measurement noise characteristics of the accelerometer and magnetometer. Represents a 3x3 identity matrix. This represents a 2x2 identity matrix.

[0039] Simultaneously, during the calibration phase, gyroscope zero-bias calibration, accelerometer deviation compensation, and initial attitude estimation are completed. After calibration, the second phase, the operational phase, begins, where attitude calculation is performed using adaptive parameters.

[0040] During the operation phase, combined with composite exercise intensity indicators To achieve adaptive adjustment of observation noise. When the shoulder is in a stable state ( When the observation data has high reliability, the constructed observation noise matrix is ​​used. This indicates observed noise to enhance posture correction; when the shoulder enters the dynamic movement phase ( When observing the noise matrix, The observation noise is represented by a scaled-up image to reduce the impact of high-frequency disturbances on filter stability. Observation noise The adjustment rules are as follows: ; in, To amplify the observation noise, the observation noise matrix is ​​increased when the motion intensity exceeds the dynamic threshold. This is multiplied by a factor of 1 to reduce the weight of inertial measurements affected by dynamic disturbances in attitude fusion. In this embodiment, The preferred value is .

[0041] In this embodiment, a two-stage extended Kalman filter architecture is used in the data preprocessing module for attitude calculation. The observation noise matrix R is updated adaptively, and data extraction during the stable time period is combined to enhance the filter's correction capability under static conditions (i.e., in a stable state) and maintain prediction smoothness during the dynamic motion phase, thereby achieving real-time 50 Hz calculation of the shoulder's three-dimensional attitude, balancing high accuracy and low latency.

[0042] This embodiment utilizes a multi-channel data acquisition module and a data preprocessing module to perform time-synchronized sampling and standard normalization preprocessing on the flexible strain sensing signals and inertial measurement signals, eliminating dimensional differences and numerical biases. Specifically, the multi-channel data acquisition module acquires 4 channels of flexible strain sensing signals from the flexible strain sensor and 9 channels of attitude data (i.e., inertial measurement signals) from the inertial measurement unit. All signals are segmented using a sliding window (50 frames in length, 50% overlap), and only clean samples with consistent labels are retained for subsequent modeling.

[0043] IV. Action Recognition Module The motion recognition module employs a multimodal feature fusion algorithm based on bidirectional cross-modal interactive attention and time-time-specific adaptive weighting mechanisms to model the dynamic correlation between flexible strain sensing signals and inertial measurement data within a unified temporal space. The bidirectional cross-modal interactive attention mechanism constructs semantic mapping and complementary relationships between modalities to adaptively adjust the dominance and interaction direction between modalities according to changes in shoulder movement characteristics. For the phased characteristics of shoulder rehabilitation movements, a time-time-specific adaptive weighting mechanism dynamically allocates modal weights based on signal strength and quality at different movement stages, thereby achieving the fusion of cross-modal temporal features. The fused temporal features are processed by a multi-layer feature encoder and a hybrid aggregation module to capture the global and local dependencies of shoulder movements, achieving high-precision recognition of multiple types of rehabilitation movements. The multi-layer feature encoder can be implemented using a Transformer encoder.

[0044] Unlike traditional one-way or static fusion methods, the fusion algorithm in the action recognition module can adaptively adjust the dominance and interaction direction of the two modalities based on changes in action stage and signal characteristics, thereby achieving multi-level information complementarity and dynamic balance. The entire action recognition process includes three core steps: bidirectional cross-modal feature interaction, time-time specific adaptive weighting, and signal quality weighting, forming a chain-like dynamic weight allocation mechanism that significantly improves the model's recognition stability and robustness in complex rehabilitation action scenarios. The main processing steps of the action recognition module include the following parts: (1) Independent encoding of multimodal features In the action recognition module, two sets of modal feature encoders, namely a flexible strain modal encoder and an inertial modal encoder, are used to independently encode the sensing signals of different modalities. The flexible strain modal encoder consists of two layers of one-dimensional convolution and one layer of feedforward network, which is used to extract the dynamic change pattern of local deformation of the shoulder to obtain flexible strain modal features; the inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer, which is used to capture the global dynamic features of posture changes to obtain inertial modal features.

[0045] (2) Modal and temporal position coding After completing the independent encoding of modal features, modal and temporal location encoding are performed to obtain flexible strain modal features. Inertial mode characteristics Inject modal identifiers and temporal location information. Specifically, construct a learnable flexible strain modal identifier vector for the two modal features. Inertial mode identification vector To distinguish the sources of features from different modalities; and to introduce sinusoidal temporal position coding. This represents the temporal order and stage changes of the sequence; then, the flexible strain mode identifier vector and the sinusoidal time position code are added step by step to the flexible strain mode feature sequence, and the inertial mode identifier vector and the sinusoidal time position code are added step by step to the inertial mode feature sequence.

[0046] Through the aforementioned integrated injection process, the features of the two modalities are uniformly mapped to the temporal feature space of the same dimension, enabling the model to be modeled under a shared semantic structure and providing a consistent feature form basis for subsequent cross-modal interactive attention, thereby realizing dynamic information association and interaction between modalities.

[0047] (3) Cross-modal dynamic feature interaction Finally, the action recognition module achieves dynamic feature interaction between flexible strain modes and inertial modes through a bidirectional cross-modal interactive attention mechanism. For flexible strain mode features... With inertial modal characteristics The interaction representation between the two is obtained through cross-modal attention computation: ; ; in , , These represent the learnable parameter matrices used to generate queries, keys, and values ​​for flexible strain modes; subspace dimension. Total feature dimension Attention count ; This represents the feature representation of flexible strain modes after semantic enhancement by inertial modes. The feature representation of inertial modes after semantic enhancement by flexible strain modes (i.e., deformation semantic enhancement); , , These represent the learnable mapping matrices used by the inertial modes to generate queries, keys, and values, respectively, with the superscript T indicating the transpose of the matrix.

[0048] After bidirectional cross-modal interaction, modal features are updated via residual connections: ; ; In the formula, This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update.

[0049] This update method preserves the original features of each modality while introducing complementary information, enabling the model to automatically adjust the dominant mode according to the action stage: strengthening the flexible strain mode in deformation-dominant actions such as abduction and flexion-extension, while increasing the dependence on the inertial mode in stages with significant rotation or attitude changes, thus achieving dynamic switching of the dominant mode.

[0050] (4) Time-specific adaptive weighting Even after cross-modal fusion, the contributions of different modalities in the action phase still vary over time. To further address the time-varying characteristics of the action phase, the action recognition module introduces a time-specific gating weighting mechanism at the time series modeling level, i.e., a time-time-specific weighting mechanism, dynamically adjusting the modal fusion ratio based on changes in action phase characteristics. At each time step... Modal weights are calculated through linear mapping and normalization: ; in, Represents the modal weights of flexible strain. Represents the inertial mode weights. This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update. This represents the temporal characteristics after weighted fusion. This indicates a feature concatenation operation. For the mapping matrix, For the bias vector, the coefficients This mechanism balances the smoothness and sensitivity of the weight distribution. It enables the model to enhance flexible strain modes during the initial and exertion phases, and inertial modes during the maintenance phase, achieving adaptive fusion in the time dimension.

[0051] (5) Signal quality weighting To further avoid information bias caused by noise interference in a single mode, this embodiment designs an adaptive fusion mechanism based on signal quality at the global level. This mechanism uses signal-to-noise ratio and channel reliability as core indicators to dynamically adjust the fusion weights of the two modes, resulting in a global feature vector after signal quality-weighted fusion.

[0052] First, the global confidence score of the flexible strain mode is calculated. Global confidence score of inertial modes They are defined as follows: ; in, The signal-to-noise ratio of the flexible strain mode is represented by... The signal-to-noise ratio (SNR) represents the inertial mode and is used to measure signal sharpness. The channel reliability represents the flexible strain mode. The channel reliability, representing the inertial mode, is used to evaluate the stability of the sensor during the current action phase. and These are all learnable parameters used to balance the impact of signal-to-noise ratio and channel reliability on fusion decisions.

[0053] Subsequently, the global confidence score is normalized using the Softmax function to generate the final modal fusion weights, and then weighted fusion modal features based on signal quality are obtained: ; in, For the global fusion weights of flexible strain modes, For the global fusion weights of inertial modes, satisfying ; This represents a feature aggregation operation along the time dimension; This represents the global feature vector after weighted fusion based on signal quality.

[0054] Through this normalization operation design, the system can adaptively allocate weights based on the signal quality and stability of each modality during the action process: when the signal noise of a certain modality is high or its reliability decreases, its weight is automatically reduced; while when another modality exhibits higher signal integrity, its weight is correspondingly increased, thereby achieving signal quality-driven modal balance under different subjects, action amplitudes, and postures. This mechanism effectively improves the stability and robustness of the model in complex action recognition scenarios, while reducing information bias and feature redundancy caused by fixed-ratio fusion.

[0055] (6) Global semantic aggregation and average pooling The global feature vector, obtained through signal quality weighted fusion, is input to a four-layer stacked Transformer encoder. Each Transformer encoder layer includes an eight-head self-attention module and a feedforward network structure. The encoder captures the continuous features of the action's evolution over time and the correlation between consecutive frames through progressive attention calculation and nonlinear feature transformation. It extracts features from the temporal trajectory, stage division, and cross-modal correlations throughout the entire action process, and constructs a unified global semantic representation to achieve feature aggregation and form a fixed-dimensional action representation vector.

[0056] In the feature aggregation stage, the action recognition module employs a dual-channel aggregation strategy consisting of a classification token (CLS Token) and global average pooling. The classification token is placed at the beginning of the sequence to aggregate complete temporal semantics during attention propagation and extract key dynamic features. Global average pooling performs statistical aggregation in the time dimension, preserving the overall trend, modal energy, and state change information of the action process. By synergistically integrating the semantic expressive power of the classification token with the global statistical characteristics of average pooling, the stability and anti-interference ability of feature representation are significantly improved while preserving complete action dynamic information, providing a more reliable feature foundation for subsequent action representation and recognition results.

[0057] After feature aggregation, the semantic vector of the classification identifier is combined with the pooled features to form a fixed-dimensional action representation vector. This vector integrates key features such as cross-modal fusion results, action stage changes, posture trajectory, and local strain signals to uniformly describe the complete action process.

[0058] After the motion representation vector is mapped to a preset motion category space through a fully connected layer, the recognition results of multiple types of shoulder rehabilitation motions can be output.

[0059] In the application phase of this system, a real-time analysis platform was designed to achieve real-time monitoring and feedback of shoulder rehabilitation movements. This platform can be deployed directly after model training and performs real-time processing and classification of multi-channel flexible strain sensing signals and nine-axis inertial measurement signals. The system uses a 3–5s sliding window for data segmentation and performs standardization and filtering operations consistent with the training phase to ensure that real-time data characteristics are consistent with training data. The processed data is input into a multimodal Transformer encoder, which fuses local shoulder deformation and global posture information through a multi-head self-attention mechanism, ultimately outputting the movement category of the current window. The host computer platform supports real-time visualization of movement waveforms and posture changes and can record data for model retraining. This system achieves low-latency recognition and feedback under multimodal signal fusion, providing immediate guidance and personalized optimization for rehabilitation training.

[0060] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-sensor fusion motion recognition system for shoulder rehabilitation assessment, characterized in that, It includes a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module, and a motion recognition module; The flexible strain sensor includes multiple sensing areas distributed around the rotator cuff region, and a connecting area connecting two adjacent sensing areas or the ends of the sensing areas; the multiple sensing areas are respectively located at the top of the rotator cuff, the anterior clavicle, the posterior deltoid muscle, and the axilla. During the multi-channel data acquisition process, the multiple sensing areas are connected in series; the multi-channel data acquisition module acquires the voltage divider signals of the multiple sensing areas, and also simultaneously acquires the inertial measurement data of the inertial measurement unit, and encapsulates the acquired voltage divider signals and inertial data into multi-channel sensing signals; wherein the inertial measurement data includes the accelerometer's rate of change of acceleration, the gyroscope's angular velocity, and the magnetometer's magnetic field change; The data preprocessing module is used to preprocess multi-channel sensor signals to eliminate dimensional differences and numerical biases. The motion recognition module employs a multimodal feature fusion algorithm based on bidirectional cross-modal interactive attention and time-time specific adaptive weighting mechanism for the preprocessed multi-channel sensor signals. This algorithm enables dynamic correlation modeling between flexible strain sensor signals and inertial measurement data within a unified temporal space. The bidirectional cross-modal interactive attention mechanism constructs semantic mapping and complementary relationships between modalities to adaptively adjust the dominance and interaction direction between modalities according to changes in shoulder movement characteristics. For the phased characteristics of shoulder rehabilitation movements, a time-time specific adaptive weighting mechanism dynamically allocates modal weights based on signal strength and quality at different movement stages, thereby achieving the fusion of cross-modal temporal features. The fused temporal features are then processed by a multi-layer feature encoder and a hybrid aggregation module to capture the global and local dependencies of shoulder movements, achieving high-precision recognition of various rehabilitation movements.

2. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The flexible strain sensor is made of a composite material of carbon black and organosilicon elastomer, which is screen-printed onto a fabric substrate and then cured in a vacuum environment. Each sensing area is a linear structure with a winding rectangular trajectory at the turning points; Each connecting region is a sheet-like or strip-like structure; The fabric base area where multiple connection areas are located, combined with the fabric base area where multiple sensing areas are located, is integrated as part of the garment's shoulder sleeve and embedded into the sewn area of ​​the garment's shoulder sleeve.

3. The multi-sensor fusion motion recognition system according to claim 2, characterized in that, The connecting area covers the corresponding fabric base, forming an irregular area.

4. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The data preprocessing module performs preprocessing on the multi-channel sensor signals, including: Adaptive differentiated low-pass filtering adaptively determines the filter order and cutoff frequency based on the spectral distribution characteristics of the sensing signals from different channels, thereby achieving independent filtering processing for the sensing signals from each channel. Data extraction for stable time periods: Construct a composite motion intensity index that integrates acceleration change rate, gyroscope angular velocity, and magnetometer magnetic field change. By combining the composite motion intensity index with an adaptive threshold, identify stable segments in multi-channel sensor signals and extract data for stable time periods. A two-stage extended Kalman filter architecture is used for attitude calculation. In the calibration stage, inertial data in a stationary state is collected and statistically analyzed to estimate the noise characteristics of the inertial measurement unit, thereby achieving noise characteristic modeling and obtaining the process noise matrix used to characterize the gyroscope integration error and zero-bias drift. And the observation noise matrix reflecting the measurement noise characteristics of the accelerometer and magnetometer. During the operation phase, adaptive adjustment of observation noise is achieved by combining composite motion intensity indicators. When the shoulder is in a stable state, the constructed observation noise matrix is ​​used. This represents the observation noise; when the shoulder enters the dynamic movement phase, the observation noise matrix will be... The observed noise is represented by the scaled-up representation.

5. The multi-sensor fusion motion recognition system according to claim 4, characterized in that, When extracting data during stable time periods, the 25th percentile of the composite exercise intensity index sequence is used. As a statistical benchmark, the dynamic threshold Defined as: ； When the composite exercise intensity index is continuously lower than the dynamic threshold and the duration exceeds the first preset time, the corresponding continuous time interval is determined to be a stable time period.

6. The multi-sensor fusion motion recognition system according to claim 4, characterized in that, When extracting data for a stable time period, if the interval between adjacent stable time periods is less than a second preset time, the adjacent stable time periods will be automatically merged into one stable time period.

7. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The processing steps of the action recognition module include: Multimodal features are independently encoded using a flexible strain modal encoder and an inertial modal encoder to independently encode the sensing signals of different modes. The flexible strain modal encoder consists of two layers of one-dimensional convolution and one layer of feedforward network to extract the dynamic change pattern of local deformation of the shoulder and obtain flexible strain modal features. The inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer to capture the global dynamic features of attitude change and obtain inertial modal features. Modal and temporal location encoding injects modal identifiers and temporal location information into flexible strain modal features and inertial modal features respectively, and maps flexible strain modal features and inertial modal features to a temporal feature space of the same dimension. Cross-modal dynamic feature interaction is achieved through a bidirectional cross-modal interactive attention mechanism to realize dynamic feature interaction between flexible strain modes and inertial modes; Time-specific adaptive weighting: The action recognition module introduces a time-specific gating weighting mechanism at the time series modeling level, which dynamically adjusts the modality fusion ratio according to the changes in action stage features. Signal quality weighting is used to design an adaptive fusion mechanism based on signal quality at the global level. With signal-to-noise ratio and channel reliability as the core indicators, the fusion weights of flexible strain mode and inertial mode are dynamically adjusted to obtain a global feature vector after signal quality weighted fusion. Global semantic aggregation and average pooling are used to input the global feature vector, which is obtained by weighted fusion of signal quality, into a multi-layer stacked encoder. Each encoder layer includes a multi-head self-attention module and a feedforward network structure. The encoder captures the continuous features of the action evolution over time and the correlation between previous and subsequent frames through progressive attention calculation and nonlinear feature transformation. It extracts features from the temporal trajectory, stage division and cross-modal correlation of the entire action process and constructs a unified global semantic representation to achieve feature aggregation and form a fixed-dimensional action representation vector. After the action representation vector is mapped to the preset action category space through a fully connected layer, the recognition results of multiple shoulder rehabilitation actions are output.

8. The multi-sensor fusion motion recognition system according to claim 7, characterized in that, The interaction between flexible strain modal features and inertial modal features obtained through cross-modal dynamic feature interaction is represented as follows: ； ； in, Describe the characteristics of flexible strain modes. Indicates inertial modal characteristics; , , These represent the learnable parameter matrices used to generate queries, keys, and values ​​for flexible strain modes, respectively. For subspace dimension, For the total feature dimension, For the number of attention heads; This represents the feature representation of flexible strain modes after semantic enhancement by inertial modes. This represents the feature representation of inertial modes after semantic enhancement by flexible strain modes; , , These represent the learnable mapping matrices used by the inertial modes to generate queries, keys, and values, respectively, with the superscript T indicating the transpose of the matrix; After bidirectional cross-modal interaction, modal features are updated via residual connections: ； ； In the formula, This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update.

9. The multi-sensor fusion motion recognition system according to claim 8, characterized in that, In the time-time specific adaptive weighting process, at each time step Modal weights are calculated through linear mapping and normalization: ； in, Represents the modal weights of flexible strain. Represents the inertial mode weights. This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update. This represents the temporal characteristics after weighted fusion. This indicates a feature concatenation operation. For the mapping matrix, For the bias vector, the coefficients Used to balance the smoothness and sensitivity of the weight distribution.

10. The multi-sensor fusion motion recognition system according to claim 8, characterized in that, The signal quality weighting process includes: First, define the global confidence score for the flexible strain modes. Global confidence score of inertial modes They are respectively: ； in, The signal-to-noise ratio of the flexible strain mode is represented by... The signal-to-noise ratio (SNR) represents the inertial mode. The channel reliability represents the flexible strain mode. The reliability of the channel represents the inertial mode; and All are learnable parameters; Subsequently, the global confidence score is normalized using the Softmax function to generate the final modal fusion weights, and then weighted fusion modal features based on signal quality are obtained: ； in, For the global fusion weights of flexible strain modes, For the global fusion weights of inertial modes, satisfying ; This represents a feature aggregation operation along the time dimension; This represents the global feature vector after weighted fusion based on signal quality.

Images