A multi-sensor fusion motion recognition system for shoulder rehabilitation assessment

Abstract

The application belongs to biological signal recognition and data processing technology, and is a multi-sensor fusion motion recognition system for shoulder rehabilitation evaluation. The system comprises a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module and a motion recognition module. The flexible strain sensor comprises a plurality of sensing areas and connecting areas distributed around the rotator cuff area. The multi-channel data acquisition module acquires multi-channel sensing signals, which are preprocessed by the data preprocessing module. The motion recognition module adopts a multi-modal feature fusion algorithm based on bidirectional cross-modal interaction attention and time period-specific adaptive weight mechanism to realize dynamic correlation modeling between strain sensing signals and inertial measurement data in a unified time sequence space, fuse cross-modal time sequence features, capture global and local dependency of shoulder movement, and recognize multi-class rehabilitation actions. The application effectively overcomes the static limitation of the prior art and significantly improves the accuracy and robustness of shoulder rehabilitation evaluation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of biological signal recognition and data processing, and particularly relates to a multi-sensor fusion motion recognition system for shoulder rehabilitation evaluation. BACKGROUND

[0002] With the acceleration of population aging and the increasing number of patients with upper limb dysfunction caused by diseases such as stroke, periarthritis of shoulder, and rotator cuff tear, shoulder rehabilitation evaluation and training have become a key problem to be solved in the field of rehabilitation medicine. Traditional shoulder function evaluation mainly relies on the subjective judgment of doctors and simple measurement tools, which has limitations such as non-uniform evaluation standard, poor reproducibility, lack of objective quantitative indicators, and difficulty in providing accurate basis for the development of individualized rehabilitation programs.

[0003] In recent years, multi-modal sensor fusion technology has been widely used in the research of upper limb and shoulder motion recognition. The invention patent CN105963926A disclosed on September 28, 2016, proposes a multi-modal hand function rehabilitation evaluation system based on electromyography, force and posture signals, which realizes multi-dimensional motion monitoring through multi-source parallel acquisition, but the fusion method is only feature splicing, lacking of semantic interaction and stage weight self-adaptive mechanism between modalities, which is difficult to depict the complex multi-axis coupling characteristics of shoulder joint. The invention patent CN106691478A disclosed on May 24, 2017, uses pressure, tactile and inertial signals in parallel for hand function evaluation, and uses fixed or semi-static weight for weighted scoring, which can fuse different modalities, but cannot dynamically adjust with the change of motion stage or signal quality, and lacks robustness to noise and poor contact. The invention patent CN120564254A disclosed on August 29, 2025, proposes a motion recognition system and method for upper limb exoskeleton, which fuses visual and inertial data through heterogeneous spatio-temporal graph convolution, and achieves breakthrough in exoskeleton control, but its fusion mechanism relies on fixed graph structure and static attention mechanism, and does not design adaptive weight updating strategy for the stage characteristics and signal volatility of shoulder rehabilitation.

[0004] Existing multi-modal fusion methods generally use simple feature splicing or fixed weighted average, which belongs to static fusion and lacks the mechanism of dynamic weight adjustment according to the motion stage, signal quality and individual differences. This kind of static fusion method cannot establish cross-modal deep association, and cannot fully play the information complementarity between sensors, which has shortcomings in capturing complex motion characteristics such as shoulder multi-axis rotation and muscle strain. SUMMARY

[0005] In order to solve the problems existing in the prior art, the application provides a multi-sensor fusion action recognition system for shoulder rehabilitation evaluation, introduces a bidirectional cross-modal interaction attention and time period specific adaptive weight algorithm, can realize dynamic correlation modeling of flexible strain and inertial signals at the feature layer, and can distribute modal weights in real time according to action stages and signal quality, so as to realize high-precision, real-time and automatic recognition and quantitative evaluation of shoulder rehabilitation actions, effectively overcome the static limitation of the existing fusion technology, and significantly improve the precision and robustness of shoulder rehabilitation evaluation.

[0006] The application achieves the purpose by the following technical scheme: a multi-sensor fusion action recognition system for shoulder rehabilitation evaluation, comprising a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module and an action recognition module.

[0007] The flexible strain sensor comprises a plurality of sensing areas distributed around the rotator cuff area and a connecting area connected between adjacent two sensing areas or the end of the sensing area; the plurality of sensing areas are arranged at the top of the rotator cuff, the anterior clavicle, the posterior deltoid and the axillary position respectively.

[0008] In the multi-channel data acquisition process, the plurality of sensing areas are connected in series; the multi-channel data acquisition module acquires the partial pressure signals of the plurality of sensing areas, also synchronously acquires the inertial measurement data of the inertial measurement unit, encapsulates the acquired partial pressure signals and inertial data into multi-channel sensing signals; wherein the inertial measurement data comprises the acceleration rate of the accelerometer, the angular velocity of the gyroscope and the magnetic field change of the magnetometer.

[0009] The data preprocessing module is used for preprocessing the multi-channel sensing signals to eliminate dimensional differences and numerical biases.

[0010] The action recognition module adopts a multi-modal feature fusion algorithm based on the bidirectional cross-modal interaction attention and the time period specific adaptive weight mechanism to realize dynamic correlation modeling between the flexible strain sensing signals and the inertial measurement data in a unified time sequence space; the bidirectional cross-modal interaction attention mechanism is used to construct the semantic mapping and complementary relationship between the modes, so as to adaptively adjust the dominance and interaction direction between the modes according to the shoulder action feature change; the time period specific adaptive weight mechanism is used to dynamically distribute the modal weights according to the signal intensity and quality at different action stages, so as to realize the fusion of cross-modal time sequence features; the fused time sequence features are processed by a multi-layer feature encoder and a hybrid aggregation module, the global and local dependency relationship of the shoulder movement is captured, and high-precision recognition of multi-class rehabilitation actions is realized.

[0011] Preferably, the flexible strain sensor is prepared by using a carbon black and silicone elastomer composite material, coated on the fabric substrate by screen printing and cured in a vacuum environment.

[0012] Each sensing region is a linear structure, and is a rectangular meandering track at the turning point.

[0013] Each connection region is a sheet structure or a strip structure.

[0014] The fabric substrate region where the plurality of connection regions are located, combined with the fabric substrate region where the plurality of sensing regions are located, as a whole, is a part of the shoulder sleeve of the garment, embedded in the sewing area of the shoulder sleeve of the garment.

[0015] Preferably, the data preprocessing module preprocesses the multi-channel sensing signal, including:

[0016] Adaptive differential low-pass filtering, according to the spectral distribution characteristics of different channel sensing signals, adaptively determines the order and cutoff frequency of the filter, and realizes independent filtering processing of each channel sensing signal;

[0017] Stable time period data extraction, construct a composite motion intensity index that fuses acceleration change rate, gyroscope angular velocity and magnetometer magnetic field change, identify the stable segment in the multi-channel sensing signal through the composite motion intensity index combined with the adaptive threshold, and extract the data of the stable time period;

[0018] Adopting a two-stage extended Kalman filter architecture for attitude calculation, in the calibration stage, collecting inertial data in the static state and performing statistical analysis, estimating the noise characteristics of the inertial measurement unit, realizing noise characteristic modeling, obtaining the process noise matrix reflecting the measurement noise characteristics of the accelerometer and the magnetometer ; In the running stage, combined with the composite motion intensity index, the observation noise is adaptively adjusted, when the shoulder is in a stable state, the observation noise is represented by the constructed observation noise matrix , when the shoulder enters the dynamic motion stage, the observation noise matrix is scaled up to represent the observation noise.

[0019] Further, in the stable time period data extraction, the 25th percentile of the composite motion intensity index sequence is taken as the statistical reference, and the dynamic threshold is defined as:

[0020] ;

[0021] When the composite motion intensity index continuously falls below the dynamic threshold and the duration exceeds the first preset time, it is determined that the corresponding continuous time interval is a stable time period.

[0022] Preferably, the processing procedure of the action recognition module comprises:

[0023] Multi-modal feature independent coding, using a flexible strain modal encoder and an inertial modal encoder, independently encodes the sensing signals of different modalities; the flexible strain modal encoder includes two one-dimensional convolution layers and a feedforward network layer, which is used to extract the dynamic change pattern of the local deformation variable of the shoulder, and obtain the flexible strain modal feature; the inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer, which is used to capture the global dynamic feature of the posture change, and obtain the inertial modal feature;

[0024] Modal and time position coding, respectively, injects modal identification and time position information into the flexible strain modal feature and the inertial modal feature, and uniformly maps the flexible strain modal feature and the inertial modal feature to the same dimension of the time sequence feature space;

[0025] Cross-modal dynamic feature interaction, through a bidirectional cross-modal interaction attention mechanism, the dynamic feature interaction between the flexible strain modal and the inertial modal is realized;

[0026] Time period-specific adaptive weighting, the action recognition module introduces a time-specific gating weighting mechanism at the time sequence modeling level, and dynamically adjusts the modal fusion ratio according to the action phase feature change;

[0027] Signal quality weighting, a signal quality-based adaptive fusion mechanism is designed at the global level, taking the signal-to-noise ratio and the channel reliability as the core indicators, and dynamically adjusting the fusion weights of the flexible strain modal and the inertial modal to obtain a global feature vector after signal quality weighted fusion;

[0028] Global semantic aggregation and average pooling, the global feature vector after signal quality weighted fusion is input into a multi-layer stacked encoder, each layer of the encoder includes a multi-head self-attention module and a feedforward network structure; the encoder captures the continuity feature of the action evolution over time and the correlation between the previous and the next frames through progressive attention calculation and nonlinear feature transformation, extracts the features of the time sequence trajectory, phase division and cross-modal correlation in the whole action process, and constructs a unified global semantic representation, realizes feature aggregation, and forms a fixed-dimensional action representation vector; after the action representation vector is mapped to the preset action category space through a fully connected layer, the recognition results of multiple shoulder rehabilitation actions are output.

[0029] The present application has the following advantages and effects compared with the prior art

[0030] 1. More comprehensive and stable action monitoring: The system adopts flexible strain sensors integrated with clothing, combined with the physiological movement rules of the shoulder, to achieve a high degree of fit with the human body. The sensors are distributed in multiple points along the shoulder sleeve, fully covering the main movement directions of the shoulder joint, significantly reducing the measurement blind area and improving the monitoring accuracy. The sensor material is treated with carbon black-silicone elastomer composite and polydimethylsiloxane (PDMS) curing, with high sensitivity, flexibility, and resistance to moisture and sweat, maintaining signal stability and low noise output in long-term wear and multi-action scenarios.

[0031] 2. More efficient and reliable multi-modal data acquisition: A multi-channel acquisition platform is built around a microcontroller to achieve simultaneous acquisition and real-time transmission of flexible strain and inertial measurement data. The system effectively suppresses noise through analog-to-digital converter sampling and active filtering, and transmits data to the host computer through Bluetooth, ensuring data continuity and accuracy, providing high-quality input for action recognition.

[0032] 3. More adaptive and robust data processing: The system fully introduces adaptive mechanisms in signal processing and attitude solving. Adaptive multi-channel filtering automatically determines the cutoff frequency of each channel, and adaptive stable segment extraction algorithm identifies high-confidence intervals based on dynamic thresholds to ensure data purity and continuity. Attitude solving uses two-stage adaptive extended Kalman filter (AEKF) to update process and observation noise matrix in real time during operation, achieving 50 Hz attitude estimation. The triple adaptive strategy enables the system to dynamically adjust parameters with signal characteristics and environmental changes, significantly improving accuracy, stability, and robustness.

[0033] 4. More accurate multi-modal fusion recognition: The system is based on a multi-modal Transformer architecture with bidirectional cross-modal interaction attention and time-specific adaptive weight mechanism, which dynamically models the association between flexible strain sensor signals and inertial measurement signals in a unified time-space. The algorithm establishes semantic mapping and complementary relationship between the two modalities through bidirectional attention mechanism, and adaptively adjusts information interaction direction and modality dominance according to the action phase, accurately capturing multi-dimensional features of complex shoulder movements. The time-specific adaptive weight module dynamically allocates modality weights based on signal quality and phase changes, allowing the model to strengthen key modality responses in different action phases. The fusion phase combines signal quality-driven global weight optimization to effectively suppress noise interference and enhance model robustness. The hybrid aggregation strategy of classification token (CLS Token) and global average pooling further improves the global feature expression ability, achieving high-precision recognition of subtle actions and multi-axis rotation. The system exhibits excellent stability and generalization performance under complex postures and different subjects, providing accurate and reliable technical support for real-time monitoring and quantitative evaluation of shoulder rehabilitation training.

[0034] 5. The upper computer interacts in real time and intelligently: the platform can directly load the trained model to classify and visually feedback the collected data in real time, and output strain waveform, posture curve and recognition result. The system supports data storage and online iteration of the model, and builds an intelligent closed loop of "collection-processing-recognition-feedback-optimization", providing low-delay, quantitative and expandable intelligent evaluation and individualized intervention support for rehabilitation training. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a structural block diagram of a multi-sensor fusion motion recognition system for shoulder rehabilitation evaluation in the embodiment of the present application;

[0036] Figure 2 is a layout schematic diagram of a flexible strain sensor and an inertial measurement unit in the rotator cuff region;

[0037] Figure 3 is a structural design diagram of a flexible strain sensor;

[0038] Figure 4 is a schematic diagram of a flexible strain sensor printed on a fabric substrate;

[0039] Figure 5 is an equivalent circuit schematic diagram of the sensing area when collecting sensor signals. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in conjunction with the embodiments and drawings, but the implementation of the present application is not limited thereto.

[0041] EMBODIMENT

[0042] The embodiment provides a multi-sensor fusion motion recognition system for shoulder rehabilitation evaluation, which realizes shoulder motion recognition and evaluation based on multi-modal fusion of a flexible strain sensor and an inertial measurement unit (IMU), and is used for motion monitoring and quantitative evaluation in the rehabilitation training process.

[0043] As shown in Figure 1 , Figure 2 , the system of the embodiment specifically includes a flexible strain sensor 1, an inertial measurement unit 2, a multi-channel data acquisition module, and an upper computer, wherein the upper computer is provided with a data preprocessing module and a motion recognition module.

[0044] I. Flexible strain sensor

[0045] The flexible strain sensor includes a plurality of independent sensing areas 11 distributed around the rotator cuff region, and a connecting area 12 connected between adjacent two sensing areas or the end of the sensing area, as shown in Figure 3The sensing areas of the embodiment are provided with four independent sensing areas arranged at the top of the shoulder sleeve, the front clavicle, the rear deltoid, and the underarm position. The arrangement of the sensing areas is based on the physiological structure and movement characteristics of the shoulder joint, and can sense the main movements of the shoulder joint, such as abduction, lifting, forward bending, internal rotation, stretching, and external rotation, so as to comprehensively cover the multi-directional movement of the shoulder, effectively reduce the blind area of recognition, and improve the accuracy and stability of movement monitoring.

[0046] Further, the sensing areas and the connecting areas of the flexible strain sensor can be optimized in length and aspect ratio to reduce the influence of the resistance of the connecting areas on the overall sensing accuracy. The specific optimization design is that the sensing area is designed as a rectangular meandering track with a length of 30 mm, a width of 1.5 mm, and a pitch of 1 mm, and the larger aspect ratio significantly improves the effective resistance change ratio of the sensing area, thereby improving the strain sensitivity; the connecting area is designed as a length of 80 mm and a width of 35 mm, and the smaller aspect ratio effectively reduces the resistance value of the connecting area, thereby reducing the interference of the non-sensing area and improving the measurement accuracy and stability.

[0047] The flexible strain sensor is prepared by using a carbon black and silicone elastomer composite material, coated on a fabric substrate by silk screen printing, and cured and formed in a vacuum environment. The conductive material can be mixed in the following mass ratio: carbon black 3.0 g, silicone elastomer 30.3 g, silicone oil 45.0 g, and curing agent 1.6 g. After stirring at 400 rpm for 30 minutes, the slurry is coated on the fabric substrate by silk screen printing, and cured in a 100°C vacuum oven for 1 hour. See Figure 3 、 Figure 4 Each sensing area 11 is a linear structure, and the linear structure of the sensing area is preferably 30 mm x 1.5 mm and a rectangular meandering track at the turning point, so as to increase the conductive path and strain accumulation effect of the sensing area, reduce interference, and improve signal stability, thereby improving sensitivity and resolution. Each connecting area is a sheet structure or a strip structure, and the connecting area 12 can completely cover the corresponding fabric substrate 13 to form Figure 3 、 Figure 4 a black irregular area in the middle. The black irregular area where the plurality of connecting areas are located, in combination with the fabric substrate area where the plurality of sensing areas are located, is overall a part of the shoulder sleeve of the clothing and is embedded in the stitched area of the shoulder sleeve of the clothing.

[0048] In the clothing process, the flexible strain sensor adopts a flexible cutting piece design that conforms to the shoulder sleeve curve in overall shape, and the edge shape matches the garment splicing line, which can be naturally embedded in the sewing area of the clothing shoulder sleeve, without being directly compressed or cut by the sewing line, avoiding performance damage. The high-elasticity compression fabric is used as the substrate, which can make the sensor closely conform to the shoulder surface and maintain a stable position during movement, reducing measurement errors caused by sliding or wrinkling. It takes into account physiological adaptability, comfort and engineering feasibility, ensuring the stability of data acquisition and the practicality of wearing.

[0049] To further improve the performance, the prepared flexible strain sensor is immersed in a 1% PDMS (polydimethylsiloxane) ethanol solution and cured step by step to form a uniform and dense protective layer on the surface. The protective layer has excellent electrical insulation and flexibility, which can effectively isolate humidity and sweat interference, stabilize the conductive network, reduce hysteresis effect, and significantly improve signal stability and long-term durability.

[0050] II. Multi-channel data acquisition module

[0051] The multi-channel data acquisition module takes a microcontroller as the core, and through four channels ADC1-ADC4 of an analog-to-digital converter (ADC), it acquires the voltage division signals (i.e. strain electric signals) of four independent sensing areas, and synchronously obtains the inertial measurement data of the inertial measurement unit through the IIC bus (Inter-Integrated Circuit bus), including the acceleration rate of the accelerometer, the angular velocity of the gyroscope, and the magnetic field change of the magnetometer, realizing real-time fusion of dual-mode signals. The flexible strain sensor is used to perceive the local deformation of the shoulder, and the inertial measurement unit provides the overall posture information, and the combination of the two can realize comprehensive monitoring of shoulder movement. To ensure signal quality, an active low-pass filter with a cutoff frequency of 5 Hz is connected in series at the output end of the sensor to suppress high-frequency noise. The microcontroller connects Bluetooth through a serial port, and encapsulates the acquired voltage division signals and inertial data into multi-channel sensing signals at a sampling frequency of 50 Hz, and transmits the multi-channel sensing signals to the data preprocessing module of the upper computer after analog low-pass filtering and synchronous timestamp encapsulation, realizing real-time fusion of local and overall movement information of the shoulder.

[0052] In the multi-channel data acquisition process, the four sensing areas are connected in series and constitute a voltage division circuit with a reference resistor , which reflects the local deformation in real time through voltage changes, as shown in Figure 5 . The series structure can superimpose signal strength and suppress noise interference, ensuring the stability and reliability of the measurement. The reference resistor can be 500 Ω.

[0053] The equivalent resistance value of the sensing area of the flexible strain sensor increases with the increase of tensile deformation and decreases with the increase of compression deformation, a voltage dividing circuit is formed with the reference resistance, the resistance value change of each sensing area in the flexible strain sensor is calculated based on the voltage dividing principle and the reference resistance, and the calculation formula is as follows:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] wherein represents an external power supply, represents the voltage of the four connection areas of the flexible strain sensor to ground after being sampled by an analog-to-digital converter ADC and taking 20-point average; and the four resistance values ∼ characterize the deformation degree of the four sensing areas of the flexible strain sensor under the current posture.

[0059] III. Data preprocessing module

[0060] The data preprocessing module is used for preprocessing the multi-channel sensing signals, including adaptive differentiated Butterworth low-pass filtering, stable time period data extraction and extended Kalman filter attitude solution, for filtering noise, fusing multi-source data and extracting stable time period data, adaptively adjusting observation noise, to eliminate dimension difference and numerical bias, and improve the accuracy and stability of subsequent analysis and recognition. The processing process of the data preprocessing module mainly includes the following parts:

[0061] (1) Adaptive differentiated low-pass filtering

[0062] For multi-channel sensing signals, the data preprocessing module adopts a Butterworth low-pass filtering method combining differentiation and adaptive mechanism. According to the spectral distribution characteristics of different channel sensing signals, the order and cutoff frequency of the filter are adaptively determined, realizing independent filtering processing of each channel sensing signal, so as to effectively suppress high-frequency noise while maximizing the retention of low-frequency effective components of shoulder movement.

[0063] The data preprocessing module first performs fast Fourier transform (FFT) on each channel sensing signal and calculates the power spectral density:

[0064] ;

[0065] wherein, represents the power spectral density of the signal at frequency point , denotes the frequency index in the fast Fourier transform corresponding frequency point, is the complex frequency spectrum sequence after fast Fourier transform of the original time sequence (i.e. original channel sensing signal); denotes the normalization coefficient, is the number of FFT sampling points, is the sampling frequency.

[0066] The power spectrum curvature function is defined in the logarithmic coordinate for detecting the energy change inflection point:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] In the preset frequency band [0.2, 10] Hz, the frequency corresponding to the maximum point of the curvature is selected as the adaptive cutoff frequency:

[0072] ;

[0073] wherein, denotes the power spectrum curvature value for detecting the energy change inflection point, measuring the local bending degree of the curve; denotes the first-order difference approximation (slope) of the power spectrum curve in the logarithmic coordinate, denotes the second-order difference approximation (curvature change rate) of the power spectrum curve in the logarithmic coordinate; denotes the frequency index corresponding frequency point in the logarithmic coordinate, denotes the frequency index corresponding frequency point in the logarithmic coordinate, denotes the frequency index corresponding frequency point in the logarithmic coordinate; denotes the logarithmic power spectrum value of the signal at the frequency index corresponding frequency point, denotes the logarithmic power spectrum value of the signal at the frequency index corresponding frequency point, denotes the logarithmic power spectrum value of the signal at the frequency index corresponding frequency point; denotes the frequency and power spectrum curve point in the corresponding logarithmic coordinate system, for obtaining the smooth energy change trend; denotes the adaptive cutoff frequency.

[0074] The data preprocessing module can automatically adjust the filtering parameters according to the signal energy distribution, without manual threshold setting. Experimental results show that the optimal cutoff frequencies of the flexible strain sensor, accelerometer, gyroscope and magnetometer are 0.5 Hz, 2 Hz, 3 Hz and 1 Hz respectively, which can effectively balance the signal smoothness and dynamic response. The cutoff frequencies of each channel sensing signal are set adaptively by FFT, specifically, a second-order low-pass filter structure is used for the multi-channel sensing signal detected by the flexible strain sensor to maintain smoothness, and a fourth-order filter structure is used for the overall attitude information detected by the inertial measurement unit to enhance the cutoff characteristics; in addition, in order to ensure the consistency of signal phase and time alignment, bidirectional zero-phase filtering is used, that is, zero-phase delay is realized by bidirectional operation in the filtering process.

[0075] (2) Stable time period data extraction

[0076] To eliminate short-term disturbances and improve the reliability of attitude calculation, the data preprocessing module constructs a composite motion intensity index that integrates the acceleration rate of change, gyroscope angular velocity and magnetometer magnetic field change. The composite motion intensity index integrates the acceleration, angular velocity and magnetic field change characteristics of the inertial measurement unit, and comprehensively represents the overall motion state of the shoulder. By combining the adaptive threshold with the composite motion intensity index, the stable segments in the multi-channel sensing signal are identified, and then the data of the stable time period (which can be referred to as stable segment) is extracted, realizing effective selection of low-noise and high-confidence data.

[0077] For each sampling time , three types of feature quantities are defined: acceleration rate of change , angular velocity amplitude and magnetic field rate of change ; the standard deviations of the three types of feature quantities, i.e. acceleration rate of change, angular velocity amplitude and magnetic field rate of change, are calculated respectively , , , and normalized to obtain the composite motion intensity index :

[0078] ;

[0079] The composite motion intensity index can comprehensively reflect the overall motion intensity of the shoulder at any time, and the lower the value, the more stable the attitude.

[0080] To realize adaptive judgment under different experimental conditions, this embodiment introduces a dynamic threshold mechanism based on percentile to determine whether a stable time period appears in the multi-channel sensing signal. The 25th percentile of the composite motion intensity index sequence is taken as the statistical reference, and the dynamic threshold is defined as:

[0081] ;

[0082] When the composite motion intensity index is continuously lower than the dynamic threshold and the duration exceeds the first preset time (for example, 5s), it is determined that the corresponding continuous time interval is a stable time period.

[0083] To avoid excessive segmentation caused by short-term fluctuations or slight posture adjustment, the embodiment further introduces a timing constraint mechanism, requiring the duration of the stable time period to be no less than the first preset time (for example, 5s), to filter false stable segments caused by sensor noise or short-term stillness; when the interval between adjacent stable time periods is less than the second preset time (for example, 0.3s), the adjacent stable time periods are automatically merged into one stable time period to reduce excessive segmentation and maintain timing continuity. The data of the stable time period will be labeled and saved independently for subsequent posture solving and rehabilitation motion recognition. The stable time period recognition mechanism used in this embodiment can automatically adjust the threshold according to the signal fluctuation characteristics, without the need for manual threshold setting, and maintains stable recognition performance under different subjects, motion amplitudes and environmental conditions.

[0084] (3) Adaptive Extended Kalman Filter

[0085] Finally, the data preprocessing module uses a two-stage extended Kalman filter architecture for posture solving, dynamically updating the observation noise matrix during the filtering process according to the motion state to improve the robustness and stability of the posture estimation.

[0086] The first stage is the calibration stage, which collects inertial data in the static state and performs statistical analysis to estimate the noise characteristics of each flexible strain sensor and inertial measurement unit, realize noise characteristic modeling, obtain the process noise matrix for characterizing the gyroscope integral error and zero bias drift, and the observation noise matrix reflecting the measurement noise characteristics of the accelerometer and the magnetometer.

[0087] For the gyroscope, the theoretical output in the static state should be close to zero angular velocity, assuming that the collected N samples are , whose mean is , then the gyroscope noise variance calculation formula is:

[0088] ;

[0089] For the accelerometer, the theoretical output in the static state should be , , where g is the acceleration of gravity, assuming that the collected N samples are , whose mean is , then the accelerometer noise variance calculation formula is:

[0090] ;

[0091] Based on the aforementioned noise variance, the process noise matrix is ​​constructed by adaptively setting the variance of static data. With observation noise matrix :

[0092] ;

[0093] Among them, the process noise matrix Characterizing gyroscope integration error and zero bias drift, observation noise matrix This reflects the measurement noise characteristics of the accelerometer and magnetometer. Represents a 3x3 identity matrix. This represents a 2x2 identity matrix.

[0094] Simultaneously, during the calibration phase, gyroscope zero-bias calibration, accelerometer deviation compensation, and initial attitude estimation are completed. After calibration, the second phase, the operational phase, begins, where attitude calculation is performed using adaptive parameters.

[0095] During the operation phase, combined with composite exercise intensity indicators To achieve adaptive adjustment of observation noise. When the shoulder is in a stable state ( When the observation data has high reliability, the constructed observation noise matrix is ​​used. This indicates observed noise to enhance posture correction; when the shoulder enters the dynamic movement phase ( When observing the noise matrix, The observation noise is represented by a scaled-up image to reduce the impact of high-frequency disturbances on filter stability. Observation noise The adjustment rules are as follows:

[0096] ;

[0097] in, To amplify the observation noise, the observation noise matrix is ​​increased when the motion intensity exceeds the dynamic threshold. This is multiplied by a factor of 1 to reduce the weight of inertial measurements affected by dynamic disturbances in attitude fusion. In this embodiment, The preferred value is .

[0098] In this embodiment, a two-stage extended Kalman filter architecture is used in the data preprocessing module for attitude calculation. The observation noise matrix R is updated adaptively, and data extraction during the stable time period is combined to enhance the filter's correction capability under static conditions (i.e., in a stable state) and maintain prediction smoothness during the dynamic motion phase, thereby achieving real-time 50 Hz calculation of the shoulder's three-dimensional attitude, balancing high accuracy and low latency.

[0099] The embodiment synchronously samples and pre-processes the flexible strain sensing signals and the inertial measurement signals by a multi-channel data acquisition module and a data pre-processing module, and eliminates dimensional differences and numerical biases. The multi-channel data acquisition module acquires 4-channel flexible strain sensing signals from the flexible strain sensor and 9-channel attitude data, i.e., inertial measurement signals, from the inertial measurement unit. All signals are segmented in a sliding window (length 50 frames, overlap rate 50%) and only pure samples with consistent labels are retained for subsequent modeling.

[0100] IV. Action recognition module

[0101] The action recognition module uses a multi-modal feature fusion algorithm based on bidirectional cross-modal interaction attention and time period-specific adaptive weight mechanism to model the dynamic correlation between the flexible strain sensing signals and the inertial measurement data in a unified time sequence space. The bidirectional cross-modal interaction attention mechanism is used to construct the semantic mapping and complementary relationship between modalities, so as to adaptively adjust the dominance and interaction direction between modalities according to the changes in shoulder action features. For the phased features of shoulder rehabilitation actions, the time period-specific adaptive weight mechanism is used to dynamically allocate modal weights according to signal strength and quality at different action stages, thereby realizing the fusion of cross-modal time sequence features. The fused time sequence features are processed by a multi-layer feature encoder and a hybrid aggregation module, which can capture the global and local dependency relationships of shoulder movements and realize high-precision recognition of multiple rehabilitation actions. The multi-layer feature encoder can be implemented by a Transformer encoder.

[0102] Unlike traditional one-way or static fusion methods, the fusion algorithm of the action recognition module can adaptively adjust the dominance and interaction direction of the two modalities according to the action stage and signal characteristic changes, thereby realizing multi-level information complementation and dynamic balance. The entire action recognition process includes three core steps of bidirectional cross-modal feature interaction, time period-specific adaptive weighting, and signal quality weighting, forming a chain-like dynamic weight distribution mechanism, which significantly improves the recognition stability and robustness of the model in complex rehabilitation action scenarios. The main processing process of the action recognition module includes the following parts:

[0103] (1) Independent encoding of multi-modal features

[0104] In the action recognition module, two sets of modal feature encoders, i.e., a flexible strain modal encoder and an inertial modal encoder, are respectively used to independently encode the sensing signals of different modalities. The flexible strain modal encoder includes two one-dimensional convolution layers and one feedforward network, which is used to extract the dynamic change pattern of the local deformation variable of the shoulder and obtain the flexible strain modal feature. The inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer, which is used to capture the global dynamic characteristics of the posture change and obtain the inertial modal feature.

[0105] (2) Modal and time position encoding

[0106] After completing the independent encoding of the modal features, modal and time position encoding is performed on the flexible strain modal feature and the inertial modal feature to inject modal identifiers and time position information. Specifically, a learnable flexible strain modal identifier vector and an inertial modal identifier vector are constructed for the two modal features to distinguish the sources of different modal features. A sinusoidal time position encoding is introduced to represent the time order and stage change of the sequence. The flexible strain modal identifier vector and the sinusoidal time position encoding are fused into the flexible strain modal feature sequence by adding them step by step, and the inertial modal identifier vector and the sinusoidal time position encoding are fused into the inertial modal feature sequence by adding them step by step.

[0107] Through the above fusion injection process, the two modal features are uniformly mapped to the same dimensional time sequence feature space, enabling the model to be modeled under a shared semantic structure and providing a consistent feature form basis for subsequent cross-modal interaction attention, realizing dynamic information association and interaction between modalities.

[0108] (3) Cross-modal dynamic feature interaction

[0109] Finally, the action recognition module realizes the dynamic feature interaction between the flexible strain modal and the inertial modal through a bidirectional cross-modal interaction attention mechanism. For the flexible strain modal feature and the inertial modal feature , the interaction representation of the two is obtained through cross-modal attention calculation:

[0110] ;

[0111] ;

[0112] wherein , , denote the learnable parameter matrices of flexible strain modality for generating query Query, key Key and value Value, respectively; subspace dimension total feature dimension number of attention heads ; denote the feature representation of flexible strain modality after being semantically enhanced by inertial modality, denote the feature representation of inertial modality after being semantically enhanced by flexible strain modality (i.e., deformation variable semantic enhancement); denote the learnable mapping matrices of inertial modality for generating query Query, key Key and value Value, respectively, and the superscript T denotes the transpose of the matrix.

[0113] After bidirectional cross-modal interaction, the modal features are updated through residual connection:

[0114] ;

[0115] ;

[0116] In the formula, denote the flexible strain modality feature updated by residual connection, denote the inertial modality feature updated by residual connection.

[0117] This updating method not only preserves the original features of each modality, but also introduces complementary information, so that the model can automatically adjust the dominant modality according to the action phase: in the action phase dominated by deformation such as abduction and flexion, the flexible strain modality is strengthened, while in the action phase with significant rotation or posture change, the dependence on the inertial modality is enhanced, realizing the dynamic switching of the dominant modality.

[0118] (4) Time period-specific adaptive weighting

[0119] After cross-modal fusion, the contribution of different modalities in the action phase still changes over time. To further cope with the time-varying characteristics of the action phase, the action recognition module introduces a time-specific gating weighting mechanism, i.e., a time period-specific weighting mechanism, at the time series modeling level, which dynamically adjusts the modal fusion ratio according to the feature changes in the action phase. At each time step , the modal weight is calculated through linear mapping and normalization:

[0120] ;

[0121] Among them, denote the flexible strain modality weight, denote the inertial modality weight, denote the flexible strain modality feature updated by residual connection, ​​denotes the updated inertia modal feature, denotes the weighted fused temporal feature, denotes the feature concatenation operation, is a mapping matrix, is a bias vector, and is used to balance the smoothness and sensitivity of the weight distribution. This mechanism enables the model to strengthen the flexible strain modal in the initial and force stage, and to enhance the inertia modal in the maintenance stage, achieving adaptive fusion in the time dimension.

[0122] (5) Signal quality weighting

[0123] To further avoid information bias caused by single modal interference by noise, the embodiment designs an adaptive fusion mechanism based on signal quality at the global level. This mechanism dynamically adjusts the fusion weights of the two modalities based on the signal-to-noise ratio and channel reliability as the core indicators, and obtains the global feature vector after signal quality weighted fusion.

[0124] First, the global confidence score of the flexible strain modal is denoted as and the global confidence score of the inertia modal is denoted as , which are defined as:

[0125]

[0126] wherein, denotes the signal-to-noise ratio of the flexible strain modal, denotes the signal-to-noise ratio of the inertia modal, used to measure the signal clarity; denotes the channel reliability of the flexible strain modal, denotes the channel reliability of the inertia modal, used to evaluate the stability of the sensor in the current action stage; and are learnable parameters used to balance the influence of signal-to-noise ratio and channel reliability in the fusion decision.

[0127] Subsequently, the global confidence is normalized using the Softmax function to generate the final modal fusion weight, and further obtain the weighted fusion modal feature based on signal quality:

[0128]

[0129] wherein, is the global fusion weight of the flexible strain modal, is the global fusion weight of the inertia modal, satisfying ; denotes the feature aggregation operation in the time dimension; denotes the global feature vector after signal quality weighted fusion.​​

[0130] Through this normalization operation design, the system can adaptively assign weights according to the signal quality and stability of each modality during the action process: when the signal noise of a certain modality is high or the reliability decreases, its weight automatically decreases; and when another modality shows higher signal integrity, its weight is correspondingly increased, thereby realizing signal quality-driven modality balance under different subjects, action amplitudes and posture conditions. This mechanism effectively improves the stability and robustness of the model in complex action recognition scenarios, while reducing information bias and feature redundancy caused by fixed proportion fusion.

[0131] (6) Global semantic aggregation and average pooling

[0132] The global feature vector fused based on signal quality is input into a four-layer stacked Transformer encoder. Each layer of the Transformer encoder includes an eight-head self-attention module and a feedforward network structure. The encoder captures the continuity features of the action over time and the correlation between the previous and subsequent frames through progressive attention calculation and nonlinear feature transformation, extracts features of the time sequence, stage division and cross-modal correlation of the entire action process, and constructs a unified global semantic representation, realizes feature aggregation, and forms a fixed-dimensional action representation vector.

[0133] In the feature aggregation stage, the action recognition module adopts a dual-channel aggregation strategy composed of a classification identifier (CLS Token) and global average pooling. The classification identifier is set at the beginning of the sequence, used to converge the complete temporal semantic in the attention propagation and extract key dynamic features; the global average pooling performs statistical aggregation in the time dimension, retaining the overall trend, modality energy and state change information of the action process. By integrating the semantic expression ability of the classification identifier and the global statistical characteristics of the average pooling, the stability and anti-interference ability of feature expression are significantly improved while retaining the complete action dynamic information, providing a more reliable feature basis for subsequent action representation and recognition results.

[0134] After feature aggregation, the semantic vector of the classification identifier and the pooled features are combined to form a fixed-dimensional action representation vector. This vector integrates key features such as cross-modal fusion results, action stage changes, posture motion trajectories and local strain signals, and is used to uniformly describe the complete action process.

[0135] The action representation vector is mapped to the preset action category space through a fully connected layer, and the recognition results of multiple shoulder rehabilitation actions can be output.

[0136] In the application stage of the system, a real-time analysis platform of the upper computer is designed to realize real-time monitoring and feedback of shoulder rehabilitation movements. The platform can be directly deployed after the model training is completed, and can process and classify the multi-channel flexible strain signals and nine-axis inertial measurement signals in real time. The system uses a 3-5 s sliding window to segment the data, and performs the same standardization and filtering operations as in the training stage to ensure that the real-time data features are consistent with the training data. The processed data is input into the multi-modal Transformer encoder, which fuses the local deformation and global posture information of the shoulder through the multi-head self-attention mechanism, and finally outputs the action category of the current window. The upper computer platform supports real-time visualization of action waveforms and posture changes, and can record data for model retraining. The system realizes low-latency recognition and feedback under multi-modal signal fusion, providing immediate guidance and personalized optimization for rehabilitation training.

[0137] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be within the scope of protection of the present application.

Claims

1. A multi-sensor fusion motion recognition system for shoulder rehabilitation assessment, characterized in that, It includes a flexible strain sensor, an inertial measurement unit, a multi-channel data acquisition module, a data preprocessing module, and a motion recognition module; The flexible strain sensor includes multiple sensing areas distributed around the rotator cuff region, and a connecting area connecting two adjacent sensing areas or the ends of the sensing areas; the multiple sensing areas are respectively located at the top of the rotator cuff, the anterior clavicle, the posterior deltoid muscle, and the axilla. During the multi-channel data acquisition process, the multiple sensing areas are connected in series; the multi-channel data acquisition module acquires the voltage divider signals of the multiple sensing areas, and also simultaneously acquires the inertial measurement data of the inertial measurement unit, and encapsulates the acquired voltage divider signals and inertial data into multi-channel sensing signals; wherein the inertial measurement data includes the accelerometer's rate of change of acceleration, the gyroscope's angular velocity, and the magnetometer's magnetic field change; The data preprocessing module is used to preprocess multi-channel sensor signals to eliminate dimensional differences and numerical biases. The motion recognition module employs a multimodal feature fusion algorithm based on bidirectional cross-modal interactive attention and time-time specific adaptive weighting mechanism for the preprocessed multi-channel sensor signals. This algorithm enables dynamic correlation modeling between flexible strain sensor signals and inertial measurement data within a unified temporal space. The bidirectional cross-modal interactive attention mechanism constructs semantic mapping and complementary relationships between modalities to adaptively adjust the dominance and interaction direction between modalities according to changes in shoulder movement characteristics. For the phased characteristics of shoulder rehabilitation movements, a time-time specific adaptive weighting mechanism dynamically allocates modal weights based on signal strength and quality at different movement stages, thereby achieving the fusion of cross-modal temporal features. The fused temporal features are then processed by a multi-layer feature encoder and a hybrid aggregation module to capture the global and local dependencies of shoulder movements, achieving high-precision recognition of multiple types of rehabilitation movements. The processing steps of the action recognition module include: Multimodal features are independently encoded using a flexible strain modal encoder and an inertial modal encoder to independently encode the sensing signals of different modes. The flexible strain modal encoder consists of two layers of one-dimensional convolution and one layer of feedforward network to extract the dynamic change pattern of local deformation of the shoulder and obtain flexible strain modal features. The inertial modal encoder is based on a multi-head self-attention structure and a linear projection layer to capture the global dynamic features of attitude change and obtain inertial modal features. Modal and temporal location encoding injects modal identifiers and temporal location information into flexible strain modal features and inertial modal features respectively, and maps flexible strain modal features and inertial modal features to a temporal feature space of the same dimension. Cross-modal dynamic feature interaction is achieved through a bidirectional cross-modal interactive attention mechanism to realize dynamic feature interaction between flexible strain modes and inertial modes; Time-specific adaptive weighting: The action recognition module introduces a time-specific gating weighting mechanism at the time series modeling level, which dynamically adjusts the modality fusion ratio according to the changes in action stage features. Signal quality weighting is used to design an adaptive fusion mechanism based on signal quality at the global level. With signal-to-noise ratio and channel reliability as the core indicators, the fusion weights of flexible strain mode and inertial mode are dynamically adjusted to obtain a global feature vector after signal quality weighted fusion. Global semantic aggregation and average pooling are used to input the global feature vector, which is obtained by weighted fusion of signal quality, into a multi-layer stacked encoder. Each encoder layer includes a multi-head self-attention module and a feedforward network structure. The encoder captures the continuous features of the action evolution over time and the correlation between previous and subsequent frames through progressive attention calculation and nonlinear feature transformation. It extracts features from the temporal trajectory, stage division and cross-modal correlation of the entire action process and constructs a unified global semantic representation to achieve feature aggregation and form a fixed-dimensional action representation vector. After the action representation vector is mapped to the preset action category space through a fully connected layer, the recognition results of multiple shoulder rehabilitation actions are output.

2. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The flexible strain sensor is made of a composite material of carbon black and organosilicon elastomer, which is screen-printed onto a fabric substrate and then cured in a vacuum environment. Each sensing area is a linear structure with a winding rectangular trajectory at the turning points; Each connecting region is a sheet-like or strip-like structure; The fabric base area where multiple connection areas are located, combined with the fabric base area where multiple sensing areas are located, is integrated as part of the garment's shoulder sleeve and embedded into the sewn area of ​​the garment's shoulder sleeve.

3. The multi-sensor fusion motion recognition system according to claim 2, characterized in that, The connecting area covers the corresponding fabric base, forming an irregular area.

4. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The data preprocessing module performs preprocessing on the multi-channel sensor signals, including: Adaptive differentiated low-pass filtering adaptively determines the filter order and cutoff frequency based on the spectral distribution characteristics of the sensing signals from different channels, thereby achieving independent filtering processing for the sensing signals from each channel. Data extraction for stable time periods: Construct a composite motion intensity index that integrates acceleration change rate, gyroscope angular velocity, and magnetometer magnetic field change. By combining the composite motion intensity index with an adaptive threshold, identify stable segments in multi-channel sensor signals and extract data for stable time periods. A two-stage extended Kalman filter architecture is used for attitude calculation. In the calibration stage, inertial data in a stationary state is collected and statistically analyzed to estimate the noise characteristics of the inertial measurement unit, thereby achieving noise characteristic modeling and obtaining the process noise matrix used to characterize the gyroscope integration error and zero-bias drift. And the observation noise matrix reflecting the measurement noise characteristics of the accelerometer and magnetometer. During the operation phase, adaptive adjustment of observation noise is achieved by combining composite motion intensity indicators. When the shoulder is in a stable state, the constructed observation noise matrix is ​​used. This represents the observation noise; when the shoulder enters the dynamic movement phase, the observation noise matrix will be... The observed noise is represented by the scaled-up representation.

5. The multi-sensor fusion motion recognition system according to claim 4, characterized in that, When extracting data during stable time periods, the 25th percentile of the composite exercise intensity index sequence is used. As a statistical benchmark, the dynamic threshold Defined as: ； When the composite exercise intensity index is continuously lower than the dynamic threshold and the duration exceeds the first preset time, the corresponding continuous time interval is determined to be a stable time period.

6. The multi-sensor fusion motion recognition system according to claim 4, characterized in that, When extracting data for a stable time period, if the interval between adjacent stable time periods is less than a second preset time, the adjacent stable time periods will be automatically merged into one stable time period.

7. The multi-sensor fusion motion recognition system according to claim 1, characterized in that, The interaction between flexible strain modal features and inertial modal features obtained through cross-modal dynamic feature interaction is represented as follows: ； ； in, Describe the characteristics of flexible strain modes. Indicates inertial modal characteristics; , , These represent the learnable parameter matrices used to generate queries, keys, and values ​​for flexible strain modes, respectively. For subspace dimension, For the total feature dimension, For the number of attention heads; This represents the feature representation of flexible strain modes after being enhanced with inertial mode semantics. This represents the feature representation of inertial modes after semantic enhancement by flexible strain modes; , , These represent the learnable mapping matrices used by the inertial modes to generate queries, keys, and values, respectively, with the superscript T indicating the transpose of the matrix; After bidirectional cross-modal interaction, modal features are updated via residual connections: ； ； In the formula, This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update.

8. The multi-sensor fusion motion recognition system according to claim 7, characterized in that, In the time-time specific adaptive weighting process, at each time step Modal weights are calculated through linear mapping and normalization: ； in, Represents the modal weights of flexible strain. Represents the inertial mode weights. This represents the flexible strain mode characteristics after the residual connection update. This represents the inertial mode characteristics after the residual connection update. This represents the temporal characteristics after weighted fusion. This indicates a feature concatenation operation. For the mapping matrix, For the bias vector, the coefficients Used to balance the smoothness and sensitivity of the weight distribution.

9. The multi-sensor fusion motion recognition system according to claim 7, characterized in that, The signal quality weighting process includes: First, define the global confidence score for the flexible strain modes. Global confidence score of inertial modes They are respectively: ； in, The signal-to-noise ratio of the flexible strain mode is represented. The signal-to-noise ratio (SNR) represents the inertial mode. The channel reliability represents the flexible strain mode. The reliability of the channel represents the inertial mode; and All are learnable parameters; Subsequently, the global confidence score is normalized using the Softmax function to generate the final modal fusion weights, and then weighted fusion modal features based on signal quality are obtained: ； in, For the global fusion weights of flexible strain modes, For the global fusion weights of inertial modes, satisfying ; This represents a feature aggregation operation over the time dimension; This represents the global feature vector obtained by weighted fusion based on signal quality.

Images