Knee joint multi-sensor fusion perception method and system based on flexible fabric

Abstract

The present application relates to sensing technology, specifically to a knee joint multi-sensor fusion sensing method and system based on flexible fabric. The system includes a flexible fabric sensor array, a signal acquisition module, a fusion sensing module and a classification module. The sensor units of the flexible fabric sensor array are evenly distributed in each sub-area of the front part of the knee joint. The signal acquisition module is connected to the flexible fabric sensor array for real-time acquisition of multi-channel sensing signals and transmission to the fusion sensing module. The fusion sensing module extracts local time sequence features using a convolutional neural network, fuses global time sequence features through a bidirectional long short-term memory network, and weights key time steps by an attention module to extract multi-dimensional features, realizing recognition of sitting leg lifting, standing and walking. The classification module is connected to the fusion sensing module and is used to output the classification results of the knee joint movement mode. The present application realizes efficient feature extraction and adaptive dynamic fusion of multi-channel signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensing, in particular to a knee joint multi-sensor fusion sensing method and system based on flexible fabric. BACKGROUND

[0002] With the rapid development of wearable technology and biomedical monitoring means, real-time monitoring of knee joint motion state has gradually become a research hotspot in rehabilitation training, sports performance evaluation and health management of the aging population. The existing knee joint monitoring methods mainly rely on imaging analysis (such as X-ray, MRI) or three-dimensional motion capture systems based on laboratory environment, which can provide relatively accurate data, but the equipment is expensive, bulky and the application scene is limited, which is difficult to popularize to daily and real-time use. Therefore, to realize the low-cost, comfortable and high-precision monitoring of the knee joint motion state has become a key problem to be solved in this field.

[0003] In recent years, wearable sensor technology has been gradually applied to knee joint monitoring. The invention patent CN105852866A disclosed on August 17, 2016 proposes a wearable sensing system for knee joint adduction torque measurement, which collects plantar pressure and predicts joint mechanical parameters through a neural network model, but it mainly relies on a single pressure signal and the monitoring dimension is limited; the invention patent CN112336339A disclosed on February 9, 2021 discloses a knee joint stress monitoring sports pants based on flexible sensing, which integrates a conductive coating flexible sensor in the fabric, improving the wearing comfort, but its signal processing and data fusion capability are still insufficient.

[0004] Overall, the existing technology still has the following defects: first, most devices only rely on accelerometers, pressure sensors or other single modal sensing elements, which are difficult to fully reflect the multi-dimensional motion and stress characteristics of the knee joint; second, some systems use rigid sensing units or simple fixed structures, which are not flexible and comfortable, and the long-term wearing experience is poor; third, multi-sensor systems are mostly modularized and stacked, lacking flexible and integrated design at the fabric level, and lacking efficient multi-modal fusion mechanism in the data processing link, resulting in insufficient accuracy and robustness of the monitoring results; fourth, some monitoring devices rely on external terminals for processing, which is limited in real-time and portability, and is difficult to meet the needs of daily rehabilitation and sports training.

[0005] In summary, the existing knee joint monitoring methods have obvious deficiencies in flexible integration, multi-sensor fusion and real-time application, and it is urgent to propose a knee joint multi-sensor fusion sensing method and system based on intelligent flexible fabric to improve the wearing comfort, monitoring accuracy and application universality. SUMMARY

[0006] The application provides a knee joint multi-sensor fusion sensing method and system based on a flexible fabric, which realizes real-time monitoring and fusion sensing identification of multi-dimensional motion states such as knee joint flexion, rotation and lateral swing by efficient integration of a flexible fabric multi-sensor array, combination of signal acquisition and deep learning fusion algorithm, thereby meeting the multi-scene application requirements of rehabilitation training, motion monitoring and health management, improving the wearing comfort while improving the precision of multi-sensor data fusion.

[0007] In one aspect, the application provides a knee joint multi-sensor fusion sensing system based on a flexible fabric, which comprises a flexible fabric sensor array, a signal acquisition module, a fusion sensing module and a classification module.

[0008] The flexible fabric sensor array comprises a plurality of independent and functionally non-interfering flexible fabric sensing units, which are uniformly distributed in each subzone of the front part of the knee joint, i.e. the upper edge of the patella, the quadriceps tendon insertion area, the tibial tuberosity and the patellar ligament area.

[0009] The signal acquisition module is connected to the flexible fabric sensor array and is used for real-time acquisition of multi-channel sensing signals and transmission to the fusion sensing module.

[0010] The fusion sensing module adopts a deep learning model combining a convolutional neural network, a bidirectional long short-term memory network and an attention module, uses the convolutional neural network to extract local time sequence features, fuses global time sequence features through the bidirectional long short-term memory network, and then weights key time steps by the attention module, so as to extract multi-dimensional features in the multi-channel sensing signals and realize identification of three types of knee joint motion modes, i.e. sitting leg lifting, standing and walking.

[0011] The classification module is connected to the fusion sensing module and is used for outputting the classification results of the knee joint motion mode.

[0012] Preferably, the fusion sensing module uses a one-dimensional convolutional layer and two residual blocks in the convolutional neural network to extract local time sequence features, inputs the local time sequence features after channel splicing into the bidirectional long short-term memory network to fuse forward and backward time sequence information, obtains a global time sequence feature sequence to capture the time sequence correlation characteristics of the knee joint sitting leg lifting, standing and walking, and uniformly maps the three types of knee joint motion modes and the local time sequence features of each subzone of the knee joint to an interpretable dynamics representation space; and the attention module is used for attention weighting of the global time sequence feature sequence to highlight the key time step features with large contribution to the discrimination of the knee joint motion mode.

[0013] Further, in the dynamics characterization space, the sitting leg-lifting behavior presents a one-time short-time strong activation trajectory, and the time positions of the lifting, peak, and recovery of the leg-lifting action are clearly distinguishable; the standing presents a long-time period slow drift trajectory, and the short-time shaking is encoded as a weak disturbance; the walking corresponds to a set of repeated quasi-periodic trajectories, and the cycle length and cycle similarity have obvious geometric structures.

[0014] Further, the attention module applies a self-defined weighted attention layer on the global time sequence feature sequence output by the bidirectional long short-term memory network to learn the importance of different time steps to the knee joint motion pattern discrimination.

[0015] The feature vector dimension output by the bidirectional long short-term memory network at each time step is dim, and a trainable weight matrix of shape (dim, dim) is defined in the self-defined weighted attention layer. The global time sequence feature sequence input into the attention module is linearly mapped by the weight matrix to output a context vector of shape (dim, 1), realizing the weighted aggregation of the global time sequence feature sequence.

[0016] Further, the global time sequence feature sequence is weighted in the attention module to highlight the key time step features that contribute more to the discrimination of the knee joint motion pattern, which specifically includes:

[0017] For the sitting leg-lifting posture, the attention module automatically assigns higher weights to the key time steps in the starting and peak stages of the action to accurately identify the starting and ending stages of the sitting leg-lifting;

[0018] For the standing posture, the attention module will reduce the attention to the stable stage and focus more attention on the signals with larger changes to distinguish standing from the other two types of knee joint motion patterns;

[0019] For the walking posture, the attention module automatically enhances the attention to the periodic fluctuations and assigns higher weights to the key time steps in the fluctuation stages of the steps.

[0020] Further, the attention module calculates the attention weights in the following way:

[0021] ;

[0022] wherein, is the weight matrix, is the bias term; is a nonlinear activation function used to map the global time sequence feature sequence to the dynamics characterization space; is the calculated attention weight;

[0023] ;

[0024] is a trainable weight vector, denotes the transpose of the weight vector , denotes the importance weight of the t-th time step;

[0025] The weights of all time steps are normalized by the softmax function to ensure that the sum is 1:

[0026] ;

[0027] is the final context vector, that is, the comprehensive feature representation obtained by weighting the global time sequence feature sequence according to its attention weight.

[0028] Preferably, the flexible fabric sensor array adopts a double-layer fabric circuit structure, a plurality of sensing units integrated in the upper layer fabric circuit are divided into an upper region and a lower region, the sensing units in the upper region cover the upper edge of the patella and the tendon attachment area of the quadriceps femoris muscle, and are used for monitoring the dynamic strain when the quadriceps femoris muscle contracts; the sensing units in the lower region are positioned in the tibial tuberosity and the patellar ligament area, and are used for capturing the patellar ligament tension change in real time; and the lower layer fabric circuit forms a connection wire.

[0029] In another aspect, the embodiment of the present application also provides a flexible fabric-based knee joint multi-sensor fusion perception method, which is realized by using the above-mentioned knee joint multi-sensor fusion perception system; the method comprises the following steps:

[0030] S1, pre-processing the multi-channel sensing signals of the knee joint movement to obtain multi-channel time sequence signals unified to the same reference frame in the time axis and amplitude scale;

[0031] S2, extracting local time sequence features from the pre-processed multi-channel time sequence signals by using a convolutional neural network;

[0032] S3, after the local time sequence features output by the convolutional neural network are spliced according to the channels, the forward and backward time sequence information is fused by using a two-layer bidirectional LSTM structure of a bidirectional long short-term memory network to obtain a global time sequence feature sequence;

[0033] S4, attention weighting is performed on the global time sequence feature sequence to highlight the key time step features with large contribution to the discrimination of the knee joint movement mode;

[0034] S5, outputting the classification result of the knee joint movement mode.

[0035] The present application has the following advantages and effects relative to the prior art:

[0036] 1. Due to the different response characteristics of the sensor array in different locations: the sensors placed on the lateral side of the knee joint experience greater deformation during flexion, extension, and lateral swing, and their sensitivity is significantly better than that of the medial region, enabling them to more accurately capture movement characteristics such as eversion and rotation; while the sensor array placed on the upper side of the knee joint, being close to the quadriceps insertion point and the upper edge of the patella, exhibits higher response intensity during the initiation of joint flexion and extension and angle changes, and its sensitivity is generally higher than that of the sensors placed on the lower side of the knee joint, enabling it to reflect movement trends earlier and more clearly; although the sensors placed on the lower side of the knee joint have slightly lower overall sensitivity, they still play an important role in joint stability and impact load monitoring.

[0037] This invention employs multi-dimensional coverage and high-sensitivity monitoring: by uniformly covering the key areas of the knee joint with an eight-sensor array, it effectively avoids the blind spot problem of single-point sensors, and achieves comprehensive monitoring of complex movements such as flexion, extension, rotation and lateral swing. The signal continuity and accuracy are significantly better than existing single-sensor solutions.

[0038] 2. This invention achieves intelligent fusion perception and high-precision recognition: By establishing an improved deep learning model as the fusion perception module, efficient feature extraction and adaptive dynamic fusion of multi-channel signals are performed. The improved deep learning model, i.e., the fusion perception module, includes a convolutional neural network (CNN), a bidirectional long short-term memory network (Bi-LSTM), and an attention module. This invention inputs the raw signals collected by multi-channel sensors into the deep learning model. The CNN effectively extracts local spatial features and suppresses noise, while the Bi-LSTM models long-term dependencies in time-series data through forward and backward information flows. The attention module further assigns weights to key channels and important time segments, thereby achieving adaptive feature fusion and dynamic enhancement.

[0039] 3. This invention improves comfort and environmental adaptability: This invention uses a highly elastic, breathable and wear-resistant flexible fabric base, which can maintain good fit during large knee flexion and extension, sudden stops and turns or long-term load exercises, ensuring that the sensor fits tightly against the skin under large movements, avoiding signal distortion and wearing discomfort caused by slippage or wrinkles. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of the flexible fabric sensor array used in the embodiments of the present invention;

[0041] Figure 2 This is a flowchart of the fusion sensing process for multi-channel sensor signals in an embodiment of the present invention;

[0042] Figure 3a result schematic for visualization by confusion matrix;

[0043] Figure 4 a result schematic for visualization by T-distributed Stochastic Neighbor Embedding of two-dimensional features;

[0044] Figure 5 a result schematic for visualization by T-distributed Stochastic Neighbor Embedding of three-dimensional features. DETAILED DESCRIPTION

[0045] The application will be described in further detail below with reference to embodiments and drawings, but the embodiments of the application are not limited thereto. EMBODIMENT

[0046] The embodiment provides a knee joint multi-sensor fusion sensing system based on a flexible fabric, which comprises a flexible fabric sensor array, a signal acquisition module, a fusion sensing module and a classification module.

[0047] The flexible fabric sensor array comprises eight independent flexible fabric sensing units which are not interfered with each other in function and are uniformly distributed in four key biomechanics monitoring areas of the front part of the knee joint, i.e., the upper edge of the patella, the tendon insertion area of the quadriceps femoris muscle, the tibial tuberosity and the patellar ligament area, so as to realize multi-angle and all-around coverage monitoring of the motion state of the knee joint and provide multi-dimensional information of the motion state of the knee joint.

[0048] As shown in Figure 1 The flexible fabric sensor array adopts a double-layer intelligent fabric circuit structure, and a fabric base 3 is arranged between an upper-layer fabric circuit 1 and a lower-layer fabric circuit 2. The upper-layer fabric circuit integrates eight sensing units S1-S8, and the eight sensing units are divided into two groups of upper and lower areas. The four sensing units in the upper area cover the upper edge of the patella and the tendon insertion area of the quadriceps femoris muscle and are used for monitoring the dynamic strain when the quadriceps muscle contracts. The four sensing units in the lower area are positioned in the tibial tuberosity and the patellar ligament area and are used for capturing the patellar ligament tension change in real time. The lower-layer fabric circuit forms a connecting wire. The two-layer fabric circuits are connected through vertical interconnection holes filled with high-viscosity conductive paste, that is, a through hole is formed at a preset position of the fabric circuit through precise punching, and high-viscosity carbon black-silicone rubber composite paste is filled in the through hole to realize conductive connection, so as to ensure stable transmission of the signal and interlayer coordination.

[0049] Each sensing unit is made of conductive composite material, supported and encapsulated by a fabric substrate, which is an integrated carbon black-silicone rubber composite conductive layer, capable of sensing the mechanical deformation of the fabric material, including stretching, bending and pressure, and converting it into an electrical signal proportional to the deformation amplitude. The composite conductive layer that constitutes the sensing unit includes a sensing area and a connecting wire, which are formed simultaneously in the same printing step; the sensing area is used to sense the local strain change of the knee joint in the process of flexion, extension, rotation and lateral swing in real time, and the equivalent resistance of the sensing area is ≥100 kΩ to improve the sensitivity to small strain changes; the equivalent resistance of the connecting wire is ≤50 kΩ to ensure the signal integrity in the long-distance transmission process. Printing the sensing unit on a flexible fabric, a highly elastic base material, makes the sensor array closely fit the skin during knee movement, avoiding sliding and compression; and the fabric material has good air permeability, is suitable for long-term wear, and reduces sweat accumulation and skin irritation. In addition, the 8 sensing units are about 23 mm long and 19 mm wide, and are evenly distributed in the key area of the front of the knee joint, ensuring multi-angle and multi-dimensional coverage of the knee joint movement state, which can fully capture various complex motion forms of the knee joint and avoid the blind area and data discontinuity that may be caused by single-point sensors.

[0050] In the present embodiment, each sensing unit of the flexible fabric sensor array is made of carbon black-silicone rubber (CB-SE) composite material, and the preparation method is as follows:

[0051] (1) Material ratio: mix carbon black (CB), room temperature vulcanized silicone rubber (SE) and dimethyl silicone oil (SO) according to the weight ratio of 3 g: 30.3 g: 45 g;

[0052] (2) Stirring and dispersion: stir at 400 rpm for 30 minutes to form a uniform conductive slurry;

[0053] (3) Screen printing: use a 1000-mesh screen to print the slurry on a knitted fabric substrate composed of 80% nylon and 20% lycra;

[0054] (4) Curing treatment: cure in a 100°C vacuum oven for 1 hour to form a conductive pattern;

[0055] (5) Structure design: the sensing area is designed in a meandering shape, with an effective length of 221 mm, a width of 2 mm, an interval of 1 mm, and an equivalent resistance of ≥100 kΩ; the connecting wire has a width of 20 mm and an equivalent resistance of ≤50 kΩ.

[0056] The signal acquisition module is connected to the flexible fabric sensor array, and is used for acquiring multi-channel sensing signals in real time and transmitting to the fusion perception module. In the embodiment, the signal acquisition module adopts a microcontroller as a master control unit, and is configured with a six-channel synchronous data acquisition circuit to ensure real-time parallel acquisition of multi-source signals. The acquisition port of the signal acquisition module is connected with an external AD sampling module to improve signal quantization precision and dynamic range, so as to meet the application requirements of high sensitivity and high stability; the acquisition port is connected with the sensor array through a metal button to ensure the stability of signal transmission. Specifically, in the signal acquisition link, the six-channel synchronous data acquisition not only ensures complete coverage of sensing signals in different regions of the knee joint, but also lays a foundation for subsequent multi-dimensional information fusion. Moreover, multi-channel data acquisition can ensure that information is not lost, and multi-channel layout can not only significantly reduce the risk of signal loss and measurement blind area, but also enhance the stability of the system in complex motion scenarios.

[0057] In the fusion processing link, the system first extracts features from the multi-channel sensor data, including mean, variance, extreme value, skewness and kurtosis, to characterize the dynamic change law of the output of different sensors and identify the motion patterns of the knee joint, such as sitting leg lifting, standing and walking; then a multi-channel attention mechanism is introduced to adaptively model the feature contribution degree of each sensor, so as to highlight key signals and suppress redundant information. By introducing the multi-channel attention mechanism, the signal channels that contribute more to the recognition of the motion patterns of the knee joint are highlighted, the signal contribution degree of each channel is dynamically weighted and fused, the importance of the sensors can be dynamically allocated, key information can be highlighted and redundant noise can be suppressed, so that the overall feature representation has better discriminability. For example, in the flexion and extension action, the impedance change of part of the channels can better reflect the actual motion state, and the multi-channel attention mechanism will increase the weight of this part of the channels, thereby enhancing the discriminability of the overall signal.

[0058] In the embodiment, the fusion processing link is mainly realized by the fusion perception module. The fusion perception module adopts a deep learning model combining a convolutional neural network (CNN), a bidirectional long short-term memory network (Bi-LSTM) and an attention module, i.e., an improved deep learning model. The CNN is used to extract local time sequence features, the Bi-LSTM is used to fuse global time sequence features, and the attention module is used to weight key time steps, so as to extract multi-dimensional features from multi-channel sensing signals and realize accurate recognition of knee joint motion patterns such as sitting leg lifting, standing and walking. The classification module is used to output the classification results of the knee joint motion patterns.

[0059] The input of the fusion perception module is the multi-channel sensing signal collected by the signal collection module. The multi-channel sensing signal is first preprocessed, and then enters the feature extraction stage. One-dimensional convolution (Conv1D) and residual block are used to extract local time sequence features, and dimension reduction and feature compression are performed through the pooling layer (MaxPool). The data after feature extraction is input into the bidirectional long short-term memory network Bi-LSTM to model the time sequence dependence relationship before and after. On this basis, the attention module weights the features of the key time steps to enhance the signal segments that contribute more to the posture discrimination. The signal processed by the attention module is output to the classification module. The classification module includes a fully connected layer, which accurately identifies the output signal of the fusion perception module as three types of knee joint motion modes: "sitting and lifting legs", "standing" and "walking".

[0060] Specifically, the sitting and lifting legs action is a non-periodic transient process. Its signal feature is a sharp change in signal amplitude within a short time window , which is approximately a pulse response:

[0061] , t ;

[0062] where A is the amplitude coefficient, k is the change rate, is the peak time of the action, is the noise, and the start, peak and end stages of the action contain the most critical discrimination information, and the sensor signal shows a rapid unidirectional change pulse. The sitting and lifting legs action is observed to capture the "start, reach peak, recover" complete process of the lifting legs action, reflecting the skin strain change caused by the contraction of the quadriceps femoris.

[0063] The walking action is a quasi-periodic cyclic process. Its signal can be modeled as the superposition of a fundamental frequency (step frequency) and its harmonics:

[0064] ;

[0065] where is the step frequency, and are the amplitudes and phases of each harmonic, and the model needs to capture the periodic envelope and key phase points (such as the moment when the heel touches the ground and the toe leaves the ground) of the signal. These points usually correspond to the extreme points or zero-crossing points in each gait cycle. Physically, the amplitude modulation reflects the "flexion, extension, and flexion again" double-peak characteristics of the knee joint during walking, where the high-frequency component corresponds to the alternation of single-leg support and swing periods.

[0066] The standing action can be regarded as a stationary random process, whose signal fluctuates slightly around a steady-state mean value, and the main energy is concentrated in the low frequency or even close to the direct current component:

[0067] ;

[0068] wherein, is the steady-state mean value, is the low-variance random fluctuation mainly caused by slight body sway and measurement noise. For the standing posture, the characteristic distribution of the entire time series is relatively uniform, and there is no obvious key time point. It describes the low-amplitude random fluctuation caused by slight body adjustment and center of gravity shift when the knee joint is at a relatively fixed angle during standing.

[0069] Based on the above differences in the time steps of the three types of knee joint motion modes, the embodiment combines the fusion perception module and the classification module to realize end-to-end processing and recognition from the original signal to the motion mode recognition, ensuring real-time and accuracy. As shown in Figure 2 , the specific fusion perception method includes the following steps:

[0070] S1, data preprocessing: standardizing the multi-channel sensing signals of the knee joint motion.

[0071] For the multi-channel sensing signals (including strain, acceleration and other modalities) collected by the 8-channel sensors in the four partitions of the knee joint, namely the upper edge of the patella, the tibial tuberosity, the lateral side and the medial side, first, the multi-channel time synchronization alignment is performed to solve the time delay caused by the difference in sensor installation position, and to ensure the time consistency of the signals of the four partitions at the same motion time; then, 5Hz low-pass filtering is used to suppress muscle tremor high-frequency noise, and sliding window denoising is used to filter the slow drift noise caused by sweat corrosion, wherein the window size is preferably 50ms to adapt to the minimum response time of the knee joint motion; finally, each partition signal is normalized separately to eliminate the amplitude difference between the high-amplitude signal of the upper edge of the patella and the low-amplitude signal of the medial side, and to avoid that a certain partition signal masks the key features of other partitions, and the standardized multi-channel time sequence signals are output, with a dimension of the product of the time step and the number of sensor channels.

[0072] After preprocessing, the multi-channel time sequence signals obtained by the embodiment are unified to the same reference frame in the time axis and amplitude scale, and the signal differences between different knee joint partitions and different modalities are mainly preserved in their inherent time sequence patterns and relative change trends. Therefore, on the one hand, the influence of external factors such as individual differences, tightness of wearing, and inconsistency of sensor sensitivity on the original signal is weakened, and on the other hand, the key dynamic features in the knee joint flexion process are preserved, providing a stable and comparable consistent input basis for subsequent feature extraction and mode recognition, to improve the robustness and generalization ability of multi-sensor fusion perception.

[0073] S2, local time sequence feature extraction: for the pre-processed multi-channel time sequence signal, in the convolutional neural network CNN, the local time sequence feature is extracted using one-dimensional convolution layer and two residual blocks, wherein the convolution kernel of the one-dimensional convolution layer is 3, and the filter number is 64.

[0074] In the process of extracting features by CNN, the input is the multi-channel time sequence signal after data preprocessing, denoted as , is the time step, is the number of sensor channels, and the formula of convolution operation is as follows:

[0075] ;

[0076] wherein, is the output feature value of the first filter at time step ; is the length of the convolution kernel, is the convolution kernel weight, is the bias term; is the index inside the convolution kernel, representing the position of the convolution kernel in the time dimension; represents the value of the input signal at time step , channel c, which facilitates symmetric convolution with time step as the center to capture local time sequence features. The residual block performs residual connection on the output signal after convolution operation, as follows:

[0077]

[0078] ; wherein Y represents the output feature tensor of the residual block, and is the final output of the entire residual calculation, which will be used as the input of the next layer network;

[0079] represents the output signal after convolution operation; represents the activation function, BN() represents the batch normalization function, represents the convolution operation, and S() is the shortcut mapping function. Since the input of the convolutional neural network CNN is the multi-channel time sequence signal processed by sliding window segmentation, the time step is

[0080] , thus the time sequence feature extracted by the convolutional neural network CNN is a local time sequence feature, that is, it does not cover the complete time step.

[0081] ​In the embodiment, the feature tensor output by the convolutional neural network CNN has a dimension of L'xF, where L' is the number of time steps after downsampling, and F is the number of convolution output channels; the local time sequence features output by the CNN include strain extreme values, transient change trends, and steady-state distribution characteristics, providing high-quality time sequence feature inputs for a subsequent bidirectional long short-term memory network Bi-LSTM, and realizing smooth transition from local features to global time sequence patterns.

[0082] S3, Bi-LSTM time sequence modeling: after the local time sequence features output by the convolutional neural network CNN are spliced by channel, the features are input into the bidirectional long short-term memory network Bi-LSTM, the forward and backward time sequence information is fused through the two-layer bidirectional LSTM structure of the bidirectional long short-term memory network Bi-LSTM, and a global time sequence feature sequence is obtained.

[0083] The bidirectional LSTM structure fuses the forward hidden state and the backward hidden state to obtain an output vector of the bidirectional LSTM structure.

[0084] wherein, is the output vector of the bidirectional LSTM structure; is the hidden state of the forward LSTM at time step t, which is calculated by the forward LSTM unit, the information flow direction of which is consistent with the order of the time sequence, and all historical information from the initial point of the sequence to the current time step t is captured and encoded; is the hidden state of the backward LSTM at time step t, which is calculated by the backward LSTM unit, the information flow direction of which is opposite to the order of the time sequence, and all future information from the end point of the sequence to the current time step t is captured and encoded.

[0085] The bidirectional LSTM structure combines the information on both sides to obtain more comprehensive time sequence features, i.e., a global time sequence feature sequence. In the two-layer bidirectional LSTM structure, the first layer has 128 units, and the second layer has 64 units; finally, an output feature tensor with a shape of (L', 2H) is obtained, where L' is the number of time steps after downsampling, H is the number of hidden units of the current layer, and H multiplied by 2 represents the splicing of the forward hidden state and the backward hidden state.

[0086] Specifically, in the Bi-LSTM network, the time sequence correlation characteristics of the knee joint posture leg lifting, standing, and walking are captured, and modeling logic is customized according to the time sequence characteristics of the three posture motions of the knee joint:

[0087] ​1) Adopt a gating mechanism to adapt to the timing signals of three postures. For the timing signal of sitting posture leg lifting, it is a short timing with a duration of 500-800 ms. The forward LSTM strengthens the forward dependence of the timing signal from the start to the peak, and the backward LSTM captures the reverse dependence of the timing signal from the peak to the recovery. For the timing signal of standing, it is a long timing with a duration of ≥3s. The high forgetting gate weight is used to retain the long-term steady-state characteristics and suppress the edge shaking noise. For the timing signal of walking, its quasi-period is 500-1000 ms. Through double-layer LSTM, the first layer LSTM is 128 units, which is used to capture the local dependence of the foot from touching the ground to leaving the ground within a single cycle, and the second layer LSTM is 64 units, which is used to capture the cycle correlation signal across cycles.

[0088] 2) Global parameter extraction, output global timing dependence features, including timing peak interval sequence (periodic distribution for walking, transient distribution for leg lifting), mutation time step position sequence (corresponding to knee joint flexion and extension start and inflection point), and bidirectional timing dependence strength (walking bidirectional correlation ≥0.8, leg lifting forward dependence dominant).

[0089] On the basis of the above Bi-LSTM structure, the global dependence modeling layer essentially maps the three types of knee joint motion modes and the local timing features of each partition of the knee joint to an interpretable dynamic representation space; in this dynamic representation space: sitting leg lifting shows a one-time short-time strong activation trajectory, and the time positions of lifting, peak, and recovery of the leg lifting action are clearly divided; standing presents a long period of near steady-state slow drift trajectory, and short-time shaking is only coded as a weak disturbance; walking corresponds to a set of repeated quasi-periodic trajectories, and the cycle length and cycle similarity have obvious geometric structure.

[0090] Through bidirectional gate accumulation and multi-layer recursive compression, Bi-LSTM integrates the scattered start, peak, recovery, steady state, and cycle rhythm information in the original multi-channel time series into a set of compact and directional global dependence features, providing a clear timing skeleton for the subsequent attention module to weight process the key time steps, and laying a foundation for the final classification module to distinguish the three types of motion and pathological differences of sitting leg lifting, standing, and walking in the same feature space.

[0091] S4, attention weighting processing: attention weighting is performed on the global timing feature sequence output by Bi-LSTM to highlight the key time step features that contribute greatly to the discrimination of knee joint motion patterns.

[0092] In this embodiment, the introduction of the attention module not only weights the key time steps in the feature extraction stage, but also adaptively adjusts the attention degree of different time features in the classification stage. Among them, the weighting mechanism can highlight the most critical time steps for posture recognition, and enhance the robustness and accuracy of the model by suppressing unimportant time step features. Weighting is achieved by calculating the attention score of each time step, converting these scores into attention weights, and then performing weighted summation on the input features to obtain the final feature representation.

[0093] The attention module is implemented through a self-defined weighted attention layer, i.e., applying a self-defined weighted attention layer to the global time sequence features output by Bi-LSTM to learn the importance of different time steps in discriminating knee movement patterns, so that the model can automatically focus on the time that contributes most to the final classification.

[0094] In this embodiment, in the self-defined weighted attention layer, a trainable weight matrix W of shape (dim, dim) is defined. Since the feature vector output by Bi-LSTM at each time step has a dimension of dim, the linear mapping of the attention layer must be consistent with the input, and the weight matrix W is set to have a shape of (dim, dim). The global time sequence features input into the attention module are linearly mapped through the weight matrix W, and are activated through a nonlinear activation function . Then, the self-defined weighted attention layer outputs a context vector u with a shape of (dim, 1), which realizes the weighted aggregation of the global time sequence features. The shape of the context vector is set to (dim, 1) to calculate the scalar score of the linearly mapped features and to compare the importance of the time steps. In the attention module, the global time sequence features output by Bi-LSTM are weighted with attention to highlight the time step features that contribute more to the discrimination of knee movement patterns. Specifically, for the sitting and lifting posture, the signal changes greatly in the starting stage and peak stage, and these time points contain the most critical discriminative information. The attention module automatically assigns higher weights to these key time steps, thereby helping the model to more accurately identify the starting and ending stages of the sitting and lifting posture. For the standing posture, the signal is relatively stable and has a small change amplitude, and the attention module will reduce the attention to this stable stage and focus more attention on the signals with larger changes, thereby helping to distinguish standing from the other two types of knee movement patterns. For the walking posture, since the walking action has periodicity, the attention module automatically enhances the attention to the periodic fluctuations and assigns higher weights to the key time steps where the fluctuation stages of the steps are located, such as the footstep landing time or the turning point of the gait, thereby enabling the model to focus on the key signals of walking.

[0095] More specifically, the attention module in this embodiment receives the time series feature sequence of the bidirectional LSTM output where vector represents the feature representation after fusing the time series information before and after the t-th time step, and the context vector is obtained after calculation by the self-defined weighted attention layer. In practical applications, the attention module can adaptively assign attention weights to different time steps, so that the model can focus on capturing the most critical motion moments for pose recognition.

[0096] The calculation method of the attention module for attention weights is as follows:

[0097] ;

[0098] wherein, is a weight matrix, is a bias term; is a nonlinear activation function, which is used to map the global time series feature sequence to a dynamic representation space; is the calculated attention weight.

[0099] ;

[0100] is a trainable weight vector, represents the transpose of the weight vector , and represents the importance weight of the t-th time step.

[0101] The weights of all time steps are normalized by the softmax function to ensure that their sum is 1:

[0102] ;

[0103] is the final context vector, i.e., the comprehensive feature representation obtained by weighting the global time series feature sequence according to its attention weight.

[0104] The output dimension of the self-defined weighted attention layer is consistent with the LSTM hidden layer. In addition to the trainable weight matrix, bias term, and trainable weight vector, the attention module can also include a trainable multi-head attention weight, and multiple attention heads are used to calculate and process different time scale feature dependencies in parallel, i.e., each attention head has an independent trainable weight matrix and context vector , so that various dynamic dependency features of the input sequence can be captured in different subspaces. Therefore, in order to enhance the modeling ability of the model for different time scale dependencies, the self-defined weighted attention layer is expanded to a multi-head attention structure, i.e., each attention head has an independent trainable weight matrix and context vector , so as to capture dynamic dependency features of different levels in the sequence in parallel. The outputs of each attention head can be integrated into the final feature representation by weighted summation.

[0105] wherein the single-head attention calculates a query-key-value mapping, and the final context vector is obtained by calculating the weight and weighted summation. The multi-head attention is an extension of the single-head attention, each head has independent query, key and value mapping; the input of each attention head is the same, which is the tensor from the input of the previous layer, i.e. the time series features obtained after the previous layer processing, with a shape of , L is the time step, and dim is the feature dimension of each time step.

[0106] Each attention head linearly transforms the input through a trainable weight matrix to obtain the query, key and value; the output of each attention head is a feature representation obtained by calculating the attention score and weighted sum of the value. The multi-head attention output is the final output space obtained by linear transformation after weighted summation of the outputs of multiple attention heads.

[0107] The multi-head attention output is:

[0108] ;

[0109] The dimension of the final output space is , wherein is the feature dimension after weighted summation.

[0110] The multi-channel attention mechanism can process multiple attention heads in parallel, so that each head can independently calculate and learn in different subspaces, dynamically adjust the attention degree of each time step according to the characteristics of the input data, and capture the feature dependency relationship of different levels or different time scales of the input data.

[0111] The global timing features processed by the attention module will highlight different key time step features: for the global timing feature of sitting and lifting legs, the overall attention trajectory presents a clear double-peak focusing form on the time axis, and the response near the starting moment and peak value is continuously amplified; for the global timing feature of standing, the attention weight presents a platform-like distribution in the middle steady-state interval, and the front and rear transition zones and short-time disturbances are actively compressed; for the global timing feature of walking, the touch-down and take-off inflection points in multiple cycles are repeatedly highlighted in the feature space, and the remaining slow-changing segments are weakened as background information. Therefore, the self-defined attention layer realizes efficient aggregation of sequence features through a trainable weight matrix, a context vector and a multi-head attention mechanism, so as to highlight the dynamic mode of the key time step and maintain an interpretable sequence representation consistent with the output dimension of LSTM.

[0112] S5, classification output: the classification result of the knee joint motion mode is output through the full connection layer of the classification module.

[0113] The full connection layer is completely independent of the attention module and is part of the classification module outside the attention module. The full connection layer is responsible for converting the weighted features into specific classification results. The full connection layer focuses on classification decision-making, realizing attention point separation.

[0114] Specifically, the first full connection layer further maps the abstract action features to a "separable space", allowing the model to learn a nonlinear decision boundary and enhance the discriminability between classes. The second full connection layer maps the features to three posture categories, and finally outputs the probabilities of the three postures.

[0115] The system can also analyze the knee joint motion mode based on real-time recognition results and historical motion data to determine whether there are abnormal actions or potential risks (such as excessive flexion and extension, excessive lateral twisting, etc.).

[0116] The knee joint multi-sensor fusion perception system of the embodiment is arranged on a wearable fabric, and the steps used mainly include the following:

[0117] Step one, wear the sensor: attach the flexible fabric sensor array along the key areas of the knee joint (the upper edge of the patella, the tendon insertion area of the quadriceps femoris muscle, the tibial tuberosity and the patellar ligament area) on the knee joint to ensure that the sensor is tightly attached and comfortable. Adjust the position of the fabric base to ensure that the eight sensing units evenly cover the specified area, avoiding sensor sliding or lifting.

[0118] Step two, connect the signal acquisition module: connect the sensor array and the signal acquisition module through a connecting line to ensure good circuit connection and confirm that the electrical signals of the multi-channel sensor can be normally collected.

[0119] Step three, signal calibration and preprocessing, before formal measurement, the initial calibration of the electrical signal of each sensing unit, eliminate zero drift and environmental interference; the collected multi-channel time sequence signal is standardized.

[0120] Step four, action execution and data collection, the subject completes the target action as required, and the system collects multi-channel signals of the knee joint during flexion, extension, rotation and lateral swing in real time. The flexible fabric sensor ensures continuous and stable signal without blind area, and the signal acquisition module transmits data to the fusion perception module.

[0121] Step five, data processing and posture recognition, the system inputs the collected signals into the fusion perception module to extract local time sequence features, fuse global time sequence features, and weight key time steps, and generates real-time recognition results of three postures of sitting up, standing and walking.

[0122] Step six, abnormal action detection and prompt, the system analyzes the knee joint motion pattern according to the real-time recognition result and historical motion data, and judges whether there is abnormal action or potential risk (such as excessive flexion, lateral twist, etc.).

[0123] The three posture motion signals are classified by the fusion perception of the embodiment, and are visualized by confusion matrix, T-distributed stochastic neighborhood embedding two-dimensional feature visualization, and T-distributed stochastic neighborhood embedding three-dimensional feature visualization, respectively, which can directly evaluate the accuracy of the classification of the three postures of the knee joint, as shown in Figures 3-5 , wherein Figure 3 is a result schematic diagram of visualization by confusion matrix, Figure 4 is a result schematic diagram of visualization by T-distributed stochastic neighborhood embedding two-dimensional feature, Figure 5 is a result schematic diagram of visualization by T-distributed stochastic neighborhood embedding three-dimensional feature.

[0124] The intelligent fusion perception technology of the present application has significantly higher accuracy and stability than traditional methods in real-time recognition of knee flexion, lateral swing and compound action. It not only can maintain high-precision output in rapidly changing dynamic motion state, but also can maintain good robustness and anti-interference ability in long-term wearing and complex environmental interference conditions. Compared with the existing schemes relying on single sensing signal or traditional statistical feature extraction method, the present application has substantial improvement in real-time, accuracy and adaptability, and provides more reliable technical support for knee joint motion monitoring, rehabilitation evaluation and intelligent medical treatment.

[0125] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A flexible fabric based knee joint multi-sensor fusion perception system, characterized in that, The application relates to a knee joint motion mode recognition method based on a flexible fabric sensor array. The flexible fabric sensor array comprises a plurality of independent and functionally non-interfering flexible fabric sensor units which are uniformly distributed in the front part of a knee joint and are in the regions of the upper edge of a patella, a quadriceps tendon attachment area, a tibial tuberosity and a patellar ligament. The signal acquisition module is connected to the flexible fabric sensor array and is used for collecting multi-channel sensing signals in real time and transmitting the signals to the fusion perception module. The fusion perception module adopts a deep learning model combining a convolutional neural network, a bidirectional long short-term memory network and an attention module, uses the convolutional neural network to extract local time sequence features, fuses global time sequence features through the bidirectional long short-term memory network, and weights key time steps through the attention module, so that multi-dimensional features are extracted from the multi-channel sensing signals, and three types of knee joint motion modes, i.e. a sitting leg-lifting mode, a standing mode and a walking mode, are recognized. The classification module is connected to the fusion perception module and is used for outputting classification results of the knee joint motion modes. In the convolutional neural network, one-dimensional convolution layers and two residual blocks are used to extract local time sequence features, the local time sequence features are spliced according to channels and then input into the bidirectional long short-term memory network to fuse forward and backward time sequence information, so that a global time sequence feature sequence is obtained to capture the time sequence correlation characteristics of the sitting leg-lifting mode, the standing mode and the walking mode of the knee joint, and the three types of knee joint motion modes and the local time sequence features of the regions of the knee joint are all mapped to an interpretable dynamic representation space; in the attention module, the global time sequence feature sequence is weighted through attention to highlight key time step features which have a large contribution to the discrimination of the knee joint motion modes. The bidirectional long short-term memory network comprises a two-layer bidirectional LSTM structure, the bidirectional LSTM structure splices a forward hidden state with a backward hidden state : ; wherein, is an output vector of a bidirectional LSTM structure; is a hidden state of a forward LSTM at time step t; is a hidden state of a backward LSTM at time step t; The bidirectional long short-term memory network customizes modeling logic according to the time sequence characteristics of the three types of posture motions of the knee joint, i.e. the sitting leg-lifting mode, the standing mode and the walking mode: 1) a gating mechanism is adopted to adapt to the time sequence signals of the three types of posture motions, for the time sequence signal of the sitting leg-lifting mode, a short time sequence is used, a forward LSTM is used to strengthen the forward dependence of the time sequence signal from starting to reaching a peak value, and a backward LSTM is used to capture the backward dependence of the time sequence signal from the peak value to recovery; for the time sequence signal of the standing mode, a long time sequence is used, a high forgetting gate weight is used to retain long time stable characteristics and suppress edge shaking noise; for the time sequence signal of the walking mode, a first layer bidirectional LSTM structure is used to capture local dependence of a foot from touching the ground to leaving the ground in a single period, and a second layer bidirectional LSTM structure is used to capture cross-period cyclic correlation signals; 2) global parameters are extracted to output global time sequence dependence features, the global time sequence dependence features include a time sequence peak interval sequence, a mutation time step position sequence and bidirectional time sequence dependence strength; when the posture motion is walking, the time sequence peak interval sequence is periodically distributed, when the posture motion is leg-lifting, the time sequence peak interval sequence is transiently distributed, and the mutation time step position sequence corresponds to knee flexion and extension starting and inflection points. In the dynamic characterization space, the seated leg raise is a one-time short-term strong activation trajectory, and the time position of the leg raise, peak and recovery are clearly distinguishable; the standing posture presents a slow drift trajectory with a near steady state over a long period of time, and the short-term swaying motion is encoded as a weak disturbance; walking corresponds to a set of repetitive quasi-periodic trajectories, and the similarity of the period length and period has obvious geometric structure.

2. The knee joint multi-sensor fusion perception system of claim 1, wherein, The attention module applies a custom weighted attention layer to the global temporal feature sequence output by the bidirectional long short-term memory network to learn the importance of different time steps in the discrimination of knee joint movement patterns. Let the dimension of the feature vector output by the bidirectional long short-term memory network at each time step be dim. The custom weighted attention layer defines a trainable weight matrix of shape (dim, dim). The global temporal feature sequence input to the attention module is linearly mapped through the weight matrix, and a context vector of shape (dim, 1) is output, thereby realizing the weighted aggregation of the global temporal feature sequence.

3. The knee joint multi-sensor fusion perception system of claim 2, wherein, In the attention module, the global temporal feature sequence is weighted by attention to highlight key time step features that contribute significantly to the discrimination of knee joint movement patterns. Specifically, this includes: For seated leg raise posture, the attention module automatically assigns higher weights to the key time steps where the start and peak phases of the movement are located in order to accurately identify the start and end phases of seated leg raise. For standing posture, the attention module will reduce its focus on the steady phase and concentrate more on signals with significant changes in order to distinguish standing from the other two types of knee joint movement patterns. For walking posture, the attention module automatically enhances its focus on periodic fluctuations, assigning higher weights to key time steps where the undulation phase of the gait occurs.

4. The knee joint multi-sensor fusion perception system of claim 3, wherein, The attention module calculates attention weights as follows: ； wherein, is a weight matrix, is a bias term; is a non-linear activation function for mapping the global temporal feature sequence to a dynamics representation space; is the computed attention weight; ； is a trainable weight vector, represents the transpose of the weight vector represents the importance weight at the t-th time step;​ The weights at all time steps are normalized using the softmax function to ensure that their sum is 1. ； The final context vector, i.e., the comprehensive feature representation obtained by weighting the global temporal feature sequence according to its attention weight.

5. The knee joint multi-sensor fusion perception system of claim 1, wherein, The flexible fabric sensor array adopts a double-layer fabric circuit structure. The upper fabric circuit integrates several sensing units, which are divided into upper and lower zones. The upper zone sensing units cover the upper edge of the patella and the quadriceps tendon attachment area of ​​the knee joint, and are used to monitor the dynamic strain during quadriceps contraction. The lower zone sensing units are located in the tibial tuberosity and patellar ligament area, and are used to capture changes in patellar ligament tension in real time. The lower fabric circuit forms connecting wires.

6. The knee joint multi-sensor fusion perception system of claim 5, wherein, The two-layer fabric circuitry is connected via vertical interconnect holes filled with a high-viscosity conductive paste.

7. A flexible fabric-based knee joint multi-sensor fusion perception method, characterized in that, The method is implemented using the knee joint multi-sensor fusion sensing system according to any one of claims 1-6; the method includes the following steps: S1. Preprocess the multi-channel sensing signal of knee joint movement to obtain a multi-channel time-series signal that is unified to the same reference frame in terms of time axis and amplitude scale. S2. Extract local temporal features from the preprocessed multi-channel temporal signal using a convolutional neural network; S3. After concatenating the local temporal features output by the convolutional neural network by channel, the forward and backward temporal information is fused through a two-layer bidirectional LSTM structure of a bidirectional long short-term memory network to obtain a global temporal feature sequence. S4, attention weighting is performed on the global time sequence feature sequence to highlight key time step features that have a large contribution to the discrimination of the knee joint motion pattern; S5, outputting the classification result of the knee joint motion pattern.

Images