Daily life activity ability assessment method and system based on multi-branch space-time fusion network

Abstract

The invention discloses a daily life activity ability assessment method and system based on a multi-branch space-time fusion network. The method comprises the steps of collecting motion data of a to-be-evaluated object, performing segmentation, noise reduction and standardization preprocessing, and extracting time-frequency domain features and motion deviation degree, motion similarity index and bilateral symmetry index provided by the invention; obtaining a uniform Bhattacharyya index scale score; and taking the motion feature data set and the clinical score as input, performing training based on a multi-branch space-time fusion algorithm, constructing a multi-task prediction model, and realizing automatic evaluation of the ADL. According to the method, the multi-branch space-time fusion network is innovatively introduced, the relevance between tasks is fully utilized, the spatial feature extraction capability is enhanced in combination with the graph neural network, adaptive fusion of the features is realized through a cross-layer interactive attention mechanism, the model prediction precision and evaluation reliability are remarkably improved, and the method is suitable for large-scale popularization and application. And an effective technical support is provided for rehabilitation evaluation and personalized treatment of patients with dyskinesia and disability and dementia.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of ability evaluation, and particularly relates to a daily life activity ability evaluation method and system based on a multi-branch space-time fusion network. BACKGROUND

[0002] Limb motor dysfunction caused by stroke, brain trauma, and the like, and disabled and demented old people, will most leave different degrees of Activities of Daily Living (ADL) disorders, ADL covers basic self-care activities such as eating, dressing, and toilet, and moving functions such as walking and going up and down stairs, and seriously affects the ability of returning to the family and society. The recovery of ADL function is the core task of patient rehabilitation, and its accurate evaluation has multiple key significances for clinical decision-making. First, accurate evaluation can identify the specific functional defects of patients, and provide objective basis for developing individualized rehabilitation plans; second, continuous monitoring of ADL function changes can track the recovery process, and guide the clinical team to dynamically adjust intervention measures; finally, standardized evaluation results provide scientific reference for discharge arrangement, community support, and medical resource allocation. Therefore, accurate and reliable evaluation of ADL is the core basis for developing effective rehabilitation strategies, tracking the recovery process, and optimizing resource allocation in the later stage.

[0003] At present, the ADL evaluation tools (such as Barthel index, Functional Independence Measure scale FIM) widely used in clinical practice mainly rely on the subjective observation and scoring of rehabilitation therapists. Such methods have three significant limitations: first, strong subjectivity, low inter-rater reliability. Second, low evaluation efficiency, unable to realize high-frequency monitoring. A complete evaluation usually takes 20-30 minutes, resulting in limited evaluation frequency, making it difficult to capture the daily fluctuations and subtle progress of patient function, causing rehabilitation intervention lag. Third, single evaluation dimension, lack of refined information. Traditional scales only provide qualitative grades such as “complete dependence” and “partial dependence”, and cannot quantify the quality, pattern or symmetry of the action, making it difficult to guide precise and individualized rehabilitation treatment.

[0004] With the development of technology, sensor-based automated assessment systems provide a new path to overcome the above limitations. Systems based on a single inertial measurement unit (IMU, usually worn on the wrist or waist) are of interest due to their low cost and portability. However, such systems have a fundamental flaw: their single sensing dimension cannot capture the coordinated movement of multiple joints and limbs necessary to complete ADL tasks. Essentially, single-sensor systems sacrifice the comprehensiveness of information for portability, resulting in a low ceiling for prediction accuracy and an inability to attribute functional impairment. On this basis, multi-sensor fusion systems significantly improve data integrity by deploying IMUs at multiple key points on the human body. Related research shows that increasing the number of sensors and model complexity (such as using traditional machine learning algorithms such as SVM, random forest, etc.) can improve prediction accuracy to some extent, especially in capturing limb coordination. However, these systems still face two major challenges: (1) single model architecture, making it difficult to effectively capture both the complex spatial relationships (such as inter-limb coordination) and temporal dynamics (such as action sequences) in ADL tasks; (2) poor feature interpretability, with most using general time-frequency statistical features, which lack clinical interpretability and are difficult for clinicians to understand and trust, making it difficult to effectively guide rehabilitation treatment;

[0005] Systematic analysis shows that there are four major technical gaps in existing research: First, data representation is incomplete, ignoring the coordination between the trunk-limb and bilateral limbs, affecting the accuracy of complex task assessment (such as standing-sitting transitions and stair climbing); Second, there is a lack of clinical interpretability, and general features (such as mean and variance) cannot directly map the "patient-healthy person" or "affected side-healthy side" differences; Third, there is a lack of spatio-temporal feature fusion, and single models cannot capture both spatial layout and temporal dynamics; Fourth, there is a lack of real-world scenario validation, and most systems have not been tested in home or community environments.

[0006] Therefore, there is an urgent need to develop an ADL intelligent assessment system that can overcome the above limitations and provide objective, efficient and accurate assessment. SUMMARY

[0007] The present application aims to solve the problems existing in the prior art and provides a daily life activity ability assessment method and system based on a multi-branch spatio-temporal fusion network.

[0008] The technical solution to achieve the purpose of the present application is a daily life activity ability assessment method based on a multi-branch spatio-temporal fusion network, the method comprising the following steps:

[0009] Step 1, deploy a multi-sensor system to maximize the capture of key motion information while minimizing the number of sensors and wearing discomfort;

[0010] Step 2, setting a standardized daily life activity ability evaluation task, namely a standardized ADL evaluation task, including a plurality of ADL evaluation sub-tasks;

[0011] Step 3, the subject to be tested completes the standardized ADL evaluation task, and in this process, the data of the multi-sensor system are synchronously collected;

[0012] Step 4, pre-processing the data collected in step 3, realizing denoising while making the data meet the requirements of subsequent network input, and performing feature extraction to obtain a feature vector;

[0013] Step 5, constructing and training a multi-branch spatio-temporal fusion network, inputting the feature vector into the trained multi-branch spatio-temporal fusion network to generate a score of each ADL evaluation sub-task and an ADL total score; the multi-branch spatio-temporal fusion network respectively captures spatial features and temporal features, and adaptively fuses the two types of features through a learnable attention mechanism, while generating a score of each ADL evaluation sub-task and an ADL total score.

[0014] Further, in step 1, the multi-sensor system includes a trunk sensor, an upper limb sensor and a lower limb sensor, the trunk sensor as a main sensor is placed in the middle of the sternum, the upper limb sensor is placed on both sides of the upper arm and the wrist, and the lower limb sensor is placed on both sides of the thigh and the ankle.

[0015] Further, in step 2, setting the standardized ADL evaluation task needs to follow the following principles: (a) clinical relevance - directly reflecting daily function; (b) measurability - reliable capture by sensors; (c) safety - minimization of execution risk; (d) time efficiency - compact and efficient evaluation process;

[0016] In step 2, the standardized ADL evaluation task is shown in Table 1 as follows:

[0017] Table 1 Standardized ADL evaluation task

[0018] Further, in step 4, feature extraction specifically includes:

[0019] (1) extracting traditional time-frequency domain features, including:

[0020] Time domain features: including mean, variance, peak value, kurtosis;

[0021] Frequency domain features: including dominant frequency, spectral energy, bandwidth, spectral entropy, harmonic ratio;

[0022] (2) extracting clinically interpretable features, including:

[0023] Action Deviation (ADD): Defined as the Euclidean distance between the motion trajectory of the object under test and the health reference template; the health reference template refers to the motion trajectory of the healthy object.

[0024] Action Similarity Index (ASI): Defined as the cosine similarity between the motion pattern of the test object and the health reference pattern; the health reference pattern refers to the motion pattern of a healthy object.

[0025] Affected-Healthy Side Symmetry Index (AUSI): Defined as the symmetry between the affected and healthy sides of a test subject, measured by comparing corresponding sensor data.

[0026] (3) Extracting spatial attitude features: Spatial attitude information is obtained from magnetometer data through quaternion conversion, including pitch angle, roll angle and heading angle of the trunk and upper / lower limbs;

[0027] Combine all the features from (1) to (3) above into a feature vector. N is the total number of samples, and M is the dimension of the feature vector.

[0028] Furthermore, the multi-branch spatiotemporal fusion network described in step 5 specifically includes:

[0029] (1) Input layer

[0030] Input data tensor X∈ N×T×C×S It contains N samples, each sample has T time steps, each sensor has C channels, and there are a total of S sensors;

[0031] Before processing can be performed on the network, the input data tensor needs to be reshaped into... This involves merging the sensor dimension and the channel dimension to meet the input format requirements of 1D-CNN;

[0032] At the same time, the input data tensors are normalized to ensure that they are all within a preset range;

[0033] (2) Spatial feature extraction branch

[0034] 1D-CNN is used to capture the spatial relationships and local motion patterns between sensors, while GNN is introduced to process the sensor topology and extract the spatial features of sensor motion data.

[0035] (3) Temporal feature extraction branch

[0036] A bidirectional LSTM model is used to model the temporal dependencies and action stages, and the temporal features of sensor motion data are extracted.

[0037] (4) Joint feature layer

[0038] This is used to fuse engineering features and deep learning features; the engineering features are the feature vectors obtained by feature extraction in step 4, and the deep learning features are the features extracted by the spatial feature extraction branch and the temporal feature extraction branch.

[0039] (5) Output layer

[0040] The output structure is hierarchical, generating scores for each ADL evaluation subtask and the total ADL score.

[0041] Furthermore, the spatial feature extraction branch is specifically implemented as follows:

[0042] (1) Extracting spatial feature vectors using 1D-CNN : ;

[0043] in,

[0044]

[0045]

[0046]

[0047] In the formula, This represents the convolution operation. For activation function, For convolution kernel, , For bias terms, This indicates a global average pooling operation; This is a max pooling operation; The output of the second convolution layer is represented as a feature slice of all samples and all channels of the second layer feature map at time step t. This is the pooling window size; This represents the pooling step size; Batch size; The result after normalizing the input data tensor X′;

[0048] (2) Use graph neural networks (GNNs) to model the topological relationships between sensors and obtain global topological features. : ;

[0049] in,

[0050]

[0051]

[0052]

[0053] In the formula, For the normalized adjacency matrix, For the first Hidden layer state; This represents the second layer of the graph neural network (GNN) and all features of all batches of samples from the s-th sensor.

[0054] Furthermore, the temporal feature extraction branch is specifically implemented as follows:

[0055] Extracting time feature vectors:

[0056]

[0057] in,

[0058]

[0059]

[0060]

[0061] In the formula, , These are bidirectional LSTMs with 128 and 256 hidden units, respectively. , These represent the final hidden states of the first and second BiLSTM layers, respectively. These are the weights and biases of the fully connected layer, respectively. This is a random deactivation operation used to prevent model overfitting during training; This represents the hidden state output of the second-layer BiLSTM at the last time step across all batches of samples; This represents the probability of neuron inactivation during the Dropout operation. This is the result of normalizing the input data tensor X′.

[0062] Furthermore, the joint feature layer is specifically implemented as follows:

[0063] (1) Project the features extracted from the spatial feature branch and the temporal feature branch onto the same dimension D:

[0064]

[0065] In the formula, These are spatial feature vectors. , , Features after projection;

[0066] (2) Perform interactive attention fusion:

[0067]

[0068] in,

[0069]

[0070]

[0071]

[0072]

[0073] In the formula, For the fused feature representation, This represents a multi-source feature vector fused through a concatenation operation. This represents the operation of concatenating multiple vectors along the feature dimension; These are the query, key, and value projection matrices, respectively. Scaling factor For gating unit signals, For the Sigmoid function; It is a linear transformation layer with weights and biases; A normalized exponential function is used to convert the input values ​​into a probability distribution, which is then used to generate attention weights;

[0074] (3) Integrating engineering features and deep learning features:

[0075]

[0076] In the formula, The output features of the joint feature layer, The feature vector representing the characteristics of an engineering project.

[0077] Furthermore, the formula for calculating the score of each ADL evaluation task generated by the output layer is as follows:

[0078]

[0079] in,

[0080]

[0081] In the formula, This represents the prediction score for r ADL subtasks; , For the corresponding bias term;

[0082] The formula for calculating the total ADL score generated by the output layer is as follows:

[0083]

[0084] In the formula, The overall score for ADL. The weights and biases are used to optimally combine the scores of each ADL subtask into the total score.

[0085] Furthermore, the training of the multi-branch spatiotemporal fusion network described in step 5 specifically includes:

[0086] Step 5-1: Using the multi-sensor system, collect motion data from several subjects with ADL impairments and several healthy controls, and score them using the Barthel Index to construct a daily living movement dataset; wherein, each subject completes a standardized ADL assessment task several times, and the current score is used as a label;

[0087] Step 5-2: Divide the daily life movement dataset into a training set, a validation set, and a test set using stratified sampling.

[0088] Step 5-3: Construct the weighted composite loss function Together, we will optimize the evaluation of ADL assessment sub-tasks and the prediction of the total ADL score.

[0089]

[0090] in,

[0091]

[0092]

[0093]

[0094]

[0095] In the formula, For scoring the damage function of sub-items, Let ADL total score loss function be used. The regularization loss function is... For consistency constraint loss function, They are respectively , , , The weight, This represents the actual score of the i-th ADL evaluation subtask. Let i be the predicted score for the i-th ADL evaluation subtask. This is the actual value of the total ADL score. This is the predicted value for the total ADL score. Let i be the weight matrix in the model;

[0096] Step 5-4: Based on the weighted composite loss function, the multi-branch spatiotemporal fusion network is trained using the training set to obtain the trained multi-branch spatiotemporal fusion network; wherein, the hyperparameters in the network model are determined by grid search combined with Bayesian optimization.

[0097] Step 5-5: Evaluate the regression performance of the trained network model. If it does not meet the requirements, return to step 5-1.

[0098] Compared with the prior art, the significant advantages of this invention are:

[0099] (1) This invention proposes a multi-branch spatiotemporal fusion network, which innovatively combines a one-dimensional convolutional neural network (1D-CNN) for extracting spatial features and a bidirectional long short-term memory network (Bi-LSTM) for extracting temporal features. It also uses graph neural network (GNN) branches to model the topological relationships between sensors and achieves adaptive weighted fusion through a cross-layer interactive attention mechanism. This design overcomes the limitations of a single architecture model and can simultaneously capture complex spatial relationships (such as limb coordination), temporal dynamics (such as action sequences), and topological structures (such as bilateral symmetry) in ADL tasks, significantly improving prediction accuracy.

[0100] (2) This invention proposes three novel clinically interpretable features: Action Deviation (ADD), Action Similarity Index (ASI), and Affected-Healthy Side Symmetry Index (AUSI). These features directly link sensor data with clinical concerns, enabling the system to transcend a simple "rating predictor" and become a clinical decision support tool capable of explaining the nature of movement abnormalities. ADD quantifies the difference between the patient's movements and a healthy reference template, ASI assesses the similarity of movement patterns, and AUSI measures the degree of bilateral limb symmetry. These indicators provide precise localization and quantification of functional deficit information for rehabilitation treatment.

[0101] (3) The ADL quantitative assessment system proposed in this invention employs multiple inertial measurement unit (IMU) sensors deployed at key locations on the patient's trunk and bilateral upper and lower limbs, enabling comprehensive capture of the patient's motion data. Combining eight standardized assessment tasks, the system achieves objective, efficient, and accurate ADL assessment through multi-dimensional feature extraction (time-frequency domain features, clinically interpretable features, and posture features).

[0102] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0103] Figure 1 This is a schematic diagram of the principle framework for assessing daily living activities based on a multi-branch spatiotemporal fusion network in one embodiment.

[0104] Figure 2This is a schematic diagram of an IMU sensor being worn in one embodiment, wherein... Figure 2 (a) in the image indicates the upper limb task-wearing position. Figure 2 (b) Lower limb task wearing position.

[0105] Figure 3 This is a diagram of the architecture of a multi-branch spatiotemporal fusion network (MSTF-Net) in one embodiment.

[0106] Figure 4 This is a schematic diagram of the cross-layer interactive attention mechanism module structure in one embodiment.

[0107] Figure 5 This is a flowchart of a method for assessing daily living activities based on a multi-branch spatiotemporal fusion network in one embodiment. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0109] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0110] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0111] In one embodiment, combined Figure 5 This paper presents a method for assessing Activities of Daily Living (ADL) abilities based on a multi-branch spatiotemporal fusion network, aiming to address the three major technical problems of existing ADL assessment methods: high subjectivity, low efficiency, and insufficient accuracy. The method includes the following steps:

[0112] Step 1: Deploy a multi-sensor system, aiming to capture key motion information to the maximum extent while minimizing the number of sensors and wearing discomfort;

[0113] Step 2: Set up standardized daily living activities assessment tasks, i.e., standardized ADL assessment tasks, which include multiple ADL assessment sub-tasks;

[0114] Step 3: The object under test completes the standardized ADL evaluation task, and data from the multi-sensor system is collected simultaneously during this process;

[0115] Step 4: Preprocess the data collected in Step 3 to remove noise while ensuring the data meets the requirements of subsequent network input, and extract features to obtain feature vectors.

[0116] Step 5: Construct and train a multi-branch spatiotemporal fusion network. Input the feature vector into the trained multi-branch spatiotemporal fusion network to generate the score for each ADL evaluation subtask and the total ADL score. The multi-branch spatiotemporal fusion network captures spatial features and temporal features respectively, and adaptively fuses the two types of features through a learnable attention mechanism, while generating the score for each ADL evaluation subtask and the total ADL score.

[0117] Figure 1 This diagram illustrates the overall research framework of the present invention, showing all the key stages of the system's workflow: motion data acquisition, data preprocessing, multi-branch spatiotemporal fusion network model, and result output. The diagram, presented in flowchart form, illustrates the entire system's working principle and data flow, from data acquisition to the final ADL scale scoring.

[0118] Furthermore, in one embodiment, in step 1, the multi-sensor system includes a trunk sensor, an upper limb sensor, and a lower limb sensor. The trunk sensor, serving as the main sensor, is positioned in the middle of the sternum (monitoring overall posture, center of gravity shift, and core stability). The upper limb sensors are positioned on both upper arms and wrists (capturing shoulder joint movements and upper limb coordination), and the lower limb sensors are positioned on both thighs and ankles (tracking lower limb strength, gait, and balance changes). Each sensor is secured using a custom-designed elastic strap with embedded silicone anti-slip strips to ensure positional stability during movement.

[0119] Here, data can also be collected sequentially from one limb (affected side and healthy side), which can reduce the number of sensors (five are sufficient).

[0120] Figure 2 This is a schematic diagram illustrating the wearing of the IMU sensors of the present invention, detailing the placement of the five IMU sensors on the patient's body, including the trunk and both upper and lower limbs. Specifically, the diagram shows: the trunk sensor is located in the middle of the sternum (between the 4th and 5th ribs); the bilateral upper limb sensors are located in the upper arm (mid-humerus) and wrist; and the bilateral lower limb sensors are located in the thigh (mid-femur) and ankle.

[0121] Preferably, in some embodiments, each sensor in the multi-sensor system is a wireless inertial measurement unit (IMU) sensor, and each sensor integrates a three-axis accelerometer (±16g), a three-axis gyroscope (±2000° / s), and a three-axis magnetometer (±4800μT), providing complete nine-axis motion data. The data sampling rate is set to 50Hz, the sensors are powered by rechargeable lithium batteries (battery life >8 hours), communicate via the ZigBee protocol with a communication distance of approximately 10 meters, and are powered by a 150 mAh rechargeable lithium battery, supporting continuous operation for up to 8 hours.

[0122] Here, all sensors in the multi-sensor system must undergo two-step calibration before each use: (1) Static calibration: The sensors are placed in six standard directions on the horizontal plane for t seconds (15 seconds) each, and the zero bias and scaling factor are calculated; (2) Dynamic calibration: Standardized rotational motion is performed to verify the consistency between sensors. Sensor synchronization adopts timestamp alignment technology, with the master sensor (body) serving as the time reference, and the slave sensors synchronized via a wireless protocol, with time drift <10ms.

[0123] Furthermore, in one embodiment, the standardized ADL assessment task set in step 2 should follow the following principles: (a) clinical relevance—directly reflecting daily function; (b) measurability—reliable sensor capture; (c) safety—minimizing execution risk; and (d) time efficiency—compact and efficient assessment process.

[0124] Preferably, in some embodiments, in step 2, the standardized ADL evaluation tasks are as shown in Table 2 below:

[0125] Table 2 Standardized ADL Assessment Tasks

[0126] Furthermore, in one embodiment, step 4 involves preprocessing the data, specifically including:

[0127] Signal filtering: A fourth-order Butterworth low-pass filter with a cutoff frequency of 20Hz is applied to the accelerometer and gyroscope data to remove high-frequency noise and retain the main motion components. A moving average filter (window size 5) is applied to the magnetometer data to reduce interference from ambient magnetic fields.

[0128] Action segmentation: Task start and end points are identified through acceleration threshold detection; dynamic time warping (DTW) algorithm is used to align the action timeline. Using torso acceleration as the primary signal, action intervals are automatically extracted, with a 0.5-second buffer before and after, and resting periods are removed.

[0129] Time alignment: Align action phases using a dynamic time warping algorithm;

[0130] Length uniformity: By normalizing time (e.g., linear interpolation resampling to 300 time points) and padding short sequences with zeros, the time series is resampled to T time points.

[0131] Furthermore, in one embodiment, feature extraction in step 4 specifically includes:

[0132] (1) Extract traditional time-frequency domain features (dimensions can be customized), including:

[0133] Time-domain characteristics include mean, variance, peak value, and kurtosis.

[0134] Frequency domain characteristics include dominant frequency, spectral energy, bandwidth, spectral entropy, and harmonic ratio.

[0135] (2) Extract clinically interpretable features, including:

[0136] Motion Deviation ADD: Defined as the Euclidean distance between the motion trajectory of the test object and the health reference template; the health reference template refers to the motion trajectory of a healthy object; expressed as:

[0137]

[0138] in, For the duration of the task (time point). The vector represents the motion of the object under test at time t (3 accelerations + 3 gyroscope axes). A 6-dimensional vector representing the motion of a healthy subject. It is an L2 norm.

[0139] Traditional methods only calculate the Euclidean distance of a single moment or the entire trajectory. The ADD calculation formula introduces a time-series averaging mechanism. It can capture the cumulative deviation throughout the entire movement process and is more sensitive to abnormalities at different stages of the movement. ADD is used to assess the overall difference between a patient's movement and a healthy template; the smaller the value, the closer it is to healthy movement.

[0140] Action Similarity Index (ASI): Defined as the cosine similarity between the motion pattern of the test subject and the motion pattern of a healthy reference subject; the healthy reference subject refers to the motion pattern of a healthy subject; expressed as:

[0141]

[0142] in, For the duration of the task (time point). A 6-dimensional vector representing the motion of the object under test (3 accelerations + 3 gyroscope axes). A 6-dimensional vector representing the motion of a healthy subject. It is an L2 norm.

[0143] Traditional methods use DTW (Dynamic Time Warping) distance, which has high computational complexity. ASI calculation, however, employs a temporal extension of cosine similarity, resulting in lower computational complexity. Furthermore, normalization eliminates interference from differences in motion amplitude. ASI measures motion pattern similarity, ranging from 0 to 1; values ​​closer to 1 indicate higher similarity.

[0144] Affected-Healthy Side Symmetry Index (AUSI): Defined as the symmetry between the affected and healthy sides of a test subject, measured by comparing corresponding sensor data.

[0145]

[0146] in, The feature vector of the affected limb. For the feature vector of the healthy limb, It is an L2 norm.

[0147] Traditional methods of comparing symmetry simply compare the amplitude ratio of signals on the left and right sides. The AUSI calculation formula uses high-dimensional feature vectors to calculate symmetry, which can more comprehensively reflect bilateral functional differences. AUSI is used to quantify the degree of bilateral limb symmetry, ranging from 0 to 1; the closer to 1, the better the symmetry.

[0148] (3) Extracting spatial posture features: Spatial posture information is obtained by converting magnetometer data into quaternions, including pitch, roll, and yaw angles of the trunk and upper / lower limbs, which reflect the forward and backward tilting, left and right tilting, and horizontal rotation of the body, respectively. This can enhance the quantitative description of the body's spatial configuration during movement. For example, in the task of maintaining standing balance, changes in pitch angle can intuitively reflect the degree of forward or backward tilting of the patient's body, providing supplementary evidence for balance function assessment. The trunk posture angles include pitch, roll, and yaw angles, and the limb posture angles include, but are not limited to, the pitch angle, yaw angle, and roll angle of the affected limb, forming a 6-dimensional posture feature.

[0149] Combine all the features from (1) to (3) above into a feature vector. N is the total number of samples, and M is the dimension of the feature vector, such as Figure 3 As shown, the feature combination follows the "complementarity principle" to ensure that each type of feature captures unique information: traditional features provide basic motion parameters, novel features quantify clinical functional relevance, and posture features describe body spatial configuration.

[0150] Traditional single-architecture neural networks (CNNs or LSTMs) struggle to simultaneously handle the complex spatial configurations (limb relationships, pose alignment) and temporal dynamics (action sequences, rhythmic patterns) in ADL tasks. While traditional CNN architectures excel at extracting local spatial patterns, they fall short in modeling the long-range temporal dependencies required for functional action evaluation. Recurrent architectures like LSTMs effectively capture temporal patterns but lack the ability to extract multi-sensor spatial features. To address the limitation of existing models in "inability to fuse spatiotemporal features," this invention proposes a multi-branch spatiotemporal fusion network (MSTF-Net) that separately captures spatial features (limb symmetry, sensor spatial relationships) and temporal features (action rhythms, sequence dynamics), and adaptively fuses these two types of features through a learnable attention mechanism, thus overcoming the performance bottleneck of single-architecture networks.

[0151] Furthermore, in one embodiment, combined with Figure 3 The multi-branch spatiotemporal fusion network (MSTF-Net) mentioned in step 5 specifically includes:

[0152] (1) Input layer

[0153] Input data tensor X∈ N×T×C×S It contains N samples, each sample has T time steps, and each sensor has C channels (C = 6 channels, triaxial acceleration and triaxial angular velocity), for a total of S sensors;

[0154] Before processing can be performed on the network, the input data tensor needs to be reshaped into... This involves merging the sensor dimension and the channel dimension to meet the input format requirements of 1D-CNN;

[0155] At the same time, the input data tensors are normalized to ensure that they are all within a preset range;

[0156] Preferably, input normalization uses channel-level z-score normalization:

[0157]

[0158] in and The training set contains the mean and standard deviation of each channel.

[0159] (2) Spatial feature extraction branch

[0160] 1D-CNN is used to capture the spatial relationships and local motion patterns between sensors, while GNN is introduced to process the sensor topology (using human joints as nodes and the distance between sensors as edge weights) to extract the spatial features of sensor motion data.

[0161] This branch runs independently at each time step, learning spatial filters along the sensor dimension. The 1D-CNN architecture is dedicated to extracting three key spatial attributes for ADL assessment: inter-sensor correlations characterizing limb coordination, sensor-specific activity patterns quantifying unilateral limb function, and bilateral sensor spatial symmetry measuring differences between affected and healthy sides.

[0162] (3) Temporal feature extraction branch

[0163] A bidirectional LSTM model is used to model the temporal dependencies and action stages, and the temporal features of sensor motion data are extracted.

[0164] This temporal feature extraction branch is specifically used to extract and identify features such as the temporal coordination of sensor signals in the action transition phase, quantify the regularity of movement, and characterize the synchronization between limbs;

[0165] (4) Joint feature layer

[0166] This is used to fuse engineering features and deep learning features; the engineering features are the feature vectors obtained by feature extraction in step 4, and the deep learning features are the features extracted by the spatial feature extraction branch and the temporal feature extraction branch.

[0167] (5) Output layer

[0168] The output structure is hierarchical, generating scores for each ADL evaluation subtask and the total ADL score.

[0169] Preferably, in some embodiments, combined with Figure 4 The spatial feature extraction branch is specifically implemented as follows:

[0170] (1) Extracting spatial feature vectors using 1D-CNN :

[0171]

[0172] in,

[0173]

[0174]

[0175]

[0176] In the formula, This represents the convolution operation. For activation function, For convolution kernel, , For bias terms, This indicates a global average pooling operation; This is a max pooling operation; The output of the second convolution layer is represented as a feature slice of all samples and all channels of the second layer feature map at time step t. The pooling window size is set to k=2, which means that the maximum value is taken every two time points. The pooling step size, =2 means that the pooling window moves by 2 time points each time; Batch size; The result after normalizing the input data tensor X′;

[0177] (2) Use graph neural networks (GNNs) to model the topological relationships between sensors and display the features before the spatial branch gap. Treating each node as a set of node features (if there are five sensors, then each time step has feature maps of five nodes), we obtain the global topological features. :

[0178] in,

[0179]

[0180]

[0181]

[0182] In the formula, For the normalized adjacency matrix, For the first Hidden layer state; This represents the second layer of the graph neural network (GNN) and all features of all batches of samples from the s-th sensor.

[0183] Preferably, in some embodiments, the temporal feature extraction branch is specifically implemented as follows:

[0184] Extracting time feature vectors:

[0185]

[0186] in,

[0187]

[0188]

[0189]

[0190] In the formula, , These are bidirectional LSTMs with 128 and 256 hidden units, respectively. , These represent the final hidden states of the first and second BiLSTM layers, respectively. These are the weights and biases of the fully connected layer, respectively. This is a random deactivation operation used to prevent model overfitting during training; This represents the hidden state output of the second-layer BiLSTM at the last time step across all batches of samples; The probability of neuron inactivation during the Dropout operation is preferably... , This is the result of normalizing the input data tensor X′.

[0191] Preferably, in some embodiments, the joint feature layer is specifically implemented as follows:

[0192] (1) Project the features extracted from the spatial feature branch and the temporal feature branch onto the same dimension D (preferably, D is 128):

[0193]

[0194] In the formula, These are spatial feature vectors. , , Features after projection;

[0195] (2) Perform interactive attention fusion:

[0196]

[0197] in,

[0198]

[0199]

[0200]

[0201]

[0202] In the formula, For the fused feature representation, This represents a multi-source feature vector fused through a concatenation operation. This represents the operation of concatenating multiple vectors along the feature dimension; These are the query, key, and value projection matrices, respectively. Scaling factor For gating unit signals, For the Sigmoid function; It is a linear transformation layer with weights and biases; A normalized exponential function is used to convert the input values ​​into a probability distribution, which is then used to generate attention weights;

[0203] (2) Integrating engineering features and deep learning features:

[0204]

[0205] In the formula, The output features of the joint feature layer, The feature vector representing the characteristics of an engineering project.

[0206] Preferably, in some embodiments, the output layer generates a score for each ADL evaluation task using the following formula:

[0207]

[0208] in,

[0209]

[0210] In the formula, This represents the prediction score for r ADL subtasks; , For the corresponding bias term;

[0211] The formula for calculating the total ADL score generated by the output layer is as follows:

[0212]

[0213] In the formula, The overall score for ADL. The weights and biases are used to optimally combine the scores of each ADL subtask into the total score.

[0214] Here, the two-tiered structure ensures consistency between the project score and the total score, and allows the model to learn the difference in importance of each sub-project task to the overall ADL assessment.

[0215] Furthermore, in one embodiment, the training of the multi-branch spatiotemporal fusion network in step 5 specifically includes:

[0216] Step 5-1: Using the multi-sensor system, collect motion data from several subjects with ADL impairments and several healthy controls, and score them using the Barthel Index to construct a daily living movement dataset; wherein, each subject completes a standardized ADL assessment task several times, and the current score is used as a label;

[0217] Step 5-2: Divide the daily life movement dataset into a training set, a validation set, and a test set using stratified sampling.

[0218] Preferably, the stratification is based on the total ADL score range, such as 0-20, 21-40, 41-60, 61-80, 81-100, and the scoring here uses the Barthel Index.

[0219] Step 5-3: Construct the weighted composite loss function Together, we will optimize the evaluation of ADL assessment sub-tasks and the prediction of the total ADL score.

[0220]

[0221] in,

[0222]

[0223]

[0224]

[0225]

[0226] In the formula, For scoring the damage function of sub-items, Let ADL total score loss function be used. The regularization loss function is... As a consistency-constrained loss function, when the scoring system supports linear summation, this constraint loss function ensures consistency between the total score and the sub-task scores. They are respectively , , , The weight, This represents the actual score of the i-th ADL evaluation subtask. Let i be the predicted score for the i-th ADL evaluation subtask. This is the actual value of the total ADL score. This is the predicted value for the total ADL score. Let i be the weight matrix in the model; preferably, ;

[0227] Step 5-4: Based on the weighted composite loss function, train the multi-branch spatiotemporal fusion network using the training set to obtain the trained multi-branch spatiotemporal fusion network; wherein, the hyperparameters in the network model are determined by grid search combined with Bayesian optimization (preferably, batch size is 32; initial learning rate is 0.001; dropout rate is 0.3; L2 regularization strength is 1e-4 (to suppress excessive weights and avoid overfitting); maximum training epochs are 50. To avoid overfitting, in addition to dropout and L2 regularization, an early stopping mechanism is adopted: the tolerance is set to 5, if the validation set loss does not decrease for 5 consecutive epochs, training is terminated, and the model with the lowest validation loss is saved).

[0228] Step 5-5: Evaluate the regression performance of the trained network model. If it does not meet the requirements, return to step 5-1.

[0229] Here, a preferred approach is to use three indicators to comprehensively evaluate the regression performance of the model:

[0230] Coefficient of determination ( This measures the proportion of the variance in the true scores explained by the model (range: -∞ to 1). It is considered clinically significant.

[0231]

[0232] Root Mean Square Error (RMSE): Quantifies the average magnitude of prediction error (unit: 0-100). RMSE < 5 is considered clinically acceptable.

[0233]

[0234] Mean Absolute Error (MAE): Measures the average absolute deviation between predicted and actual values ​​(less sensitive to outliers than RMSE), reflecting the robustness of the model.

[0235]

[0236] in, Rate the actual ADL (Advanced Level Requirement). To predict the score, This represents the average of the actual ratings.

[0237] In one embodiment, a daily living activities assessment system based on a multi-branch spatiotemporal fusion network is provided, the system comprising:

[0238] The first module is used to deploy a multi-sensor system that aims to capture key motion information to the maximum extent while minimizing the number of sensors and wearing discomfort.

[0239] The second module is used to implement: setting standardized daily living activities ability assessment tasks, i.e., standardized ADL assessment tasks, which include multiple ADL assessment sub-tasks;

[0240] The third module is used to enable the test object to complete a standardized ADL evaluation task, and to simultaneously collect data from the multi-sensor system during the process.

[0241] The fourth module is used to: preprocess the data collected by the third module to remove noise while ensuring that the data meets the requirements of subsequent network input, and extract features to obtain feature vectors;

[0242] The fifth module is used to: construct and train a multi-branch spatiotemporal fusion network, input the feature vector into the trained multi-branch spatiotemporal fusion network, and generate a score for each ADL evaluation subtask and a total ADL score; the multi-branch spatiotemporal fusion network captures spatial features and temporal features respectively, and adaptively fuses the two types of features through a learnable attention mechanism, while generating a score for each ADL evaluation subtask and a total ADL score.

[0243] Specific limitations regarding the assessment system for activities of daily living (ADL) based on multi-branch spatiotemporal fusion networks can be found in the limitations of the ADL assessment method based on multi-branch spatiotemporal fusion networks described above, and will not be repeated here. Each module in the aforementioned ADL assessment system based on multi-branch spatiotemporal fusion networks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0244] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for assessing daily living activities based on a multi-branch spatiotemporal fusion network.

[0245] For specific limitations on each step, please refer to the limitations on the assessment method of daily living activities based on multi-branch spatiotemporal fusion networks mentioned above, which will not be repeated here.

[0246] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for assessing daily living activities based on a multi-branch spatiotemporal fusion network.

[0247] For specific limitations on each step, please refer to the limitations on the assessment method of daily living activities based on multi-branch spatiotemporal fusion networks mentioned above, which will not be repeated here.

[0248] As a specific example, in one embodiment, to verify the effectiveness and reliability of the invention, a systematic test was conducted on 60 subacute patients (Brunnstrom III–V) and 20 healthy controls. Each subject completed 8 standardized ADL tasks, each repeated 3 times, constructing an Activities of Daily Living (ADL) dataset. Stratified sampling was used to partition the dataset into training / validation / test sets in a 7:2:1 ratio (stratification based on ADL total score ranges: 0-20, 21-40, 41-60, 61-80, 81-100). This approach ensures a consistent score distribution across sets, avoiding bias caused by uneven sample representation. To evaluate the generalization ability of the MSTF-Net model on new patients, a "leave-one-patient-out" cross-validation scheme was used: the data from the 60 patients were alternately divided into 59 training and 1 testing subjects, and the average performance of the 60 results was calculated. This method is more rigorous than random partitioning and can more accurately assess the model's applicability in clinical practice.

[0249] The test results show that:

[0250] 1. Prediction Accuracy: MSTF-Net achieved excellent prediction performance on the test set, with a determination coefficient (R²) of 0.925, a root mean square error (RMSE) of 1.85, and a mean absolute error (MAE) of 2.89 for the ADL total score prediction. Under the "keep one patient out" cross-validation scheme, R² decreased by only 0.021 (from 0.925 to 0.904), validating the model's robustness on unseen new patient data.

[0251] 2. Clinical Consistency: Bland-Altman analysis assessed the consistency between model predictions and clinical scores. The results showed a mean difference (bias) of 0.08 points, close to zero, indicating no systematic bias in the predictions. The 95% bound on agreement (LOA) was [-1.2, 1.3], meaning that 95% of the prediction error falls within this range, far less than the clinically pre-defined acceptable difference threshold (±2 points).

[0252] 3. Task-specific performance: The system achieved clinically acceptable accuracy (R²>0.85) on all eight ADL tasks, with the overall predictive performance for lower limb tasks (mean R²=0.929) significantly outperforming that of upper limb tasks (mean R²=0.877). This difference is mainly attributed to the fact that lower limb tasks involve a larger range of motion, generating stronger sensor signals, more stable sensor positions, and higher scoring consistency.

[0253] In summary, the method and system for assessing daily living activities based on multi-branch spatiotemporal fusion networks proposed in this invention have the following characteristics:

[0254] 1. High Prediction Accuracy: This system achieves R²=0.925 and RMSE=1.85 in predicting the total ADL score, significantly outperforming traditional machine learning methods (such as SVM: R²=0.78) and single-branch deep learning models (such as CNN: R²=0.876). This high accuracy stems from the multi-branch spatiotemporal fusion network's precise capture of complex action patterns and the innovative features' effective representation of key clinical indicators.

[0255] 2. Objective Consistency: Bland-Altman analysis showed that the system was highly consistent with the clinical gold standard (95% concordance threshold [-1.2, 1.3]), far superior to traditional inter-rater reliability (typically 0.68-0.82). This objective consistency effectively eliminates the uncertainty caused by subjective differences among raters, ensuring high consistency of assessment results across different clinical environments and physicians.

[0256] 3. Refined Quantification: Through innovative clinically interpretable features—ADD (Action Deviation), ASI (Action Similarity Index), and AUSI (Affected and Healthy Side Symmetry Index)—the system achieves precise localization and quantification of functional deficits, supporting personalized rehabilitation. These features directly link sensor data with clinical concerns, enabling the system to transcend a simple "rating predictor" and become a "clinical decision support tool" capable of explaining the nature of movement abnormalities.

[0257] 4. Comprehensive Spatiotemporal Integration: The Multi-Branch Spatiotemporal Fusion Network (MSTF-Net) architecture innovatively combines spatial features extracted by 1D-CNN with temporal features extracted by Bi-LSTM, and utilizes topological fusion with Graph Neural Networks (GNNs). Through a cross-layer interactive attention mechanism, it achieves adaptive feature weighting fusion, significantly improving prediction accuracy. Compared to single-architecture models, this fusion method can more comprehensively capture the complex spatiotemporal dynamics of ADL tasks.

[0258] This invention represents a significant advancement in rehabilitation assessment technology, providing objective, efficient, and refined ADL (Activities of Daily Living) ability assessments. Through innovative feature design and spatiotemporal fusion architecture, it achieves high predictive accuracy while providing clinically interpretable insights, supporting personalized rehabilitation interventions and monitoring assessments.

[0259] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for assessing activities of daily living based on a multi-branch spatiotemporal fusion network, characterized in that, The method includes the following steps: Step 1: Deploy a multi-sensor system, aiming to capture key motion information to the maximum extent while minimizing the number of sensors and wearing discomfort; Step 2: Set up standardized daily living activities assessment tasks, i.e., standardized ADL assessment tasks, which include multiple ADL assessment sub-tasks; Step 3: The object under test completes the standardized ADL evaluation task, and data from the multi-sensor system is collected simultaneously during this process; Step 4: Preprocess the data collected in Step 3 to remove noise while ensuring the data meets the requirements of subsequent network input, and extract features to obtain feature vectors. Step 5: Construct and train a multi-branch spatiotemporal fusion network. Input the feature vector into the trained multi-branch spatiotemporal fusion network to generate the score for each ADL evaluation subtask and the total ADL score. The multi-branch spatiotemporal fusion network captures spatial features and temporal features respectively, and adaptively fuses the two types of features through a learnable attention mechanism, while generating the score for each ADL evaluation subtask and the total ADL score.

2. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 1, characterized in that, In step 1, the multi-sensor system includes a trunk sensor, an upper limb sensor, and a lower limb sensor. The trunk sensor, as the main sensor, is placed in the middle of the sternum. The upper limb sensors are placed on both upper arms and wrists, and the lower limb sensors are placed on both thighs and ankles.

3. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 1, characterized in that, Step 2, when setting up standardized ADL assessment tasks, should follow these principles: (a) clinical relevance—directly reflecting daily function; (b) measurability—reliable sensor capture; (c) safety—minimizing execution risks; (d) time efficiency—compact and efficient assessment process. In step 2, the standardized ADL assessment tasks are shown in the table below: 。 4. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 1, characterized in that, Step 4, feature extraction, specifically includes: (1) Extracting traditional time-frequency domain features, including: Time-domain characteristics: including mean, variance, peak value, and kurtosis; Frequency domain characteristics include dominant frequency, spectral energy, bandwidth, spectral entropy, and harmonic ratio. (2) Extract clinically interpretable features, including: Action Deviation (ADD): Defined as the Euclidean distance between the motion trajectory of the object under test and the health reference template; the health reference template refers to the motion trajectory of the healthy object. Action Similarity Index (ASI): Defined as the cosine similarity between the motion pattern of the test object and the health reference pattern; the health reference pattern refers to the motion pattern of a healthy object. Affected-Healthy Side Symmetry Index (AUSI): Defined as the symmetry between the affected and healthy sides of a test subject, measured by comparing corresponding sensor data. (3) Extracting spatial attitude features: Spatial attitude information is obtained from magnetometer data through quaternion conversion, including pitch angle, roll angle and heading angle of the trunk and upper / lower limbs; Combine all the features from (1) to (3) above into a feature vector. N is the total number of samples, and M is the dimension of the feature vector.

5. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 1, characterized in that, The multi-branch spatiotemporal fusion network mentioned in step 5 specifically includes: (1) Input layer Input data tensor X∈ N×T×C×S It contains N samples, each sample has T time steps, each sensor has C channels, and there are a total of S sensors; Before processing can be performed on the network, the input data tensor needs to be reshaped into... This involves merging the sensor dimension and the channel dimension to meet the input format requirements of 1D-CNN; At the same time, the input data tensors are normalized to ensure that they are all within a preset range; (2) Spatial feature extraction branch 1D-CNN is used to capture the spatial relationships and local motion patterns between sensors, while GNN is introduced to process the sensor topology and extract the spatial features of sensor motion data. (3) Temporal feature extraction branch A bidirectional LSTM model is used to model the temporal dependencies and action stages, and the temporal features of sensor motion data are extracted. (4) Joint feature layer This is used to fuse engineering features and deep learning features; the engineering features are the feature vectors obtained by feature extraction in step 4, and the deep learning features are the features extracted by the spatial feature extraction branch and the temporal feature extraction branch. (5) Output layer The output structure is hierarchical, generating scores for each ADL evaluation subtask and the total ADL score.

6. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 5, characterized in that, The spatial feature extraction branch is specifically implemented as follows: (1) Extracting spatial feature vectors using 1D-CNN : ； in, ； ； ； In the formula, This represents the convolution operation. For activation function, For convolution kernel, , For bias terms, This indicates a global average pooling operation; This is a max pooling operation; The output of the second convolution layer is represented as a feature slice of all samples and all channels of the second layer feature map at time step t. This is the pooling window size; This represents the pooling step size; Batch size; The result after normalizing the input data tensor X′; (2) Use graph neural networks (GNNs) to model the topological relationships between sensors and obtain global topological features. : ； in, ； ； ； In the formula, For the normalized adjacency matrix, For the first Hidden layer state; This represents the second layer of the graph neural network (GNN) and all features of all batches of samples from the s-th sensor.

7. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 6, characterized in that, The temporal feature extraction branch is specifically implemented as follows: Extracting time feature vectors: ； in, ； ； ； In the formula, , These are bidirectional LSTMs with 128 and 256 hidden units, respectively. , These represent the final hidden states of the first and second BiLSTM layers, respectively. These are the weights and biases of the fully connected layer, respectively. This is a random deactivation operation used to prevent model overfitting during training; This represents the hidden state output of the second-layer BiLSTM at the last time step across all batches of samples; This represents the probability of neuron inactivation during the Dropout operation. This is the result of normalizing the input data tensor X′.

8. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 7, characterized in that, The joint feature layer is specifically implemented as follows: (1) Project the features extracted from the spatial feature branch and the temporal feature branch onto the same dimension D: ； In the formula, These are spatial feature vectors. , , Features after projection; (2) Perform interactive attention fusion: ； in, ； ； ； ； In the formula, For the fused feature representation, This represents a multi-source feature vector fused through a concatenation operation. This represents the operation of concatenating multiple vectors along the feature dimension; These are the query, key, and value projection matrices, respectively. Scaling factor For gating unit signals, For the Sigmoid function; It is a linear transformation layer with weights and biases; A normalized exponential function is used to convert the input values ​​into a probability distribution, which is then used to generate attention weights; (3) Integrating engineering features and deep learning features: ； In the formula, The output features of the joint feature layer, The feature vector representing the characteristics of an engineering project.

9. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 8, characterized in that, The formula for calculating the score of each ADL evaluation task generated by the output layer is as follows: ； in, ； In the formula, This represents the prediction score for r ADL subtasks; , For the corresponding bias term; The formula for calculating the total ADL score generated by the output layer is as follows: ； In the formula, The overall score for ADL. The weights and biases are used to optimally combine the scores of each ADL subtask into the total score.

10. The method for assessing daily living activities based on a multi-branch spatiotemporal fusion network according to claim 9, characterized in that, Step 5, training the multi-branch spatiotemporal fusion network, specifically includes: Step 5-1: Using the multi-sensor system, collect motion data from several subjects with ADL impairments and several healthy controls, and score them using the Barthel Index to construct a daily living movement dataset; wherein, each subject completes a standardized ADL assessment task several times, and the current score is used as a label; Step 5-2: Divide the daily life movement dataset into a training set, a validation set, and a test set using stratified sampling. Step 5-3: Construct the weighted composite loss function Together, we will optimize the evaluation of ADL assessment sub-tasks and the prediction of the total ADL score. ； in, ； ； ； ； In the formula, For scoring the damage function of sub-items, Let ADL total score loss function be used. The regularization loss function is... For consistency constraint loss function, They are respectively , , , The weight, This represents the actual score of the i-th ADL evaluation subtask. Let i be the predicted score for the i-th ADL evaluation subtask. This is the actual value of the total ADL score. This is the predicted value for the total ADL score. Let i be the weight matrix in the model; Step 5-4: Based on the weighted composite loss function, the multi-branch spatiotemporal fusion network is trained using the training set to obtain the trained multi-branch spatiotemporal fusion network; wherein, the hyperparameters in the network model are determined by grid search combined with Bayesian optimization. Step 5-5: Evaluate the regression performance of the trained network model. If it does not meet the requirements, return to step 5-1.

Images