A method and device for analyzing and quantitatively evaluating limb activity

Abstract

This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for analyzing and quantifying limb activity. The method includes: extracting features from a skeleton sequence to obtain skeletal feature tensors and velocity feature tensors to form a combined feature tensor, and performing spatiotemporal alignment and normalization; pre-training an autoencoder based on the normalized combined feature tensor, learning a fusion representation of features through the autoencoder, and outputting the pre-trained weights and bottleneck feature set of the autoencoder; pre-training a limb activity regression network based on the bottleneck feature set, and outputting the pre-trained weights of the limb activity regression network; optimizing and updating the parameters of the autoencoder and the limb activity regression network through joint fine-tuning; removing the decoder of the optimized and updated autoencoder, and connecting the autoencoder and the limb activity regression network to construct an analysis and evaluation model for the inference stage, thereby achieving quantitative evaluation of limb activity during the inference stage.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for analyzing and quantifying limb activity. Background Technology

[0002] Limb activity refers to the quantitative representation of the movement efficiency of various parts of the human body per unit time, described by a continuous scale value from 0.0 (rest) to 1.0 (maximum intensity). Traditional sensor solutions use inertial sensors, surface electromyography (EMG) devices, etc., attached to human joints to process the acquired sensor signals and assess limb activity. This approach suffers from problems such as unnatural movement due to invasiveness, spatiotemporal asynchrony of data from multiple sensors, and high costs. In recent years, with the rapid development of visual computing technology, limb activity analysis based on visual computing has become a new research trend. The limb activity visual analysis scheme uses a sequence of human joint points acquired by a visual sensor capable of collecting depth information. Machine learning methods are then used to process and analyze the sequence to obtain limb activity assessment results. The human joint points follow the standard of the human joint topology model; the complete sequence contains 17 joint points, covering the head, cervical spine, thoracic spine, shoulder, elbow, wrist, hip, knee, ankle, and foot.

[0003] Current research on vision-based limb activity analysis includes the following methods. For example, the application "A Behavioral and Psychological Anomaly Detection Method Based on Limb Activity" (application number: 202111050931.9) proposes a method for predicting abnormal limb activity levels. However, this method only extracts motion features using time-series mean to construct a limb activity vector and employs an activity level prediction model based on traditional machine learning methods, resulting in insufficient extraction of feature details. Another example is the application "An Abnormal Emotion Inference System Based on Posture Feature Alignment" (application number: 202410435408.5), which proposes a method for extracting posture features and fusing multimodal features to infer emotions. Its focus is on the fusion of multimodal features, lacking discriminative analysis of limb biomechanical characteristics.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that current methods and devices for analyzing limb activity have insufficient feature detail extraction or lack of limb difference design.

[0006] The present invention adopts the following technical solution: In a first aspect, the present invention provides a method for analyzing and quantifying limb activity, comprising: Extract features from the skeleton sequence to obtain a skeleton feature tensor and a velocity feature tensor to form a combined feature tensor, and perform spatio-temporal alignment and normalization; Pre-train the autoencoder based on the normalized combined feature tensor, learn the fused representation of the features through the autoencoder, and output the pre-trained weights of the autoencoder and the bottleneck feature set; Pre-train the limb activity regression network based on the bottleneck feature set, and output the pre-trained weights of the limb activity regression network; Jointly fine-tune the autoencoder and the limb activity regression network to complete the parameter optimization and update of the autoencoder and the limb activity regression network; Remove the decoder of the autoencoder after optimization and update, and connect the autoencoder and the limb activity regression network to build an analysis and evaluation model in the inference stage, and realize the quantitative evaluation of limb activity in the inference stage.

[0007] Furthermore, the method further includes: Obtain the joint coordinate sequence {SK i,t} of the skeleton sequence, where i ∈ {1, …, V}, t ∈ {1, …, T}, V represents the number of nodes, and T represents the number of frames; Calculate the velocity feature vector S i,t =SK i,t -SK i,（t-1） to form the velocity feature tensor S, where S ∈ R V×(T-1)×3 , t ∈ {2, …, T}; Calculate the skeleton feature vector B j,t =SK j,t -SK i,t to form the skeleton feature tensor B, where B ∈ R (V-1)×T×3 , j ∈ {2, …, V}, i is the adjacent node closer to the central node, i < j, and the skeleton feature vector corresponding to the central node is initialized as a zero vector to match the adjacency matrix.

[0008] Furthermore, the method further includes: Discard the velocity of the central node to reduce the dimension of the velocity feature tensor, and discard the first frame to intercept the skeleton feature tensor to reduce the dimension of the skeleton feature tensor, so that the dimensions of the skeleton feature tensor and the velocity feature tensor are the same, align all velocity features and skeleton features in the spatial and temporal dimensions, combine the velocity feature tensor and the skeleton feature tensor into a combined feature tensor, and normalize the combined feature tensor after spatio-temporal alignment.

[0009] Furthermore, the method further includes: The human skeleton sequence obtained from public datasets and gait capture sensors is used to extract features to obtain a normalized combined feature tensor, which is then used as the input and output samples of the autoencoder. The autoencoder is pre-trained, and the trained autoencoder weights are output. The acquired human skeleton sequence was split into head, upper limb and lower limb parts. The normalized combined feature tensors of the head, upper limb and lower limb were obtained by feature extraction. The limb activity labels of the head, upper limb and lower limb were then labeled. The normalized combined feature tensors of the head, upper limbs, and lower limbs are used as the sample input of the pre-trained autoencoder. The bottleneck features of the head, upper limbs, and lower limbs output by the autoencoder are used as the sample input data of the limb activity regression network, and the limb activity labels of the head, upper limbs, and lower limbs are used as the sample output data. The limb activity regression network consists of three subnetworks corresponding to the analysis of head, upper limb, and lower limb activity. Each subnetwork is pre-trained based on its regression loss.

[0010] Furthermore, the method also includes: The pre-trained weights of the autoencoder and the limb activity regression network are loaded, the decoder in the autoencoder is frozen, and end-to-end joint fine-tuning is achieved through a progressive unfreezing strategy. The progressive unfreezing strategy includes fine-tuning the fully connected layer or the top convolutional layer of the limb activity regression network, unfreezing all layers of the limb activity regression network, unfreezing the top layer of the encoder, and unfreezing all layers of the encoder.

[0011] Furthermore, during training, the loss function of the autoencoder is: L autoencoder =αL recon +βL speed L recon = L speed = , where L recon The reconstruction loss is given by N, where N is the number of samples and F is the number of samples. orig(i) F represents the original combined feature tensor of the i-th sample. recon(i) L represents the reconstructed combined feature tensor of the i-th sample. speed The reconstruction loss is for the velocity feature tensor. Let represent the original velocity feature tensor of the i-th sample. Let α and β represent the reconstruction velocity feature tensor of the i-th sample, with α and β set as constants.

[0012] Furthermore, during training, the loss function of the limb activity regression network is: L regression =L MSE +γL smooth LMSE = L smooth = L smooth(i) = , where L MSE The mean squared error loss is given by N, where N is the number of samples. It is the true activity label value of the i-th sample. L is the predicted value of the i-th sample. smooth For time series smoothing constraints, L smooth(i) This is the temporal smoothing loss for the i-th sample, where T is the temporal length. This represents the predicted value of the i-th sample in the t-th frame. Let γ represent the predicted value of the i-th sample in the (t-1)-th frame, and let γ be a constant.

[0013] Furthermore, in the joint fine-tuning phase, the total loss function is: L joint =L regression_tri +λL recon , where L regression_tri L is the sum of the regression losses of the three activity regression networks for the head, upper limb, and lower limb. recon For the reconstruction loss, λ is set to a constant.

[0014] Secondly, the present invention provides an apparatus for analyzing and quantifying limb activity, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the above-described method for limb activity analysis and quantitative assessment.

[0015] Furthermore, the computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the aforementioned method for limb activity analysis and quantitative assessment.

[0016] The beneficial effects of this invention are as follows: by extracting the triaxial features of the skeleton and the triaxial features of velocity, the extraction of feature details is maximized; an autoencoder is used to effectively fuse the skeletal and velocity features, ensuring temporal continuity; and in the design of the limb activity regression network, limb differentiation is achieved through three sub-networks based on the different characteristics of different body parts, enabling discriminative analysis and effective quantitative evaluation of limb biomechanical characteristics. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for analyzing and quantifying limb activity provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a combined feature extraction process provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of a feature fusion and regression analysis process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the module architecture of a device for analyzing and quantifying limb activity provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the architecture of a limb activity regression network provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the architecture of a self-encoder provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the architecture of a device for analyzing and quantifying limb activity provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0021] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0022] In describing some embodiments, the terms "coupled," "coupled," and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "connected" or "coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other, such as "optical coupling," "wireless connection," etc. The embodiments disclosed herein are not necessarily limited to the scope of this invention.

[0023] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] Example 1: like Figure 1 As shown, Embodiment 1 of the present invention provides a method for analyzing and quantifying limb activity, specifically including the following steps: In step 101, features are extracted from the skeleton sequence to obtain skeletal feature tensors and velocity feature tensors to form a combined feature tensor, and spatiotemporal alignment and normalization are performed.

[0025] Based on the skeleton sequence obtained from the public dataset and the data captured by the gait capture sensor, the velocity feature tensor and the skeletal feature tensor are calculated respectively. The skeleton sequence includes V nodes and T frames. The velocity feature tensor has three channels: X, Y, and Z. The skeletal feature tensor also has three channels: X, Y, and Z. The velocity feature tensor and the skeletal feature tensor are combined to form a total of six channels.

[0026] Discard the bone feature vector at T=1 in the bone feature tensor to make the frame number of the bone feature tensor and the velocity feature tensor consistent, so as to align the time dimension of the bone feature tensor and the velocity feature tensor; discard the velocity feature vector of the center node to make the number of nodes of the velocity feature tensor and the bone feature tensor consistent, so as to align the spatial dimension of the velocity feature tensor and the bone feature tensor, so that the velocity feature vector and the bone feature vector correspond one-to-one to form a combined feature tensor, and normalize the combined feature tensor.

[0027] In step 102, the autoencoder is pre-trained based on the normalized combined feature tensor. The autoencoder learns the fusion representation of the features and outputs the pre-trained weights and bottleneck feature set of the autoencoder.

[0028] The autoencoder includes an encoder module, a bottleneck layer, and a decoder module. The encoder module receives a combined feature tensor, reduces the dimensionality of the combined feature tensor and extracts the core features, and outputs a low-dimensional and compact bottleneck feature through the bottleneck layer. The decoder module receives the bottleneck feature, reconstructs it into the original combined feature tensor through learning, and outputs a reconstructed combined feature tensor with the same dimension as the original combined feature tensor.

[0029] There is a reconstruction loss between the reconstructed combined feature tensor and the original combined feature tensor. The smaller the reconstruction loss, the closer the reconstructed combined feature tensor is to the original combined feature tensor, and the better the reconstruction effect. In order to improve the reconstruction effect, the reconstruction loss is fed back to the decoder module and the encoder module, and the weights and biases of each layer of the decoder and encoder are updated in turn. Through continuous iterative updates, the reconstruction loss is gradually reduced until it tends to stabilize, thus completing the pre-training of the autoencoder.

[0030] In step 103, the limb activity regression network is pre-trained based on the bottleneck feature set, and the pre-trained weights of the limb activity regression network are output.

[0031] Specifically, the limb activity regression network includes the head activity regression network, the upper limb activity regression network, and the lower limb activity regression network.

[0032] After the autoencoder is pre-trained, the human skeleton sequence obtained from the public dataset and data captured by the gait capture sensor is split into head, upper limb, and lower limb parts. Correspondingly, the combined feature tensors obtained through feature extraction include head combined tensor, upper limb combined vector, and lower limb combined tensor. These head combined tensor, upper limb combined vector, and lower limb combined tensor are input into the pre-trained autoencoder. The bottleneck features output by the pre-trained encoder include head bottleneck features, upper limb bottleneck features, and lower limb bottleneck features. The head bottleneck features are used as input samples for the head activity regression network, and the upper limb bottleneck features are used as input samples for the upper limb activity regression network. The input samples for the network are the lower limb bottleneck features as input samples for the lower limb activity regression network. The head activity regression network, upper limb activity regression network, and lower limb activity regression network are the three sub-networks of the limb activity regression network. The three are independent of each other. The head, upper limb, and lower limb of the original dataset are labeled with limb activity labels, which are used as the sample output data of the three sub-networks of the activity regression network. The weights and biases of the corresponding sub-network are updated based on the regression loss of each sub-network without affecting the weights and biases of the other two sub-networks. The training is repeated iteratively until the regression losses of the three sub-networks tend to stabilize, thus completing the pre-training of each sub-network.

[0033] In step 104, the autoencoder and limb activity regression network are jointly fine-tuned to complete the parameter optimization and update of the autoencoder and limb activity regression network.

[0034] After pre-training the autoencoder and limb activity regression network, the autoencoder and limb activity regression network are fine-tuned and optimized simultaneously to achieve collaborative optimization and update of the parameters of the two networks.

[0035] Furthermore, pre-trained weights of the autoencoder and limb activity regression network are loaded, and a phased, progressive unfreezing strategy is adopted to conduct end-to-end joint training of the autoencoder and limb activity regression network. The decoder of the autoencoder is always frozen and its parameters are not updated. By sequentially optimizing the top layer of the regression network, all layers of the regression network, the top layer of the encoder and all layers of the regression network, and all layers of the encoder and all layers of the regression network, the joint optimization and update of all layers of the encoder of the autoencoder and all layers of the limb activity regression network are achieved.

[0036] In step 105, the decoder of the optimized and updated autoencoder is removed, and the autoencoder is connected to the limb activity regression network to construct an analysis and evaluation model for the inference stage, thereby achieving quantitative evaluation of limb activity during the inference stage.

[0037] After optimization and update, the autoencoder no longer needs to be reconstructed and trained through the decoder, nor does it need to calculate the reconstruction loss function during joint fine-tuning. The bottleneck features output by the encoder can be directly used as input to the limb activity regression network for quantitative evaluation of limb activity. Therefore, the decoder of the optimized and updated autoencoder is removed, and the autoencoder and the limb activity regression network are connected to adapt to the inference scenario of limb activity evaluation.

[0038] By integrating an autoencoder and a limb activity regression network to construct an analysis and evaluation model for the inference stage, the acquired human skeletal sequence is split into head, upper limb, and lower limb parts and input into the analysis and evaluation model. Through feature extraction and normalization, normalized combined feature tensors for the head, upper limb, and lower limb are obtained respectively. The autoencoder reduces the dimensionality of the normalized combined feature tensors of each part to extract core features and output low-dimensional and compact bottleneck features. The bottleneck features include head bottleneck features, upper limb bottleneck features, and lower limb bottleneck features. The head bottleneck features, upper limb bottleneck features, and lower limb bottleneck features are used as inputs to the head, upper limb, and lower limb activity regression networks, respectively. The head activity regression network outputs the activity score of the head, the upper limb activity regression network outputs the activity scores of the left and right arms, and the lower limb activity regression network outputs the activity scores of the left and right legs.

[0039] like Figure 2 As shown, step 101 further includes the following steps: In step 201, the joint coordinate sequence {SK} of the skeleton sequence is obtained. i,t}, where i∈{1,…,V},t∈{1,…,T},V represents the number of nodes, andT represents the number of frames.

[0040] Timestamp alignment ensures the skeleton sequence is synchronized with video frames, and the coordinate system is uniformly transformed to the world coordinate system, where X represents the forward direction, Y represents the vertical direction, and Z represents the lateral direction.

[0041] In step 202, the velocity eigenvector S is calculated. i,t =SK i,t -SK i,（t-1） This forms a velocity feature tensor S, where S∈R V×(T-1)×3 , t∈{2,…,T}.

[0042] In step 203, the skeletal feature vector B is calculated. j,t =SK j,t -SK i,t This forms a skeletal feature tensor B, where B∈R (V-1)×T×3, where \(j\in\{2,\ldots,V\}\), \(t\in\{1,\ldots,T\}\), \(i\) is an adjacent node closer to the central node, \(i < j\), and the skeletal feature vector corresponding to the central node is initialized as a zero vector to match the adjacency matrix.

[0043] Specifically, the velocity feature vector is calculated through the inter-frame displacement change, so \(t\geq2\). In one embodiment, assume that it is necessary to calculate the velocity feature vector of the tenth node moving between the fourth frame and the fifth frame. According to the formula, we can get \(S\) 10,5 =SK 10,5 -SK 10,4 , calculate the velocity feature vectors of all nodes between all frames to form a velocity feature tensor; the skeletal feature vector is obtained by calculating the relative position relationship between different skeletal nodes in the same frame, so \(j\geq2\). In one embodiment, assume that it is necessary to calculate the skeletal feature vector between the eleventh node and the tenth node at the fourth frame. According to the formula, we can get \(B\) 11,4 =SK 11,4 -SK 10,4 , calculate the skeletal feature vectors between adjacent nodes of all frames to form a skeletal feature tensor.

[0044] In step 204, the velocity of the central node is discarded for dimensionality reduction of the velocity feature tensor, and the first frame is discarded to intercept the skeletal feature tensor for dimensionality reduction of the skeletal feature tensor, so that the dimensions of the skeletal feature tensor and the velocity feature tensor are the same, in order to align all velocity features and skeletal features in the spatial and temporal dimensions. Combine the velocity feature tensor and the skeletal feature tensor into a combined feature tensor, and perform normalization on the combined feature tensor after spatio-temporal alignment.

[0045] To adjust the combined feature tensor into a convolution-friendly format, the dimension of the combined feature tensor \(F\) is rearranged to obtain \(F\) reshape =F.permute(2,0,1). The original dimension order of the combined feature tensor is the number of nodes, the number of frames, and the number of channels, and the corresponding indices are 0, 1, 2 respectively; the dimension order of the combined feature tensor after dimension rearrangement is the number of channels, the number of frames, and the number of nodes.

[0046] Furthermore, perform normalization on the combined feature tensor after spatio-temporal alignment and dimension rearrangement. In one embodiment, the normalization formula is as follows:

[0047]

[0048] where, is the eigenvalue of the combined feature tensor after dimension rearrangement on the \(r\)th feature channel, is the arithmetic mean of all velocity features or skeletal features on the \(r\)th feature channel, Let be the standard deviation of all velocity or skeletal features on the r-th feature channel. Let r be the eigenvalue of the r-th feature channel after standardization. For all feature channels The combined feature tensor constituted for Minimum value among all channels for The maximum value among all channels, This is the final combined feature tensor after normalization and standardization.

[0049] like Figure 3 As shown, steps 102-104 further include the following steps: In step 301, the normalized combined feature tensor obtained by feature extraction from the human skeleton sequence obtained based on the public dataset and the data captured by the gait capture sensor is used as the input and output samples of the autoencoder to pre-train the autoencoder and output the trained autoencoder weights. Specifically, the pre-training objective of the autoencoder is to compress the combined features of 6 channels into bottleneck features of 3 channels while retaining key information. During training, the loss function of the autoencoder is: L autoencoder =αL recon +βL speed L recon = L speed = , where L recon The reconstruction loss is given by N, where N is the number of samples and F is the number of samples. orig(i) F represents the original combined feature tensor of the i-th sample. recon(i) L represents the reconstructed combined feature tensor of the i-th sample. speed The reconstruction loss is for the velocity feature tensor. Let represent the original velocity feature tensor of the i-th sample. Let α and β represent the reconstruction velocity feature tensor of the i-th sample, with α and β set as constants.

[0050] Where L recon To ensure the decoder can reconstruct the original input from the compressed features, mean squared error is used to measure the difference between the original input and the reconstructed output. speed The similarity between the original velocity features and the reconstructed velocity features is obtained by calculating the cosine similarity. In one embodiment, the sum of the weights α and β is 1. They can be set to α=0.7 and β=0.3 to balance the preservation of reconstruction quality and velocity information.

[0051] In step 302, the acquired human skeleton sequence is split into head, upper limb, and lower limb parts. Normalized combined feature tensors of the head, upper limb, and lower limb are obtained by feature extraction, and limb activity labels are applied to the head, upper limb, and lower limb respectively. The normalized combined feature tensors of the head, upper limb, and lower limb are used as sample inputs to the pre-trained autoencoder. The bottleneck features of the head, upper limb, and lower limb output by the autoencoder are used as sample input data for the limb activity regression network, and the limb activity labels of the head, upper limb, and lower limb are used as sample output data.

[0052] It should be noted that during the pre-training of the autoencoder, the combined feature tensor does not differentiate between different body parts. After the autoencoder is pre-trained, a normalized combined feature tensor corresponding to each independent body part is generated and used as the input of the pre-trained autoencoder. The bottleneck features of the head, upper limbs, and lower limbs output by the autoencoder are used as sample input data for the limb activity regression network, so as to independently train the three sub-networks of the limb activity regression network.

[0053] In one embodiment, the autoencoder specifically extracts features from three independent parts of the human body and outputs three sets of corresponding bottleneck features: head bottleneck features, upper limb bottleneck features, and lower limb bottleneck features. The head bottleneck features include head and neck movements and skeletal core information such as head rotation or nodding. The upper limb bottleneck features include arm, shoulder, and elbow joint movements and skeletal core information such as arm swinging or raising. The lower limb bottleneck features include stepping or knee bending and leg pushing, knee and ankle joint movements and skeletal core information. The bottleneck features of the corresponding parts output by the pre-trained autoencoder are directly used as input samples for the pre-training of the corresponding sub-regression network.

[0054] Independent labeling is performed on three separate body parts: the head, upper limbs, and lower limbs. In one embodiment, a human skeleton sequence is obtained based on a publicly available dataset and data captured by a gait capture sensor. Each skeleton sequence is then independently labeled for the head, upper limbs, and lower limbs. Specifically, head activity is labeled based on cervical spine kinematic characteristics, upper limb activity is labeled based on arm swing amplitude and frequency, and lower limb activity is labeled based on gait cycle phase. The upper limb label value includes two values ​​corresponding to the left and right upper limbs, and the lower limb label value includes two values ​​corresponding to the left and right lower limbs. The labeling results are reviewed and adjusted to ensure reliability. For example, if the left-right asymmetry error is >0.3, the labeling is reviewed; otherwise, the labeling result is directly adopted.

[0055] In step 303, the limb activity regression network includes three subnetworks corresponding to the activity analysis of the head, upper limbs, and lower limbs. Based on the regression loss of each subnetwork, each subnetwork is pre-trained.

[0056] The sample pairs, consisting of bottleneck features of the head, upper limbs, and lower limbs and corresponding limb activity labels, are input into the corresponding sub-networks. Each sub-network independently completes forward propagation and outputs its own limb activity prediction value. For each sub-network, the regression loss between its prediction value and label value is calculated independently. There is no fusion or correlation between the three regression losses, and each corresponds to the prediction effect of a sub-network.

[0057] In one embodiment, the weights and biases of the head subnetwork are updated using the regression loss of the head, the weights and biases of the upper limb subnetwork are updated using the regression loss of the upper limb, and the weights and biases of the lower limb subnetwork are updated using the regression loss of the lower limb, until the regression losses of the three subnetworks all tend to stabilize, thus completing the pre-training of each subnetwork.

[0058] During training, the loss function of the limb activity regression network is: L regression =L MSE +γL smooth L MSE = L smooth = L smooth(i) = , where L MSE The mean squared error loss is given by N, where N is the number of samples. It is the true activity label value of the i-th sample. L is the predicted value of the i-th sample. smooth For time series smoothing constraints, L smooth(i) This is the temporal smoothing loss for the i-th sample, where T is the temporal length. This represents the predicted value of the i-th sample in the t-th frame. Let γ represent the predicted value of the i-th sample in the (t-1)-th frame, and let γ be a constant.

[0059] The training objective of the limb activity regression network is to predict limb activity values ​​(continuous values ​​from 0 to 1) while maintaining temporal smoothness, where L MSE To measure the difference between the predicted and labeled values, a first-order difference penalty term L is added to the loss function to ensure temporal smoothness of the predictions and avoid abrupt changes in predicted values ​​between adjacent frames. smooth In one embodiment, γ can be set to 0.1.

[0060] Furthermore, if it is necessary to add limb activity analysis for other parts such as the torso, only the corresponding bottleneck features, labeling, and subnetworks need to be added, without changing the structure and parameters of the existing subnetworks, which is highly scalable.

[0061] In step 304, the pre-trained weights of the autoencoder and the limb activity regression network are loaded, the decoder in the autoencoder is frozen, and end-to-end joint fine-tuning is achieved through a progressive unfreezing strategy. The progressive unfreezing strategy includes fine-tuning the fully connected layer or the top convolutional layer of the limb activity regression network, unfreezing all layers of the limb activity regression network, unfreezing the top layer of the encoder, and unfreezing all layers of the encoder.

[0062] By gradually unfreezing different network layers, end-to-end joint fine-tuning of the autoencoder and limb activity regression network is achieved. The decoder of the autoencoder remains frozen, with its parameters not updated. To implement the progressive unfreezing strategy, firstly, the fully connected layers or top convolutional layers of the limb activity regression network are optimized and updated, while the encoder and other network layers of the limb activity regression network remain frozen. Next, all network layers of the limb activity regression network are unfrozen for optimization and updates, with the encoder remaining frozen. Then, the top layer of the encoder is unfrozen for joint optimization and updates of the top layer of the encoder and all network layers of the limb activity regression network, with the bottom layer of the encoder remaining frozen. Finally, the encoder is unfrozen for optimization and updates of all network layers of the encoder and all network layers of the limb activity regression network, thus achieving full unfreezing.

[0063] During the joint fine-tuning phase, the total loss function is: L joint =L regression_tri +λL recon , where L regression_tri L is the sum of the regression losses of the three activity regression networks for the head, upper limb, and lower limb. recon For the reconstruction loss, λ is set to a constant.

[0064] During the joint fine-tuning phase, the autoencoder's encoder and the limb activity regression network are fine-tuned together to optimize the overall performance of feature extraction and regression tasks. The regression loss of the head activity regression network is defined as L. regression_head The regression loss of the upper limb activity regression network is defined as L. regression_upper The regression loss of the lower limb activity regression network is defined as L. regression_lower The total loss function of the limb activity regression network is L regression_tri =L regression_head +L regression_upper +L regression_lower , where L regression_head L regression_upper L regression_lower From formula L regression =L MSE +γL smooth As can be seen, the specific calculation steps and principles are as described above, and will not be repeated here; L recon From formula Lrecon = Therefore, the specific calculation steps and principles are as described above, and will not be repeated here.

[0065] It should be noted that calculating L regression_upper At that time, the predicted value and label value of each sample both contain two values ​​corresponding to the left upper limb and the right upper limb. Calculate L. regression_lower In this case, the predicted value and label value of each sample both contain two values ​​corresponding to the left lower limb and the right upper limb. In one embodiment, λ can be set to 0.1.

[0066] Example 2: like Figure 4 As shown, in order to implement the method for limb activity analysis and quantitative assessment described in Embodiment 1, Embodiment 2 of the present invention provides a device for limb activity analysis and quantitative assessment. The device includes a combined feature extraction module, an autoencoder, and a limb activity regression network. The combined feature extraction module is used to extract skeletal features and motion features, and retain temporal feature information to form a six-channel combined feature, wherein the skeletal features include three-axis features of bones, and the motion features include three-axis features of velocity. The autoencoder is used to fuse the six-channel combined features into a three-channel bottleneck feature to adapt to the limb activity regression network. The limb activity regression network includes three sub-networks corresponding to limb activity analysis of the head, upper limb, and lower limb, respectively. Each sub-network outputs a corresponding limb activity prediction value based on the three-channel bottleneck features fused from different limb parts.

[0067] like Figure 5 As shown, based on the 3-channel features fused by the autoencoder for different limb parts, the 3 sub-networks output the corresponding limb activity prediction values. The head activity regression network outputs the limb activity prediction value of the head, the upper limb activity regression network outputs the limb activity prediction values ​​of the left and right arms, and the lower limb activity regression network outputs the limb activity prediction values ​​of the left and right legs.

[0068] like Figure 6 As shown, the network architecture of the autoencoder includes an encoder module, a bottleneck layer, and a decoder module. In one embodiment, the encoder module has an output size of 3×112×112, the bottleneck layer has a size of 3×112×112, and the decoder module has an output size of 6×224×224.

[0069] In one embodiment, the encoder module adopts a 4-layer network architecture, specifically including convolutional layer 1, pooling layer, convolutional layer 2, and convolutional layer 3, with the following specific settings: Convolutional layer 1 includes 6 input channels, 64 output channels, a 3×3 kernel, a stride of 1, padding of 1, followed by a ReLU activation function; max pooling layer includes a 2×2 pooling kernel, a stride of 2; convolutional layer 2 (residual core layer) includes a main path with a 3×3 kernel, a stride of 1, padding of 1, followed by a ReLU activation function, and a residual path with 1×1 convolutional adjustment channels (64→128). The feature fusion formula is: Fout=0.4×Fres+0.6×Fconv; where the main path and the residual path are element-wise added and then normalized; convolutional layer 3 includes 128 input channels, 3 output channels, and a 1×1 kernel.

[0070] In one embodiment, the decoder module includes a transposed convolutional layer, an upsampling layer, convolutional layer 1, and convolutional layer 2, specifically configured as follows: The transposed convolutional layer includes 3 input channels, 128 output channels, a 1×1 kernel, and a ReLU activation function; the upsampling layer uses bilinear interpolation, a scaling factor of 2, and an output size of 128×224×224; convolutional layer 1 includes 128 input channels, 64 output channels, a 3×3 kernel, a stride of 1, padding of 1, and a ReLU activation function; convolutional layer 2 includes 64 input channels, 6 output channels, a 3×3 kernel, a stride of 1, and padding of 1.

[0071] In one embodiment, the head activity regression network includes: a convolutional block 1, a convolutional block 2, a convolutional block 3 architecture, and a fully connected layer, as specifically designed below: Convolutional Block 1: Convolutional layer (3 input channels, 64 output channels, 3×3 kernel, stride 1, padding 1), ReLU, Dropout (0.2); Convolutional Block 2: Convolutional layer (64 input channels, 128 output channels, 3×3 kernel, stride 1, padding 1), ReLU, max pooling layer (pooling kernel 2×2, stride 2), Dropout (0.2); Convolutional Block 3: Two convolutional layers (128 input channels, 256 output channels, 3×3 convolutional kernel, stride 1, padding 1), each convolutional layer is followed by ReLU, and then a max pooling layer (pooling kernel 2×2, stride 2). Fully connected layers: Linear layer (input 256×56×56, output 512), ReLU, Linear layer (512, output 1).

[0072] In one embodiment, the upper limb activity regression network includes: a convolutional layer, a spatial dropout layer, and an output layer, as specifically designed below: Convolutional layer: 3 input channels, 128 output channels, 3×3 kernel, stride 1, padding 1. The upper limb network has more channels than the head network because arm movements are more complex.

[0073] Spatial Dropout layer: SpatialDropout2d is used to randomly drop the entire feature map channel with a dropout rate of 0.3 to enhance the ability to resist occlusion.

[0074] Output layer: A 1×1 convolution is used, followed by a sigmoid activation function to achieve dual-channel output, with the left and right arms corresponding to two predicted values.

[0075] In one embodiment, the lower limb activity regression network includes: a temporal convolutional layer, a spatial feature enhancement layer, an adaptive pooling layer, and an output layer, specifically designed as follows: Temporal convolutional layer: A one-dimensional temporal convolutional kernel (kernel_size=5) is used to enhance temporal features.

[0076] Spatial feature enhancement layer: Employs modules based on spatial attention mechanisms, such as the Squeeze-and-E×citation module.

[0077] Adaptive pooling layer: Global adaptive average pooling.

[0078] Output layer: Fully connected layer, then activated by the Sigmoid function, with dual-channel output, the left leg and the right leg corresponding to two predicted values.

[0079] By extracting triaxial features of the skeleton and triaxial features of velocity, the extraction of feature details is maximized. An autoencoder is used to effectively fuse the skeletal and velocity features while ensuring temporal continuity. In the design of the limb activity regression network, three subnetworks are used to achieve limb differentiation based on the different characteristics of different body parts, enabling discriminative analysis and effective quantitative evaluation of limb biomechanical characteristics.

[0080] Example 3: like Figure 7 As shown, in order to implement the method for limb activity analysis and quantitative assessment described in Embodiment 1, Embodiment 3 of the present invention provides an apparatus for limb activity analysis and quantitative assessment, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the method for limb activity analysis and quantitative assessment.

[0081] The apparatus for limb activity analysis and quantitative assessment in this embodiment includes one or more processors and a memory. Figure 7 Let's take a processor as an example. The processor and memory can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0082] The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform a method for limb activity analysis and quantitative assessment.

[0083] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs. The processor executes methods for limb activity analysis and quantitative assessment by running non-volatile software programs and instructions stored in memory.

[0084] The memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] The program instructions / modules are stored in the memory. When executed by one or more processors, they perform the limb activity analysis and quantitative assessment method described in the above embodiments, for example, performing each step of the limb activity analysis and quantitative assessment method described above.

[0086] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0087] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for analyzing and quantifying limb activity, characterized in that, include: Feature extraction is performed on the skeleton sequence to obtain the skeletal feature tensor and velocity feature tensor to form a combined feature tensor, and then spatiotemporal alignment and normalization are performed. The autoencoder is pre-trained based on the normalized combined feature tensor, and the fusion representation of features is learned through the autoencoder. The pre-trained weights and bottleneck feature set of the autoencoder are then output. The limb activity regression network is pre-trained based on the bottleneck feature set, and the pre-trained weights of the limb activity regression network are output. The autoencoder and limb activity regression network were jointly fine-tuned to optimize and update their parameters. The decoder of the optimized and updated autoencoder is removed, and the autoencoder is connected to the limb activity regression network to build an analysis and evaluation model for the inference stage, thereby achieving quantitative evaluation of limb activity during the inference stage.

2. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, The method further includes: Obtain the joint coordinate sequence {SK} of the skeleton sequence i,t }, where i∈{1,…,V},t∈{1,…,T},V represents the number of nodes, andT represents the number of frames; Calculate the velocity eigenvector S i,t =SK i,t -SK i,（t-1） This forms a velocity feature tensor S, where S∈R V×(T-1)×3 , t∈{2,…,T}; Calculate the bone feature vector B j,t =SK j,t -SK i,t , to form the bone feature tensor B, where B ∈ R (V-1)×T×3 , j ∈ {2, …, V}, i is the adjacent node closer to the central node, i < j, and the bone feature vector corresponding to the central node is initialized as a zero vector to match the adjacency matrix.

3. The method for analyzing and quantifying limb activity according to claim 2, characterized in that, The method further includes: The velocity feature tensor is reduced by discarding the velocity of the center node, and the skeletal feature tensor is reduced by discarding the skeletal feature tensor extracted from the first frame, so that the skeletal feature tensor and the velocity feature tensor have the same dimension. This is to align all velocity features and skeletal features in spatial and temporal dimensions, combine the velocity feature tensor and the skeletal feature tensor into a combined feature tensor, and normalize and standardize the combined feature tensor after spatiotemporal alignment.

4. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, The method further includes: The human skeleton sequence obtained from public datasets and gait capture sensors is used to extract features to obtain a normalized combined feature tensor, which is then used as the input and output samples of the autoencoder. The autoencoder is pre-trained, and the trained autoencoder weights are output. The acquired human skeleton sequence was split into head, upper limb and lower limb parts. The normalized combined feature tensors of the head, upper limb and lower limb were obtained by feature extraction. The limb activity labels of the head, upper limb and lower limb were then labeled. The normalized combined feature tensors of the head, upper limbs, and lower limbs are used as the sample input of the pre-trained autoencoder. The bottleneck features of the head, upper limbs, and lower limbs output by the autoencoder are used as the sample input data of the limb activity regression network, and the limb activity labels of the head, upper limbs, and lower limbs are used as the sample output data. The limb activity regression network consists of three subnetworks corresponding to the analysis of head, upper limb, and lower limb activity. Each subnetwork is pre-trained based on its regression loss.

5. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, The method further includes: The pre-trained weights of the autoencoder and the limb activity regression network are loaded, the decoder in the autoencoder is frozen, and end-to-end joint fine-tuning is achieved through a progressive unfreezing strategy. The progressive unfreezing strategy includes fine-tuning the fully connected layer or the top convolutional layer of the limb activity regression network, unfreezing all layers of the limb activity regression network, unfreezing the top layer of the encoder, and unfreezing all layers of the encoder.

6. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, During training, the loss function of the autoencoder is: L autoencoder =αL recon +βL speed L recon = L speed = , where L recon The reconstruction loss is given by N, where N is the number of samples and F is the number of samples. orig(i) F represents the original combined feature tensor of the i-th sample. recon(i) L represents the reconstructed combined feature tensor of the i-th sample. speed The reconstruction loss is for the velocity feature tensor. Let represent the original velocity feature tensor of the i-th sample. Let α and β represent the reconstruction velocity feature tensor of the i-th sample, with α and β set as constants.

7. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, During training, the loss function of the limb activity regression network is: L regression =L MSE +γL smooth L MSE = L smooth = L smooth(i) = , where L MSE The mean squared error loss is given by N, where N is the number of samples. It is the true activity label value of the i-th sample. L is the predicted value of the i-th sample. smooth For time series smoothing constraints, L smooth(i) This is the temporal smoothing loss for the i-th sample, where T is the temporal length. This represents the predicted value of the i-th sample in the t-th frame. Let γ represent the predicted value of the i-th sample in the (t-1)-th frame, and let γ be a constant.

8. The method for analyzing and quantifying limb activity according to claim 1, characterized in that, During the joint fine-tuning phase, the total loss function is: L joint =L regression_tri +λL recon , where L regression_tri L is the sum of the regression losses of the three activity regression networks for the head, upper limb, and lower limb. recon For the reconstruction loss, λ is set to a constant.

9. A device for analyzing and quantifying limb activity, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the method for limb activity analysis and quantitative assessment as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the method for limb activity analysis and quantitative assessment as described in any one of claims 1-8.

Images