A hand motion recovery method based on power maintenance

Abstract

The application discloses a hand motion recovery method based on power maintenance, first constructs a hand network based on initial pose estimation of a parameterized hand model and obtains vertex coordinates of the hand network, then extracts high and low resolution stream features and fuses to generate adaptive features, a double-stream encoder extracts features of different depths based on the adaptive features, then performs feature modulation based on a kernelized Gram matrix statistic and maps into a multi-scale conditional feature sequence, finally fuses the initial pose estimation into a conditional vector, finally a backbone network predicts a clean hand motion sequence as an output, a conditional diffusion model performs reverse denoising and fine-tuning, generates a hand motion recovery result and performs uncertainty quantification and physical reliability evaluation based on dynamics variance of Bayesian inference. The application realizes recovery of a physically reasonable three-dimensional hand motion trajectory from a maintenance monocular video and simultaneously estimates generalized force variance in a motion recovery process to quantize physical consistency and reliability of maintenance action estimation.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation of power maintenance and computer vision three-dimensional motion reconstruction technology, specifically a method for hand motion recovery based on power maintenance. Background Technology

[0002] In power maintenance scenarios, recovering hand movements from monocular cameras aims to reconstruct the three-dimensional configuration of a maintenance worker's hand over time from a single-view video sequence. This technology can be used for assessing the standardization of maintenance actions, identifying and locating abnormal behaviors, issuing early warnings of dangerous postures, providing remote maintenance guidance, and digitally reviewing maintenance processes.

[0003] With the development of deep learning technology, significant progress has been made in recovering hand meshes from monocular images or videos. Image-based methods have continuously improved frame-by-frame accuracy through large-scale pre-trained base models, but they still have significant shortcomings in long video scenarios of power maintenance: First, long video processing has high requirements for context modeling and encoding efficiency, and existing methods are prone to cross-frame jitter and drift at critical moments; Second, maintenance operations are often accompanied by insulating gloves, tool obstructions, and complex backgrounds, and the model's perception of dynamic characteristics is insufficient, which can easily lead to physically unreasonable results.

[0004] To address the above challenges, introducing physical constraints into 3D hand reconstruction is a promising direction. For power maintenance applications, existing methods have the following three main shortcomings: (1) Physical level: Existing mainstream physical fusion methods generally adopt a deterministic approach, applying physical constraints by driving the residuals of the dynamic equations to zero, rather than modeling the distribution of the residuals. This ignores the inherent uncertainties such as noise at the maintenance site, annotation bias, and model mismatch, and cannot quantify the physical reliability of motion estimation; (2) Representation and localization level: Existing methods usually adopt a single-resolution feature extraction architecture, which is difficult to simultaneously take into account global temporal consistency and fine local geometric details, resulting in insufficient localization accuracy of the key action start and end times and the tool contact area; (3) Robustness and generalization level: There are geometric singularities and imaging interferences such as self-occlusion, glove reflection, motion blur, and operation in confined spaces in the power maintenance sequence. Existing methods lack effective feature-level noise suppression mechanisms, and unreliable features are easy to propagate and reduce the stability of downstream identification and retrieval tasks. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method for hand motion recovery based on power maintenance, which realizes the recovery of physically reasonable three-dimensional hand motion trajectory from maintenance monocular video, and simultaneously estimates the generalized force variance in the motion recovery process, so as to quantify the physical consistency and reliability of maintenance action estimation.

[0006] The technical solution of this invention is as follows: A method for restoring hand movement during power maintenance includes the following steps: (1) Construct a hand network based on the initial pose estimation of the parametric hand model, constrain and correct the hand network with the physical rules of dynamic modeling, and obtain the vertex coordinates of the hand network; (2) Based on the vertex coordinates of the hand network, high-resolution enhanced features and low-resolution global features are extracted, and adaptive features are generated through a lightweight cross-attention fusion adapter; (3) The dual-stream encoder extracts features at different depths based on the adaptive features, then modulates the features based on the kernelized Gram matrix statistics, and maps them into a multi-scale conditional feature sequence. Finally, it fuses the sequence with the initial pose estimate into a conditional vector. (4) The backbone network predicts and outputs a clean hand motion sequence, and the conditional diffusion model performs reverse denoising and refinement to generate hand motion recovery results; (5) Based on the dynamic variance of Bayesian inference, the uncertainty of the hand motion recovery results is quantified and the physical reliability is assessed.

[0007] The initial pose estimation based on the parametric hand model constructs a hand network, which is then constrained and corrected using physical rules of dynamic modeling to obtain the vertex coordinates of the hand network. This process specifically includes the following steps: S11, a segment containing The maintenance monocular video image sequence is processed. Each frame of the sequence is input into the hand pose estimator to obtain the parameterized hand model parameters for each frame. The parameterized hand model parameters include the wrist rotation vector. Wrist translation vector Attitude parameters and shape parameters , Attitude parameters It includes 3D rotation vectors for 15 hand joints, and the shape parameters include 10 principal component coefficients in the shape space of the parameterized hand model, which are 10-dimensional vectors; The average shape parameter is calculated using the following formula (1). : (1); S12. Based on the parametric hand model, define the generalized coordinates of the hand network. , see the following formula (2) for details: (2); In equation (2), the generalized coordinates of the hand network It is the wrist rotation vector Wrist translation vector and attitude parameters composition; Generalized speed of hand networks and generalized acceleration It is obtained through the finite difference method, as shown in equation (3) below: (3); In equation (3), The generalized coordinates represent the (t+1)th frame of the image; The generalized coordinates represent the (t-1)th frame of the image; Represents the generalized coordinates of the image in frame t; Represents the time interval between adjacent frames; S13. Establish the Euler-Lagrange equation for the hand, as shown in equation (4) below: (4); In equation (4), Represents the generalized quality matrix; Representing the Coriolis force and centrifugal force terms, describing the non-inertial force effects caused by joint rotation; Represents the gravity term, indicating the generalized force at each joint in the gravitational field; It represents the net generalized force, which is the sum of the external force and joint torque required for hand movement; This represents a set of mass parameters for fifteen hand joints; The set of inertial tensors representing the fifteen hand joints; The vertex coordinates of the hand mesh are obtained by using the linear blending skin function of the parameterized hand model, as shown in equation (5) below: (5); In equation (5), Represents the coordinates of the hand grid vertices in the t-th frame image; Represents a linear blending skinning function; In this context, 778 represents the number of vertices in the parametric hand model, and 3 represents the three-dimensional coordinate dimension of each vertex. Using skin weights, each vertex of the hand mesh is assigned to the corresponding hand joint. The parameterized hand model defines the first... Skin weights of each vertex relative to each hand joint ,but ,and In the linear hybrid skin process, the first The deformed position of each vertex ,in, For the template posture, the first homogeneous coordinates of the vertices For the first The global transformation matrix calculated along the kinematic chain for each hand joint; To calculate the volume and inertial parameters of each hand joint, a deterministic allocation is performed using the maximum weight principle, i.e. , Representing the Each vertex is assigned an index to a hand joint; that is, for each vertex in the hand network, traverse the fifteen hand joints to find the largest weight. The corresponding hand joint, will the first The vertex is assigned to the first... For each hand joint, the hand joint to which the face belongs is determined by the majority of the three vertices corresponding to each face in the hand network; if there is no majority of the face, the face is assigned to the hand joint with the largest average skinning weight, thereby obtaining the vertex set and face set corresponding to each hand joint. The volume of each hand joint is calculated by tetrahedronizing a closed surface. , Equivalent density of hand tissue Multiply to obtain the mass corresponding to each hand joint. ; For the Each hand joint is formed by a local mesh consisting of corresponding vertices and faces. The boundaries of the mesh are closed to obtain a closed surface. Then, the closed surface is tetrahedralized to obtain several tetrahedral elements. The volume of the j-th tetrahedral unit is The center of mass is Then the first The total volume of each hand joint Center of mass , Let J be the centroid of the j-th tetrahedral element; in the... Local inertial tensor in the coordinate system of the centroid of a hand joint See the following formula (6) for details: (6); In equation (6), The coordinates represent the infinitesimal volume element; Represents a 3×3 identity matrix; Represents transpose; Represents a volumetric infinitesimal element; Calculated and As the first The mass and local inertia tensors of each hand joint are used to construct the generalized mass matrix. And the Coriolis force and centrifugal force terms ; S14. Rewrite the Euler-Lagrange equation for the hand in residual form, as shown in equation (7) below: (7); In equation (7), Let be the dynamic residual of the t-th frame image, which is treated as a virtual observation that follows a probability distribution, rather than being forced to be zero.

[0008] The process of generating adaptation features based on the vertex coordinates of the hand network specifically includes the following steps: S21. For the hand grid in each frame of the image, extract spatial feature labels using a grid convolutional neural network to generate original resolution features. , see the following formula (8) for details: (8); In equation (8), Represents a grid convolutional neural network. For the number of feature labels, For feature channel dimensions; Simultaneously, for the hand mesh in each frame of the image, after downsampling, spatial feature labels are extracted using a mesh convolutional neural network to generate low-resolution features. , see the following formula (9) for details: (9); In equation (9), Represents downsampling, This represents the number of feature labels after downsampling; S22. Extract low-resolution global features using the global transformer attention module. , see the following formula (10) for details: (10); In equation (10), Represents the global transformer attention module; S23. Extract high-resolution features using the frame-by-frame Vision Transformer module. , see the following formula (11) for details: (11); In equation (11), Represents the frame-by-frame Vision Transformer module; S24. Generate high-resolution enhanced features based on low-resolution global features and high-resolution features. , see the following formula (12) for details: (12); In equation (12), This represents a guided feature extraction network, which is a shallow convolutional network used to extract features from high-resolution data. Extract high-frequency guidance signals from them; Represents the bilinear interpolation operator; The representative interpolation and refinement network is a shallow convolutional network used for low-resolution global features. Perform upsampling and refinement; It represents splicing; This represents a fusion network, which is a shallow convolutional network used for feature splicing and fusion. S25. Generate adaptation features through a lightweight cross-attention fusion adapter. The lightweight cross-attention fusion adapter is composed of alternating stacks of cross-attention layers and self-attention layers. The processing procedure of the cross-attention layer is shown in the following equation (13), and the processing procedure of the self-attention layer is shown in the following equation (14): (13); (14); In equations (13) and (14), Representing the cross-attention layer, it enhances high-resolution features. As a query Q, low-resolution global feature As keys K and values ​​V, the global temporal context encoded in the low-resolution stream is injected into the high-resolution feature space; Represents cross-attention fusion features; Representing the self-attention layer, it fuses cross-attention features. Using query Q, key K, and value V, local associations are established between high-resolution labels and feature remodeling is completed; Represents the self-attention fusion feature; The output of the last self-attention layer of the lightweight cross-attention fusion adapter is used as the adaptation feature. .

[0009] The global transformer attention module includes alternating frame-level attention layers and a global attention layer, specifically comprising the following steps: S221, The processing procedure of the frame-level attention layer is shown in the following formula (15): (15); In equation (15), Represents the global transformer attention module. The frame-level attention features of the t-th frame image output by the frame-level attention layer; Represents the global transformer attention module. The input features of the frame-level attention layer, when hour, ; Representing the Frame-level attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function; S222. Concatenate the frame-level attention features of all frames to obtain the frame-level attention concatenated features. , see the following formula (16) for details: (16); In equation (16), This represents a splicing operation; S223, The processing procedure of the global attention layer is shown in the following formula (17): (17); In equation (17), Representing the The global attention features of all frames output by the global attention layer are split to obtain the global attention features of each frame. That is, as the first The input features of the frame-level attention layer, and the output global attention features of all frames from the last global attention layer, are decomposed to obtain low-resolution global features. ; Representing the Global attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function.

[0010] The dual-stream encoder extracts features at different depths based on adaptive features, then modulates the features based on kernelized Gram matrix statistics, and maps them into a multi-scale conditional feature sequence. Finally, it fuses this sequence with the initial pose estimate to form a conditional vector. The specific steps include: S31. The backbone Transformer of the dual-stream encoder includes a multi-layer structure. The output features of a selected layer in the first few layers of the backbone Transformer are used as shallow features, the output features of a selected layer in the middle few layers of the backbone Transformer are used as mid-layer features, and the output features of a selected layer in the last few layers of the backbone Transformer are used as deep features. The shallow features, mid-layer features, and deep features are the output features of the selected network layer set. S32. Linearly transform the shallow, middle, and deep features corresponding to the t-th frame image into the corresponding query matrix. Bond matrix Then, the query-query Gram matrix is ​​calculated according to the following formula (18). Key-key Gram matrix And query-keygram matrix ; (18); In equation (18), Represents the feature dimension; Then the adjacent frame windows The Gram matrix sequence within the matrix is ​​modeled as a matrix-valued stochastic process, and the time-series expectation operator is calculated. and temporal variance operator See equations (19) and (20) below for details: (19); (20); In equations (19) and (20), Represents the time offset index, representing the adjacent frame window. Within, the timing window between the i-th frame and the j-th frame, and the adjacent frame window. That is, a continuous frame interval centered on the i-th frame and extracted before and after it; The steady-state component of the characteristic manifold represents the desired geometric structure; The wave component of the characteristic manifold; Represents different Gram matrix types; For the temporal expectation operator, for adjacent frame windows The expected value of the Gram matrix sequence within the time dimension is taken; As a spatial expectation operator, it takes the expectation of the layer dimension of the Gram matrix within the selected set of network layers; Represents the temporal variance operator, for adjacent frame windows The variance of the Gram matrix sequence within the time dimension is taken; Based on a hierarchical Bayesian inference framework, multi-scale anomaly priors are derived from different depths, and a posterior anomaly saliency map is constructed. , see the following formula (21) for details: (twenty one); In equation (21), When representing shallow input features, and The steady-state components of the characteristic manifold calculated in real time; When representing shallow input features, and The wave component of the characteristic manifold calculated in time; When representing the mid-level features of the input, and The steady-state components of the characteristic manifold calculated in real time; When representing deep input features, and The wave component of the characteristic manifold calculated in time; When representing deep input features, and The steady-state components of the characteristic manifold calculated in real time; This represents a priori shallow anomalies. This represents a priori anomalies in the middle layer. This represents a priori deep anomalies; The saliency map represents the posterior result after fusing the three priors; This represents element-wise multiplication; Based on anomaly significance map For adaptation features Perform feature modulation to obtain modulation features , see the following formula (22) for details: (twenty two); Finally, the modulation features of all frames are combined into a modulation feature sequence. ; By setting an anomaly significance threshold For the anomaly saliency plot Exceeding the abnormal significance threshold modulation characteristics Mark; S33, Modulation feature sequence Encoding mapping is performed to obtain multi-scale conditional feature sequences. , see the following formula (23) for details: (twenty three); In equation (23), Represents the conditional coding mapping operator; S34. Fuse the initial pose estimation and multi-scale conditional feature sequence into a conditional vector. , see the following formula (24) for details: (twenty four); In equation (24), Represents the conditional feature encoding mapping operator; Represents the average shape parameter; Represents the pose parameters corresponding to all frame images. The sequence of attitude parameters.

[0011] The backbone network predicts and outputs a clean hand motion sequence, which is then refined using a conditional diffusion model to generate a hand motion recovery result. This process includes the following steps: S41. Define the global hand motion deviation sequence. , see the following formula (25) for details: (25); In equation (25), Represents the sequence of attitude parameters and shape parameters Mapping to the generalized coordinate motion state; This represents the initial motion estimation sequence corresponding to all frame images; This represents the initial motion estimate corresponding to the t-th frame image, including the wrist rotation vector. Wrist translation vector and fifteen attitude parameters ; Represents transpose; Represents a clean hand movement sequence to be restored; S42. Define the single-step diffusion offset increment as follows (26): (26); In equation (26), This represents the offset increment of the k-th diffusion step relative to the (k-1)-th diffusion step; Represents the cumulative offset scheduling coefficient at step k; This represents the cumulative offset scheduling coefficient at step k-1; when k equals 1, equal ; S43. The single-step noise addition process of the forward diffusion in the conditional diffusion model is shown in the following equation (27): (27); In equation (27), This represents the hand movement sequence at the given (k-1)th diffusion step. Conditional vector and initial motion estimation sequence Given the premise, generate the hand motion sequence for the kth diffusion step. Forward diffusion transfer distribution; Representative with For the mean, The covariance matrix is ​​a Gaussian distribution; Represents the scale of diffused noise. Represents the noise variance scale; The identity matrix is ​​represented by equation (27); Equation (27) indicates that during the forward diffusion single-step noise addition process, the hand motion sequence of the previous diffusion step is translated along the deviation direction. And add variance as Gaussian noise; The forward diffusion single-step noise addition process according to formula (27) is performed step by step, and the entire process is cumulatively integrated, as shown in formula (28) below: (28); Equation (28) indicates that after k steps of diffusion, the mean of the hand motion sequence decreases from the clean hand motion sequence to be recovered. Stepwise to the initial motion estimation sequence move; S44. Conditional diffusion model reverse denoising and restoration: First, build the overall framework of reverse diffusion, as shown in the following formula (29): (29); In equation (29), Represents in a given Under the premise of generating The reverse diffusion distribution; This represents the initial reverse diffusion distribution starting from the k-th diffusion step hand motion sequence; Represented by parameters A controllable, learnable reverse transfer kernel; K is the total number of diffusion steps; This represents the complete sequence of hand movements from the first diffusion step to the kth diffusion step; the integral is used to marginalize all intermediate states, i.e., the entire sequence of hand movements. In equation (29) Each diffusion step is determined by parameters The learnable inverse transfer kernel for control is updated as shown in equation (30) below: (30); In equation (30), The mean function representing backward diffusion; The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; The clean hand motion sequence predicted by the backbone network prediction head. Weighting coefficients; Represents the cumulative offset scheduling coefficient at step k; This represents the offset increment of k diffusion steps relative to the (k-1)th diffusion step; This represents the noise variance scale determined by the forward diffusion process; In equation (30), the prediction head of the backbone network predicts and outputs a clean hand motion sequence. , see the following formula (31) for details: (31); In equation (31), The sequence of clean hand movements predicted by the backbone network; A linear mapping function representing the backbone network prediction head; The sequence of hand movements represents the k-th diffusion step; S45. Construct a double loss function and train to iteratively update the parameters of the backbone network and the conditional diffusion model. The double loss function is shown in the following formula (32): (32); In equation (32), This represents the data-driven loss, as shown in equation (33) below. This represents the loss of physical perception, as shown in equation (34) below. , These represent the training weights for the two losses, respectively. (33); In equation (33), This represents randomly sampling within the range of 1 to K for each diffusion step k and calculating the expectation. This represents the actual hand movement sequence in the training data, i.e., the true label; The clean hand motion sequence predicted by the backbone network prediction head is the prediction result. Represents the square of the L2 norm; (34); In equation (34), Representative based on The calculated Eulerian-Lagrange dynamic residual sequence, i.e., the hand dynamic residual; This indicates that the probability distribution follows a certain probability distribution. Represents a zero-mean Gaussian distribution; The virtual observation variance representing the dynamic residuals, , This represents the total number of samples in the training set. Represents the scaling factor; Represents the identity matrix; This represents the actual hand movement sequence of the i-th sample in the training set; Representative based on The calculated Eulerian-Lagrange dynamical residual sequence; S46. Inference stage: The conditional diffusion model uses the initial motion estimation sequence... Corresponding high noise state Starting with step K, from step K to step 1, a learnable inverse transfer kernel is executed progressively, with each step using a clean hand motion sequence predicted by the backbone network. Integration and Update and Then based on the updated mean function and noise variance ,renew Iterate to At that time, output the hand motion recovery results after noise reduction and refinement. .

[0012] The uncertainty quantification and physical reliability assessment of the hand motion recovery results based on the Bayesian inference dynamic variance specifically includes the following steps: S51. Using the final layer Laplace approximation as the posterior inference step, only the posterior distribution of the backbone network's predicted head parameters is mapped to a Gaussian posterior prediction distribution, while simultaneously outputting the predicted clean hand motion sequence. The point estimation and prediction uncertainty are shown in the following equation (35): (35); In equation (35), Represents in a given , and training dataset At that time, the clean hand movement sequence to be restored The posterior predicted distribution; The linear mapping function representing the backbone network prediction head, That is, the clean hand movement sequence predicted by the backbone network. The point estimate corresponds to the mean of the Gaussian posterior prediction distribution; This represents the diagonal variance matrix obtained by the Laplace approximation of the last layer; This is a joint-level variance prediction function that outputs the variance of each hand joint, i.e., the corresponding prediction uncertainty. S52. In backdiffusion sampling, the hand motion state and its uncertainty are gradually propagated in the backdiffusion process by updating the hand motion state and propagating its variance. In the backdiffusion sampling, the update of the hand motion state is shown in the following equation (36): (36); In equation (36), This represents the hand motion sequence obtained from the (k-1)th diffusion step through reverse diffusion sampling. The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; The clean hand motion sequence predicted by the backbone network prediction head. Weighting coefficients; The standard deviation of the reverse transfer noise at the k-th diffusion step; Represents standard Gaussian noise, and satisfies ; In reverse diffusion sampling, the propagation of the variance of hand motion state is shown in the following equation (37): (37); In equation (37), A diagonal approximation representing variance; The variance of the hand motion state represents the (k-1)th diffusion step. This represents the variance of the hand motion state at the current k-th diffusion step; The variance of clean hand motion predicted by the backbone network prediction head is calculated by the following formula (38); represent and The covariance between them is calculated by the following formula (39); (38); In equation (38), This represents constructing a diagonal variance matrix from the joint-level variance vectors; Represents the joint-level variance prediction function, inputting the hand motion state at the current k-th diffusion step. Conditional vector and current diffusion steps Output the variance of each hand joint; (39); In equation (39), represent Expectations; represent Expectations; represent Expectations; Represents the number of samples in the Monte Carlo sampling; ⊙ indicates element-wise multiplication. This represents the hand motion state of the i-th sample at the current k-th diffusion step; This represents the clean hand motion sequence output after the i-th sample is predicted by the backbone network; S53. Based on the content constructed in S52, variance estimation is performed, specifically as follows: S531, Bayesian Prior Initialization: Based on the Bayesian prior distribution, rules are set for the starting point. From Gaussian distribution Once fixed after a single sampling, it is considered a deterministic conditional quantity; initial expectation. Initialize the variance of hand motion state Initialize covariance ; S532, Bayesian posterior iterative propagation: For k=K,K-1,…,1, perform the following operations in sequence: Based on the current hand movement state, sample the backbone network prediction results from the Gaussian distribution; Update the expectation and covariance of the next step according to equation (39); Update the variance of the hand movement state of the next step according to equation (37); Update the hand movement state of the next step according to formula (36); S533. After K-step Bayesian posterior iterations, the final refined hand motion state sequence is obtained. and its variance ; S54. Utilize the final hand movement state variance The variance of the generalized force is calculated by propagation through the Jacobian matrix. , see the following formula (40) for details: (40); In equation (40), Represents generalized force Relative to hand movement state The Jacobian matrix; Represents generalized force hand movement status The partial derivatives; Represents the final refined sequence of hand movement states. Calculate the Jacobian matrix at the expected value; This represents transposition.

[0013] The final output of the hand movement recovery method includes three parts: (a) the final refined hand movement state sequence. (b) Frame-by-frame, joint-by-joint generalized force variance plot, i.e., from the generalized force variance (c) Extract diagonal elements and then expand them according to the dimensions of frames and hand joints to obtain a visualization view; (d) Anomaly saliency map at the feature level, used to identify geometrically singular regions and assist in locating key anomalies.

[0014] Advantages of this invention: (1) Based on the parameterized hand model, this invention defines the generalized coordinates of the hand network, establishes the Euler-Lagrange equation of the hand, obtains the vertex coordinates of the hand mesh through the linear hybrid skinning function of the parameterized hand model, and regards the residual of the Euler-Lagrange equation of the hand as a virtual observation that follows a probability distribution, laying the foundation for the probabilistic integration of subsequent physical constraints.

[0015] (2) This invention extracts high-resolution enhanced features and low-resolution global features, and generates adaptive features through a lightweight cross-attention fusion adapter. The low-resolution stream establishes a temporally consistent global representation through an alternating frame-level-global attention mechanism. The high-resolution stream processes each frame of the hand mesh independently at the original resolution to preserve fine joint geometric details. The features of the two streams are fused through a lightweight cross-attention fusion adapter, so that the final representation has both global consistency and fine details, thereby meeting the context modeling and key action localization requirements in long video scenarios of power maintenance.

[0016] (3) The present invention uses a guided feature extraction network to extract high-frequency guided signals from high-resolution features, and uses a bilinear interpolation operator and an interpolation refinement network to upsample and refine low-resolution global features. Finally, a fusion network is used to achieve feature fusion and generate high-resolution enhanced features, which effectively compensates for the loss of high-frequency information caused by downsampling.

[0017] (4) This invention modulates features based on the nucleated Gram matrix statistics. By analyzing the higher-order statistics of the feature space, it detects unreliable feature markers in the hand motion sequence. Based on the hierarchical Bayesian inference framework, it derives multi-scale anomaly priors from different depths, constructs a posterior anomaly saliency map, eliminates noise sources caused by geometric singular regions at the feature level, and provides interpretable prior information for the localization of abnormal behavior.

[0018] (5) The present invention constructs a physical perception conditional diffusion model and constructs a dual loss function composed of data-driven loss and physical perception loss. The parameters of the backbone network and the conditional diffusion model are iteratively updated by training. The physical perception loss no longer forces the Euler-Lagrange dynamic residual to be strictly zero, but is regarded as a virtual observation that follows a Gaussian distribution. The physical constraint is probabilistically incorporated into the training target of the conditional diffusion model through negative log-likelihood.

[0019] (6) Based on the dynamic variance inferred by Bayesian inference, this invention quantifies the uncertainty and assesses the physical reliability of hand motion recovery results. It uses the final layer Laplace approximation to construct a joint-level variance prediction function. During the reverse diffusion sampling process, the uncertainty of the prediction is gradually propagated through the variance propagation formula. Combined with Monte Carlo sampling to estimate the covariance, the generalized force variance map is finally obtained frame by frame and joint by joint through Jacobi matrix propagation. This provides an interpretable index of physical consistency for the reliability of hand motion estimation and can be used to identify high-risk action segments. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] A method for restoring hand movement during power maintenance includes the following steps: (1) Construct a hand network based on the initial pose estimation of the parametric hand model, constrain and correct the hand network with the physical rules of dynamic modeling, and obtain the vertex coordinates of the hand network. The specific steps include: S11, a segment containing The maintenance monocular video image sequence is processed. Each frame of the sequence is input into a hand pose estimator (using the WiLoR model) to obtain the parameterized hand model parameters for each frame. The parameterized hand model parameters include the wrist rotation vector. Wrist translation vector Attitude parameters and shape parameters , Attitude parameters It includes 3D rotation vectors for 15 hand joints, and the shape parameters include 10 principal component coefficients in the shape space of the parameterized hand model, which are 10-dimensional vectors; The average shape parameter is calculated using the following formula (1). : (1); Calculating the average shape parameters eliminates random fluctuations in frame-by-frame shape estimation, resulting in a stable representation of the hand shape. S12. Based on the parametric hand model, define the generalized coordinates of the hand network. , see the following formula (2) for details: (2); In equation (2), the generalized coordinates of the hand network It is the wrist rotation vector Wrist translation vector and attitude parameters composition; Generalized speed of hand networks and generalized acceleration It is obtained through the finite difference method, as shown in equation (3) below: (3); In equation (3), The generalized coordinates represent the (t+1)th frame of the image; The generalized coordinates represent the (t-1)th frame of the image; Represents the generalized coordinates of the image in frame t; Represents the time interval between adjacent frames; S13. Establish the Euler-Lagrange equation for the hand, as shown in equation (4) below: (4); In equation (4), Represents the generalized quality matrix; Representing the Coriolis force and centrifugal force terms, describing the non-inertial force effects caused by joint rotation; Represents the gravity term, indicating the generalized force at each joint in the gravitational field; It represents the net generalized force, which is the sum of the external force and joint torque required for hand movement; This represents a set of mass parameters for fifteen hand joints; The set of inertial tensors representing the fifteen hand joints; The vertex coordinates of the hand mesh are obtained by using the linear blending skin function of the parameterized hand model, as shown in equation (5) below: (5); In equation (5), Represents the coordinates of the hand grid vertices in the t-th frame image; Represents a linear blending skinning function; In this context, 778 represents the number of vertices in the parametric hand model, and 3 represents the three-dimensional coordinate dimension of each vertex. Using skin weights, each vertex of the hand mesh is assigned to the corresponding hand joint. The parameterized hand model defines the first... Skin weights of each vertex relative to each hand joint ,but ,and In the linear hybrid skin process, the first The deformed position of each vertex ,in, For the template posture, the first homogeneous coordinates of the vertices For the first The global transformation matrix calculated along the kinematic chain for each hand joint; To calculate the volume and inertial parameters of each hand joint, a deterministic allocation is performed using the maximum weight principle, i.e. , Representing the Each vertex is assigned an index to a hand joint; that is, for each vertex in the hand network, traverse the fifteen hand joints to find the largest weight. The corresponding hand joint, will the first The vertex is assigned to the first... For each hand joint, the hand joint to which the face belongs is determined by the majority of the three vertices corresponding to each face in the hand network; if there is no majority of the face, the face is assigned to the hand joint with the largest average skinning weight. In this way, the vertex set and face set corresponding to each hand joint are obtained, avoiding the same vertex or face being counted repeatedly in the volume calculation. The volume of each hand joint is calculated by tetrahedronizing a closed surface. , Equivalent density of hand tissue Multiply to obtain the mass corresponding to each hand joint. ; For the Each hand joint is formed by a local mesh consisting of corresponding vertices and faces. The boundaries of the mesh are closed to obtain a closed surface. Then, the closed surface is tetrahedralized to obtain several tetrahedral elements. The volume of the j-th tetrahedral unit is The center of mass is Then the first The total volume of each hand joint Center of mass , Let J be the centroid of the j-th tetrahedral element; in the... Local inertial tensor in the coordinate system of the centroid of a hand joint See the following formula (6) for details: (6); In equation (6), The coordinates represent the infinitesimal volume element; Represents a 3×3 identity matrix; Represents transpose; Represents a volumetric infinitesimal element; Calculated and As the first The mass and local inertia tensors of each hand joint are used to construct the generalized mass matrix. And the Coriolis force and centrifugal force terms ; S14. Rewrite the Euler-Lagrange equation for the hand in residual form, as shown in equation (7) below: (7); In equation (7), Let be the dynamic residual of the t-th frame image, which is treated as a virtual observation that follows a probability distribution, rather than being forced to be zero; (2) Based on the vertex coordinates of the hand network, high-resolution enhanced features and low-resolution global features are extracted, and adaptive features are generated through a lightweight cross-attention fusion adapter. The specific steps include: S21. For the hand grid in each frame of the image, extract spatial feature labels using a grid convolutional neural network to generate original resolution features. , see the following formula (8) for details: (8); In equation (8), Represents a grid convolutional neural network. For the number of feature labels, For feature channel dimensions; Simultaneously, for the hand mesh in each frame of the image, after downsampling, spatial feature labels are extracted using a mesh convolutional neural network to generate low-resolution features. , see the following formula (9) for details: (9); In equation (9), Represents downsampling, This represents the number of feature labels after downsampling; S22. Extract low-resolution global features using the global transformer attention module. , see the following formula (10) for details: (10); In equation (10), Represents the global transformer attention module; The global transformer attention module includes alternating frame-level attention layers and a global attention layer, specifically comprising the following steps: S221, The processing procedure of the frame-level attention layer is shown in the following formula (15): (15); In equation (15), Represents the global transformer attention module. The frame-level attention features of the t-th frame image output by the frame-level attention layer; Represents the global transformer attention module. The input features of the frame-level attention layer, when hour, ; Representing the Frame-level attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function; S222. Concatenate the frame-level attention features of all frames to obtain the frame-level attention concatenated features. , see the following formula (16) for details: (16); In equation (16), This represents a splicing operation; S223, The processing procedure of the global attention layer is shown in the following formula (17): (17); In equation (17), Representing the The global attention features of all frames output by the global attention layer are split to obtain the global attention features of each frame. That is, as the first The input features of the frame-level attention layer, and the output global attention features of all frames from the last global attention layer, are decomposed to obtain low-resolution global features. ; Representing the Global attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function; The frame-level attention layer computes attention between the tags within each frame, capturing the spatial structure of a single frame; the global attention layer computes attention between all tags across all frames, establishing cross-frame temporal relationships; by performing global attention at low resolution, the computational bottleneck of quadratic complexity at full resolution is avoided, enabling efficient processing of long sequences of video images. S23. Extract high-resolution features using the frame-by-frame Vision Transformer module. , see the following formula (11) for details: (11); In equation (11), Representing the frame-by-frame Vision Transformer (ViT) module, the frame-by-frame ViT module processes each frame independently, preserving high-frequency geometric information of fine structures such as fingertips and joint folds. Since it does not involve cross-frame global attention, the computational complexity of the high-resolution stream is linearly related to the number of frames. S24. Generate high-resolution enhanced features based on low-resolution global features and high-resolution features. , see the following formula (12) for details: (12); In equation (12), This represents a guided feature extraction network, which is a shallow convolutional network used to extract features from high-resolution data. Extract high-frequency guidance signals from them; This represents the bilinear interpolation operator, which upsamples low-resolution global features to the same spatial size as high-resolution features. The representative interpolation and refinement network is a shallow convolutional network used for low-resolution global features. Perform upsampling and refinement; It represents splicing; This represents a fusion network, which is a shallow convolutional network used for feature splicing and fusion. By leveraging the spatially accurate high-frequency details in high-resolution features, low-resolution global features are refined and compensated, effectively making up for the information loss caused by downsampling and generating enhanced feature representations that are spatially aligned with high resolution. The generated high-resolution enhanced features improve detail recovery capabilities while preserving the global geometric consistency of the low-resolution flow. S25. Generate adaptation features through a lightweight cross-attention fusion adapter. The lightweight cross-attention fusion adapter is composed of alternating stacks of cross-attention layers and self-attention layers. The processing procedure of the cross-attention layer is shown in the following equation (13), and the processing procedure of the self-attention layer is shown in the following equation (14): (13); (14); In equations (13) and (14), Representing the cross-attention layer, it enhances high-resolution features. As a query Q, low-resolution global feature As keys K and values ​​V, the global temporal context encoded in the low-resolution stream is injected into the high-resolution feature space; Represents cross-attention fusion features; Representing the self-attention layer, it fuses cross-attention features. Using query Q, key K, and value V, local associations are established between high-resolution labels and feature remodeling is completed; Represents the self-attention fusion feature; The output of the last self-attention layer of the lightweight cross-attention fusion adapter is used as the adaptation feature. ; The lightweight cross-attention fusion adapter explicitly injects the global temporal context modeled in the low-resolution stream into the high-resolution stream, realizing the joint modeling of dual-stream features. It completes the semantic injection of global context into high-resolution local representation through the attention mechanism, so that the final features have both long temporal consistency and fine local structure expression capabilities. (3) The dual-stream encoder extracts features at different depths based on the adaptive features, then modulates the features based on the kernelized Gram matrix statistics, and maps them into a multi-scale conditional feature sequence. Finally, it fuses the sequence with the initial pose estimate into a conditional vector. The specific steps include: S31. The backbone Transformer of the dual-stream encoder includes a multi-layer structure. The output features of a selected layer in the first few layers of the backbone Transformer are used as shallow features, the output features of a selected layer in the middle few layers of the backbone Transformer are used as mid-layer features, and the output features of a selected layer in the last few layers of the backbone Transformer are used as deep features. The shallow features, mid-layer features, and deep features are the output features of the selected network layer set. S32. Hand motion sequences contain geometrically singular regions such as self-occluded fingers, motion blur, and joint folding. These regions generate statistically unstable markers in the feature space, propagate through the network, and reduce the accuracy of geometric inference. A feature modulation mechanism based on high-order statistical analysis is introduced to detect and suppress these unreliable markers. Specifically, the shallow, medium, and deep features corresponding to the t-th frame image are linearly transformed into the corresponding query matrix. Bond matrix Then, the query-query Gram matrix is ​​calculated according to the following formula (18). Key-key Gram matrix And query-keygram matrix ; (18); In equation (18), Represents the feature dimension; Then the adjacent frame windows The Gram matrix sequence within the matrix is ​​modeled as a matrix-valued stochastic process, and the time-series expectation operator is calculated. and temporal variance operator See equations (19) and (20) below for details: (19); (20); In equations (19) and (20), Represents the time offset index, representing the adjacent frame window. Within, the timing window between the i-th frame and the j-th frame, and the adjacent frame window. That is, a continuous frame interval centered on the i-th frame and extracted before and after it; The steady-state component of the characteristic manifold represents the desired geometric structure; The wave component of the characteristic manifold; Represents different Gram matrix types; For the temporal expectation operator, for adjacent frame windows The expected value of the Gram matrix sequence within the time dimension is taken; As a spatial expectation operator, it takes the expectation of the layer dimension of the Gram matrix within the selected set of network layers; Represents the temporal variance operator, for adjacent frame windows The variance of the Gram matrix sequence within the time dimension is taken; Based on a hierarchical Bayesian inference framework, multi-scale anomaly priors are derived from different depths, and a posterior anomaly saliency map is constructed. , see the following formula (21) for details: (twenty one); In equation (21), When representing shallow input features, and The steady-state component of the feature manifold computed at time represents the temporal expectation statistic of the bond-bond Gram matrix in the shallow selected attention layer, which is used to characterize the stable semantic structure of shallow features between adjacent frames. When representing shallow input features, and The variability component of the feature manifold calculated over time represents the temporal variance statistic of the shallow query-key Gram matrix, used to characterize the cross-frame variability of the relationship between query features and key features; When representing the mid-level features of the input, and The steady-state component of the feature manifold computed at time represents the temporal expectation statistic of the query-query Gram matrix in the selected attention layer of the middle layer, which is used to characterize the stable structure of the geometric relationship between the feature labels in the middle layer; When representing deep input features, and The variability component of the characteristic manifold calculated over time represents the temporal variance statistic of the deep query-query Gram matrix, which is used to measure the degree of cross-frame variability in deep spatial relationships. When representing deep input features, and The steady-state components of the feature manifold computed over time represent the time-series expected statistics of the deep query-query Gram matrix, used to represent the stable spatial structure in deep features; This represents a priori shallow anomalies. This represents a priori anomalies in the middle layer. This represents a priori deep anomalies; The saliency map represents the posterior result after fusing the three priors; This represents element-wise multiplication; when A lower value indicates weaker stability of shallow semantic structures; when A higher value indicates that the attention association fluctuates significantly between adjacent frames. and Element-wise multiplication can highlight semantically unstable and attention-relationship-fluctuating anomaly markers in shallow layers; when A lower value indicates a lack of stable cross-frame consistency during mid-level geometric inference, therefore... The mid-level geometric anomaly response was obtained; Used to preserve deep, stable regions with small cross-frame fluctuations, and then combined with Element-wise multiplication creates a low-frequency spatial constraint on the anomaly saliency map, avoiding oversuppression caused by shallow noise alone; Based on anomaly significance map For adaptation features Perform feature modulation to obtain modulation features , see the following formula (22) for details: (twenty two); Modulation characteristics While preserving reliable marker information, noise propagation in unreliable regions is suppressed, and finally, the modulation features of all frames are combined into a modulation feature sequence. ; By setting an anomaly significance threshold For the anomaly saliency plot Exceeding the abnormal significance threshold modulation characteristics Mark; S33, Modulation feature sequence Encoding mapping is performed to obtain multi-scale conditional feature sequences. , see the following formula (23) for details: (twenty three); In equation (23), Represents the conditional coding mapping operator; S34. Fuse the initial pose estimation and multi-scale conditional feature sequence into a conditional vector. , see the following formula (24) for details: (twenty four); In equation (24), Represents the conditional feature encoding mapping operator; Represents the average shape parameter; Represents the pose parameters corresponding to all frame images. The sequence of attitude parameters; (4) The Transformer backbone network predicts and outputs a clean hand motion sequence. The conditional diffusion model performs inverse denoising and refinement to generate the hand motion recovery result. The specific steps include: S41. Define the global hand motion deviation sequence. , see the following formula (25) for details: (25); In equation (25), Represents the sequence of attitude parameters and shape parameters Mapping to the generalized coordinate motion state; This represents the initial motion estimation sequence corresponding to all frame images; This represents the initial motion estimate corresponding to the t-th frame image, including the wrist rotation vector. Wrist translation vector and fifteen attitude parameters ; Represents transpose; Represents a clean hand movement sequence to be restored; S42. Define the single-step diffusion offset increment as follows (26): (26); In equation (26), This represents the offset increment of the k-th diffusion step relative to the (k-1)-th diffusion step; Represents the cumulative offset scheduling coefficient at step k; This represents the cumulative offset scheduling coefficient at step k-1; when k equals 1, equal ; S43. The single-step noise addition process of the forward diffusion in the conditional diffusion model is shown in the following equation (27): (27); In equation (27), This represents the hand movement sequence at the given (k-1)th diffusion step. Conditional vector and initial motion estimation sequence Given the premise, generate the hand motion sequence for the kth diffusion step. Forward diffusion transfer distribution; Representative with For the mean, The covariance matrix is ​​a Gaussian distribution; Represents the scale of diffused noise. Represents the noise variance scale; The identity matrix is ​​represented by equation (27); Equation (27) indicates that during the forward diffusion single-step noise addition process, the hand motion sequence of the previous diffusion step is translated along the deviation direction. And add variance as Gaussian noise; The forward diffusion single-step noise addition process according to formula (27) is performed step by step, and the entire process is cumulatively integrated, as shown in formula (28) below: (28); Equation (28) indicates that after k steps of diffusion, the mean of the hand motion sequence decreases from the clean hand motion sequence to be recovered. Stepwise to the initial motion estimation sequence move; S44. Conditional diffusion model reverse denoising and restoration: First, build the overall framework of reverse diffusion, as shown in the following formula (29): (29); In equation (29), Represents in a given Under the premise of generating The reverse diffusion distribution; This represents the initial reverse diffusion distribution starting from the k-th diffusion step hand motion sequence; Represented by parameters A controllable, learnable reverse transfer kernel; K is the total number of diffusion steps; This represents the complete sequence of hand movements from the first diffusion step to the kth diffusion step; the integral is used to marginalize all intermediate states, i.e., the entire sequence of hand movements. In equation (29) Each diffusion step is determined by parameters The learnable inverse transfer kernel for control is updated as shown in equation (30) below: (30); In equation (30), The mean function representing backward diffusion; The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; This represents the clean hand motion sequence predicted by the Transformer backbone network prediction head. Weighting coefficients; Represents the cumulative offset scheduling coefficient at step k; This represents the offset increment of k diffusion steps relative to the (k-1)th diffusion step; This represents the noise variance scale determined by the forward diffusion process; In equation (30), the prediction head of the Transformer backbone network predicts and outputs a clean hand motion sequence. , see the following formula (31) for details: (31); In equation (31), This represents a clean hand motion sequence predicted by the Transformer backbone network. A linear mapping function representing the prediction head of the Transformer backbone network; The sequence of hand movements represents the k-th diffusion step; S45. Construct a double loss function to train and iteratively update the parameters of the Transformer backbone network and the conditional diffusion model. The double loss function is shown in the following formula (32): (32); In equation (32), The data-driven loss is represented by the following equation (33), which is used to constrain the clean hand motion sequence predicted by the Transformer backbone network to be close to the real hand motion sequence. The physical perception loss is represented by the following equation (34), which is derived from the negative log-likelihood of zero-mean Gaussian virtual observations and is used to penalize hand motion predictions with poor physical consistency. , These represent the training weights for the two losses, respectively. (33); In equation (33), This represents randomly sampling within the range of 1 to K for each diffusion step k and calculating the expectation. This represents the actual hand movement sequence in the training data, i.e., the true label; This represents the clean hand motion sequence predicted by the Transformer backbone network prediction head, i.e., the prediction result; Represents the square of the L2 norm; (34); In equation (34), Representative based on The calculated Eulerian-Lagrange dynamic residual sequence, i.e., the hand dynamic residual; This indicates that the probability distribution follows a certain probability distribution. Represents a zero-mean Gaussian distribution; The virtual observation variance representing the dynamic residuals, , This represents the total number of samples in the training set. Represents the scaling factor; Represents the identity matrix; This represents the actual hand movement sequence of the i-th sample in the training set; Representative based on The calculated Eulerian-Lagrange dynamical residual sequence; S46. Inference stage: The conditional diffusion model uses the initial motion estimation sequence... Corresponding high noise state Starting with step K, from step K to step 1, a learnable inverse transfer kernel is executed progressively, with each step using a clean hand motion sequence predicted by the Transformer backbone network. Integration and Update and Then based on the updated mean function and noise variance ,renew Iterate to At that time, output the hand motion recovery results after noise reduction and refinement. ; (5) Based on the dynamic variance of Bayesian inference, the uncertainty quantification and physical reliability assessment of the hand motion recovery results are carried out, which specifically includes the following steps: S51. Using the final layer Laplace approximation as the posterior inference step, only the posterior distribution of the Transformer backbone network prediction head parameters is mapped to a Gaussian posterior prediction distribution, while outputting the predicted clean hand motion sequence. The point estimation and prediction uncertainty are shown in the following equation (35): (35); In equation (35), Represents in a given , and training dataset At that time, the clean hand movement sequence to be restored The posterior predicted distribution; The linear mapping function representing the prediction head of the Transformer backbone network. That is, the clean hand motion sequence predicted by the Transformer backbone network. The point estimate corresponds to the mean of the Gaussian posterior prediction distribution; This represents the diagonal variance matrix obtained by the Laplace approximation of the last layer; This is a joint-level variance prediction function that outputs the variance of each hand joint, i.e., the corresponding prediction uncertainty. S52. In backdiffusion sampling, the hand motion state and its uncertainty are gradually propagated in the backdiffusion process by updating the hand motion state and propagating its variance. In the backdiffusion sampling, the update of the hand motion state is shown in the following equation (36): (36); In equation (36), This represents the hand motion sequence obtained from the (k-1)th diffusion step through reverse diffusion sampling. The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; This represents the clean hand motion sequence predicted by the Transformer backbone network prediction head. Weighting coefficients; The standard deviation of the reverse transfer noise at the k-th diffusion step; Represents standard Gaussian noise, and satisfies ; In reverse diffusion sampling, the propagation of the variance of hand motion state is shown in the following equation (37): (37); In equation (37), A diagonal approximation representing variance; The variance of the hand motion state represents the (k-1)th diffusion step. This represents the variance of the hand motion state at the current k-th diffusion step; The variance of the clean hand motion predicted by the Transformer backbone network prediction head is calculated by the following formula (38); represent and The covariance between them is calculated by the following formula (39); (38); In equation (38), This represents constructing a diagonal variance matrix from the joint-level variance vectors; Represents the joint-level variance prediction function, inputting the hand motion state at the current k-th diffusion step. Conditional vector and current diffusion steps Output the variance of each hand joint; (39); In equation (39), represent Expectations; represent Expectations; represent Expectations; Represents the number of samples in the Monte Carlo sampling; ⊙ indicates element-wise multiplication. This represents the hand motion state of the i-th sample at the current k-th diffusion step; This represents the clean hand motion sequence output after the i-th sample is predicted by the Transformer backbone network; S53. Based on the content constructed in S52, variance estimation is performed, specifically as follows: S531, Bayesian Prior Initialization: Based on the Bayesian prior distribution, rules are set for the starting point. From Gaussian distribution Once fixed after a single sampling, it is considered a deterministic conditional quantity; initial expectation. Initialize the variance of hand motion state Initialize covariance ; S532, Bayesian posterior iterative propagation: For k=K,K-1,…,1, perform the following operations in sequence: Based on the current hand movement state, sample the prediction results of the Transformer backbone network from the Gaussian distribution; update the expectation and covariance of the next step according to equation (39); update the variance of the hand movement state of the next step according to equation (37); update the hand movement state of the next step according to formula (36); S533. After K-step Bayesian posterior iterations, the final refined hand motion state sequence is obtained. and its variance ; S54. Utilize the final hand movement state variance The variance of the generalized force is calculated by propagation through the Jacobian matrix. , see the following formula (40) for details: (40); In equation (40), Represents generalized force Relative to hand movement state The Jacobian matrix; Represents generalized force hand movement status The partial derivatives; Represents the final refined sequence of hand movement states. Calculate the Jacobian matrix at the expected value; Represents transpose; Calculating the generalized force variance quantifies the uncertainty in the motion recovery process, enabling the identification of potentially unreliable motion estimation frames and joints. High variance regions correspond to frames or joints with weak physical consistency, i.e., regions where the original image estimation is unreliable. Combined with the feature-level anomaly detection results provided by the feature modulation mechanism, the reliability of hand motion estimation can be evaluated at multiple levels.

[0023] The final output of the hand movement recovery method consists of three parts: (a) the final refined hand movement state sequence. (b) Frame-by-frame, joint-by-joint generalized force variance plot, i.e., from the generalized force variance (c) Extract diagonal elements and then expand them according to the dimensions of frames and hand joints to obtain a visualization view; (d) Anomaly saliency map at the feature level, used to identify geometrically singular regions and assist in locating key anomalies. Example

[0024] The overall model corresponding to this invention is implemented using the PyTorch deep learning framework and deployed on a server environment equipped with a high-performance GPU. The global transformer attention module for the low-resolution stream includes alternating frame-level attention layers and global attention layers. The frame-by-frame Vision Transformer module for the high-resolution stream processes each frame independently. The grid convolutional neural network contains four layers with channel dimensions of [32, 64, 64, 64] in sequence. The encoder and decoder of the global transformer attention module each contain four layers, each layer using eight attention heads with an embedding dimension of 512. The lightweight cross-attention fusion adapter consists of two layers of cross-attention layers and two layers of self-attention layers stacked alternately.

[0025] During the training phase, based on a 3D hand motion capture dataset from a power maintenance scenario, the AdamW optimizer was used for parameter updates, with an initial learning rate set to [value missing]. The decay factor is 0.8 every 10 training epochs. Training weights with data-driven loss. Weights of physical perception loss The total number of diffusion steps K in the conditional diffusion model is set to 4, and the sequence length M is set to 16 frames; the threshold for anomaly salience in feature modulation is also specified. The validation set is adaptively determined. Furthermore, geometric supervision loss and consistency regularization terms can be introduced to mitigate error accumulation in diffusion sampling.

[0026] During the inference phase, given a video sequence input from a power maintenance site, an initial pose estimate is first obtained through a pre-trained hand pose estimator. After dual-stream feature extraction and feature modulation, a conditional vector is obtained. Then, the Transformer backbone network predicts and outputs a clean hand motion sequence, and performs a reverse diffusion process to refine the hand motion. Simultaneously, a variance estimation algorithm is executed, with the Monte Carlo sampling number S set to 20. Finally, the refined hand motion state sequence, generalized force variance map, and anomaly saliency map are output, which are used for maintenance action quality assessment, abnormal behavior localization, and safety auxiliary decision-making.

[0027] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for restoring hand movement during power maintenance, characterized in that: Specifically, it includes the following steps: (1) Construct a hand network based on the initial pose estimation of the parametric hand model, constrain and correct the hand network with the physical rules of dynamic modeling, and obtain the vertex coordinates of the hand network; (2) Based on the vertex coordinates of the hand network, high-resolution enhanced features and low-resolution global features are extracted, and adaptive features are generated through a lightweight cross-attention fusion adapter; (3) The dual-stream encoder extracts features at different depths based on the adaptive features, then modulates the features based on the kernelized Gram matrix statistics, and maps them into a multi-scale conditional feature sequence. Finally, it fuses the sequence with the initial pose estimate into a conditional vector. (4) The backbone network predicts and outputs a clean hand motion sequence, and the conditional diffusion model performs reverse denoising and refinement to generate hand motion recovery results; (5) Based on the dynamic variance of Bayesian inference, the uncertainty of the hand motion recovery results is quantified and the physical reliability is assessed.

2. The hand movement recovery method based on power maintenance according to claim 1, characterized in that: The initial pose estimation based on the parametric hand model constructs a hand network, which is then constrained and corrected using physical rules of dynamic modeling to obtain the vertex coordinates of the hand network. This process specifically includes the following steps: S11, a segment containing The maintenance monocular video image sequence is processed. Each frame of the sequence is input into the hand pose estimator to obtain the parameterized hand model parameters for each frame. The parameterized hand model parameters include the wrist rotation vector. Wrist translation vector Attitude parameters and shape parameters , Attitude parameters It includes 3D rotation vectors for 15 hand joints, and the shape parameters include 10 principal component coefficients in the shape space of the parameterized hand model, which are 10-dimensional vectors; The average shape parameter is calculated using the following formula (1). : （1）； S12. Based on the parametric hand model, define the generalized coordinates of the hand network. , see the following formula (2) for details: （2）； In equation (2), the generalized coordinates of the hand network It is the wrist rotation vector Wrist translation vector and attitude parameters composition; Generalized speed of hand networks and generalized acceleration It is obtained through the finite difference method, as shown in equation (3) below: （3）； In equation (3), The generalized coordinates represent the (t+1)th frame of the image; The generalized coordinates represent the (t-1)th frame of the image; Represents the generalized coordinates of the image in frame t; Represents the time interval between adjacent frames; S13. Establish the Euler-Lagrange equation for the hand, as shown in equation (4) below: （4）； In equation (4), Represents the generalized mass matrix; Representing the Coriolis force and centrifugal force terms, describing the non-inertial force effects caused by joint rotation; Represents the gravity term, indicating the generalized force at each joint in the gravitational field; It represents the net generalized force, which is the sum of the external force and joint torque required for hand movement; This represents a set of mass parameters for fifteen hand joints; The set of inertial tensors representing the fifteen hand joints; The vertex coordinates of the hand mesh are obtained by using the linear blending skin function of the parameterized hand model, as shown in equation (5) below: （5）； In equation (5), Represents the coordinates of the hand grid vertices in the t-th frame image; Represents a linear blending skinning function; In this context, 778 represents the number of vertices in the parametric hand model, and 3 represents the three-dimensional coordinate dimension of each vertex. Using skin weights, each vertex of the hand mesh is assigned to the corresponding hand joint. The parameterized hand model defines the first... Skin weights of each vertex relative to each hand joint ,but ,and In the linear hybrid skin process, the first The deformed position of each vertex ,in, For the template posture, the first homogeneous coordinates of the vertices For the first The global transformation matrix calculated along the kinematic chain for each hand joint; To calculate the volume and inertial parameters of each hand joint, a deterministic allocation is performed using the maximum weight principle, i.e. , Representing the Each vertex is assigned an index to a hand joint; that is, for each vertex in the hand network, traverse the fifteen hand joints to find the largest weight. The corresponding hand joint, will the first The vertex is assigned to the first... For each hand joint, the hand joint to which the face belongs is determined by the majority of the three vertices corresponding to each face in the hand network; if there is no majority of the face, the face is assigned to the hand joint with the largest average skinning weight, thereby obtaining the vertex set and face set corresponding to each hand joint. The volume of each hand joint is calculated by tetrahedronizing a closed surface. , Equivalent density of hand tissue Multiply to obtain the mass corresponding to each hand joint. ; For the Each hand joint is formed by a local mesh consisting of corresponding vertices and faces. The boundaries of the mesh are closed to obtain a closed surface. Then, the closed surface is tetrahedralized to obtain several tetrahedral elements. The volume of the j-th tetrahedral unit is The center of mass is Then the first The total volume of each hand joint Center of mass , Let J be the centroid of the j-th tetrahedral element; in the... Local inertial tensor in the coordinate system of the centroid of a hand joint See the following formula (6) for details: （6）； In equation (6), The coordinates representing the infinitesimal volume element; Represents a 3×3 identity matrix; Represents transposition; Represents a volumetric infinitesimal element; Calculated and As the first The mass and local inertia tensors of each hand joint are used to construct the generalized mass matrix. And the Coriolis force and centrifugal force terms ; S14. Rewrite the Euler-Lagrange equation for the hand in residual form, as shown in equation (7) below: （7）； In equation (7), Let be the dynamic residual of the t-th frame image, which is treated as a virtual observation that follows a probability distribution, rather than being forced to be zero.

3. The hand movement recovery method based on power maintenance according to claim 2, characterized in that: The process of generating adaptation features based on the vertex coordinates of the hand network specifically includes the following steps: S21. For the hand grid in each frame of the image, extract spatial feature labels using a grid convolutional neural network to generate original resolution features. , see the following formula (8) for details: （8）； In equation (8), Represents a grid convolutional neural network. For the number of feature labels, For feature channel dimensions; Simultaneously, for the hand mesh in each frame of the image, after downsampling, spatial feature labels are extracted using a mesh convolutional neural network to generate low-resolution features. , see the following formula (9) for details: （9）； In equation (9), Represents downsampling, This represents the number of feature labels after downsampling; S22. Extract low-resolution global features using the global transformer attention module. , see the following formula (10) for details: （10）； In equation (10), Represents the global transformer attention module; S23. Extract high-resolution features using the frame-by-frame Vision Transformer module. , see the following formula (11) for details: （11）； In equation (11), Represents the frame-by-frame Vision Transformer module; S24. Generate high-resolution enhanced features based on low-resolution global features and high-resolution features. , see the following formula (12) for details: （12）； In equation (12), This represents a guided feature extraction network, which is a shallow convolutional network used to extract features from high-resolution data. Extract high-frequency guidance signals from them; Represents the bilinear interpolation operator; The representative interpolation and refinement network is a shallow convolutional network used for low-resolution global features. Perform upsampling and refinement; It represents splicing; This represents a fusion network, which is a shallow convolutional network used for feature splicing and fusion. S25. Generate adaptation features through a lightweight cross-attention fusion adapter. The lightweight cross-attention fusion adapter is composed of alternating stacks of cross-attention layers and self-attention layers. The processing procedure of the cross-attention layer is shown in the following equation (13), and the processing procedure of the self-attention layer is shown in the following equation (14): （13）； （14）； In equations (13) and (14), Representing the cross-attention layer, it enhances high-resolution features. As a query Q, low-resolution global feature As keys K and values ​​V, the global temporal context encoded in the low-resolution stream is injected into the high-resolution feature space; Represents cross-attention fusion features; Representing the self-attention layer, it fuses cross-attention features. Using query Q, key K, and value V, local associations are established between high-resolution labels and feature remodeling is completed; Represents the self-attention fusion feature; The output of the last self-attention layer of the lightweight cross-attention fusion adapter is used as the adaptation feature. .

4. The hand movement recovery method based on power maintenance according to claim 3, characterized in that: The global transformer attention module includes alternating frame-level attention layers and a global attention layer, specifically comprising the following steps: S221, The processing procedure of the frame-level attention layer is shown in the following formula (15): （15）； In equation (15), Represents the global transformer attention module. The frame-level attention features of the t-th frame image output by the frame-level attention layer; Represents the global transformer attention module. The input features of the frame-level attention layer, when hour, ; Representing the Frame-level attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function; S222. Concatenate the frame-level attention features of all frames to obtain the frame-level attention concatenated features. , see the following formula (16) for details: （16）； In equation (16), This represents a splicing operation; S223, The processing procedure of the global attention layer is shown in the following formula (17): （17）； In equation (17), Representing the The global attention features of all frames output by the global attention layer are split to obtain the global attention features of each frame. That is, as the first The input features of the frame-level attention layer, and the output global attention features of all frames from the last global attention layer, are decomposed to obtain low-resolution global features. ; Representing the Global attention layer; , , Representing the first The query matrix, key matrix, and value matrix obtained after linear transformation of the input features of the frame-level attention layer all have dimensions of 1. ; Represents the scaling factor; This represents the softmax normalization function.

5. A hand movement recovery method based on power maintenance according to claim 3, characterized in that: The dual-stream encoder extracts features at different depths based on adaptive features, then modulates the features based on kernelized Gram matrix statistics, and maps them into a multi-scale conditional feature sequence. Finally, it fuses this sequence with the initial pose estimate to form a conditional vector. The specific steps include: S31. The backbone Transformer of the dual-stream encoder includes a multi-layer structure. The output features of a selected layer in the first few layers of the backbone Transformer are used as shallow features, the output features of a selected layer in the middle few layers of the backbone Transformer are used as mid-layer features, and the output features of a selected layer in the last few layers of the backbone Transformer are used as deep features. The shallow features, mid-layer features, and deep features are the output features of the selected network layer set. S32. Linearly transform the shallow, middle, and deep features corresponding to the t-th frame image into the corresponding query matrix. Bond matrix Then, the query-query Gram matrix is ​​calculated according to the following formula (18). Key-key Gram matrix And query-keygram matrix ; （18）； In equation (18), Represents the feature dimension; Then the adjacent frame windows The Gram matrix sequence within the matrix is ​​modeled as a matrix-valued stochastic process, and the time-series expectation operator is calculated. and temporal variance operator See equations (19) and (20) below for details: （19）； （20）； In equations (19) and (20), Represents the time offset index, representing the adjacent frame window. Within, the timing window between the i-th frame and the j-th frame, and the adjacent frame window. That is, a continuous frame interval centered on the i-th frame and extracted before and after it; The steady-state component of the characteristic manifold represents the desired geometric structure; The wave component of the characteristic manifold; Represents different Gram matrix types; For the temporal expectation operator, for adjacent frame windows The expected value of the Gram matrix sequence within the time dimension is taken; As a spatial expectation operator, it takes the expectation of the layer dimension of the Gram matrix within the selected set of network layers; Represents the temporal variance operator, for adjacent frame windows The variance of the Gram matrix sequence within the time dimension is taken; Based on a hierarchical Bayesian inference framework, multi-scale anomaly priors are derived from different depths, and a posterior anomaly saliency map is constructed. , see the following formula (21) for details: （21）； In equation (21), When representing shallow input features, and The steady-state components of the characteristic manifold calculated in real time; When representing shallow input features, and The wave component of the characteristic manifold calculated in time; When representing the mid-level features of the input, and The steady-state components of the characteristic manifold calculated in real time; When representing deep input features, and The wave component of the characteristic manifold calculated in time; When representing deep input features, and The steady-state components of the characteristic manifold calculated in real time; This represents a priori shallow anomalies. This represents a priori anomalies in the middle layer. This represents a priori deep anomalies; The saliency map represents the posterior result after fusing the three priors; This represents element-wise multiplication; Based on anomaly significance map For adaptation features Perform feature modulation to obtain modulation features , see the following formula (22) for details: （22）； Finally, the modulation features of all frames are combined into a modulation feature sequence. ; By setting an anomaly significance threshold For the anomaly saliency plot Exceeding the abnormal significance threshold modulation characteristics Mark; S33, Modulation feature sequence Encoding mapping is performed to obtain multi-scale conditional feature sequences. , see the following formula (23) for details: （23）； In equation (23), Represents the conditional coding mapping operator; S34. Fuse the initial attitude estimation and multi-scale conditional feature sequence into a conditional vector. , see the following formula (24) for details: （24）； In equation (24), Represents the conditional feature encoding mapping operator; Represents the average shape parameter; Represents the pose parameters corresponding to all frame images. The sequence of attitude parameters.

6. A hand movement recovery method based on power maintenance according to claim 5, characterized in that: The backbone network predicts and outputs a clean hand motion sequence, which is then refined using a conditional diffusion model to generate a hand motion recovery result. This process includes the following steps: S41. Define the global hand motion deviation sequence. , see the following formula (25) for details: （25）； In equation (25), Represents the sequence of attitude parameters and shape parameters Mapping to the generalized coordinate motion state; This represents the initial motion estimation sequence corresponding to all frame images; This represents the initial motion estimate corresponding to the t-th frame image, including the wrist rotation vector. Wrist translation vector and fifteen attitude parameters ; Represents transposition; Represents a clean hand movement sequence to be restored; S42. Define the single-step diffusion offset increment as follows (26): （26）； In equation (26), This represents the offset increment of the k-th diffusion step relative to the (k-1)-th diffusion step; Represents the cumulative offset scheduling coefficient at step k; This represents the cumulative offset scheduling coefficient at step k-1; when k equals 1, equal ; S43. The single-step noise addition process of the forward diffusion in the conditional diffusion model is shown in the following equation (27): （27）； In equation (27), This represents the hand movement sequence at the given (k-1)th diffusion step. Conditional vector and initial motion estimation sequence Given the premise, generate the hand motion sequence for the kth diffusion step. Forward diffusion transfer distribution; Representative with For the mean, The covariance matrix is ​​a Gaussian distribution; Represents the scale of diffused noise. Represents the noise variance scale; Represents the identity matrix; Equation (27) represents the process of translating the hand motion sequence of the previous diffusion step along the deviation direction during the forward diffusion single-step noise addition process. And add variance as Gaussian noise; The forward diffusion single-step noise addition process according to formula (27) is performed step by step, and the entire process is cumulatively integrated, as shown in formula (28) below: （28）； Equation (28) indicates that after k steps of diffusion, the mean of the hand motion sequence decreases from the clean hand motion sequence to be recovered. Stepwise to the initial motion estimation sequence move; S44. Conditional diffusion model reverse denoising and restoration: First, build the overall framework of reverse diffusion, as shown in the following formula (29): （29）； In equation (29), Represents in a given Under the premise of generating The reverse diffusion distribution; This represents the initial reverse diffusion distribution starting from the k-th diffusion step hand motion sequence; Represented by parameters Controllable, learnable inverse transfer kernel; K is the total number of diffusion steps; This represents the complete sequence of hand movements from the first diffusion step to the kth diffusion step; the integral is used to marginalize all intermediate states, i.e., the entire sequence of hand movements. In equation (29) Each diffusion step is determined by parameters The learnable inverse transfer kernel for control is updated as shown in equation (30) below: （30）； In equation (30), The mean function representing backward diffusion; The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; The clean hand motion sequence predicted by the backbone network prediction head. Weighting coefficients; Represents the cumulative offset scheduling coefficient at step k; This represents the offset increment of k diffusion steps relative to the (k-1)th diffusion step; This represents the noise variance scale determined by the forward diffusion process; In equation (30), the predictor head of the backbone network predicts and outputs a clean hand motion sequence. , see the following formula (31) for details: （31）； In equation (31), The sequence of clean hand movements predicted by the backbone network; A linear mapping function representing the backbone network prediction head; The sequence of hand movements represents the k-th diffusion step; S45. Construct a double loss function and train to iteratively update the parameters of the backbone network and the conditional diffusion model. The double loss function is shown in the following formula (32): （32）； In equation (32), This represents the data-driven loss, as shown in equation (33) below. This represents the loss of physical perception, as shown in equation (34) below. , These represent the training weights for the two losses, respectively. （33）； In equation (33), This represents randomly sampling within the range of 1 to K for each diffusion step k and calculating the expected value. This represents the actual hand movement sequence in the training data, i.e., the true label; The clean hand motion sequence predicted by the backbone network prediction head is the prediction result. Represents the square of the L2 norm; （34）； In equation (34), Representative based on The calculated Eulerian-Lagrange dynamic residual sequence, i.e., the hand dynamic residual; This indicates that the probability distribution follows a certain probability distribution. Represents a Gaussian distribution with zero mean; The virtual observation variance representing the dynamic residuals, , This represents the total number of samples in the training set. Represents the scaling factor; Represents the identity matrix; This represents the actual hand movement sequence of the i-th sample in the training set; Representative based on The calculated Eulerian-Lagrange dynamical residual sequence; S46. Inference stage: The conditional diffusion model uses the initial motion estimation sequence... Corresponding high noise state Starting with step K, from step K to step 1, a learnable inverse transfer kernel is executed progressively, with each step using a clean hand motion sequence predicted by the backbone network. Integration and Update and Then based on the updated mean function and noise variance ,renew Iterate to At that time, output the hand motion recovery results after noise reduction and refinement. .

7. A method for restoring hand movement based on power maintenance according to claim 6, characterized in that: The uncertainty quantification and physical reliability assessment of the hand motion recovery results based on the Bayesian inference dynamic variance specifically includes the following steps: S51. Using the final layer Laplace approximation as the posterior inference step, only the posterior distribution of the backbone network's predicted head parameters is mapped to a Gaussian posterior prediction distribution, while simultaneously outputting the predicted clean hand motion sequence. The point estimation and prediction uncertainty are shown in the following equation (35): （35）； In equation (35), Represents in a given , and training dataset At that time, the clean hand movement sequence to be restored The posterior predicted distribution; The linear mapping function representing the backbone network prediction head, That is, the clean hand movement sequence predicted by the backbone network. The point estimate corresponds to the mean of the Gaussian posterior prediction distribution; This represents the diagonal variance matrix obtained by the Laplace approximation of the last layer; This is a joint-level variance prediction function that outputs the variance of each hand joint, i.e., the corresponding prediction uncertainty. S52. In backdiffusion sampling, the hand motion state and its uncertainty are gradually propagated in the backdiffusion process by updating the hand motion state and propagating its variance. In the backdiffusion sampling, the update of the hand motion state is shown in the following equation (36): （36）； In equation (36), This represents the hand motion sequence obtained from the (k-1)th diffusion step through reverse diffusion sampling. The noise variance represents the back diffusion in the k-th diffusion step; Represents the identity matrix; This represents the k-th diffusion step; Represents the hand movement sequence of the kth diffusion step. Weighting coefficients; The clean hand motion sequence predicted by the backbone network prediction head. Weighting coefficients; The standard deviation of the reverse transfer noise at the k-th diffusion step; Represents standard Gaussian noise, and satisfies ; In reverse diffusion sampling, the propagation of the variance of hand motion state is shown in the following equation (37): （37）； In equation (37), A diagonal approximation representing variance; The variance of the hand motion state represents the (k-1)th diffusion step. This represents the variance of the hand motion state at the current k-th diffusion step; The variance of clean hand motion predicted by the backbone network prediction head is calculated by the following formula (38); represent and The covariance between them is calculated by the following formula (39); （38）； In equation (38), This represents constructing a diagonal variance matrix from the joint-level variance vectors; This represents the joint-level variance prediction function, with the hand motion state at the current k-th diffusion step as input. Conditional vector and current diffusion steps Output the variance of each hand joint; （39）； In equation (39), represent Expectations; represent Expectations; represent Expectations; Represents the number of samples in the Monte Carlo sampling; ⊙ indicates element-wise multiplication. This represents the hand motion state of the i-th sample at the current k-th diffusion step; This represents the clean hand motion sequence output after the i-th sample is predicted by the backbone network; S53. Based on the content constructed in S52, variance estimation is performed, specifically as follows: S531, Bayesian Prior Initialization: Based on the Bayesian prior distribution, rules are set for the starting point. From Gaussian distribution Once fixed after a single sampling, it is considered a deterministic conditional quantity; initial expectation. Initialize the variance of hand motion state Initialize covariance ; S532, Bayesian posterior iterative propagation: For k=K,K-1,…,1, perform the following operations in sequence: Based on the current hand movement state, sample the backbone network prediction results from the Gaussian distribution; Update the expectation and covariance of the next step according to equation (39); Update the variance of the hand movement state of the next step according to equation (37); Update the hand movement state of the next step according to formula (36); S533. After K-step Bayesian posterior iterations, the final refined hand motion state sequence is obtained. and its variance ; S54. Utilize the final hand movement state variance The variance of the generalized force is calculated through Jacobian matrix propagation. , see the following formula (40) for details: （40）； In equation (40), Represents generalized force Relative to hand movement state Jacobian matrix; Represents generalized force hand movement status The partial derivatives; Represents the final refined sequence of hand movement states. Calculate the Jacobian matrix at the expected value; This represents transposition.

8. A method for restoring hand movement based on power maintenance according to claim 7, characterized in that: The final output of the hand movement recovery method includes three parts: (a) the final refined hand movement state sequence. (b) Frame-by-frame, joint-by-joint generalized force variance plot, i.e., from the generalized force variance (c) Extract diagonal elements and then expand them according to the dimensions of frames and hand joints to obtain a visualization view; (d) Anomaly saliency map at the feature level, used to identify geometrically singular regions and assist in locating key anomalies.

Images