A multi-modal action recognition and fusion evaluation method and system based on HGAT

Abstract

This application provides a method and system for multimodal motion recognition and fusion evaluation based on HGAT. The method includes: acquiring multimodal data of a target object; wherein the target object is a person requiring evaluation among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data; based on the visual images and sensor data, performing node extraction and cross-modal association to determine a heterogeneous graph; based on the heterogeneous graph, calculating attention coefficients; based on the attention coefficients, performing weight allocation and feature fusion to obtain fused features; based on the fused features, performing observation noise covariance prediction and filtering processing through a recurrent neural network to obtain high-precision skeletal data; based on the high-precision skeletal data, calculating ground truth joint range of motion, motion smoothness, and stability limits; based on the ground truth joint range of motion, motion smoothness, and stability limits, performing fusion evaluation to obtain the evaluation result. The above scheme can improve the robustness and accuracy of motion capture.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a multimodal action recognition and fusion evaluation method and system based on HGAT. Background Technology

[0002] With the deepening application of multimodal sensing technology in the field of intelligent rehabilitation, high-precision motion capture has become the core of quantitative assessment. However, existing single-modal technologies face insurmountable bottlenecks in complex clinical and home rehabilitation scenarios. In existing intelligent rehabilitation systems, motion capture mainly relies on two technical paths: one is based on computer vision (such as smart mirrors), which can obtain global coordinates of joint points without wearing devices, but is prone to key point recognition failure, jitter, or trajectory loss when faced with common equipment occlusion (such as rehabilitation frames and wheelchairs), complex background interference, and changes in lighting conditions in rehabilitation scenarios; the other is based on wearable sensors (such as smart knee braces), which, although having a high acquisition frequency and unaffected by line-of-sight occlusion, is limited by the physical characteristics of inertial measurement units, and will inevitably produce integral drift over long periods of operation, leading to cumulative errors in pose calculation, and lacking absolute position reference in Euclidean space.

[0003] Traditional fusion methods (such as weighted averaging or extended Kalman filtering) often struggle to achieve adaptive complementarity when dealing with severely nonlinear, burst-like occlusion, or heterogeneous data (image spatial data and temporal physical data). For example, when a patient's leg is obscured by a table, causing visual impairment, the algorithm cannot automatically detect and switch to a reliable sensor data source, resulting in poor robustness and accuracy of motion capture. Summary of the Invention

[0004] This application aims to provide a multimodal motion recognition and fusion evaluation method and system based on HGAT, which can improve the robustness and accuracy of motion capture.

[0005] The technical solution of this application is implemented as follows: In a first aspect, embodiments of this application provide a multimodal action recognition and fusion evaluation method based on HGAT, the method comprising: Acquire multimodal data of the target object; wherein, the target object is the person who needs to be assessed among multiple rehabilitation trainees; the multimodal data includes visual images and sensor data; Based on the visual image and the sensor data, node extraction and cross-modal association are performed to determine the heterogeneous graph; Based on the heterogeneous graph, attention coefficients are calculated; and weight allocation and feature fusion are performed based on the attention coefficients to obtain fused features. Based on the fusion features, observation noise covariance prediction and filtering are performed using a recurrent neural network to obtain high-precision skeletal data; and based on the high-precision skeletal data, the true joint range of motion, motion smoothness, and stability limit are calculated. Based on the true joint range of motion, the smoothness of motion, and the stability limit, a fusion evaluation is performed to obtain the evaluation result.

[0006] In the above scheme, the step of performing node extraction and cross-modal association based on the visual image and the sensor data to determine the heterogeneous graph includes: Node extraction is performed on the visual image and the sensor data to obtain visual nodes and sensing nodes; wherein, the visual node includes multiple sub-visual nodes; and the sensing node includes multiple sub-sensing nodes. Based on the multiple sub-visual nodes, connect the sub-visual nodes that conform to human anatomy to obtain the skeletal physical edges; Connect the sub-visual nodes and sub-sensing nodes corresponding to the same anatomical position among the multiple sub-visual nodes to obtain cross-modal virtual edges; The heterogeneous graph is determined based on the visual node, the sensing node, the skeletal physical edge, and the cross-modal virtual edge.

[0007] In the above scheme, calculating the attention coefficient based on the heterogeneous graph includes: Based on the multiple sub-visual nodes and multiple sub-sensing nodes in the heterogeneous graph, determine the initial visual feature vector corresponding to each of the multiple sub-visual nodes and the initial sensing feature vector corresponding to each of the multiple sub-sensing nodes. Based on the initial visual feature vector and the initial sensing feature vector, mapping processing is performed to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes and the sensing feature vectors corresponding to each of the multiple sub-sensing nodes. Based on the visual feature vector and the sensing feature vector, a splicing attention mechanism is used to calculate the sub-attention coefficients between each of the multiple sub-visual nodes and the multiple sub-sensing nodes; and based on the sub-attention coefficients, the attention coefficients are determined.

[0008] In the above scheme, the step of performing mapping processing based on the initial visual feature vector and the initial sensing feature vector to obtain the visual feature vector corresponding to each of the plurality of sub-visual nodes and the sensing feature vector corresponding to each of the plurality of sub-sensing nodes includes: Obtain the first weight matrix and the first bias vector corresponding to the initial visual feature vector, and the second weight matrix and the second bias vector corresponding to the initial sensing feature vector; Based on the first weight matrix and the first bias vector, the initial visual feature vector is mapped to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes. Based on the second weight matrix and the second bias vector, the initial sensing feature vector is mapped to obtain the sensing feature vectors corresponding to each of the multiple sub-sensing nodes.

[0009] In the above scheme, the step of weight allocation and feature fusion based on the attention coefficient to obtain fused features includes: Based on the multiple sub-attention coefficients corresponding to each sub-visual node in the attention coefficients, the fusion weight of each sub-visual node is determined; Determine the neighborhood features of the neighboring nodes of each sub-visual node; Based on the fusion weight of each sub-visual node, the neighborhood features are weighted and fused to obtain the initial sub-fusion features; The initial sub-fusion features are processed by an activation function to obtain the sub-fusion features of each sub-visual node; The fusion feature is determined based on the sub-fusion features of each sub-visual node.

[0010] In the above scheme, the step of obtaining high-precision skeletal data by predicting and filtering observation noise covariance through a recurrent neural network based on the fused features includes: Based on the fusion features, calculate the variance of the fusion features; Based on the fusion features, the variance, the pre-acquired historical fusion features, and the angular velocity, the observation noise covariance is predicted to obtain the observation noise matrix; wherein, the historical fusion features are the fusion features of the previous time step. Determine the prior estimate of the covariance and the state at the next time step; The Kalman gain is determined based on the prior estimated covariance, the observation noise matrix, and the pre-acquired observation matrix. Based on the Kalman gain, the next time-step state, and the observation matrix, filtering is performed to obtain the optimal state; and based on the optimal state, the high-precision skeletal data is determined.

[0011] In the above scheme, determining the prior estimated covariance and the state at the next time step includes: Determine the state transition matrix, the optimal state at time t-1, the posterior estimated covariance, the control input matrix, the control input vector, and the noise covariance matrix; Based on the state transition matrix, the optimal state at time t-1, the control input matrix, and the control input vector, calculations are performed to determine the state at the next time step. Based on the state transition matrix, the posterior estimated covariance, and the noise covariance matrix, calculations are performed to determine the prior estimated covariance.

[0012] Secondly, embodiments of this application provide a multimodal action recognition and fusion evaluation system based on HGAT. The HGAT-based multimodal action recognition and fusion evaluation system includes: an acquisition module, a determination module, a calculation module, and an evaluation module, wherein... The acquisition module is used to acquire multimodal data of the target object; wherein, the target object is a person who needs to be assessed among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data; The determining module is used to perform node extraction and cross-modal association based on the visual image and the sensor data to determine the heterogeneous graph; The calculation module is used to calculate the attention coefficient based on the heterogeneous graph; and to perform weight allocation and feature fusion based on the attention coefficient to obtain fused features; based on the fused features, to perform observation noise covariance prediction and filtering through a recurrent neural network to obtain high-precision skeletal data; and to calculate the true joint mobility, motion smoothness and stability limit based on the high-precision skeletal data. The evaluation module is used to perform a fusion evaluation based on the true joint range of motion, the motion smoothness, and the stability limit to obtain the evaluation result.

[0013] Thirdly, embodiments of this application provide a multimodal action recognition and fusion evaluation device based on HGAT, comprising: a processor and a memory; wherein, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for causing a processor to perform the method described in the first aspect.

[0015] This application provides a multimodal motion recognition and fusion evaluation method and system based on HGAT. The method includes: acquiring multimodal data of a target object; wherein the target object is a person requiring evaluation among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data; based on the visual images and the sensor data, performing node extraction and cross-modal association to determine a heterogeneous graph; calculating attention coefficients based on the heterogeneous graph; and performing weight allocation and feature fusion based on the attention coefficients to obtain fused features; based on the fused features, performing observation noise covariance prediction and filtering processing through a recurrent neural network to obtain high-precision skeletal data; and calculating ground truth joint range of motion, motion smoothness, and stability limit based on the high-precision skeletal data; and performing fusion evaluation based on the ground truth joint range of motion, motion smoothness, and stability limit to obtain an evaluation result. In the above scheme, a heterogeneous graph is determined by extracting nodes and performing cross-modal association on visual images and sensor data. Attention coefficients are calculated using the heterogeneous graph, and based on these coefficients, weight allocation and feature fusion are performed on visual images and sensor data across different spatiotemporal dimensions to obtain fused features. In environments with visual occlusion or changes in lighting, the weights of sensor data are automatically enhanced to compensate for visual loss. When sensor data drifts, calibration is performed using absolute visual coordinates. Furthermore, based on the fused features, observation noise covariance prediction and filtering are performed using a recurrent neural network to obtain high-precision skeletal data. This high-precision skeletal data enables the output of high-precision, blind-spot-free whole-body posture and biomechanical indicators, supporting accurate assessment of rehabilitation movements and fall prevention prediction, and improving the robustness and accuracy of motion capture. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0018] Figure 1 An optional flowchart illustrating an HGAT-based multimodal action recognition and fusion evaluation method provided in this application embodiment. Figure 1 ; Figure 2An optional flowchart illustrating an HGAT-based multimodal action recognition and fusion evaluation method provided in this application embodiment. Figure 2 ; Figure 3 A schematic diagram of an optional fusion feature generation method for a multimodal action recognition and fusion evaluation method based on HGAT provided in an embodiment of this application; Figure 4 A schematic diagram illustrating an optional optimal state generation method for a multimodal action recognition and fusion evaluation method based on HGAT, provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a multimodal action recognition and fusion evaluation system based on HGAT provided in this application embodiment; Figure 6 This is a schematic diagram of the structure of a multimodal action recognition and fusion evaluation device based on HGAT, provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0020] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0021] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0022] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0023] This application provides a multimodal action recognition and fusion evaluation method based on HGAT. Figure 1An optional flowchart illustrating an HGAT-based multimodal action recognition and fusion evaluation method provided in this application embodiment. Figure 1 , will combine Figure 1 The steps shown are explained.

[0024] S101. Acquire multimodal data of the target object; wherein, the target object is the person who needs to be assessed among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data.

[0025] In some embodiments of this application, the target group is a person requiring assessment among multiple rehabilitation trainees. Multimodal data includes visual images and sensor data. Visual images are acquired through a smart mirror deployed in the rehabilitation setting, and sensor data is acquired through a smart knee brace deployed on the target person.

[0026] In some embodiments of this application, a multimodal action recognition and fusion evaluation method based on HGAT is adapted to action recognition and fusion evaluation scenarios.

[0027] In some embodiments of this application, a multimodal action recognition and fusion evaluation method based on HGAT is adapted to a multimodal action recognition and fusion evaluation system based on HGAT.

[0028] In some embodiments of this application, the binocular visual images (i.e., visual images) of the smart mirror and the physical quantities (joint angles, angular velocities, and displacements) of the smart knee brace are acquired simultaneously, i.e., sensor data.

[0029] It should be noted that the Heterogeneous Graph Attention Network (HGAT) is a deep learning model specifically designed for processing heterogeneous graphs. Its core lies in capturing the complex relationships between different types of nodes and edges through an attention mechanism, thereby improving the performance and interpretability of graph neural networks in real-world complex scenarios.

[0030] S102. Based on visual images and sensor data, perform node extraction and cross-modal association to determine the heterogeneous graph.

[0031] In some embodiments of this application, visual nodes and sensor nodes are extracted from visual images and sensor data respectively to obtain visual nodes and sensor nodes. Each visual node includes multiple sub-visual nodes, and each sensor node includes multiple sub-sensor nodes. Based on the multiple sub-visual nodes, sub-visual nodes conforming to human anatomy are connected to obtain skeletal physical edges. Sub-visual nodes and sub-sensor nodes corresponding to the same anatomical location among the multiple sub-visual nodes and multiple sub-sensor nodes are connected to obtain cross-modal virtual edges. Based on the visual nodes, sensor nodes, skeletal physical edges, and cross-modal virtual edges, a heterogeneous graph is determined.

[0032] S103. Based on the heterogeneous graph, calculate the attention coefficient; and based on the attention coefficient, perform weight allocation and feature fusion to obtain fused features.

[0033] In some embodiments of this application, based on multiple sub-visual nodes and multiple sub-sensing nodes in a heterogeneous graph, initial visual feature vectors corresponding to each of the multiple sub-visual nodes and initial sensing feature vectors corresponding to each of the multiple sub-sensing nodes are determined; based on the initial visual feature vectors and initial sensing feature vectors, mapping processing is performed to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes and the sensing feature vectors corresponding to each of the multiple sub-sensing nodes; based on the visual feature vectors and sensing feature vectors, a concatenation attention mechanism is used to calculate the sub-attention coefficients between each of the multiple sub-visual nodes and the multiple sub-sensing nodes; and based on the sub-attention coefficients, the attention coefficients are determined.

[0034] In some embodiments of this application, the fusion weight of each sub-visual node is determined based on multiple sub-attention coefficients corresponding to each sub-visual node in the attention coefficients; the neighborhood features of the neighboring nodes of each sub-visual node are determined; the neighborhood features are weighted and fused based on the fusion weight of each sub-visual node to obtain initial sub-fusion features; the initial sub-fusion features are processed by an activation function to obtain the sub-fusion features of each sub-visual node; and the fusion features are determined based on the sub-fusion features of each sub-visual node.

[0035] S104. Based on the fusion features, observation noise covariance prediction and filtering are performed through a recurrent neural network to obtain high-precision skeletal data; and based on the high-precision skeletal data, the true joint range of motion, motion smoothness and stability limit are calculated.

[0036] In some embodiments of this application, the variance of the fusion features is calculated based on the fusion features; based on the fusion features, variance, pre-acquired historical fusion features, and angular velocity, the observation noise covariance is predicted to obtain the observation noise matrix; wherein, the historical fusion features are the fusion features of the previous time step; the prior estimated covariance and the state at the next time step are determined; based on the prior estimated covariance, the observation noise matrix, and the pre-acquired observation matrix, the Kalman gain is determined; based on the Kalman gain, the state at the next time step, and the observation matrix, filtering is performed to obtain the optimal state; and based on the optimal state, high-precision skeletal data is determined. Based on the high-precision skeletal data, the true joint range of motion, motion smoothness, and stability limit are calculated.

[0037] S105. Based on the true value of joint range of motion, motion smoothness and stability limit, a fusion evaluation is performed to obtain the evaluation results.

[0038] In some embodiments of this application, true joint range of motion is based on denoised true joint angles, which can calculate the maximum range of motion of joint angles within a specified movement cycle and assess the degree of joint stiffness, adhesion, or limited movement; motion smoothness is based on fused angular velocity curves, which can analyze whether there is tremor or pause in the movement and is used to assess abnormal muscle tone; dynamic stability limit is based on fused absolute coordinates and velocity of the whole body center of mass, which can calculate the dynamic stability limit between the extrapolated center of mass and the support boundary in real time and determine the risk of fall in static or moving states. A fusion assessment is performed based on true joint range of motion, motion smoothness, and stability limit to obtain the assessment result.

[0039] For example, Figure 2 An optional flowchart illustrating an HGAT-based multimodal action recognition and fusion evaluation method provided in this application embodiment. Figure 2 The specific steps of the HGAT-based multimodal action recognition and fusion evaluation method are as follows: A. Multimodal data acquisition.

[0040] The system primarily collects visual and sensor data, requiring simultaneous acquisition and timestamp alignment. It simultaneously acquires binocular visual images from the smart mirror and physical quantities (joint angles, angular velocities, displacement) from the smart knee brace, and aligns these with timestamps.

[0041] B. Heterogeneous graph construction.

[0042] It mainly includes visual nodes and sensor nodes, as well as physical connection edges and cross-modal association edges. A heterogeneous graph containing visual nodes and sensor nodes is constructed, physical connection edges and cross-modal association edges are established, and node alignment is performed.

[0043] C. Calculation of attention coefficient.

[0044] The main approach is to improve data quality through HGAT and output fused features. Specifically, the heterogeneous graph is input into the HGAT network, the attention coefficients between visual nodes and sensor nodes are calculated, weights are dynamically allocated based on the attention coefficients, and fused features are output.

[0045] D. Neural filtering correction.

[0046] The algorithm uses RNN / Kalman filtering to predict R+temporal smoothing and drift elimination. The input is the fused features. The RNN is used to predict the observation noise covariance (R) in the current environment. The Kalman filter is then used to smooth the fused features output by HGAT in the temporal domain and eliminate drift to obtain the optimal state (i.e., the fused high-precision skeleton data).

[0047] E. Calculation of fusion index.

[0048] Based on the fused skeletal data (i.e., the fused high-precision skeletal data), the joint range of motion, motion smoothness, and MoS (i.e., stability limit) are calculated.

[0049] F. Feedback and Control.

[0050] Feedback and fusion results are used to adjust device stiffness and generate guiding voice commands. The calculation results (i.e., joint range of motion, motion smoothness, and stability limits) are used to drive exoskeleton stiffness adjustment or generate rehabilitation guidance voice commands.

[0051] It is understood that the embodiments of this application determine a heterogeneous graph by extracting nodes and performing cross-modal association on visual images and sensor data; calculate attention coefficients through the heterogeneous graph, and based on the attention coefficients, perform weight allocation and feature fusion of visual images and sensor data in different spatiotemporal dimensions to obtain fused features. In environments with visual occlusion or changes in lighting, the weights of sensor data are automatically enhanced to compensate for visual loss; when sensor data drifts, calibration is performed using visual absolute coordinates. Furthermore, based on the fused features, observation noise covariance prediction and filtering are performed through a recurrent neural network to obtain high-precision skeletal data. This allows for the output of high-precision, blind-spot-free whole-body posture and biomechanical indicators, supporting accurate assessment of rehabilitation movements and fall prevention prediction, and improving the robustness and accuracy of motion capture.

[0052] In some embodiments of this application, S102 can be implemented by S201-S204, as follows: S201. Extract nodes from the visual image and sensor data respectively to obtain visual nodes and sensing nodes; wherein, the visual node includes multiple sub-visual nodes; the sensing node includes multiple sub-sensing nodes.

[0053] S202. Based on multiple sub-visual nodes, connect the sub-visual nodes that conform to human anatomy to obtain the physical edges of the skeleton.

[0054] S203. Connect the sub-visual nodes and sub-sensor nodes corresponding to the same anatomical position among multiple sub-visual nodes and multiple sub-sensor nodes to obtain cross-modal virtual edges.

[0055] S204. Determine the heterogeneous graph based on visual nodes, sensor nodes, skeletal physical edges, and cross-modal virtual edges.

[0056] For example, a heterogeneous graph G=(V,E) describing the state of the human body is constructed. The node set contains two types of heterogeneous nodes, representing different data sources. Specifically, there are visual nodes V_vis (N skeletal key points extracted by the smart mirror's vision algorithm, such as the 3D coordinates (x,y,z) of the hip, knee, and ankle) and sensor nodes V_sen (M physical state quantities provided by the high-precision sensor module built into the smart knee brace; the specific feature vectors include joint angle θ, angular velocity ω, and displacement d). The edge set contains skeletal physical edges E_phy (connecting visual nodes that conform to human anatomy, such as hip-knee) and cross-modal virtual edges E_cross (connecting visual nodes and sensor nodes at the same anatomical location, such as the "visual left knee node" and the "knee brace left knee sensor node").

[0057] In some embodiments of this application, the calculation of attention coefficients based on heterogeneous graphs in S103 can be implemented through S301-S303, as follows: S301. Based on multiple sub-visual nodes and multiple sub-sensing nodes in the heterogeneous graph, determine the initial visual feature vector corresponding to each of the multiple sub-visual nodes and the initial sensing feature vector corresponding to each of the multiple sub-sensing nodes.

[0058] S302. Based on the initial visual feature vector and the initial sensing feature vector, perform mapping processing to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes and the sensing feature vectors corresponding to each of the multiple sub-sensing nodes.

[0059] S303. Based on visual feature vectors and sensor feature vectors, a splicing attention mechanism is used to calculate the sub-attention coefficients between each of the multiple sub-visual nodes and the multiple sub-sensor nodes; and the attention coefficients are determined based on the sub-attention coefficients.

[0060] In some embodiments of this application, a first weight matrix and a first bias vector corresponding to the initial visual feature vector, and a second weight matrix and a second bias vector corresponding to the initial sensing feature vector are obtained; based on the first weight matrix and the first bias vector, the initial visual feature vector is mapped to obtain visual feature vectors corresponding to multiple sub-visual nodes; based on the second weight matrix and the second bias vector, the initial sensing feature vector is mapped to obtain sensing feature vectors corresponding to multiple sub-sensing nodes.

[0061] For example, the visual node features in the heterogeneous graph are 3D coordinates (x, y, z), and the sensor node features are... The combined vectors determine the initial visual feature vectors corresponding to each of the multiple sub-visual nodes based on the visual node features; and determine the initial sensing feature vectors corresponding to each of the multiple sub-sensing nodes based on the sensing node features.

[0062] The initial visual feature vector and the initial sensing feature vector are mapped to a hidden space of the same dimension using a projection matrix of a specific modality. : in, It is the visual feature vector mapped from visual child node i; It is the sensing feature vector mapped from sensing child node j, representing the node's representation in the hidden layer space; , It is the first weight matrix; This is the second weight matrix. Both the first and second weight matrices are learnable weight matrices. Used to extract deep features of visual modalities; Used to extract deep features of sensing modes; It is the initial visual feature vector of the visual child node. It is the initial sensing feature vector of the sensing sub-node; It is the first bias vector corresponding to the visual modality type; It is the second bias vector corresponding to the sensing mode type. and It is used to compensate for the systematic differences in physical dimensions, zero-point definitions and statistical distributions of different modal data, thereby improving the alignment and comparability of heterogeneous features in a unified latent space. and , It is obtained through training a neural network.

[0063] For what was obtained , An attention coefficient is calculated using a concatenation-based attention mechanism. : in, This represents a vector concatenation operation; It is a learnable attention vector (during training, this vector learns what combinations of features are "high-quality"). This is used to introduce nonlinearity, enabling the model to handle complex compensatory action relationships.

[0064] In some embodiments of this application, the weight allocation and feature fusion based on the attention coefficient in S103 to obtain the fused features can be achieved through S401-S405, as follows: S401. Based on the multiple sub-attention coefficients corresponding to each sub-visual node in the attention coefficients, determine the fusion weight of each sub-visual node.

[0065] S402. Determine the neighborhood features of the neighboring nodes of each sub-visual node.

[0066] S403. Based on the fusion weight of each sub-visual node, the neighborhood features are weighted and fused to obtain the initial sub-fusion features.

[0067] S404. The initial sub-fusion features are processed by the activation function to obtain the sub-fusion features of each sub-visual node.

[0068] S405. Determine the fusion features based on the sub-fusion features of each sub-visual node.

[0069] For example, Figure 3 This application provides an embodiment of an optional fusion feature generation diagram for a multimodal action recognition and fusion evaluation method based on HGAT. The fusion feature generation specifically includes the following steps: S1, Heterogeneous graph input.

[0070] S2. Modal feature projection and alignment, hidden space mapping , .

[0071] S3, Cross-modal attention calculation of attention coefficients .

[0072] S4, Weight Allocation.

[0073] Using Softmax normalization to achieve fusion weights The calculations specifically include: The coefficients within the neighborhood are normalized using the Softmax function to obtain the final fusion weights. : in, visual child node The set of neighboring nodes, visual child node and its neighboring child nodes Attention coefficient between them.

[0074] at this time, This refers to the system's "confidence" in the data source; when the features of visual child node i are affected by factors such as occlusion... When an anomaly occurs, after projection This "abnormal pattern" is carried by the vectors trained by the network through backpropagation. It will output a very small dot product value for combinations containing this anomalous pattern, thereby reducing the self-attention coefficient of that node. ( = (At that time), according to the characteristics of the Softmax normalization function, a certain coefficient (here, ) A decrease in the sum of its components will directly lead to a corresponding decrease in the total denominator. Under this premise, at this time, the sum of its components corresponding to reliable neighbor child nodes (such as sensor child nodes) will be reduced. The fusion weights remain unchanged. Correspondingly, the "trust" in data source j increases. This process is entirely data-driven, requiring no manual threshold setting, thus automatically and smoothly switching the trust level from "trusting vision" to "trusting sensors" at the feature level, ensuring the system's robustness under occlusion.

[0075] S5. Neighborhood feature weighted aggregation, through activation function. Output fusion features .

[0076] The calculated fusion weights are used to weight and aggregate the neighborhood features, and then an activation function is applied. This yields the final visual child nodes. Fusion features : in, Corresponding to node type The learnable linear transformation matrix (weight matrix) is used to extract deep features of each modality. For nodes The set of neighboring nodes, Neighboring nodes The initial sensing feature vector.

[0077] In some embodiments of this application, the high-precision skeletal data obtained in S104 by predicting and filtering observation noise covariance using a recurrent neural network based on fused features can be achieved through S501-S505, as follows: S501. Based on the fusion features, calculate the variance of the fusion features.

[0078] S502. Based on the fusion features, variance, pre-acquired historical fusion features and angular velocity, predict the observation noise covariance to obtain the observation noise matrix; where the historical fusion features are the fusion features of the previous moment.

[0079] S503. Determine the prior estimate of covariance and the state at the next time step.

[0080] In some embodiments of this application, the state transition matrix, the optimal state at time t-1, the posterior estimated covariance, the control input matrix, the control input vector, and the noise covariance matrix are determined; based on the state transition matrix, the optimal state at time t-1, the control input matrix, and the control input vector, calculations are performed to determine the state at the next time step; based on the state transition matrix, the posterior estimated covariance, and the noise covariance matrix, calculations are performed to determine the prior estimated covariance.

[0081] S504. Determine the Kalman gain based on the prior estimated covariance, the observation noise matrix, and the pre-acquired observation matrix.

[0082] S505: Based on Kalman gain, the next time step state, and the observation matrix, filtering is performed to obtain the optimal state; and based on the optimal state, high-precision skeletal data is determined.

[0083] For example, Figure 4 This application provides an embodiment of an optimal state generation method for multimodal action recognition and fusion evaluation based on HGAT. The optimal state generation specifically includes the following steps: S11. Obtain HGAT data.

[0084] HGAT data includes hidden states. Fusion characteristics and angular velocity Hidden state That is, the fusion characteristics of the previous moment.

[0085] S12. Temporal reasoning through neural perception pathways ( ), constraint activation Output dynamic observation noise matrix .

[0086] S13, Physical prediction path, i.e., prior state and prior covariance .

[0087] It should be noted that the prior state That is, the state at the next moment. Prior covariance, i.e., prior estimate of covariance. S14, Calculate Kalman gain .

[0088] S15. Optimal state estimation, output the optimal estimate x. t .

[0089] Optimal estimate x t That is, the optimal state .

[0090] To eliminate high-frequency noise after fusion and address potential nonlinear drift in the displacement data, a Neural Kalman Filter is introduced at the HGAT output.

[0091] Using an LSTM network, the observation noise matrix is ​​predicted in real time based on the temporal context of the fused features (including the fused feature information of each child node). The details are as follows: in, It is a diagonalization operator that transforms a vector output by an LSTM network into a diagonal matrix; The activation function forces the output to be a positive number, thus ensuring mathematical validity. It is a long short-term memory network used for temporal memory and pattern recognition to determine the current noise state; The fusion characteristics of the previous moment. This refers to the fusion feature (i.e., the overall state representation after integrating the fusion features of all nodes). The variance represents the fused features; the larger the variance, the more uncertain the HGAT itself (e.g., due to severe occlusion), and the more likely the LSTM will output a larger observation noise matrix after receiving this signal. ; Angular velocity is used to characterize the intensity of motion; when the motion is intense, the model tends to predict a larger observation noise matrix. .

[0092] Predicting the state at the next moment using a kinematic model (combined with angular velocity integral) And calculate the prior estimate covariance. : in, This is the state transition matrix; It is the optimal state at time t-1, and it is the starting point for the current prediction; It is the control input matrix, which defines the control input. How does it affect the state? It is the control input vector, which represents the known external control quantity applied to the system at time t-1, and is used to help calculate the change in position or attitude.

[0093] in, Let be the posterior estimate of the covariance, representing the optimal state estimate at time t-1. The remaining uncertainty; The process noise covariance matrix represents the uncertainty of the model itself.

[0094] Introducing the observation noise matrix predicted by a neural network Calculate Kalman gain : in, The observation matrix; To estimate the covariance a priori; This is the observation noise matrix.

[0095] The final output is the optimal state: in, The observation matrix; These are the observations output by the HGAT network. By introducing... When HGAT is uncertain about its own output (attention weights are dispersed), Get bigger As the value decreases, the filter will place more trust in the prediction value from the previous moment, thus achieving smoothness and disturbance rejection.

[0096] The embodiments of this application have the following beneficial effects: This application acquires human visual skeletal data through a smart mirror and obtains high-frequency joint angle, angular velocity, and displacement data through peripheral devices such as smart knee braces, thereby constructing a heterogeneous graph containing visual and sensor nodes. Utilizing a heterogeneous graph attention mechanism, the confidence weights of visual and sensor features are dynamically calculated across different spatiotemporal dimensions: in environments with visual occlusion or changes in lighting, the weights of sensor nodes are automatically enhanced to compensate for visual loss; and when sensor data drifts, absolute visual coordinates are used for calibration. Furthermore, a deep learning-based neural Kalman filter is introduced to predict the observation noise matrix in real time using features such as angular velocity to optimize state estimation. The final output is a high-precision, blind-spot-free whole-body posture and biomechanical index, supporting accurate assessment of rehabilitation movements and fall prevention prediction, significantly improving the robustness and clinical usability of posture estimation in complex rehabilitation scenarios.

[0097] Based on the above embodiments, this application also provides a multimodal action recognition and fusion evaluation system based on HGAT, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of a multimodal action recognition and fusion evaluation system based on HGAT provided in an embodiment of this application. The HGAT-based multimodal action recognition and fusion evaluation system 5 includes: an acquisition module 501, a determination module 502, a calculation module 503, and an evaluation module 504. The acquisition module 501 is used to acquire multimodal data of the target object; wherein, the target object is a person who needs to be assessed among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data; The determining module 502 is used to perform node extraction and cross-modal association based on the visual image and the sensor data to determine the heterogeneous graph; The calculation module 503 is used to calculate the attention coefficient based on the heterogeneous graph; and to perform weight allocation and feature fusion based on the attention coefficient to obtain fused features; and to perform observation noise covariance prediction and filtering processing through a recurrent neural network based on the fused features to obtain high-precision skeletal data; and to calculate the true joint mobility, motion smoothness and stability limit based on the high-precision skeletal data. The evaluation module 504 is used to perform a fusion evaluation based on the true joint range of motion, the motion smoothness, and the stability limit to obtain an evaluation result.

[0098] Based on the above embodiments, this application also provides a multimodal action recognition and fusion evaluation device based on HGAT, such as... Figure 6 As shown, Figure 6This is a schematic diagram of the structure of a multimodal action recognition and fusion evaluation device based on HGAT, provided in an embodiment of this application. The HGAT-based multimodal action recognition and fusion evaluation device 6 includes a processor 601 and a memory 602. The memory 602 is used to store computer programs; the processor 601 is used to call and run the computer programs from the memory to execute the HGAT-based multimodal action recognition and fusion evaluation method as described in the above embodiment.

[0099] In the embodiments of this application, the processor 601 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0100] This application provides a computer-readable storage medium storing a computer program for implementing, when executed by a processor, a multimodal action recognition and fusion evaluation method based on HGAT as described in any of the above embodiments.

[0101] For example, the program instructions corresponding to the HGAT-based multimodal action recognition and fusion evaluation method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to the HGAT-based multimodal action recognition and fusion evaluation method in the storage media are read or executed by an electronic device, the HGAT-based multimodal action recognition and fusion evaluation method described in any of the above embodiments can be implemented.

[0102] Furthermore, in the embodiments of this application, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0103] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the embodiments in this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, these will not be repeated here.

[0105] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0107] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0108] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0109] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0110] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0111] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A multimodal action recognition and fusion evaluation method based on HGAT, characterized in that, The method includes: Acquire multimodal data of the target object; wherein, the target object is the person who needs to be assessed among multiple rehabilitation trainees; the multimodal data includes visual images and sensor data; Based on the visual image and the sensor data, node extraction and cross-modal association are performed to determine the heterogeneous graph; Based on the heterogeneous graph, attention coefficients are calculated; and weight allocation and feature fusion are performed based on the attention coefficients to obtain fused features. Based on the fusion features, observation noise covariance prediction and filtering are performed using a recurrent neural network to obtain high-precision skeletal data; and based on the high-precision skeletal data, the true joint range of motion, motion smoothness, and stability limit are calculated. Based on the true joint range of motion, the smoothness of motion, and the stability limit, a fusion evaluation is performed to obtain the evaluation result.

2. The method according to claim 1, characterized in that, The step of extracting nodes and performing cross-modal association based on the visual image and the sensor data to determine the heterogeneous graph includes: Node extraction is performed on the visual image and the sensor data to obtain visual nodes and sensing nodes; wherein, the visual node includes multiple sub-visual nodes; and the sensing node includes multiple sub-sensing nodes. Based on the multiple sub-visual nodes, connect the sub-visual nodes that conform to human anatomy to obtain the skeletal physical edges; Connect the sub-visual nodes and sub-sensing nodes corresponding to the same anatomical position among the multiple sub-visual nodes to obtain cross-modal virtual edges; The heterogeneous graph is determined based on the visual node, the sensing node, the skeletal physical edge, and the cross-modal virtual edge.

3. The method according to claim 1, characterized in that, The calculation of the attention coefficient based on the heterogeneous graph includes: Based on the multiple sub-visual nodes and multiple sub-sensing nodes in the heterogeneous graph, determine the initial visual feature vector corresponding to each of the multiple sub-visual nodes and the initial sensing feature vector corresponding to each of the multiple sub-sensing nodes. Based on the initial visual feature vector and the initial sensing feature vector, mapping processing is performed to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes and the sensing feature vectors corresponding to each of the multiple sub-sensing nodes. Based on the visual feature vector and the sensing feature vector, a splicing attention mechanism is used to calculate the sub-attention coefficients between each of the multiple sub-visual nodes and the multiple sub-sensing nodes; and based on the sub-attention coefficients, the attention coefficients are determined.

4. The method according to claim 3, characterized in that, The process of mapping based on the initial visual feature vector and the initial sensing feature vector to obtain the visual feature vector corresponding to each of the plurality of sub-visual nodes and the sensing feature vector corresponding to each of the plurality of sub-sensing nodes includes: Obtain the first weight matrix and the first bias vector corresponding to the initial visual feature vector, and the second weight matrix and the second bias vector corresponding to the initial sensing feature vector; Based on the first weight matrix and the first bias vector, the initial visual feature vector is mapped to obtain the visual feature vectors corresponding to each of the multiple sub-visual nodes. Based on the second weight matrix and the second bias vector, the initial sensing feature vector is mapped to obtain the sensing feature vectors corresponding to each of the multiple sub-sensing nodes.

5. The method according to claim 1, characterized in that, The weight allocation and feature fusion based on the attention coefficients to obtain fused features include: Based on the multiple sub-attention coefficients corresponding to each sub-visual node in the attention coefficients, the fusion weight of each sub-visual node is determined; Determine the neighborhood features of the neighboring nodes of each sub-visual node; Based on the fusion weight of each sub-visual node, the neighborhood features are weighted and fused to obtain the initial sub-fusion features; The initial sub-fusion features are processed by an activation function to obtain the sub-fusion features of each sub-visual node; The fusion feature is determined based on the sub-fusion features of each sub-visual node.

6. The method according to claim 1, characterized in that, Based on the fused features, observation noise covariance prediction and filtering are performed using a recurrent neural network to obtain high-precision skeletal data, including: Based on the fusion features, calculate the variance of the fusion features; Based on the fusion features, the variance, the pre-acquired historical fusion features, and the angular velocity, the observation noise covariance is predicted to obtain the observation noise matrix; wherein, the historical fusion features are the fusion features of the previous time step. Determine the prior estimate of the covariance and the state at the next time step; The Kalman gain is determined based on the prior estimated covariance, the observation noise matrix, and the pre-acquired observation matrix. Based on the Kalman gain, the next time-step state, and the observation matrix, filtering is performed to obtain the optimal state; and based on the optimal state, the high-precision skeletal data is determined.

7. The method according to claim 6, characterized in that, The determination of the prior estimated covariance and the next time-step state includes: Determine the state transition matrix, the optimal state at time t-1, the posterior estimated covariance, the control input matrix, the control input vector, and the noise covariance matrix; Based on the state transition matrix, the optimal state at time t-1, the control input matrix, and the control input vector, calculations are performed to determine the state at the next time step. Based on the state transition matrix, the posterior estimated covariance, and the noise covariance matrix, calculations are performed to determine the prior estimated covariance.

8. A multimodal action recognition and fusion evaluation system based on HGAT, characterized in that, The HGAT-based multimodal action recognition and fusion evaluation system includes: an acquisition module, a determination module, a calculation module, and an evaluation module, wherein... The acquisition module is used to acquire multimodal data of the target object; wherein, the target object is a person who needs to be assessed among multiple rehabilitation training personnel; the multimodal data includes visual images and sensor data; The determining module is used to perform node extraction and cross-modal association based on the visual image and the sensor data to determine the heterogeneous graph; The calculation module is used to calculate the attention coefficient based on the heterogeneous graph; and to perform weight allocation and feature fusion based on the attention coefficient to obtain fused features; based on the fused features, to perform observation noise covariance prediction and filtering through a recurrent neural network to obtain high-precision skeletal data; and to calculate the true joint mobility, motion smoothness and stability limit based on the high-precision skeletal data. The evaluation module is used to perform a fusion evaluation based on the true joint range of motion, the motion smoothness, and the stability limit to obtain the evaluation result.

9. A multimodal action recognition and fusion evaluation device based on HGAT, characterized in that, include: Processor and memory, of which, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the method according to any one of claims 1 to 7.

Images