A real-time adaptive human musculoskeletal modeling and muscle force simulation system

Abstract

The application discloses a real-time adaptive human musculoskeletal modeling and muscle force simulation system and belongs to the field of computer vision.The system comprises a multi-sensor data acquisition module, a space-time calibration module, a three-dimensional force prediction module, a real-time modeling and parameter adjustment module, a muscle force simulation module and a data visualization module, collects position and pressure distribution data of human joints, realizes space-time calibration fusion, establishes a joint graph structure and a body segment graph structure, obtains space-time features through feature extraction and fusion, predicts three-dimensional foot hip bottom pressure, constructs a human musculoskeletal model containing a muscle geometric path, optimizes model parameters in real time, establishes a joint objective function to calculate muscle activation, and completes muscle force simulation.The system provided by the application can directly establish the connection between muscles and bones, improves individual adaptability, improves the physiological rationality of muscle activation distribution, and realizes high-precision real-time muscle force simulation analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biomedical engineering, and particularly relates to a real-time adaptive human musculoskeletal modeling and muscle force simulation system. BACKGROUND

[0002] In recent years, with the in-depth development of human biomechanics research, musculoskeletal modeling and muscle force analysis technology has been widely used in the fields of clinical rehabilitation, sports science, and human engineering. Measuring and analyzing muscle force data is of great value and necessity for users in the fields of rehabilitation medicine, scientific sports, and human-computer interaction.

[0003] Traditional musculoskeletal modeling methods mainly rely on expensive optical motion capture systems (such as Vicon) combined with force platforms for kinematic and dynamic data acquisition, and muscle force simulation analysis through software platforms such as OpenSim. This method mainly relies on multi-body dynamics simulation and optimization algorithms, and its process usually includes collecting human kinematic data based on optical motion capture systems in a laboratory environment, combining with the dynamic data collected synchronously during the movement, establishing a model of the human musculoskeletal system through multi-body dynamics equations, and solving the muscle force of each muscle block using static or dynamic optimization algorithms. This method uses optimization solving to solve the problem of muscle redundancy in human physiology, and has good rationality and accuracy based on physiological anatomy, but the data acquisition relies on high-precision laboratory equipment, which is not only cumbersome to operate and limited by the site, but also has high usage cost.

[0004] Data-driven methods represented by deep learning can directly establish a mapping from kinematic data to muscle force, thereby bypassing the complex simulation process. However, such methods are mostly for scientific research purposes, and their accuracy is heavily dependent on the amount of data, which requires the collection of human kinematic and dynamic data for each case, consuming a large amount of manpower and resources, and limiting the data size in model training.

[0005] In muscle force simulation calculations, traditional static optimization methods usually use an objective function that minimizes the sum of squares of muscle activation, which often leads to a few muscles bearing the main load, while other muscle activations are close to zero. This phenomenon does not match the actual muscle coordination pattern of the human body. In addition, the tendon parameters in existing models are usually fixed and cannot be adaptively adjusted according to individual differences and actual measurement data, which affects the accuracy and physiological rationality of muscle force calculation and hinders the in-depth application of this technology in personalized rehabilitation assessment and sports guidance.

[0006] The technical solution based on the depth camera combined with the pressure sensor array has the advantages of low cost, small space occupation and easy deployment, and provides new possibilities for musculoskeletal modeling. The depth camera directly obtains the three-dimensional coordinate information of the human joint point through infrared light or structured light technology, and the pressure sensor array can obtain the contact force information, and the combination of the two has the potential to realize real-time muscle force simulation. However, the existing research on musculoskeletal modeling based on depth cameras is extremely limited, mainly due to the lack of a musculoskeletal model specially constructed for depth camera data, and the technical difficulties of multi-sensor data fusion. SUMMARY

[0007] To solve the above technical problems, the present application provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, which directly establishes a musculoskeletal model based on depth camera data, and further combines adaptive parameter adjustment to improve the individual adaptability of the model. By constructing a heterogeneous multi-modal model with a graph structure conversion layer, the three-dimensional force of the contact surface is predicted, and when muscle force simulation is performed, a joint objective function is established to improve the physiological reasonableness of muscle activation distribution, realizing low-cost, high-precision real-time muscle force simulation analysis.

[0008] To achieve the above-mentioned purposes, the embodiment provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, which comprises:

[0009] A multi-sensor data acquisition module, which includes a depth camera, a plantar pressure sensor array, a hip pressure sensor array and a data acquisition end, is used to synchronously acquire three-dimensional coordinates and pressure distribution data of human joints;

[0010] A space-time calibration module adopts a space calibration algorithm to establish a coordinate system mapping relationship by clicking on the feature points of the pressure sensor array in the depth camera field of view, realizing the space calibration fusion of multiple coordinate systems;

[0011] A three-dimensional force prediction module extracts pressure distribution features from the pressure distribution data, constructs joint graph structure and body segment graph structure based on the pressure distribution features and the position of the human joint, and extracts spatial features and time sequence features through graph attention network and time convolution network respectively, and performs feature fusion to predict three-dimensional foot and hip pressure;

[0012] A real-time modeling and parameter adjustment module is used to construct a human musculoskeletal model containing muscle geometric path based on the three-dimensional coordinates of human joints, and to output reasonable muscle length and muscle speed by optimizing the parameters of the human musculoskeletal model in real time according to the actual measured muscle length range;

[0013] The muscle force simulation module is used for calculating the human joint moment through the three-dimensional coordinates of the human body joint and the three-dimensional foot-hip bottom pressure, establishing a joint objective function based on the calculated human joint moment and combining reasonable muscle length and speed to calculate muscle activation and obtain corresponding muscle force, and completing muscle force simulation.

[0014] In one embodiment, the real-time adaptive human musculoskeletal modeling and muscle force simulation system further comprises a data visualization module for providing muscle activation, muscle force characteristic curve and real-time animation of the human musculoskeletal model, and indicating the activation intensity of the muscle through color depth.

[0015] In one embodiment, in the multi-sensor data acquisition module, the human joint points collected by the depth camera include head, neck, pelvis, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, right hand, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot and right foot.

[0016] In one embodiment, in the multi-sensor data acquisition module, the foot bottom pressure sensor array and the hip bottom pressure sensor array are both matrix arrays composed of multiple rows and multiple columns of flexible pressure sensor units, which are used for collecting vertical pressure distribution.

[0017] In one embodiment, in the space-time calibration module, the mapping relationship of the coordinate system is established by clicking the feature points of the pressure sensor array in the field of view of the depth camera, which includes: taking the sensor units of the foot bottom sensor array and the hip bottom sensor array as feature points, obtaining the coordinate system position of the feature points in the field of view of the depth camera, calculating the rotation matrix of the pressure array coordinate system to the depth camera coordinate system, and establishing the mapping relationship.

[0018] In one embodiment, in the three-dimensional force prediction module, the joint graph structure is constructed based on the pressure distribution characteristics and the human joint position, and the body segment graph structure is obtained through the joint graph structure conversion, which includes:

[0019] The pressure distribution characteristics containing the foot bottom pressure distribution and the hip bottom pressure distribution extracted through the convolution layer and the self-attention mechanism are modeled into a joint graph structure with the human joint position , wherein is a set of human joint nodes, containing human joint position information and pressure distribution characteristics, is a set of human joint connection edges;

[0020] The joint graph structure is converted into a body segment graph structure through an edge-node conversion layer , wherein is a set of body segment nodes, is a set of body segment connection edges.

[0021] In one embodiment, when converting the graph structure through the edge-node conversion layer, the following conversion rules are followed:

[0022] Each edge in the joint graph structure becomes a node in the volume segment graph structure, and the node features are a weighted combination of the features of the two end nodes of the original edge:

[0023] ,

[0024] wherein, is the feature of the i-th node in the volume segment graph structure, is the weight parameter, is the feature of the i-th node in the joint graph structure, is the feature of the i-th node in the joint graph structure, is the feature of the i-th node in the joint graph structure; In the joint graph structure, if two edges share a joint node, then there is an undirected edge connection between the corresponding two nodes in the volume segment graph structure.

[0025] The present application realizes a learnable feature fusion operation through the edge-node conversion layer for graph structure conversion, which effectively converts the structural information (defined by two joints) implied in an edge in the joint graph into feature information of a node in the volume segment graph, thereby providing a high-quality data basis for subsequent volume segment-based mechanical analysis.

[0026] In one embodiment, in the three-dimensional force prediction module, spatial features and temporal features are extracted based on the joint graph structure and the volume segment graph structure through the graph attention network and the temporal convolution network respectively, and feature fusion is performed to predict the three-dimensional foot and hip pressure, including:

[0027] The joint graph structure and the volume segment graph structure are sequentially subjected to spatial feature extraction and temporal feature extraction through the graph attention network and the temporal convolution network respectively, to capture the importance relationship between nodes and the dynamic changes in the joint graph structure and the volume segment graph structure; wherein the attention coefficient of the graph attention network is:

[0028]

[0029] ,

[0030] wherein, is the weight matrix, is the attention vector, is the node feature, and || represents the splicing operation;

[0031] The spatio-temporal features extracted from the joint graph structure and the volume segment graph structure are input into the feature fusion layer for feature fusion, and then sequentially subjected to the temporal convolution network, the global pooling layer and the fully connected layer, and the output of the three-dimensional force vector containing the double foot and hip pressure is the three-dimensional foot and hip pressure.​​

[0032] The features extracted by processing the joint graph and the body segment graph twice using a time convolution network, the first time using a local time convolution network to ensure capturing the independent motion patterns of each joint and each body segment, and the second time using a convolution network after fusing the features of all joints and body segments to capture the global overall motion trend, greatly enhancing the understanding of dynamic actions, thereby ultimately achieving high-precision prediction of three-dimensional forces.

[0033] In an embodiment, in the real-time modeling and parameter adjustment module, the three-dimensional coordinates of the human joint nodes obtained based on the depth camera are used to construct a human musculoskeletal model containing muscle geometric paths, which includes:

[0034] Based on the human joint nodes collected by the depth camera and combined with the muscle system, the geometric path of each muscle is defined, and each muscle is defined by multiple path points to establish a direct connection between the muscle position and the human joint node; wherein the muscle system includes: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique, external oblique, and the corresponding left and right sides.

[0035] In an embodiment, in the real-time modeling and parameter adjustment module, the real-time optimization of the human musculoskeletal model parameters includes preliminary parameter adjustment and parameter fine-tuning.

[0036] The preliminary parameter adjustment includes: preliminary adjustment of the human musculoskeletal model parameters through a linear scaling preprocessing algorithm, and the calculation formula of linear scaling is:

[0037] ,

[0038] ,

[0039] ,

[0040] wherein, is the measured muscle length of the first muscle, is the predicted muscle length of the first muscle;

[0041] The parameter fine-tuning includes: establishing a multi-objective optimal function containing a maximum length normalized deviation term, a minimum length normalized deviation term, a uniform distribution term and a regularization term to fine-tune the muscle parameters, which is used to make the normalized muscle length and speed within a reasonable physiological range and maintain the biological reasonableness of muscle mechanics characteristics, and the calculation is as follows:

[0042] ,

[0043] in, For the first The optimal fiber length for a muscle mass. For the first The length of the relaxed tendon of a muscle. For the first The optimal initial fiber length for a muscle mass in the original human musculoskeletal model. For the first Initial values ​​of tendon relaxation length for a muscle group in a primitive human musculoskeletal model. To optimize the total number of muscles, , , and These are the weight coefficients for each item. Let Variance be the variance.

[0044] In one embodiment, the joint objective function in the muscle strength simulation module includes: an overall activation term, a maximum activation penalty term, an activation range term, and a time smoothing term; the overall activation term is used to minimize the square of the activation of all muscles, reducing overall muscle energy consumption; the maximum activation penalty term is used to suppress excessively high activation of a single muscle; the activation range term is used to promote a uniform distribution of muscle activation; the time smoothing term is used to suppress drastic fluctuations in muscle activation over time, improving physiological plausibility, and is calculated as follows:

[0045] ,

[0046] in, For the first The activation level of a mass muscle. This represents the maximum activation level for all muscles. This represents the difference between the maximum and minimum muscle activation levels. Adjacent time points Changes in the activation level of the same muscle , , These are the weighting coefficients for each item.

[0047] In one embodiment, when establishing the joint objective function to calculate muscle activation, corresponding constraints are set, and the constraints are calculated as follows:

[0048] ,

[0049] in, R For lever arm matrix, For the maximum isometric contraction force, , , tension-length, force-velocity and passive tension coefficient, respectively, is the muscle fiber length, is the muscle fiber velocity, is the fiber length at the maximum muscle force, is the maximum fiber velocity, is the joint torque.

[0050] In an embodiment, in the muscle force simulation module, the calculated muscle activation is substituted into the Hill-type muscle model to obtain the corresponding muscle force, and the muscle force calculation formula is:

[0051] ,

[0052] wherein, is the muscle force.

[0053] In an embodiment, after obtaining the muscle force, a signal noise adaptive sensing filtering technology is further used to realize real-time smoothing processing of the muscle force signal.

[0054] Compared with the prior art, the present application has at least the following beneficial effects:

[0055] (1) The traditional method uses general muscle model parameters, which is easy to cause the muscle force characteristic curve to have extreme values, and the optimization result is poor. The space-time calibration module and the real-time modeling and parameter adjustment module are introduced, the multi-sensor data fusion problem is solved through the space calibration algorithm, and a human muscle and bone model based on depth camera data is further constructed, so that the muscle position is directly proportional to the joint position of the human body instead of the general model, without manual scaling and calibration operation of the traditional method, real-time processing can be realized, and individual adaptability is significantly improved.

[0056] (2) The heterogeneous multi-modal three-dimensional force prediction module is introduced, the joint graph is converted into a body segment graph which is more in line with the mechanical transmission law through graph structure conversion, the space-time feature modeling is performed in combination with the graph attention network and the time convolution network, and finally the high-precision three-dimensional force prediction is realized, thereby providing a high-quality data basis for subsequent body segment-based mechanical analysis.

[0057] (3) The joint objective function is further established to calculate the muscle activation and obtain the corresponding muscle force, so that the muscle activation distribution is more uniform and more consistent with the actual situation, and the physiological rationality of the muscle activation distribution is improved.

[0058] (4) The real-time adaptive human muscle and bone modeling and muscle force simulation system provided by the present application not only improves the performance, but also enhances the structural rationality and explainability of the model, and is suitable for human motion analysis, rehabilitation engineering, robot control and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced.

[0060] Figure 1 is a structural schematic diagram of a real-time adaptive human musculoskeletal modeling and muscle force simulation system provided by the present application.

[0061] Figure 2 is a scene of a multi-sensor data acquisition module.

[0062] Figure 3 is a structural schematic diagram of a space-time calibration module.

[0063] Figure 4 is a structural schematic diagram of a three-dimensional force prediction module.

[0064] Figure 5 is a human musculoskeletal model schematic diagram.

[0065] Figure 6 is a flowchart of human musculoskeletal model parameter adjustment.

[0066] Figure 7 is a structural schematic diagram of a muscle force simulation module.

[0067] Figure 8 is a structural schematic diagram of a data visualization module.

[0068] Figure 9 is a muscle force comparison diagram of different standing strategies.

[0069] Figure 10 is a muscle activation pattern comparison diagram of different optimization methods. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0071] In order to realize low-cost, high-precision real-time muscle force simulation analysis, adapt to individual differences, the embodiment provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, as shown in the accompanying Figure 1 As shown in the structural diagram of the overall design scheme of the system, it includes a multi-sensor data acquisition module, a space-time calibration module, a three-dimensional force prediction module, a real-time modeling and parameter adjustment module, a muscle force simulation module and a data visualization module.

[0072] In an embodiment, a multi-sensor data acquisition module, comprising a depth camera, a plantar pressure sensor array, a buttock pressure sensor array, and a data acquisition terminal, is used to synchronously acquire position and pressure distribution data of human body joints.

[0073] Specifically, as shown in Figure 2 , the depth camera uses Microsoft Kinect Azure, is directly opposite to the front of the human body, is within a distance of 1.5-3 meters from the human body, and has a sampling frequency of 30 Hz. The plantar pressure sensor array and the buttock pressure sensor array are both matrix arrays composed of 32×32 flexible pressure sensor units, have an effective sensing area of 400 mm×400 mm, and have a unit resolution of 11.5 mm, and are used to acquire vertical direction pressure distribution. The feet and buttocks of the human body are respectively located above the corresponding pressure sensor array, and test actions such as static standing and standing up are performed, and joint kinematics data and pressure distribution data of the human body are synchronously recorded at the data acquisition terminal.

[0074] In an embodiment, a space-time calibration module is used to realize space-time calibration fusion of multiple coordinate systems by establishing a coordinate system mapping relationship through clicking feature points of the pressure sensor array in the field of view of the depth camera.

[0075] Specifically, as shown in Figure 3 , signals from multiple monitoring devices are time-aligned, and a unified time range is selected for interception, so as to realize time-domain calibration. Space calibration is realized by clicking eight points, which are four corner sensor units of the plantar sensor array and the buttock sensor array in the field of view of the depth camera software, as feature points, obtaining the positions of these feature points in the depth camera coordinate system, calculating a rotation matrix of the pressure array coordinate system to the depth camera coordinate system based on these feature points, and establishing a mapping relationship. Finally, the data is unified in the depth camera coordinate system to realize space calibration.

[0076] In an embodiment, a three-dimensional force prediction module is used to extract pressure distribution features from the pressure distribution data, construct joint graph structures and body segment graph structures based on the pressure distribution features and the positions of the human body joints, and extract spatial features and time sequence features through graph attention networks and time convolution networks respectively, and perform feature fusion to predict three-dimensional foot and buttock pressure.

[0077] Specifically, as shown in Figure 4 , the three-dimensional force prediction module builds a novel heterogeneous multi-modal fusion architecture. First, a convolution network combined with a self-attention mechanism is used to extract pressure distribution features from each frame of 64×32 pixel pressure distribution data.

[0078] Subsequently, the extracted pressure distribution features and 19 joint position features are constructed into a joint graph structure , wherein A joint set of human body, containing position information and pressure distribution characteristics of the joint set of human body, A joint connection edge set of human body;

[0079] Through the edge-node conversion layer, the joint graph structure is converted into a body segment graph structure , wherein is a body segment node set, is a body segment connection edge set.

[0080] The edge-node conversion layer converts the graph structure into a new graph structure , wherein the edge in the graph corresponds to a node in the new graph ,

[0081] 1. Node characteristics: , wherein and are the characteristics of the two end points of the corresponding edge in , and is a learnable weight.

[0082] 2. Connection relationship of nodes: for two nodes and in , if their corresponding two edges and share an end point, there is an undirected edge connection between them. By using the above edge-node conversion layer for edge-node conversion, the following purposes can be achieved:

[0083] (1) Conversion of perspective from joint center to body segment center: the joint graph structure takes the joint of human body as the node, and the edge represents the bone; while the body segment graph structure takes the bone (body segment) as the node, and the edge represents the joint. This conversion is more consistent with the force transmission path in biomechanics, and the force is transmitted through the bone (body segment), not the joint.

[0084] (2) Enhanced feature representation capability: by learning the weight

[0085] , the characteristics of the two joints are fused, so that each body segment node can adaptively absorb the information of the two joints, which is more conducive to force modeling. (3) Structural induction bias: the body segment graph more directly reflects the human biomechanical structure, which is conducive to learning the generation and transmission mechanism of human muscle force.

[0086] ​​

[0087] Then, the joint graph structure and the body segment graph structure are respectively subjected to graph attention coding and time sequence feature extraction through a graph attention network and a time convolution network, so as to capture the importance relationship between nodes in the joint graph structure and the body segment graph structure and the dynamic change of the action; wherein the attention coefficient of the graph attention network is:

[0088]

[0089] wherein, is a weight matrix, is an attention vector, is a node feature, and || represents a splicing operation;

[0090] The spatial feature obtained through the graph attention coding and the time sequence feature extracted are input into a feature fusion layer for feature fusion, and then sequentially subjected to a time convolution network, a global pooling layer and a full connection layer, so as to output a three-dimensional force vector containing nine components of the double foot bottom and the hip bottom. The network training data set contains 464 standing up action samples, and the prediction accuracy is above 95% in the vertical component and above 85% in the horizontal component.

[0091] Through the heterogeneous multi-modal fusion and the graph structure conversion, the joint graph is converted into a body segment graph which is more in line with the mechanical transmission law, the space-time feature modeling is performed through the graph attention network and the time convolution network, and finally the high-precision three-dimensional foot and hip bottom pressure prediction is realized.

[0092] In the embodiment, the real-time modeling and parameter adjustment module is configured to construct a human musculoskeletal model containing muscle geometric paths based on the positions of the human body joint nodes, and to output reasonable muscle length and speed by optimizing the parameters of the human musculoskeletal model in real time according to the actually measured muscle length range.

[0093] Specifically, as shown in Figure 5 the constructed human musculoskeletal model containing muscle geometric paths, the 19 joint node positions obtained based on the depth camera include head, neck, pelvis, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, right hand, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot.

[0094] The model contains 18 main muscles: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique, external oblique, and respectively corresponding to the left and right sides.

[0095] Each muscle is defined by 2-3 path points to define the geometric path, and the muscle position directly corresponds to the joint node position instead of the general model, without the need for scaling operation in OpenSim, and real-time processing can be achieved.

[0096] ​Further, as shown in Figure 6 , a double-layer optimization strategy is constructed to cope with individual differences in muscle parameters, which is used to real-time optimize human musculoskeletal model parameters including preliminary parameter adjustment and parameter fine-tuning.

[0097] Firstly, the human musculoskeletal model parameters are preliminarily adjusted by a linear scaling preprocessing algorithm, and the calculation formula of linear scaling is:

[0098] ,

[0099] ,

[0100] ,

[0101] wherein Lmeas is the measured muscle length of the i-th muscle, Lpred is the predicted muscle length of the i-th muscle. Subsequently, the muscle parameters are fine-tuned by using a multi-objective optimization algorithm, and the objective function of the multi-objective optimization algorithm is established as:

[0102] ,

[0103] ,

[0104] wherein Lopt is the optimal fiber length of the i-th muscle, Lrelax is the tendon slack length of the i-th muscle, Lopt0 is the initial value of the optimal fiber length of the i-th muscle in the original human musculoskeletal model, Lrelax0 is the initial value of the tendon slack length of the i-th muscle in the original human musculoskeletal model, N is the total number of optimized muscles, , , and are the corresponding weight coefficients of each term, σ is the variance. The composition of the objective function: the first term is the maximum length normalized deviation term, which makes the measured maximum muscle length normalized value and the model maximum muscle length normalized value as close as possible; the second term is the minimum length normalized deviation term, which makes the measured minimum muscle length normalized value and the model minimum muscle length normalized value as close as possible; the third term is the uniform distribution term, which makes the optimal fiber length distribution of all muscles more uniform (smaller variance); the fourth term is the regularization term, which prevents the optimization result from deviating too much from the original model parameters and ensures physiological reasonableness.

[0105] ​​​​​​​

[0106] and set constraints , , by restricting the length parameter of the muscle must be positive, under the premise of meeting this condition to find the optimal solution. By real-time optimization of human musculoskeletal model parameters, output reasonable muscle length and speed.

[0107] By the adjustment, the normalized muscle length and speed are within the reasonable physiological range, and the biological rationality of muscle mechanical characteristics is maintained.

[0108] In the embodiment, the muscle force simulation module is configured to calculate human joint moments through the positions of human joint nodes and three-dimensional foot hip pressure, establish a joint objective function based on the calculated human joint moments and in combination with reasonable muscle length and speed, calculate muscle activation and obtain corresponding muscle force, and complete muscle force simulation.

[0109] Specifically, as shown in Figure 7 , in order to solve the problem of unreasonable muscle activation distribution, an optimization framework is constructed to improve the muscle force simulation effect.

[0110] Firstly, based on the human musculoskeletal model, the human joint node data collected by the depth camera and the three-dimensional foot pressure and three-dimensional hip pressure data predicted by the three-dimensional force prediction module are input, the human joint moments are calculated according to the Newton-Euler equation, and the joints are associated with the corresponding muscles controlling the flexion and extension functions of the joints.

[0111] Based on the calculated human joint moments and in combination with reasonable muscle length and speed, a joint objective function is established to calculate muscle activation, and the objective function is:

[0112]

[0113] Among them, is the activation of the i-th muscle, is the maximum value of all muscle activations, is the difference between the maximum and minimum values of muscle activation, is the change of the same muscle activation at adjacent time. The first term (overall activation term) of the joint objective function is used to minimize the square of all muscle activations, and to reduce the overall muscle energy consumption; the second term (maximum activation penalty term) is used to suppress the over-high activation of single muscle; the third term (activation range term) is used to encourage uniform distribution of activation; and the fourth term (time smoothing term) is used to suppress the sharp fluctuation of activation over time, and to improve the physiological rationality. 、

[0114] 、 , , For each weight coefficient, 1, 5 and 100 are taken in this embodiment.

[0115] The corresponding constraint condition is the joint torque balance equation when the joint torque function is established to calculate the muscle activation, and the calculation is as follows:

[0116]

[0117] Wherein, R is the force arm matrix, is the maximum isometric contraction force, , , tension-length, force-velocity and passive tension coefficients respectively, is the muscle fiber length, is the muscle fiber velocity, is the fiber length when the muscle force is maximum, is the maximum fiber velocity, is the joint torque.

[0118] Then, the optimized muscle activation is substituted into the Hill-type muscle model to obtain the muscle force The muscle force calculation formula is as follows:

[0119] .

[0120] In the muscle force simulation module of this embodiment, the Hill-type muscle model includes a plurality of selectable mechanical property curve models, such as tension-length relationship curve, force-velocity relationship curve and passive tension curve, and the user can select a specific curve model for each curve according to different application requirements.

[0121] Further, an adaptive filtering technique is used to filter the simulated muscle force signal, and the filtering parameters are dynamically adjusted according to the noise level and frequency characteristics of the signal. The filter is a 4th order Butterworth low-pass filter, and the basic cutoff frequency is 8 Hz, which is adaptively adjusted in the range of 4-12 Hz according to the signal characteristics. The noise threshold is set to 0.15, and when the signal noise level exceeds the threshold, the cutoff frequency is automatically reduced to enhance the filtering effect.

[0122] The data visualization module is used to provide real-time animation of muscle activation, muscle force characteristic curve and human muscle and skeletal model, and the muscle activation intensity is represented by color depth. Specifically, as shown in Figure 8 In order to better show the real-time adaptive human muscle and skeletal modeling and muscle force, the real-time animation of muscle activation, muscle force characteristic curve and human muscle and skeletal model is visualized.

[0123] To verify the effectiveness of the real-time adaptive human musculoskeletal modeling and muscle force simulation system proposed in this invention, tests were conducted on healthy subjects.

[0124] In the experiment, healthy subjects adopted three different standing strategies: (1) Momentum Transfer (MT) standing strategy, (2) Exaggerated Trunk Flexion (ETF) standing strategy, and (3) Dominant Vertical Rise (DVR) standing strategy. Data were recorded using a high-precision Vicon motion capture system and a three-dimensional force table, and compared with actual measurements.

[0125] like Figure 9 The figure shows a comparison of the average trend of muscle force of a typical muscle (tibialis anterior) in multiple tests under different stand-up strategies. The muscle force calculated by the method of this invention is consistent with the overall trend of muscle force obtained by the traditional Vicon+OpenSim method. Moreover, compared with the traditional Vicon+OpenSim method, the muscle force curve of the method of this invention shows more detailed changes, proving the accuracy and reliability of this invention. The real-time adaptive human musculoskeletal modeling and muscle force simulation system of this invention can clearly identify the significant differences in the tibialis anterior muscle force change patterns of the three strategies. The ETF stand-up strategy shows more drastic fluctuations in muscle force changes, which is consistent with the actual situation of the ETF strategy, namely, excessive dorsiflexion of the ankle joint caused by the trunk leaning forward. This result verifies the effectiveness of this invention in distinguishing different movement strategies.

[0126] like Figure 10 As shown, this diagram compares muscle activation patterns using different optimization methods. Compared to the traditional method of minimizing the sum of squares of muscle activation, the muscle activation distribution obtained by this invention using a real-time modeling and parameter adjustment module and establishing a joint objective function is more even. In particular, the muscle activation of key muscles such as the hamstrings and the short head of the biceps femoris changes from near zero to significant, which is more consistent with the actual situation.

[0127] Different from the prior art, the application constructs a muscle and bone model specially for depth camera data, the muscle position directly corresponds to the joint position instead of a general model, without scaling operation in OpenSim, real-time processing can be realized. For the problem that the traditional method uses general muscle model parameters, which is easy to cause the muscle force characteristic curve to have extreme value and the optimization result is poor, the space-time calibration module and real-time modeling and parameter adjustment module are introduced, the multi-sensor data fusion problem is solved through the space calibration algorithm, and the human muscle and bone model based on depth camera data is further constructed, so that the muscle position directly corresponds to the joint position in a proportional relationship instead of a general model, without manual scaling and calibration operation, real-time processing can be realized, and individual adaptability is significantly improved. Further, the three-dimensional force prediction module is introduced, the joint graph is converted into a body segment graph which is more in line with the mechanical transmission law through heterogeneous multi-modal fusion and graph structure conversion, the space-time feature modeling is performed by combining the graph attention network and the time convolution network, and finally high-precision three-dimensional force prediction is realized, which provides a high-quality data basis for subsequent body segment-based mechanical analysis. The joint objective function is established to calculate the muscle activation and obtain the corresponding muscle force, so that the muscle activation distribution is more uniform, which is more consistent with the actual situation, the physiological rationality of the muscle activation distribution is improved, the flexibility of the model is enhanced, the combination of the low-cost depth camera and the pressure sensor array can realize high-precision real-time muscle force simulation analysis, and can be widely applied to human motion analysis, rehabilitation engineering, robot control and other fields.

[0128] The specific embodiments described above have explained the technical solutions and beneficial effects of the application, and it should be understood that the above description is only the most preferred embodiment of the application, and is not used to limit the application, and any modification, supplement and equivalent replacement, etc. within the principle range of the application should be included in the protection scope of the application.

Claims

1. A real-time adaptive human musculoskeletal modeling and muscle force simulation system, characterized in that, include: The multi-sensor data acquisition module includes a depth camera, a plantar pressure sensor array, a hip pressure sensor array, and a data acquisition terminal, which are used to simultaneously acquire the position and pressure distribution data of human joint points. The spatiotemporal calibration module employs a spatial calibration algorithm, which establishes a coordinate system mapping relationship by clicking on the feature points of the pressure sensor array in the depth camera's field of view, thereby achieving spatial calibration fusion of multiple coordinate systems. The three-dimensional force prediction module extracts pressure distribution features from pressure distribution data, and constructs joint graph structure and body segment graph structure based on pressure distribution features and the position of human joint points. The joint graph structure and body segment graph structure extract spatial features and temporal features through graph attention network and temporal convolutional network, respectively, and perform feature fusion to predict three-dimensional plantar pressure. The real-time modeling and parameter adjustment module is used to construct a human musculoskeletal model containing muscle geometry based on the position of human joints. Based on the actual measured range of muscle length, the module optimizes the parameters of the human musculoskeletal model in real time and outputs reasonable muscle length and speed. The real-time optimization of the human musculoskeletal model parameters includes preliminary parameter adjustment and fine-tuning. The preliminary parameter adjustment includes: performing preliminary adjustments to the parameters of the human musculoskeletal model using a linear scaling preprocessing algorithm. The calculation formula for linear scaling is: New optimal fiber length = original optimal fiber length × k + b Length ratio Length intercept in, Let be the measured muscle length of the i-th muscle. The predicted muscle length for the i-th muscle; The parameter fine-tuning includes: establishing a multi-objective optimal function containing a maximum length normalization deviation term, a minimum length normalization deviation term, a uniform distribution term, and a regularization term to finely adjust muscle parameters, so that the normalized muscle length and velocity are within a reasonable physiological range and the biological rationality of muscle mechanical characteristics is maintained. The calculation is as follows: Among them, l o，i For the optimal fiber length of the i-th muscle, l s，i Let be the tendon relaxation length of the i-th muscle. Let be the initial value of the optimal fiber length for the i-th muscle in the original human musculoskeletal model. Let be the initial value of the tendon relaxation length of the i-th muscle in the original human musculoskeletal model, N be the total number of muscles to be optimized, α1, α2, β, and γ be the weight coefficients of each item, and Var be the variance. The muscle force simulation module is used to calculate human joint moments by using the position of human joint points and three-dimensional foot and buttock pressure. Based on the calculated human joint moments and combined with reasonable muscle length and velocity, a joint objective function is established to calculate muscle activation and obtain the corresponding muscle force, thus completing the muscle force simulation.

2. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, The real-time adaptive human musculoskeletal modeling and muscle strength simulation system further includes: a data visualization module, which provides real-time animations of muscle activation, muscle strength characteristic curves, and human musculoskeletal models, and uses color depth to represent muscle activation intensity.

3. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, In the multi-sensor data acquisition module, the plantar pressure sensor array and the buttock pressure sensor array are both matrix arrays composed of multiple rows and columns of flexible pressure sensor units, used to collect pressure distribution in the vertical direction.

4. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, In the spatiotemporal calibration module, the step of establishing a coordinate system mapping relationship by clicking on the feature points of the pressure sensor array in the depth camera's field of view includes: clicking on the sensor units of the foot sensor array and the buttock sensor array in the depth camera's field of view as feature points, obtaining the coordinate system position of the feature points in the depth camera, calculating the rotation matrix from the pressure array coordinate system to the depth camera coordinate system, and establishing a mapping relationship.

5. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, In the three-dimensional force prediction module, the construction of joint diagram and body segment diagram structures based on pressure distribution characteristics and the positions of human joints includes: The pressure distribution features, including plantar pressure and gluteal pressure distributions, extracted through convolutional layers and self-attention mechanisms, are modeled with human joint locations to form a joint graph structure G. joint =(V joint E joint ), where V joint E is a set of human joints, containing information on the location and pressure distribution characteristics of these joints. joint This is the set of edges connecting the joints of the human body. The joint graph structure is transformed into a body segment graph structure G through an edge-node transformation layer. segment =(V segment E segment ), where V segment E is the set of body segment nodes. segment It is the set of edges connecting body segments.

6. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 5, characterized in that, When performing graph structure transformations using an edge-node transformation layer, the following transformation rules apply: In the joint graph structure, each edge becomes a node in the body segment graph structure. The node features are a weighted combination of the features of the nodes at both ends of the original edge. in, Let w be the feature of the i-th node in the segment graph structure, and w be the weight parameter. Let m be the feature of the m-th node in the articulation diagram structure. The feature of the nth node in the joint graph structure; In a joint graph structure, if two edges share a joint node, then there is an undirected edge connecting the corresponding two nodes in the body segment graph structure.

7. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 5, characterized in that, In the 3D force prediction module, spatial and temporal features are extracted and fused using graph attention networks and temporal convolutional networks based on joint graph and body segment graph structures, respectively, to predict 3D plantar pressure, including: The joint graph structure and the body segment graph structure are sequentially passed through a graph attention network and a temporal convolutional network to extract spatial and temporal features, respectively, to capture the importance relationships and dynamic changes in motion between nodes in the joint graph structure and the body segment graph structure; wherein, the attention coefficient of the graph attention network is: Where W is the weight matrix, a is the attention vector, h is the node feature, and || represents the concatenation operation; The spatiotemporal features extracted from the joint diagram and body segment diagram are input into the feature fusion layer for feature fusion. Then, the features pass through the temporal convolutional network, the global pooling layer, and the fully connected layer in sequence. The output is a three-dimensional force vector containing the soles of both feet and the buttocks, which is the three-dimensional plantar pressure.

8. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, In the real-time modeling and parameter adjustment module, the construction of a human musculoskeletal model containing muscle geometric paths based on the position of human joints includes: Based on the human joint points acquired by the depth camera and combined with the muscle system, the geometric path of each muscle is defined. Each muscle defines a geometric path through multiple path points, establishing a direct connection between the muscle position and the human joint point. The muscle system includes: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique, external oblique, and their corresponding left and right sides.

9. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that, In the muscle strength simulation module, the joint objective function includes: an overall activation term, a maximum activation penalty term, an activation range term, and a time smoothing term. The overall activation term minimizes the square of the activation of all muscles, reducing overall muscle energy consumption. The maximum activation penalty term suppresses excessively high activation in individual muscles. The activation range term promotes uniform distribution of muscle activation. The time smoothing term suppresses drastic fluctuations in muscle activation over time, improving physiological plausibility. The calculation is as follows: Among them, a i Let max(a) represent the activation level of the i-th muscle. i ) represents the maximum activation level of all muscles, range(a) i ) represents the difference between the maximum and minimum muscle activation values, a i (t)-a i (t-1) represents the change in the activation level of the same muscle at adjacent time t, and λ1, λ2, and λ3 are the weight coefficients of each item.

Images